TALIS 2013 Field Trial Response to OMB Passback

Date: December 16, 2011

To: Shelly Martinez, OMB

Through: Kashka Kubzdela, NCES

From: Patrick Gonzales, NCES

Erin Roth, NCES

Subject: Responses to OMB’s Passback of December 2, 5, 8, and 15, 2011, for TALIS 2013 Field Test Data Collection (OMB# 1850-NEW v.1)

NCES received 15 passback questions from OMB related to the TALIS 2013 field trial clearance package. Responses to each question are detailed below. Changes to the clearance package made in response to a question are indicated below as well as in Supporting Statements A and B (in tracked changes).

Question 1: In B1, how is teacher sampling handled in schools with only some (requirements indicate only a minimum of one is required) ISCED level 2 grades represented, yet ALL teachers within the school are randomly sampled? It would seem likely that some non-ISCED level 2 teachers would be sampled. Is that the intent? How will their data be used?

The international procedures call for the designated school coordinator in each school to prepare a list of all eligible teachers (using a standardized teacher sampling form). Eligible teachers for the purpose of TALIS are those teachers who conduct at least one class (subject matter not limited) of 7^th, 8^th, and/or 9^th graders (ISCED Level 2, as defined in the United States). As long as a teacher conducts at least one class with students in this grade range, s/he is considered an ‘eligible teacher.’ Any teacher who does not teach any class at these grade levels is out of scope and will not be sampled for TALIS. The intent, then, is to only include teachers of ISCED Level 2 students (grades 7, 8, and 9 in the United States). We have updated Supporting Statement Part B to make this as clear as possible.

The teacher sampling form includes: identification number (number generated by sampling software); name of teacher; birth year; sex; main teaching domain (main subject taught); exclusion status (i.e., excluded or not); and email address. Each completed TSF is submitted to NCES (or its national contractor; see Q9 below). This information is entered into the sampling software by NCES or its national contractor (WinW3S, proprietary sampling software developed by the IEA and used in other international studies such as TIMSS and PIRLS). Based on the list of teachers as well as the total number of eligible ISCED Level 2 teachers in each school, teachers will be selected to participate. In schools identified as having up to 30 eligible ISCED Level 2 teachers, all ISCED Level 2 teachers will be selected to participate. In schools identified as having more than 30 eligible ISCED Level 2 teachers, 30 teachers will be randomly selected for participation. We will sample a larger number of teachers within schools than the minimum international specifications to ensure that the United States obtains at least 4,000 sampled teachers after refusals are taken into account. We have updated this information in Supporting Statement Part B to make this as clear as possible. We have also updated the response burden estimates for the main study only in Supporting Statement Part A.

The information collected from the TSF remains with NCES and its national contractor. The names and email addresses of teachers are not associated with the TALIS database in any way; these are collected to allow for follow-up, should it be necessary. Once it is determined that no further follow-up is needed, all TSFs will be destroyed.

Question 2: In B2, what is the U.S. specific criteria for excluding remote and very small schools?

The international definition of remote and very small schools is defined as schools with three or fewer ISCED Level 2 teachers. Based on discussions with the international sampling contractor (Jean Dumais and Sylvie LaRoche of Statistics Canada), the intention of this definition is to allow countries to avoid extremely remote schools where it would be overly burdensome for data collection staff to visit (e.g., no passable roads or very small student populations). We are in discussions with the international sampling contractor about this topic because, based on our analysis of the CCD and PSS datasets, we believe that the application of this rule in the United States would eliminate approximately 29 percent of schools across the country. Though this appears to be a substantial percentage of all schools, this is only a very small percentage of our target teacher population. We believe that the implementation of the international definition will still result in very good coverage of the target teacher population (see response to Q4, below).

Question 3: Is this exclusion typical for NCES surveys? We do not recall ever encountering this type of exclusion before. If not, why is it necessary to exclude any such schools in TALIS?

Each of the other international studies in which the United States participates—PIRLS, PISA, and TIMSS—allows for the exclusion of very small or remote schools. For these three studies, the exclusion of very small or remote schools is generally a non-issue because the sampling of schools is based on student enrollment in the school (that is, student enrollment is the measure of size, with the sampling probability set proportional to size), with larger schools being more likely to be sampled than small schools. In TALIS, the measure of size for sampling purposes is the number of ISCED Level 2 teachers (the target unit of analysis), which is a far smaller number than student enrollment. In addition, due to our estimation procedure for determining the number of ISCED level 2 teachers (described in response to Question 4 below) we anticipate capturing many more than the “typical” school in which a 7^th, 8^th, or 9^th grade teacher works. Therefore, the exclusion criteria, in this case, allow us to better specify and reach our target population. More details on this issue are provided below.

Question 4: Will NCES be able to provide the “preferable” variables on number of ISCED level 2 teachers?

After extensive consultation within NCES, we believe we can provide a fairly good estimate of the number of ISCED Level 2 teachers in each school, which is the preferred measure of size in TALIS. CCD and PSS do not include the number of teachers in each grade, or further, the number of ISCED Level 2 teachers in each school. These datasets do include the total number of full-time equivalent teachers in a school and the total number of students enrolled in each grade. Our steps for creating an estimate of the number of ISCED Level 2 teachers in each school are outlined below. We then verified these estimates by comparing them to the distribution of teachers by grade using the Schools and Staffing Survey. We found that our estimates of teachers by grade level (using student distributions) were almost identical to those using teacher reported grade levels.

We estimated the total number of 7^th, 8^th, and/or 9^th grade teachers using the distribution of 7^th, 8^th, and 9^th grade students. More specifically, in each school we computed the percentage of students enrolled in each of grades 7, 8 and 9, out of the total school enrollment. We then applied these student-by-grade percentages to the total number of teachers per school in order to create an estimate for the total number of 7^th, 8^th, and/ or 9^th grade teachers per school. This resulted in a range of totals per school, from 0 to about 160, with many schools only having a fraction of a 7^th, 8^th, or 9^th grade teacher. Many schools have only a very small percentage of their total student body enrolled in any of grades 7, 8, or 9. For example, if a school traditionally offers grades K-6 but has 10 students out of 400 enrolled in an advanced track, they may actually be enrolled as 7^th grade students. Therefore, using our estimation procedure, this school, with 20 total teachers, will have an estimated 0.5 of an ISCED Level 2 teacher. As schools with very small percentages of any one grade of student occur quite frequently we anticipate capturing schools in our sampling frame that may actually have very few, or even less than 1, ISCED Level 2 teacher(s).

Further, it is important to remember that TALIS has a fairly small total minimum sample size of approximately 4,000 teachers. The sample and sampling strata need to be judiciously determined in order to allow for representative samples of the most common types of ISCED Level 2 teachers and teaching environments (i.e. teachers in middle schools, high schools, junior high schools, and combined grades schools). Excluding schools with 3 or fewer ISCED Level 2 teachers helps to ensure that we will maintain high quality, representative samples of the large majority of ISCED Level 2 teachers.

This estimation method resulted in the following outputs:

Percentage of schools with 3 or fewer ISCED 2 teachers: .28803774 (=29%)

Percentage of schools with 20 or fewer ISCED 2 teachers: .75891572 (=76%)

Percentage of schools with less than 1, but greater than 0 ISCED 2 teachers: .11867903 (=12%)

However, since the unit of analysis in TALIS is teachers, not schools, we also examined the percentage of teachers in schools meeting the thresholds specified above. Based on this analysis, if the United States were to implement the international definition of very small or remote schools, the resulting sample would still cover 97.7% of ISCED Level 2 teachers in schools. (That is, even if we were to eliminate schools with 3 or fewer ISCED Level 2 teachers, the remaining schools in the frame employ nearly 98% of the target teacher population.)

Percent of ISCED 2 teachers in schools with more than 1 ISCED 2 teacher: 99.6%

Percent of ISCED 2 teachers in schools with more than 3 ISCED 2 teachers: 97.7%

Percent of ISCED 2 teachers in schools with more than 20 ISCED 2 teachers: 64%

Question 5: Which version of CCD and PSS will be used (i.e., what will be the “latest available” at the time they are needed)?

For the field trial, we will use the 2009-2010 CCD and PSS datasets; for the main study, we will use the 2011-2012 CCD and PSS datasets. This has been added to Supporting Statement Part B.

Question 6: Where are all of the details for how and when the lists for the stage 2 sampling will be acquired? Please provide.

See the response to Q1, above, which explicates how ISCED Level 2 teachers will be identified and sampled in each school. In the case of principals, the principal of each sampled school for which there are eligible and participating ISCED Level 2 teachers will be asked to complete the principal questionnaire. This information has been added to Supporting Statement Part B.

Question 7: Does NCES have to send teacher lists from individual schools to Statistics Canada? If so, please discuss the confidentiality implications and arrangements.

Information from the teacher sampling forms will be entered by NCES or its national contractor into the sampling software (WinW3S, proprietary sampling software designed by the primary international contractor and used in the TIMSS and PIRLS studies). Statistics Canada (the international sampling contractor) requires the teacher sampling forms to be submitted to verify that the resulting teacher sample for each school was correctly done. However, NCES will eliminate the name and email address of all teachers on the sampling forms before these are forwarded to Statistics Canada. Teachers will only be identified by a unique teacher ID, as generated by the sampling software. NCES or its national contractor will retain the original teacher sampling forms should there be a need for follow-up. However, as soon as data collection has been completed, NCES or its national contractor will destroy all forms that include the teachers’ names and email addresses. No copies, electronic or otherwise, will be retained by NCES, its national contractor, or any of the international contractors.

As with other international studies, NCES will enter into a “collaborative understanding” with the OECD and its international contractors (the IEA and Statistics Canada). The purpose of the collaborative understanding is to protect the confidentiality of U.S. data. NCES currently has a collaborative understanding with its international partners for the PIRLS, PISA, and TIMSS studies (i.e., with the OECD and the IEA). The collaborative understanding document was developed and reviewed by the chief statistician, the (then) chair of the IES Disclosure Review Board, and Commissioner, and was deemed the most appropriate approach to securing the confidentiality of U.S. respondents in international studies. In addition, the U.S. dataset for TALIS will undergo the same disclosure review and analysis that applies to all NCES datasets. The U.S. TALIS dataset will undergo masking and data perturbation, based on the outcomes of the disclosure review, before it is submitted to the OECD and its international contractors for weighting, analysis, and reporting.

Question 8: Please describe NCES’s 2^nd stage sampling plans in more detail (e.g., there are several options internationally permissible for schools with less than 20 ISCED level 2 teachers).

After consultation with the international sampling referee, and taking into account expected refusal on the part of teachers, we have updated the teacher sampling plan as follows. In schools identified as having more than up to 30 eligible ISCED Level 2 teachers, all eligible ISCED Level 2 teachers will be selected to participate. In schools identified as having more than 30 eligible ISCED Level 2 teachers, 30 eligible teachers will be randomly selected to participate. We will sample a larger number of teachers within schools than the minimum international specifications to ensure that the United States obtains at least 4,000 sampled teachers after refusals are taken into account. We have updated this information in Supporting Statement Part B.

Question 9: Who is the data collection contractor for NCES?

NCES is still in the process of identifying a qualified contractor for the national data collection. Our efforts to identify a qualified and capable small business were unsuccessful, so we have had to start the process over to include a larger number of businesses, both small and large. It is our desire to have a national contract in place by the start of the field trial.

Question 10: We are quite concerned about having a 60 minute questionnaire, which is longer than almost any other survey we’ve seen in an NCES or NCEE study. What is the target duration of the questionnaire for full scale?

For purposes of the field trial, we have estimated the time to respond as 60 minutes (teacher questionnaire). However, there was evidence of a wide range of estimated teacher survey completion times (from 25 minutes to 70 minutes) provided by other countries who participated in the TALIS pilot study. In order to better understand respondent burden, we have added an item to the field trial questionnaires asking both teachers and principals how much time it took them to complete the questionnaire. We anticipate that, on average, the teacher survey may take less than 60 minutes. In addition to collecting this time use data, all participating countries have agreed that the goal is to reduce the number of items, with completion time in the main teacher survey to take no more than 45 minutes, on average. The principal questionnaire is shorter and is currently estimated to take no more than 45 minutes response time.

Question 11: To what degree does question 30 match up with similar lines of questioning used in other NCES surveys? It seems shorter and less well-rounded that some sets we seem to recall reviewing in the past and therefore wonder about its utility.

There are a number of items on the teacher and principal questionnaires which NCES has argued are of little or no analytic utility. Question 30 is one of them. Based on discussions we have had at international TALIS meetings, Q30 is intended to reveal whether teachers have a ‘constructivist’ approach to teaching. We questioned why the ‘constructivist’ approach is being targeted when other approaches to teaching are not included, but were informed that this item is of particular interest in many European countries. We will continue to advocate for its elimination at future international meetings.

Question 12: Conversely, question 32 seems way too long. Will there be factor analysis coming out of the field test designed to subset these items for full scale? And what is the planned utility of this question?

The OECD intends to examine teacher self-efficacy through Q32. A factor analysis of items will be conducted to determine whether an efficacy scale can be created through these items. We will continue to examine the advisability of retaining this item once the field trial data has been examined.

Question 13: We do not understand the policy relevance or hypothesis underlying many of the questions in the “Your Teaching in the <Target Class>” and “Your Teaching a Unit in the <Target Class>” sections, particularly as self-reports. Can you provide information about these questions as a set and individually?

NCES has strongly and repeatedly objected to the inclusion of the items included under the sections “Your Teaching in the Target Class” and “Your Teaching a Unit in the Target Class.” The purported intention is to obtain a snapshot of teaching practices without having to conduct observational studies. We do not believe that these items will be of analytic use and worry that socially desirable responses will provide misleading data. In spite of our strong objections, the OECD and a number of other countries have argued for their inclusion because they have found them “useful” in TALIS 2008. NCES will continue to advocate for the elimination of all questions under these two sections at future international meetings.

Question 14: We would like to know whether NCES has considered pushing for some of the most objectionable questions to be optional for countries, if they are retained at all for full scale. We realize that this is beyond the "norm," but in this case, it seems warranted.

Unfortunately, while participating TALIS countries do agree that some of the field trial items are controversial, all countries do not view the same items as problematic. Therefore, if we argue that some of the items (that we do not fully support) become optional we run the very significant risk that other countries will argue that different items (that they do not fully support) also become optional. We have expended great time and effort to add to and modify the teacher educational background and teacher time-use items. And while we feel that these particular items are now much improved, not all countries agree. It is possible that these questions, specifically, would be the items other countries may want to make optional, which would be a big problem for us, and for the survey in general.

 

We will continue to work on modifying and/or deleting the teaching practice and belief questions. However, it is important to note that many countries like these items and will also continue to advocate for their inclusion. Our best argument will be made later with data. These items produced problematic data in the last round of TALIS and will likely do the same in this round. We hope to combine new results from the social desirability scale with the teacher practice and belief data to argue for the modification or removal of these items.

 

In the long run for TALIS, participating countries have discussed having a ‘core’ set of questions and rotating, possibly optional, modules on specific topics of interest. Of course, we have yet to firmly decide on our long-term involvement, however, this structure would help in negotiating problems like this that we may face in the future.

Additional Comment:

Since we have just received the final approval for the U.S. adaptations to the survey instruments, we are updating the package to include the final approved version and reflect the fact that we are doing so.

Question 15: for item #9 in the Teacher questionnaire, should we define what is meant by the choices or is that changing the question too much?

There are a few occurrences in both the teacher and principal questionnaires where response categories are not well specified – such as in item 9 in the teacher questionnaire. This is an issue we plan to rectify in future iterations of TALIS.

However, other countries felt that teachers, and sometimes principals, would not be able to provide a more specific percentage of a given student subgroup and therefore preferred using these response categories that are a bit more open to interpretation. In the future we will argue for percentage ranges instead of the current response categories, and are optimistic that this change will be acceptable. We decided, at this point, to allow teachers in the US equal flexibility as teachers in other countries in answering item 9. It is also important to remember that in our context we can compare teacher responses regarding the general amount of students in various subgroups in their classes to similar CCD data. While this data is at the school and grade level (instead of classroom level) it still offers a good comparison, additional specificity, and the comparison may also be of analytical interest.