2018
Teaching and Learning International Survey
(TALIS 2018)
Principals Focus Group
OMB # 1850-0803 v. 167
National Center for Education Statistics
Institute Of Education Sciences
U.S. Department Of Education
Washington, D.C.
August 2016
Table of Contents
Chapters
1 Background and Study Rationale 2
3 Consultations oOutside NCES 4
4 Assurances of Confidentiality 4
7 Estimate of Cost to Federal Government 5
Tables
Burden calculations for TALIS 2018 Principal Focus Group 5
Schedule of Activities for TALIS 2018 Principal Focus Group 6
Appendix A: Principal Focus Group Materials A-1
Appendix B: TALIS 2018 Field Test Materials for Principal Review (under review OMB# 1850-0888 v.4) B-1
1 Background and Study Rationale
As part of a continuing cycle of international education studies, the U.S., through the National Center for Education Statistics (NCES), within the U.S. Department of Education, is currently and in the coming years participating in several international assessments and surveys. The Teaching and Learning International Survey (TALIS), sponsored by the Organization for Economic Cooperation and Development (OECD), is one of these studies. TALIS 2018 is part of the larger international program that NCES has actively participated in through collaboration with, and representation at, the OECD. Through this active participation, NCES has sought to strengthen the quality, consistency, and timeliness of international data. To continue this effort, the U.S. must follow through with well-organized and executed data gathering activities within the nation’s boundaries. These efforts allow NCES to build a comprehensive data network that reports on the status of education in the U.S. and informs decision-making at the national, state, and local levels.
TALIS, as an international survey of teachers and principals, focuses on the working conditions of teachers and the teaching and learning practices in schools. TALIS was first administered in 2008 and is conducted every five years. Having participated in 2013 but not in 2008, the United States will administer TALIS for the second time in 2018. Participating in TALIS 2018 will allow the U.S. to continue to compare with and learn from other countries working to improve their education systems and using a variety of approaches. Survey responses from school principals and teachers cover subjects such as professional development they have received; their teaching beliefs and practices; the review of teachers’ work and the feedback and recognition they received about their work; and various other school leadership, management, and workplace issues.
TALIS 2018 questionnaire development began in January 2016 and continued into summer of 2016. A pilot study began in May 2016 (the U.S. does not participate in the pilot study), and the field test will begin in February 2017. The main study will occur in the Southern hemisphere from October through December 2017 and in the Northern hemisphere from March through May 2018. The initial report from the main study data collection is due to be released in June 2019.
This request is to conduct a focus group with school principals to better understand both the barriers and benefits schools tend to associate with participation in surveys like TALIS and to identify communication strategies that will help overcome those barriers to participation. This information will guide recruitment strategies and materials development for TALIS 2018. We will arrange a focus group with principals from different states and school districts, in public and private schools, to understand their perceptions of the study, of international assessments in general, and how they would respond if selected for TALIS.
For principals agreeing to participate in the focus group, we will address the following questions:
What value do international surveys and comparisons have in education?
Does the format of TALIS 2018 (online survey of teachers and principals) influence principals’ motivations to participate?
What information do principals use when deciding whether or not to participate in a survey like TALIS? Who do they consult?
What factors might motivate teachers to participate in a voluntary survey such as TALIS?
What are the perceived benefits of participation for teachers and principals? How do the principals value TALIS?
What current or potential TALIS messages or materials do principals find relevant, useful, and informative?
What would be the most effective channels, formats, and materials to use to communicate with principals?
What other groups (e.g., district, other administrators) would be central to the decision making process and what would be the best way to reach those influencers?
Using the suggestions and information collected from this conversation, we will incorporate the principals’ feedback into the development of materials and strategies for engaging schools in TALIS 2018 recruitment. The following materials, which are proposed for use in TALIS 2018, will be presented to principals prior to and during the focus group to elicit feedback and suggestions (see appendix B for copies of all materials; these are currently awaiting OMB approval under 1850-0888 v.4).
Materials for principal review (to be discussed during the focus group)
Westat and Hager Sharp, NCES’ TALIS 2018 national data collection contractors, will work together to administer these research activities. Westat will identify a pool of principals for participation from different states and school districts. Recruited participants for the focus group will be sent a package of TALIS 2018 field test materials in advance of the discussion (see appendix B). The discussion will be conducted by a trained researcher working from a facilitator guide (see appendix A). Table 1 provides a schedule of activities.
To the extent possible, participants will be from schools with characteristics of the schools that tend to decline participation: high- or low-performing schools and large suburban schools, as well as some private schools. This group discussion will take place via WebEx with eight to nine principals—a number that allows for in-depth collection of information and buy-in from this audience. Topics of the discussion will focus on identifying the benefits principals associate with TALIS participation, the barriers they perceive to participating in a study like TALIS, which TALIS recruitment strategies and advance materials they believe would be most and least useful, and their suggestions for other types of recruitment strategies or materials. The focus group will last up to two hours.
During the focus group, using the Facilitator’s Guide (appendix A), a facilitator will lead the participating principals through a discussion via WebEx, and one to two Hager Sharp team members will observe and take notes on the participants’ comments and suggestions, which they will later compile into a summary report. The session will also be recorded.
The report will describe the methodology, participants’ responses to the research questions outlined above, the recommendations heard, and suggestions for the next steps. This report will inform potential revisions to TALIS recruitment strategies to more effectively recruit schools and teachers for participation in the survey. For the field test, elements can be added to the training of the recruitment staff to inform them of materials that resonate with principals and help them focus discussions with schools on key ideas presented in the materials as well as ways to overcome objections to participation. For the main study, the findings from the focus group will be used, together with feedback from the field test schools, to revise recruitment materials.
Consultations outside NCES include contractors with expertise in designing and administering this focus group, who will also take part in administering the study: David Kastberg, Westat Project Manager (1600 Research Blvd., Rockville, MD 20850); Lauren Salay, Hager Sharp Account Manager (1030 15th Street NW, Suite 600E, Washington, DC 20005); and Maria Ivancin, Market Research Bureau LLC President (1090 Vermont Ave, NW, Suite 300, Washington, DC 20005).
4 Assurances of Confidentiality
All contractor and subcontractor staff working on the TALIS 2018 focus group will sign the NCES Affidavit of Nondisclosure. Once these affidavits have been signed and received by NCES, Westat will share the names of a subset of schools with Hager Sharp for the purpose of forming the focus group. Personal information (e.g., name, address) will be collected for recruitment purposes, but on the data file used for analyses respondents will be identified only by a unique study ID number assigned to each participant. Within 48 hours of respondents’ participation in the focus group, the discussion notes will be edited, organized, and cleaned, and all identifiers will be stripped from the data set. The files containing the participants’ personal information will not be linked to focus group or survey data files. All computer files will be password protected and hard copies will be locked in secure locations (e.g., data will be in locked file cabinets within locked offices). Only Westat and contract staff working directly on the data analysis portion of the project will have access to the data files. Once the final report is created, all personally identifiable information will be destroyed. All presentations of data in reports will be in aggregate form, with no links to individuals.
The statement below will be presented in all written materials (e.g., letters, emails) and read at the start of the focus group session. Participants will also be informed that they can leave the discussion at any time.
NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). Your participation is voluntary and your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, § 9573).
Recruitment of principals is estimated to take up to 30 minutes per principal, and we anticipate needing to contact up to 30 principals to form the desired focus group of 9 diverse participants. The focus group is anticipated to take up to 2 hours. There is no cost to participants beyond the participation burden time. Table 1 provides the burden estimate for this study.
Table 1. Burden estimates for TALIS 2018 principal focus group
Respondent group |
Hours per respondent |
Number of respondents |
Number of responses |
Total burden hours |
Principal – Recruitment |
0.5 |
30 |
30 |
15 |
Principal ‒ Discussion |
2 |
9* |
9 |
18 |
Total Burden |
|
30 |
39 |
33 |
* Subset of initial contact group, not double counted in the total number of respondents.
During recruitment, principals will be offered a $50 incentive for participation. A monetary incentive is deemed necessary both to thank participants for their time and effort and to encourage select types of principals to take part in the focus group, so that the focus group sample can be as representative of the types of schools that tend to decline participation in TALIS as possible.
7 Estimate of Cost to Federal Government
The estimated cost to prepare for, administer, and report the results of the TALIS 2018 focus group is approximately $10,200. This cost includes salaried labor for contractor staff and other direct costs associated with organization of the meeting.
8. Schedule
The schedule of activities for the TALIS 2018 principal focus group is provided in table 2.
Table 2. Schedule of activities for the TALIS 2018 principal focus group
Activity |
Tasks |
Date ranges |
Data collection |
Recruit participants |
By early September, 2016 |
Web-Ex based focus group |
Mid- September, 2016 |
|
Analysis |
Produce Report |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |