Volume 1 NTPS TFS PFS Cognitive Testing-OMB_11.18.20

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (such as pilot tests, cognitive interviews, and usability studies) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and procedures.

Background and Study Rationale

The National Teacher and Principal Follow Up Surveys, the Teacher Follow Up Survey (TFS) and Principal Follow Up Survey (PFS), are both administered one year after the National Teacher and Principal Surveys (NTPS) are conducted. The surveys are administered only to teachers and principals who respond to the preceding year’s NTPS. The main purpose of these follow up surveys, which are much shorter than the NTPS, is to collect information about teachers’ and principals’ current employment status. It is crucial that decision makers learn about rates of attrition in the teaching and principal professions as well as the reasons that may be influencing this attrition.

The TFS and PFS were previously fielded after the last NTPS survey cycle in the 2015-16 school year.

This request is to conduct cognitive and usability testing of several important items related to both of the Follow Up Surveys and the NTPS. The purpose of this study is to evaluate the communications and procedures relating to the follow up questionnaire for the NTPS, in addition to the questionnaire itself. We will primarily be testing the wording and usability of a text survey invitation and the collection of short survey responses via text message for the TFS and PFS. We want to learn if participants understand the text messages, if they are able to respond to the survey in a text mode, and they are able to log into a survey from a link texted to them. The results of this testing will help provide input and text message wording for an experiment that will be conducted as part of the 2021-22 TFS and PFS. For teachers, we will also cognitively test several new items related to student debt and how debt may affect teacher attrition.

The 2020-21 school year has introduced unique challenges and changes to the way that schools and classrooms are structured. We believe these changes may affect some responses to the NTPS questionnaires, in particular the teacher questionnaire and the school questionnaire that are currently in the field. We would like to use these interviews to ask participants to answer questions from the questionnaire. These items have been identified as potentially challenging for participants due to impacts of coronavirus. This information will provide important context to NTPS estimates for this school year. We would like to learn more about how coronavirus affected teachers and principals and their answers to questions about:

Teacher’s classroom structure, class sizes, and time spent delivering instruction;
Attendance and enrollment in remote learning situations;

Interviews will be conducted remotely using audio or audio/visual software such as Skype for Business or Teams.

During testing, we will keep track of spontaneous comments participants make about the text messages and questions on the questionnaire. We will also note any response errors (i.e., either missing data or incorrect responses) based on participant verbalizations. We will debrief each participant session, focusing on the text messages, new question wording, and NTPS questions.

The survey questions will be evaluated in terms of participants’ understanding of the questions and their ability to respond to the questions and/or access the survey from a text mode. The text messages will be evaluated based on spontaneous comments and responses to probes about individual questions and/or messages. The primary deliverable from this study will be a report about the participant results.

Recruitment and Data Collection

Recruitment

We will recruit principals and teachers in K-12 schools. We will attempt to recruit participants from a variety of grade levels, public and private schools, and from different states. There appears to be a lot of geographic variation in how schools have responded to the coronavirus pandemic, and there are consistent differences in education policy and terminology that may also impact how they respond to the messages and questions. There is no ideal ratio or split between teachers and principals for this study.

Participants will be recruited by the U.S. Census Bureau, Center for Behavioral Science Methods (CBSM), using multiple sources. We will post an invitation on online neighborhood listservs, craigslist, a Census Bureau email blast, and social media. We will also use personal contacts. Recruitment contact materials including information about remote sessions being conducted by phone/computer are included in Attachment 1, as are the questions used to screen participants.

We expect it will be difficult to recruit participants for this study due to the disruptions of the coronavirus pandemic on education this school year. Educators and school staff have had to spend significant amounts of time this school year modifying traditional in-person learning and other educational plans. To ensure that we can recruit participants from all desired populations, and to thank them for completing the interview, each participant will be offered an incentive. Incentives will offset the costs of participation in this research, such as childcare, SMS or text message charges, and internet or phone use. We will offer $40 for teachers and $75 for principals for participation in a ninety-minute interview. Principals are a rarer and harder-to-recruit population than teachers and in the past have been offered the higher compensation level for NCES surveys.

Data Collection

We will conduct remote user sessions using phone and audio/visual software with up to 45 English-speaking participants over two rounds. Testing will be conducted iteratively. Interviewers will use the scripted protocol seen in Volume II. Participants will electronically sign a consent form through a link emailed to them before the interview (see Attachment 1). They will also give oral consent at the start of the interview.

We have budgeted 90 minutes for these interviews to allow time to test both the text messaging plan and the survey questions, but we will prioritize the text messages over the survey questions, especially for principals. For teachers, we will instruct interviewers to cut the text messages testing with at least 20 minutes left in order to have enough time for at least the new student debt questions to be tested.

After giving them some background on the study and the introduction to the protocol:

We will send text messages to participants’ phones. We will walk TFS and PFS participants through these text messages and ask them to answer text messages as they would if they received them as part of the TFS/PFS. We will ask scripted and spontaneous questions about the text messages. All text messages are in Volume II.
Next, the TFS participants will be asked to log into the landing page of a survey from their phone, using information from the text messages. They will not complete the survey on their phone. This will help us evaluate the usability of the text messages for the survey login.
After 1 and 2, participants will be asked to switch from their phone to a laptop/desktop computer and to share their screens with Skype software that was previously installed. The remainder of the interview will be conducted using participants’ computers. Teachers will be given the new TFS questions on student debt and the NTPS teacher survey items. Principals will also receive the NTPS school survey items on a computer. Participants will be asked to think aloud (verbalizing what they are thinking) as they complete the survey. Think aloud data helps identify problems and the causes of those problems.
After they complete the survey interviewers will ask probing questions as needed. After the completion of the survey, participants will be debriefed about their experience answering the survey. Each session is expected to last ninety minutes. See Volume II for the full interview protocol for both teachers and principals.

We will recruit up to 45 English-speaking participants.

The following data collection methods will be used to collect participants’ performance data:

think-aloud protocol;
real-time verbal observation by the researcher;
targeted probes when necessary (e.g., “What were you thinking when you answered that question?”);
retrospective debriefing;
video recording of their computer screen; and
audio recording.

Analysis of the data will include qualitative analysis of behavioral observations, spontaneous verbalizations, and answers to debriefing questions in order to identify problems.

The cognitive interview protocol is included in Volume II. The questions that are being tested in this part of the project are in Attachment 2.

The remote sessions will occur with the CBSM interviewer in his/her home and the test participant in his/her home. The test participant will need to have access to a phone that can receive multiple text messages, regular internet access, and a laptop or desktop computer. The consent and voucher forms can be signed before the session through a link that we will e-mail. This link can be accessed and signed on a smartphone or computer. In the event participants don’t sign before the session, we will obtain consent orally on the recording and ask they sign the voucher before we can mail the $40/$75. Each session will be conducted one-on-one, i.e., one participant and one researcher.

Consultations Outside the Agency

Census will prepare the OMB packet for cognitive testing, conduct cognitive testing, summarize interviews, and analyze results. They will use these results to make final recommendation to NCES for the wording of the text messages and questions. They will also provide information about how participants are interpreting and answering questions in the current NTPS 2020-21 survey. NCES will make the final decision about the wording for the text messages and questions.

Justification for Sensitive Questions

Throughout the item and interview protocols development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive.

Paying Respondents

We will offer $40 cash for teachers and $75 cash for principals for participation in a ninety-minute interview. Principals are rarer in the population and are more difficult to recruit than teachers, and in the past have been offered a higher compensation level for NCES surveys. Cash will be mailed via USPS priority mail flat rate envelopes. These envelopes have tracking and delivery confirmation.

Assurance of Confidentiality

Participation in this cognitive study is voluntary. Participants will be emailed the consent form and will provide oral consent, which will be recorded. If contact and recruitment are all by phone, consent will be requested at the outset of the interview session and recorded. The confidentiality statement and consent form are provided in Attachment 1. Data entered into the survey will be stored in the GovCloud in a secured FEDRamp Moderate environment with a Census Authority To Operate (ATO) to collect personally identifiable information (PII).

The interviews will be audio and video-recorded. Participants will be assigned a unique identifier (ID), which will be created solely for data file management. The participant ID will not be linked to the participant’s name. The audio and video-recorded files will be secured for the duration of the study – with access limited to key U.S. Census Bureau and NCES project staff. Interviews may also be listened in on or otherwise observed by key project staff. Participants will be informed when observers attend.

The Paperwork Reduction Act requires respondents to be informed about the authorities under which the information is collected and protected, even if they are not selected to participate. To meet this requirement everyone who calls in to be screened will be sent an email with text explaining their privacy protections (please see Attachment 1). In the event they do not have an e-mail, a CBSM employee will read it to them after screening them.

Estimate of Hour Burden

Screening potential participants will require 10 minutes per screening, and we anticipate needing to conduct 90 screening interviews to yield 45 participants for sessions. We expect the email with instructions to sign the consent/voucher form and the installation of the software to share their screen to take approximately 10 minutes or less. Finally, we expect each cognitive interview to last approximately ninety minutes. The burden on participants is a total of 91 hours, as seen in the table below. There is no direct cost to respondents.

Estimated response burden for NTPS TFS PFS Cognitive Testing

Respondents	Number of Respondents		Number of Responses	Burden Hours per Respondent	Total Burden Hours (in hours)
Recruitment Screener	90		90	0.17	15
Consent Procedure/ Installation of Software	45	*	45	0.17	8
Cognitive Interview	45	*	45	1.5	68
Total	90		180		91

*A subset of all recruited; does not contribute to the total number of respondents. Some numbers have been rounded.

Cost to the Federal Government

The estimated cost to prepare for, administer, and report the results of these cognitive interviews is approximately $168,221. The cost includes salaried labor for staff and other direct costs associated with the organization of the interviews.

Project Schedule

Recruitment will begin upon OMB approval. Interviewing is expected to be completed within 14 weeks of OMB approval. Please see the estimated project schedule below.

Estimated Project Schedule for 2020 CPS School Enrollment Cognitive Testing

Start	Activity	End
10/19/20	OMB approves package	11/13/20
11/16/20	Begin recruiting	2/8/21
11/16/20	Cognitive testing	2/12/21
1/4/21	Draft and deliver findings and recommendations	3/1/21
1/4/21	NCES and ADDP review findings and recommendations	3/1/21
3/8/21	Prepare final report	4/5/21

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	andy
File Modified	0000-00-00
File Created	2021-01-12