Justification

National Center for Education Statistics

Volume I

Supporting Statement

2020-21 National Teacher and Principal Survey (NTPS)

Screener and Respondent Portal Usability Testing

OMB# 1850-0803 v.251

Attachments:

Attachment 1 – Communication Materials: Contact, Consent, and Receipt Acknowledgement, Mailing Materials

Attachment 2 – Recruitment Screener

Attachment 3 – Protocol Materials

Attachment 4 – Screener and Portal Instrument Screen Shots to Test

June 2019

Background

The National Teacher and Principal Survey (NTPS), conducted every two or three years by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), is a system of related questionnaires that provides descriptive data on the context of elementary and secondary education. Redesigned from the Schools and Staffing Survey (SASS) with a focus on flexibility, timeliness, and integration with other ED data, the NTPS system allows for school, principal, and teacher characteristics to be analyzed in relation to one another. SASS, the predecessor to NTPS, was conducted by NCES seven times between 1987 and 2011. After 2011-12, NCES redesigned SASS and named it NTPS to reflect the redesigned study’s focus on the teacher and principal labor market and on the state of K-12 school staff. NCES conducted NTPS during the 2015-16 and 2017-18 school years. The next iteration of the NTPS will be in the 2020-21 school year. NTPS data collection is administered for NCES by the U.S. Census Bureau (Census).

The following materials are being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting pilot tests, focus groups, cognitive interviews, and usability testing.

This request is to conduct usability testing of the Screener and Respondent Portal internet instruments designed for the 2020-21 NTPS. The screener questions were embedded within the portal in the 2017-18 NTPS. The 2020-21 NTPS will use these pieces as separate instruments.

Design

The sample for 2020-21 NTPS will include approximately: 10,600 public schools and principals, 53,400 public school teachers, 3,000 private schools and principals, and 6,800 private school teachers. Data collection will begin with the sampled school. A notification letter, intended to inform the school of their selection for the survey and to verify their mailing address, will be sent to sampled schools early in the summer of 2020. An overview of this collection is provided in the National Teacher and Principal Survey of 2020-21 (NTPS 2020-21) Preliminary Field Activities (OMB# 1850-0598 v.26) request currently pending OMB approval.

In July 2020, all schools will receive an advance letter addressed to the principal at the school address. The letter will include instructions for completing a brief screener interview online using the NTPS Screener internet instrument. The purpose of the screener interview is to determine the school’s eligibility for NTPS and to establish a survey coordinator. The survey coordinator will be asked to facilitate the completion of NTPS questionnaires within their school, and materials will be mailed to him or her throughout data collection.

In September 2020, all schools will receive an initial school package addressed to the survey coordinator or principal at the school address. The package will contain a letter to the survey coordinator or principal and three individual sealed envelopes that contain login information for completing the Teacher Listing Form (TLF), Principal Questionnaire, and School Questionnaire via the Respondent Portal internet instruments. Principals and survey coordinators will also be contacted by email around the same time the initial packages are mailed to the sampled schools. The emails will contain the appropriate hyperlinks and User IDs to complete the relevant NTPS questionnaire(s) online.

The TLF is designed to obtain information about each teacher working in the school – including the name, email address, and general subject area taught (either by asking the school to verify pre-populated information when available, or to populate the TLF). The primary purpose of the TLF is to build the public and private school teacher sampling frames. Teachers will be sampled from the teacher roster verified by or obtained from their school through the TLF or, in schools that do not verify or complete the TLF, directly from teacher lists obtained from commercial vendor data and/or clerical lookup operations. Teachers are ineligible for NTPS if they are short-term substitutes, student teachers, teacher’s aides, or do not teach any of grades K-12 or comparable ungraded levels.

The TLF is included in the NTPS Respondent Portal instrument and offers the respondent three options for completing their TLF electronically. Respondents may: (1) download an Excel template, populate it with their TLF data, and upload it to the Portal; (2) enter their TLF information directly into the Portal; or when available (3) access a pre-populated teacher list in the Portal which the coordinator then needs to review and update. The first two options were available to respondents in the 2017-18 NTPS Respondent Portal instrument and the pre-populated data is a new option for the 2020-21 NTPS Respondent Portal (in the 2017-18 NTPS, pre-populated data were provided to respondents on paper forms only).

Other than this pre-populated data and the removal of the screener to being a separate instrument, the functionality of the Respondent Portal is nearly the same as in the last data collection cycle (2017-18). Adding the pre-loaded data is meant to reduce the burden respondents incur when providing the teacher list. Removing the screener task from the Portal is meant to make the tasks associated with the Portal more clear to respondents.

This request is to conduct a formative usability testing of the Screener and Respondent Portal instruments developed for 2020-21 NTPS. During the test, the participants’ performance will be investigated using a think-aloud protocol. In order to identify usability problems and to better understand and address the causes of such problems, the priorities and goals of the usability test are to observe participants: (1) providing TLF data electronically using their own equipment or working with pre-populated TLF data, and (2) completing the screener interview.

We will use public data available on the web to provide the pre-populated teacher data for this study. For schools for which the public data are available, we will manually create a file for each school recruited and upload the file into a survey that will be used in the session with the particular school.

Census will administer the usability testing interviews on behalf of NCES. Prior to beginning the primary task in the usability session, test administrators (TAs) will ask questions to gather information about the teacher list data available at the school. During the testing, TAs will observe the participant receiving mocked-up mailing materials about the survey, accessing the NTPS Screener, and then the Respondent Portal using their own device on site at the school.

TAs will observe the participant completing the screener electronically, and then completing the TLF electronically as s(he) uses the method that (s)he arrives at naturally. Participants will use their real school data because only in doing so can the nuances of how the records are stored or coded be used to identify usability issues with data entry into the TLF. Each of the TLF methods needs to be tested. TAs will suggest to participants to use a different method if too many participants are choosing one method over the others, especially when pre-populated TLF data are available. TAs will ask probing questions as needed. Due to the nature of the information collected on the TLF, usability sessions ideally should be conducted onsite at the participant’s school, using the participant’s computer so that (s)he is able to access the necessary school records to complete the TLF as a respondent would during the 2020-21 NTPS full-scale data collection. Each TA will bring a project laptop to the interview site as a back-up, but every effort should be made to have participants use their own equipment and school data during the usability testing.

After completion of the TLF, the TA will debrief the participant about his/her experience with the instrument. As time permits, the participant will be asked to complete their TLF using a different method. This might require a different User ID. Each entire usability session is expected to last ninety minutes.

The following data collection methods will be used to collect participants’ performance data:

• Think-aloud protocol with minimal probing such as “Keep talking;” “What are you thinking?” and acknowledgement tokens (linguists refer to this as backchannels) such as “Um-hum?”

• Observation notes;

• Satisfaction questionnaire;

• Vignettes;

• Retrospective debriefing; and

• Audio and video recording.

Analysis of the data will include behavioral observations, spontaneous verbalizations, and answers to debriefing questions in order to identify problems. Overall satisfaction ratings will also be produced.

Recruiting and Paying Respondents

To conduct usability testing on the NTPS Screener and Respondent Portal, we will recruit 15 individuals who work at schools in the metropolitan DC area that offer at least one of grades 1-12. The participants may work at either public or private schools and must have access to the list of teachers for their school, including information such as teacher name, email address, and subject(s) taught. We will attempt to recruit school staffs from schools of varied sizes, as measured by the number of students enrolled in the school, in order to test the effect of school size responses (e.g. a principal in a small school vs. school secretary in a large school) on TLF completion method. While the goal is to recruit a sample of small schools (<350 students), medium-sized schools (350-750 students), and large schools (750+ students), there are no set quotas for each school size.

As was done in the 2017-2018 NTPS Portal Usability Testing (OMB# 1850-0803 v.189), to assure that we are able to recruit participants from all desired populations and to thank them for completing the interview, each respondent will be offered $75. The $75 amount is requested because the target population is notoriously difficult to recruit, the task is much more burdensome than a simple survey administration, and we will be asking for ninety minutes of their time. The 90 minutes are necessary to ensure adequate time for a participant to provide a TLF, complete a screener interview, and go through as many of the TLF methods as possible. By including all of these components in each usability testing session, we will be able to fully observe and evaluate any challenges participants encounter when accessing various parts of the instrument as a school would during the 2020-21 NTPS full-scale data collection.

Census will recruit participants using multiple sources, including: (1) an email sent to all Census employees explaining the study being conducted and the type of participants being sought (this ‘social media’ way of recruiting has been very effective in the past), (2) flyers posted in local libraries, (3) social media/Craigslist announcements, and (4) personal and professional contacts. We will not recontact schools who participated in the last round of usability testing for the 2017-18 NTPS, but principals at schools who participated in cognitive testing of the principal questionnaire are in scope for testing of the portal and screener because it is a different task. Contact, consent, and receipt acknowledgment materials are provided in Attachment 1; the questions used to screen respondents for participation in Attachment 2; the usability interview protocols in Attachment 3; and draft screenshots of the screener and portal instruments in Attachment 4. A usability session will be carried out either onsite at the participant’s school or in Census’s usability lab. The session will be conducted one-on-one, i.e., one participant and one TA, except, space permitting, a note taker and/or observer may be included.

Assurance of Confidentiality

Participation is voluntary, and participants will read and sign a consent form (see Attachment 1) before their interview is conducted. No personally identifiable information will be maintained after the usability testing interview analyses are completed. Primary interview data will be destroyed on or before December 31, 2026. Data recordings will be stored on Census’s secure data servers.

The usability sessions will be audio and video-recorded. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The only identification included in the video and audio files will be the participant ID. The recorded files will be secured for the duration of the study – with access limited to key Census and NCES project staff – and will be destroyed seven years after completion of the testing. Usability sessions (live or recorded) may also be observed by key project staff. Participants will be informed when observers attend.

Estimate of Hour Burden

We expect the usability interviews to last approximately ninety minutes. Screening potential participants will require 10 minutes per screening. We anticipate needing to conduct 20 screening interviews to yield 15 participants for usability testing. This will result in an estimated total of 27 hours of respondent burden for this usability testing.

Table 1. Estimated response burden for 2020-21 NTPS Screener and Respondent Portal Usability Testing

Participants	Number of Respondents*	Number of Responses	Time Estimate (minutes)	Total Burden Hours
School staff recruitment	20	20	10	4
Usability interviews	15	15	90	23
Total	20	35	-	27

* Duplicate counts of the same respondents are not included in the total number of respondents.

Project Schedule

Recruitment will begin in July 2019 in order to contact and make appointments before schools are out for the summer. Interviewing is expected to be completed within 3 months of OMB approval. The Screener and Portal will be revised after the completion of usability testing.

Table 2. Estimated Project Schedule

Activity	Month
Participant recruitment	July- August 2019
Usability testing	August- September 2019
Quick report	September 2019
Final report	November 2019

Cost to the Federal Government

The cost to the federal government for this usability testing laboratory study is approximately $62,486.

5