Volume I - Cogintive Interviews

National Center for Education Statistics

2020-2021 National Teacher and Principal Survey (NTPS)

Cognitive Interviews

OMB# 1850-0803 v.259

Volume I

Supporting Statement

November 2019

Attachments:

Attachment 1 – Recruitment Advertisements

Attachment 2 – Recruitment Protocols with Eligibility Screeners

Attachment 3 – NTPS Cognitive Interviews FAQs

Attachment 4 – Participant Consent Form

Attachment 5 – Cognitive Interview Protocol

Attachment 6 – Questionnaires to be Cognitively Tested

Attachment 7 – Initial Contact Materials to be Cognitively Tested

Submittal-Related Information

The following materials are being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, usability testing, and cognitive interviews.

Background

The National Teacher and Principal Survey (NTPS), conducted every two or three years by the National Center for Education Statistics (NCES), is a system of related questionnaires that provides descriptive data on the context of elementary and secondary education. Redesigned from the Schools and Staffing Survey (SASS) with a focus on flexibility, timeliness, and integration with other ED data, the NTPS system allows for school, principal, and teacher characteristics to be analyzed in relation to one another. NTPS is an in-depth, nationally representative survey of first through twelfth grade public school teachers, principals, and schools. Kindergarten teachers in schools with at least a first grade are also surveyed. NTPS utilizes core content and a series of rotating modules to allow timely collection of important education trends as well as trend analysis. Topics covered include characteristics of teachers, principals, schools, teacher training opportunities, retention, retirement, hiring, and shortages. The 2020-21 national data collection will be the third administration of NTPS (SASS was conducted from 1987 until 2011, and NTPS was conducted in 2015-16 and 2017-18). The NTPS teacher, principal, and school questionnaires each include core sections of content that are included in every survey administration. NTPS questionnaires also include modular, rotating content that is fielded in selected administrations, usually every other survey cycle. Although the modular survey content varies by administration, survey questions and sections are presented in a logical order and organized to maximize clarity.

This request is to conduct two rounds of cognitive testing of the 2020-21 NTPS School questionnaire with principals and school staff to test new and revised items about virtual schools and online course offerings. The emergence of these schools has led to issues in data processing when calculating key statistics, such as the average student to teacher ratios. The cognitive interviews will enable the NTPS team to identify problems with question wording and suggest revisions to problematic questions.

Study Design, Context, and Respondent Burden

The following versions of the 2020-21 NTPS instruments will be evaluated using cognitive testing:

• School Questionnaire (public and private questionnaire versions).

Cognitive testing will be used to evaluate wording and content of both core and rotating question modules that were previously included in the 2015-16 or 2017-18 administrations, as well as the new items. For the majority of questions, testing will evaluate the same wording as included in the 2015-16 and 2017-18 instruments, or wording that is modified only slightly based on expert review prior to testing. The cognitive testing will use an iterative process. Data from the interviews will be analyzed after each round to identify any issues and each questionnaire and its items will be revised accordingly and then tested in the subsequent round (using revised probes, as needed).

Both rounds of testing will be conducted using paper instruments. Testing will utilize one or more of the following techniques: a think-aloud protocol to identify any comprehension or usability concerns, retrospective probing and debriefing, audio and/or video recording, and observation notes as recorded by the interviewer, when required.

To test these instruments, we will recruit principals and school staff members who in a national NTPS administration would be responsible for filling out their appropriate questionnaire in primary and secondary public and private schools. Exhibit 1 summarizes the estimated response burden associated with recruiting for and conducting the cognitive interviews. We will strive to recruit a mix of public and private schools including fully virtual, traditional brick and mortar with some online offerings, and brick and mortar schools only. Because this study does not aim to be fully representational at the individual level, we will not focus on recruiting a demographically representative sample of respondents.

Across interview rounds, our goal is to conduct 21 interviews. From prior experience, we anticipate needing to screen approximately two school staff for each cognitive interview of these types, to yield the desired number of completed interviews for each round (during the NTPS Portal Usability Testing in 2019, in response to our recruitment advertisements for the study, we received far fewer calls expressing interest in participation from principals as compared to teachers and school staff. However, of the principals that did call, many were eligible for participation based on the study screener). For the 2020-21 NTPS Cognitive Interview study, we anticipate needing to screen a total of 42 individuals in order to yield 21 completed interviews (accounting for ineligible respondents and cancellations), for a total respondent burden of 28 hours.

Exhibit 1. Estimated Respondent Burden by Round of Interview

Activity	Sample size	Number of respondents	Number of responses	Time per respondent (minutes)	Total burden (hours)
Recruitment
Round 1 – School Staff/Principal	18	18	18	10	3
Round 2 – School Staff/Principal	24	24	24	10	4
Recruitment Burden	-	42	42		7
Interview Data Collection
Round 1 – School Staff/Principal	18	9	9	60	9
Round 2 – School Staff/Principal	24	12	12	60	12
Data Collection Burden	-	21	21		21
Total burden		42	63		28

Note: Numbers have been rounded and therefore may affect totals. Duplicate counts of the same respondents contribute to the total number of responses but are not included in the total number of respondents. Previous experience suggests that about half of those who respond to recruitment outreach will participate in the cognitive interviews.

Because a relatively small percentage of schools meet our requirements of being fully or partially virtual we will recruit schools that appear to be at least partially online from the Common Core Data and NTPS 2017-18 responses. Respondents will also be recruited using educational website ads, listservs, Craigslist, other professional networks, and personal networks. Any schools that are in special clearance districts will be excluded.

In addition, because education is heavily influenced by state policy, we will recruit and interview educators from a number of states including, but not limited to, the District of Columbia, Maryland, Virginia, and West Virginia, and both public and private schools.

See Attachment 1 for the language to be used in recruitment advertisements and for a sample recruitment script.

Researchers from the Center for Behavioral Science Measurement (CBSM) at the U.S. Census Bureau will carry out the cognitive interviews. The interviews will be conducted either (1) in person at Census headquarters in Suitland, MD; (2) at a secure location that is convenient for the respondent, such as a library meeting room; or (3) in the event we cannot recruit enough respondents for in person interviews, virtually over Skype. Between four and eight staff members will be trained to conduct the interviews to allow the greatest flexibility possible in scheduling. Interviewers will vary somewhat in their level of experience, however all will be required to demonstrate strong understanding of the interview protocol before beginning interviews. With respondent permission, the interviews will be audio- and/or video-recorded so that the responses may be reviewed as needed during analysis. The CBSM project lead will conduct interviewer training, supervise staff, and monitor data quality throughout the field period.

During both rounds of testing, respondents will be asked to complete a paper version of the NTPS School questionnaire. After completing the questionnaire they will be shown the introductory letter and Teacher Listing Form, they will not be asked to fill it out. Instead we will probe on what kinds of teachers they would list and if they would include virtual teachers, or not. Respondents will be trained on the think-aloud technique and will be instructed to complete portions of the instrument while thinking aloud so that their thought process can be recorded when they are first answering the questions. The interviewer will observe the respondent as they complete the survey and record any relevant information on their behavior, including any demonstrated difficulties. If needed, the interviewer will spontaneously probe on any problems or hesitation respondents may experience with particular questions. After the respondent has completed the survey, the interviewer will then review the instrument with the respondent using scripted and, when relevant, spontaneous probes to address any additional unforeseen issues that arose for the respondent (the cognitive interview protocol is provided in Attachment 5). We will also show participants the school coordinator letter and instructions for filling out the Teacher Listing Form (Attachment 7); they will then be asked if they would include online teachers in the TLF if they were filling it out. Respondents will not be expected to actually fill out the TLF. The questionnaire items will be iteratively tested within each of the two rounds, with the protocol and question wording being adapted for the next round based on the results from the previous round.

Cognitive interviewing techniques allow researchers to evaluate potential response error and to assure that the survey provides valid data. In general, the goal of cognitive testing is to assess the respondents’ comprehension of the questionnaire items, including question intent and the meaning of specific words and phrases in the items. Data from cognitive interviews can identify potentially problematic survey questions that are not understood as intended. This testing can also examine the respondents’ retrieval of relevant information from memory, decision processes involved with answering a question, and questions that are difficult to answer due to cognitive burden.

The cognitive interviews will assess issues such as:

• The respondent’s understanding of terms in the survey;

• How confident the respondent is in their response;

• How they recalled the information they provided in factual questions;

• Whether they found a response choice that fit their answer;

• How easy or difficult it is for the respondent to answer each question;

• Issues with sensitive questions; and

• Consistency of answers within the questionnaire and in comparison to the expected range of answers.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that all of the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Participants will also be advised that the interview will be recorded and that the audio recording will only be reviewed for the purposes of report writing. Lastly, participants will be advised that direct quotes may be used in research papers and professional presentations, but names will never be attributed to anything a respondent says.

Participants will be assigned a unique participant identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant in any way or form. The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the cognitive testing study and will be destroyed after the final report is released. A consent form for each questionnaire is provided in Attachment 4.

Estimate of Costs for Recruiting and Paying Respondents

Because principals and school staff who are knowledgeable enough about the school are a small population that is difficult to recruit for participation, especially as their schedules are often filled with school-related activities, as in testing of the 2019-20 questionnaires, they will be offered $75 to participate. With regards to testing the school questionnaire, a $40 incentive was offered to school staff capable completing that survey during testing of the 2017-18 NTPS content, but CBSM experienced great difficulty recruiting participants for that aspect of testing. At the center of the problem is that often the only staff members that are qualified to answer the school questionnaire are principals, vice principals, and/or assistant principals and to recruit them successfully, we need to use the same approach as for recruiting principals for testing the principal questionnaire. We will thus offer the same amount to prospective participants in testing the school questionnaire as in testing the principal questionnaire, an incentive of $75. Participants in cognitive testing will be paid in cash after completion of the interview.

Estimate of Cost Burden

There are no direct costs to participants.

Cost to the Federal Government

The cost of conducting the cognitive interviews will be $151,850, under the NCES contract to CBSM at the U.S. Census Bureau.

Project Schedule

Recruit participants through networks and advertisements	January 2020
Conduct round 1 of cognitive testing of school questionnaires	January-February 2020
Revision to school questionnaire item wording	February 2020
Conduct round 2 of cognitive testing of school questionnaires	February-March 2020
Analysis of cognitive testing results and final recommendations	March 2020
Develop final wording for the NTPS 2020-21 Virtual School items on the school questionnaires	April 2020