2020-21 National Teacher and Principal Survey (NTPS 2020-21)
Volume 1 Supporting Statement
Testing Questions on the Teacher Questionnaire on Sexual Orientation and Gender Identity (SOGI), and Branding Changes
OMB# 1850-0803 v.277
National Center for Education Statistics (NCES) U.S. Department of Education
November 2020
|
The National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), requests OMB approval under the NCES system clearance for Cognitive, Pilot and Field Test studies (OMB #1850-0803) to conduct an experiment within the 2020-21 National Teacher and Principal Survey. This experiment is for an independent oversample of teachers to test Sexual Orientations and Gender Identity (SOGI) questionnaire items (for public school teachers only), crossed with IES/ED Branding conditions (for both public and private school teachers). NCES is interested in measuring item non-response, breakoffs, and unit non-response in the various conditions. The goal of the experiment is to determine the feasibility of including the SOGI items, and the impact of the different branding conditions.
The National Teacher and Principal Survey (NTPS), conducted every two or three years by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), is a system of related questionnaires that provides descriptive data on the context of elementary and secondary education. Redesigned from the Schools and Staffing Survey (SASS) with a focus on flexibility, timeliness, and integration with other ED data, the NTPS system allows for school, principal, and teacher characteristics to be analyzed in relation to one another. NTPS recruitment and data collection is administered for NCES by the U.S. Census Bureau. OMB clearance for the 2020-21 NTPS data collection was approved in 2020 (OMB #1850-0598 v.28-30, revised in v.31-34). NCES is authorized to conduct NTPS by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). NCES conducts NTPS in close consultation with other offices and organizations within and outside the U.S. Department of Education.
During the public comment period for the 2017-18 NTPS, the Williams Institute (a research center at the UCLA School of Law) suggested that collecting data on the sexual orientation and gender identy of teachers and principals would further the ability of policymakers, researchers, and other stakeholders to understand the population and characteristics of LGBTQ teachers. In particular, the inclusion of SOGI questions on NTPS would allow data users to explore whether these teachers differ from their colleagues in attrition, barriers to remaining in their profession, training and professional support, satisfaction at their school or with their career, threats from students, and other professional experiences. The public comment noted that information on sexual orientation and/or gender identity is collected on other federal surveys, including the Department of Health and Human Services, Department of Justice, and Department of Education1, and that there is a Federal Interagency Working Group on Improving Measurement of Sexual Orientation and Gender Identity in Federal Surveys. NCES’ response to the public comment indicated that they would consider testing items on sexual orientation and gender identity for the next administration of the NTPS.
In preparation for the 2019-20 NTPS (subsequently postponed to 2020-21), cognitive testing of the school, principal, and teacher questionnaires was conducted in 2017 and 2018. As proposed in the OMB package for this cognitive testing (OMB# 1850-0803 v.213), three new questions on sexual orientation and gender identity were included on the public school principal and teacher questionnaires. It should be noted that these questions were tested with respondents from public schools only; feedback from leaders of private school associations indicated that both item and unit nonresponse could increase if the questions were included on questionnaires sent to private school principals or teachers. These data have utility even if collected only from the public sector; NTPS provides nationally-representative descriptive information about public school teachers. Comparable private sector data is collected when possible, but is not required for the utility of public sector data (for example, the 2015-16 NTPS surveyed public schools, principals, and teachers only).
From December 2017 to August 2018, four rounds of cognitive testing were conducted with teachers, and three rounds of cognitive testing were conducted with principals across multiple states. In total, items were tested with 23 public school principals and 39 public school teachers, for a total of 59 cognitive interview participants. Respondents generally did not have difficulty understanding or answering the tested questions on sexual orientation and gender identity. Cognitive testing focused on respondents’ reactions to the questions and their perceptions of how other respondents would react to their presence. While item nonresponse was minimal and most participants provided responses, some speculated that others may not feel comfortable answering truthfully. As a result, the SOGI items are not being included in the NTPS 2020-21 questionnaires. However, NCES would like to continue to explore, through the separate research proposed here, whether their inclusion in the NTPS Public School Teacher Questionnaire would increase item or unit nonresponse rates. Empirical research has shown that item nonresponse is low for questions on sexual orientation, though item nonresponse may be higher for respondents with certain characteristics, for example, adults ages 65 and older, non-White respondents, and respondents with less than a high school degree or no college education2. By comparison, item nonresponse for questions on sexual orientation and gender identity may be lower than other topics considered sensitive, such as wages and income3. This earlier research was conducted in the context of household surveys so the findings may not generalize to a survey such as NTPS, in which teachers are contacted through their place of employment. NCES would like to determine whether respondents to establishment surveys are willing to report this information (as respondents to household surveys are) or if their inclusion in the questionnaire would harm data quality. Analysis will compare item response rates and survey breakoffs (for surveys completed online) between the two questionnaire versions to determine whether response rates are negatively impacted. Analysis will also compare item nonresponse rates for SOGI questions to other questions that may be considered sensitive (e.g., teaching salary and supplemental income) as a proxy for respondent discomfort4. In addition, NCES will review text provided by respondents (in response to “another gender (please specify)” and “another sexual orientation (please specify”) to determine whether respondents provided comments on SOGI questions or response options. Given the utility of measuring SOGI for teachers, if their inclusion does not adversely impact unit response rates and do not seem to cause discomfort for respondents, they may be included in a future NTPS administration
NCES is also interested in examining whether there are differences in response associated with using NCES/ED branding compared with IES/ED branding in letterheads and logos. NCES has conducted other experiments to determine the impact of federal agency branding. For example, on the 2012 administration of the National Household Education Survey (NHES), NCES examined differences is response rates when study materials were branded with ED logos and letterhead compared with U.S. Census Bureau logos and letterhead (NHES is administered for NCES by the Census Bureau).5 Similarly, the 2015-16 Private School Survey (PSS) examined differences in envelope branding between the ED and U.S. Census Breau logos (the PSS is administered by the Census Bureau).6 While NHES found increased response rates from households with the Census Bureau branding, the PSS found no differences in response rates from private schools by the branding. However, these previous experiments compared ED and Census branding, while this proposed experiment compares different entities within ED (emphasis on ED and NCES or ED and IES), which is the comparison proposed here. The results of this experiment will inform how NCES presents itself to schools sampled for NTPS and another NCES studies in order to best persuade respondents to complete surveys.
Experimental Design
To understand how the SOGI items work in a production environment and what impact, if any, they have on overall response, NCES is proposing to conduct an experiment within the 2020-21 National Teacher and Principal Survey. This experiment is for an independent oversample of teachers to test Sexual Orientation and Gender Identity (SOGI) questionnaire items. In addition to the SOGI items, NCES is also interested in examining whether there are differences in response associated with using NCES/ED branding compared with IES/ED branding in letterheads and logos. Thus, the experiment proposed here crosses inclusion of the SOGI items with IES/ED Branding conditions. NCES is interested in measuring item non-response, breakoffs, and unit non-response in the various conditions. The goal of the experiment is to determine the feasibility of including the SOGI items, and the impact of the different branding conditions, not to produce survey estimates.
The sample for the main NTPS 2020-21 data collection will include approximately 9,920 public schools (both charter and traditional) and principals and 49,250 public school teachers. The respondent universe for public schools will consist of all public schools in the 50 U.S. states and the District of Columbia (DC) that offer instruction in any of grades 1-12. The NTPS 2020-21 will also include a private school sample, with approximately 3,000 private schools and principals and approximately 6,300 private school teachers. The large NTPS sample allows extensive disaggregation of data according to important characteristics of teachers, principals, and schools.
The independent oversample for the experiment will include approximately 7,635 public schools (both charter and traditional) and 5,040 private schools, with one teacher selected per sampled school. Public and private schools will be selected from the same sampling frames as the main NTPS sample, and teacher eligibility criteria for the experiement will also be the same as for the main NTPS sample of teachers. Teachers will be selected from vendor lists rather than from a school-completed Teacher Listing Form, in order to avoid burdening oversampled schools. All teachers selected for this experiment will be the only teachers selected from their schools. Teachers may be selected from schools in special districts, if those districts have already given consent for participation in NTPS 2020-21.
The SOGI questions will only be included for public school teachers, while the IES/ED branding experiment will include both public and private school teachers. This design yields the following treatment groups, with teachers randomly assigned to the groups:
Public School Teachers (4):
NCES/ED branded, no SOGI questions (the materials used for the 2020-21 NTPS)
NCES/ED branded, SOGI questions added
IES/ED branded, no SOGI questions
IES/ED branded, SOGI questions added
Private School Teachers (2):
NCES/ED branded, no SOGI questions (the materials used for the 2020-21 NTPS)
IES/ED branded, no SOGI questions
Sample sizes were designed with the goal of sampling approximately 1,600 teachers per treatment group. For public school teachers, the branding and SOGI experiments are assumed to be independent. With one teacher sampled per school, α=0.05 and β=0.80, this sample size would detect differences in response rates of 3.51% for public school teachers and 4.96% for private school teachers. In order to avoid burdening schools unnecessarily by asking them to complete a Teacher Listing Form (TLF), teachers will be sampled from vendor lists, however, these lists may not have been updated to reflect recent teacher attrition. In order to ensure differences are detectable, sample sizes were inflated to account for out of scope teachers that may have been mistakenly sampled.
Data collection for the independent sample of teachers in the experiment will begin with mailout on March 1, 2021, concurrent with other teacher data collection activities. All teachers in the experiment will follow a “simple” data collection path with no incentives, and no telephone or field follow-up. The data collection strategy will mimic production NTPS data collection, with two mail and email invitations to complete the web survey, followed by two invitations to complete a paper questionnaire. All communications will be directly with teachers, since no school coordinator will have been recruited, and no school or principal questionnaires or Teacher Listing Form are being requested from the independent sample of schools selected for the experiment.
SOGI Questions
The questions below will be added to Section 9. Teacher Demographic Information in the NTPS Teacher Questionnaire for the SOGI experimental treatment. See Attachment 1 for a copy of the NTPS Teacher Questionnaire, along with notes about how the questionnaire will be manipulated for this experiment. Note that, as stated above, the SOGI experimental treatment will only include public school teachers, so the questionnaire seen in Attachment 1 is the 2020-21 NTPS Teacher Questionnaire (NTPS-4a).
The experimental questions will replace the current gender item, item 9-1 on the non-experimental Teacher Questionnaire. Therefore, the question ordering in the demographic section will vary on the experimental versus control versions of the TQ, as follows:
TQ (Control; non-SOGI):
9-1: Gender
9-2: Hispanic/Latino origin
9-3: Race
9-4: Birth year
TQ (SOGI):
9-1: Hispanic/Latino origin
9-2: Race
9-3: Birth year
9-4: Sex at birth
9-5: Gender
9-6: Sexual orientation
The questions below will be added, along with the introductory text.
The next few questions will help us better understand the experiences of teachers of all sexual orientations and gender identities.
Q9-4. What sex were you assigned at birth (what the doctor put on your birth certificate)?
o Male
o Female
Q9-5. What is your gender?
o Male
o Female
o Another gender (please specify) ______
Q9-6. Which of the following do you think of yourself as?
o Straight, that is, heterosexual
o Lesbian or gay, that is, homosexual
o Bisexual
o Another sexual orientation (please specify) _____
IES/ED Branding Materials
The IES/ED branding experiment will use selected letters and emails to teachers that have been previously reviewed and approved by OMB for the main NTPS study (OMB #1850-0598), but will manipulate certain portions as the letter as part of the experiment.
The previously approved materials are NCES/ED branded and constitute the control materials. The experimental materials will vary in the following ways:
The letterhead and logos on these materials will be changed from NCES/ED to IES/ED.
The letters, FAQs, and emails will change “The National Center for Education Statistics and the U.S. Census Bureau” to “The National Center for Education Statistics in the Institute of Education Sciences, and the U.S. Census Bureau,” in the text.
The Institute of Education Sciences will be added to the signature block of the NCES Commissioner on the IES/ED branded letters, and to the closing block for the emails.
Please see Volume II for the full collection of materials for this experiment. The selected letters and emails that will be used are listed in Exhibit 1. The differences detailed above are illustrated in Exhibits 2-5. A version of the initial teacher mailout letter on NCES/ED letterhead is shown in Exhibit 2, and IES/ED branded version of the same letter is shown in Exhibit 3. The initial teacher mailout email with the different text versions are shown in Exhibits 4 and 5. Finally, Exhibit 6 contains the full text of all letters as seen in the NTPS 2020-21 package (OMB #1850-0598 v.35).
The U.S. Census Bureau is the data collector for the 2020-21 NTPS, and will also be the data collector for this oversample experiment. Westat supports NCES for the 2020-21 NTPS, and will be doing the same for this experiment. For example, Census is drafting the sampling plan, and Westat is assisting NCES in review of these materials.
In this experiment, some versions of the Public School Teacher questionnaires will ask respondents about their sexual orientation and gender identity. Cognitive tests in 2018 indicated that some teachers thought others may be uncomfortable answering these items. By comparing unit nonresponse rates when these questions are included or excluded, as well as item response rates compared to other questions that may be considered sensitive (such as data on salary and supplemental income), this experiment will provide empirical data to show whether public school teachers do, in fact, consider these items to be sensitive.
Respondents will not receive any monetary incentive for their participation.
Data security and confidentiality protection procedures have been put in place for the NTPS to ensure that all contractors and agents comply with all privacy requirements including, as applicable:
The Inter-agency agreement with NCES for this study and the statement of work of NTPS contract;
Privacy Act of 1974 (5 U.S.C. §552a);
Privacy Act Regulations (34 CFR Part 5b);
Computer Security Act of 1987;
U.S.A. Patriot Act of 2001 (P.L. 107-56);
Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9573);
Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151);
Foundations of Evidence-Based Policymaking Act of 2018, Title III, Part B, Confidential Information Protection
The U.S. Department of Education General Handbook for Information Technology Security General Support Systems and Major Applications Inventory Procedures (March 2005);
The U.S. Department of Education Incident Handling Procedures (February 2009);
The U.S. Department of Education, ACS Directive OM: 5-101, Contractor Employee Personnel Security Screenings;
NCES Statistical Standards; and
All new legislation that impacts the data collected through the inter-agency agreement and contract for this study.
The U.S. Census Bureau will collect data under an interagency agreement with NCES, and maintain the individually identifiable questionnaires per the agreement, including:
Provisions for data collection in the field;
Provisions to protect the data-coding phase required before machine processing;
Provisions to safeguard completed survey documents;
Authorization procedures to access or obtain files containing identifying information; and
Provisions to remove printouts and other outputs that contain identification information from normal operation (such materials will be maintained in secured storage areas and will be securely destroyed as soon as practical).
U.S. Census Bureau and contractors working on NTPS 2020-21 will comply with the Department of Education’s IT security policy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance, as well as IT security requirements in the Federal Information Security Management Act (FISMA), Federal Information Processing Standards (FIPS) publications, Office of Management and Budget (OMB) Circulars, and the National Institute of Standards and Technology (NIST) standards and guidance. All data products and publications will also adhere to: the revised NCES Statistical Standards, as described at the website: https://nces.ed.gov/statprog/2012/.
By law (20 U.S.C. §9573), a violation of the confidentiality restrictions is a felony, punishable by imprisonment of up to 5 years and/or a fine of up to $250,000. All government or contracted staff working on the NTPS study and having access to the data, including NTPS field staff, are required to sign an NCES Affidavit of Nondisclosure and have received public-trust security clearance. These requirements include the successful certification and accreditation of the system before it can be implemented. Appropriate memoranda of understanding and interconnection security agreements will be documented as part of the certification and accreditation process.
From the initial contact with the participants in this survey through all of the follow-up efforts, potential survey respondents will be informed that (a) the U.S. Census Bureau administers NTPS on behalf of NCES; (b) NCES is authorized to conduct NTPS by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543); (c) all of the information they provide may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151); and (d) that their participation is voluntary.
The following language will be included in respondent contact materials and on data collection instruments:
The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts NTPS as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).
All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated at [XX] minutes per response based on the mean and median times in the previous collection, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or comments or concerns about the contents or the status of your individual submission of this questionnaire, please e-mail: [email protected], or write directly to: National Teacher and Principal Survey (NTPS), National Center for Education Statistics, Potomac Center Plaza, 550 12th Street SW, Washington, DC 20202.
Invitations to complete the teacher questionnaires will be sent to teachers by email and mailed paper letters. Teachers who do not respond to the initial survey request will receive up to four additional requests by both mail and email. Teachers will receive a paper version of the questionnaire in the third and fourth mailings, as needed. The estimated average burden time per teacher is 40 minutes. There are no costs to respondents beyond their time to participate.
The standard procedure for estimating burden time response cost is to multiply the estimated average length of time it takes to complete the survey by the average salary. The estimated average hourly earnings of teachers is $30.387 in elementary and secondary schools in the May 2017 National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS). The total estimated burden hours for teachers is 6,251 hours, and the associated total estimated burden time cost to respondents is $189,905.
Table 1. Estimates of respondent burden for the 2020-21 NTPS SOGI/IES Branding Experiment
Activity |
Sample Size |
Estimated Response Rate |
Estimated Number of Respondents |
Estimated Number of Responses |
Average Burden Time per Respondent (Minutes) |
Total Burden Hours |
Public School Teachers |
|
|
|
|
|
|
Teacher Questionnaire |
7,635 |
70% |
5,345 |
5,345 |
40 |
3,563 |
Private School Teachers |
|
|
|
|
|
|
Teacher Questionnaire |
5,040 |
80% |
4,032 |
4,032 |
40 |
2,688 |
TOTAL |
-- |
-- |
9,377 |
9,377 |
-- |
6,251 |
NOTE: Due to rounding, numbers presented in this table may not sum precisely to the totals provided.
The cost to the federal government for this study is estimated at $400,000, including time, estimated contract costs, and mail/labor costs.
Table 2 provides the overall schedule for the experiment.
Table 2: Schedule for the 2020-21 NTPS SOGI/IES Branding Experiment
Activity |
Dates |
School sample selected |
Late November 2020 |
Teacher sample selected |
Mid-December 2020 |
Data collection starts |
Mid-February 2021 |
Analysis report of experimental results provided to NCES |
Early November 2021 |
1 Federal Interagency Working Group on Improving Measurement of Sexual Orientation and Gender Identity in
Federal Surveys, Current Measures of Sexual Orientation and Gender Identity in Federal Surveys (2016).
2 Federal Interagency Working Group on Improving Measurement of Sexual Orientation and Gender Identity in
Federal Surveys, Evaluations of Sexual Orientation and Gender Identity Survey Measures: What Have We Learned? (2016). https://nces.ed.gov/FCSM/pdf/Evaluations_of_SOGI_Questions_20160923.pdf
3 Ortman, Jennifer M., Nancy Bates, Anna Brown, and R. Chase Sawyer. "Optimizing Self and Proxy Response to Survey Questions on Sexual Orientation and Gender Identity." In Annual Conference of the Population Association of America, Chicago, IL. https://paa.confex.com/paa/2017/mediafile/ExtendedAbstract/Paper13962/Ortman%20Bates%20Brown%20Sawyer%20PAA%202017.pdf
4 For public school teachers who responded to the 2017-18 NTPS, unweighted item response rates were 91 percent for base teaching salary, 95 percent on whether they held an outside job, and 85 percent on the type of outside job held and the income earned from that outside job. Unweighted item response rates were 97 percent for marital status, 96 percent for living arrangements, 96 percent for Hispanicity, and 95 percent for race.
5 Carroll, S.H., and Zukerberg, A. (2013). Do Names Matter? Experiements Comparing Different Branding and Levels of Personally Identifiable Information in a Mail Questionnaire? Proceedings of the 2013 Federal Committee on Statistical Methodology (FCSM) Research Conference. (https://nces.ed.gov/FCSM/pdf/F3_Carroll_2013FCSM_AC.pdf)
6 Broughman, S.O., Rettig, A., and Peterson, J. (2017). Characteristics of Private Schools in the United States: Results From the 2015-16 Private School Universe Survey First Look (NCES 2017-073). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved September 27, 2020, from http://nces.ed.gov/pubsearch.
7 The average hourly earnings of primary and secondary teachers in the May 2018 National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS) is $30.38. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ data type: Occupation code: Elementary and Middle School Teachers (25-2020) and Secondary School Teachers (25-2030); accessed on September 4, 2019.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Laurie Lewis |
| File Modified | 0000-00-00 |
| File Created | 2021-01-12 |