Survey of Doctorate Recipients Sexual Orientation and Gender Identity Question Experiment Plan

APPENDIX D.5

Survey of Doctorate Recipients

Sexual Orientation and Gender Identity Question Experiment Plan

2023 SDR SOGI Data Collection Experiment Plan

In the 2023 Survey of Doctorate Recipients (SDR), NCSES will continue its research on how to collect sexual orientation and gender identity (SOGI) data from doctorate recipients that will yield the most accurate and usable data while attempting to address privacy and confidentiality concerns. The experimental design described below will test three batteries of questions for measuring biological sex, gender identity, and sexual orientation.

The three different versions of the SOGI items are based on OMB’s recommendations on best practices, concurrent SOGI data collection experiments planned for the National Survey of College Graduates (NSCG) and the Survey of Earned Doctorates (SED), and recommendations from research conducted by NCSES on the SED. The rationale for testing these three different batteries of SOGI items is as follows:

OMB Recommendations on Best Practices – This battery of questions is taken from the OMB SOGI recommendations on best practices document¹ and would result in data comparable to other federal survey data collected. The GI item from this set will also be tested in the 2024 SED, and the SO item will be tested in the 2023 NSCG bridge panel.

SOGI Minority-Yes/No Option – This battery of questions will test the possibility of collecting minimally sufficient information to identify the SOGI minority for reporting. This approach has simple Yes or No response options to present the lowest specificity in response options and is less burdensome. This set of questions will also be tested in the 2024 SED, and the GI item from this set will be tested in the 2023 NSCG bridge panel.

Higher-level Specificity Option – This version of the SOGI items has more extensive answer options for both the gender identity question and the sexual orientation question. Recent SED cognitive tests suggested greater inclusivity by offering most response options may positively influence likelihood of responding to SOGI survey questions. The list of answer options was developed by the SED research and recommended for further testing. This high-level specificity approach for both the gender identity and the sexual orientation questions will be tested only by the SDR given the differences in confidentiality and privacy protections between the SED and SDR, and may offer unique information to refine current SOGI best practices for the SDR survey target population.

Experimental Design

To ensure adequate power to assess the performance of these questions, while minimizing any negative effect of the experiment on the 2023 SDR production sample, the experiment will include approximately 12,000 sample cases, which is approximately 10% of sample in the 2023 SDR. The use of a subsample of cases for this experiment minimizes the burden on the overall SDR longitudinal sample and lessens the potential adverse impact on response in subsequent SDR cycles. Stratification will be used in the experimental design to ensure adequate sample for key subgroups. The crossing of a three-level age variable and a three-level doctoral field of degree will be used to define the blocks in the design:

• Age group (<40, 40-64, 65+)

• Doctoral field of degree (science, engineering, or health)

Within each stratum, a systematic sample will be drawn for each of the three batteries of questions using the following sort order:

• Gender (based on existing response to the gender question from the SED or prior cycles of SDR)

• Race/ethnicity group (Hispanic, any race; Non-Hispanic [NH] American Indian or Alaska Native; NH Asian; NH Black; NH Native Hawaiian or Other Pacific Islander; NH White; or NH Multi-race)

• Predicted location (U.S. or out of U.S.)

• 77-level doctorate field of degree

• Age

This sort order will ensure that the sample will be well distributed across these covariates.

Key aspects of the design and analysis plan

• The SOGI experimental questions will be asked after the disability questions near the end of the SDR questionnaire to mitigate any concerns about data loss due to potential survey breakoff.

• Only respondents that respond through the web instrument will be asked the SOGI experimental questions to minimize the impact on the paper and CATI data collections. Note that it is expected that higher than 90% of respondents will participate via the web in the 2023 SDR so limiting the experiment to the web mode will have a minimal impact on the generalizability of the resulting experiment data.

• The biological sex at birth question and the gender identity question will be asked on the same page for each of the three batteries and the sexual orientation question will be on a second page.

• The design also will test survey functionality added to address concerns respondents may have about data confidentiality, privacy, and relevance of the SOGI data. The SED cognitive tests results suggest that some SOGI minorities would weigh the relevance of SOGI data and concerns about data confidentiality and privacy when deciding whether to answer SOGI questions. SOGI screens will feature a link providing an explanation of the relevance of the SOGI questions and how the data will be protected.

• Finally, respondents who are presented experimental questions will be asked about privacy concerns around the SOGI questions they were asked using Likert scales and mark all that apply questions.

• The following data will be examined to assess the relative efficacy of the SOGI questions:

• Response distribution

• Item nonresponse

• Reliability of estimates at the response option level

• Level of disclosure rick at response option level

• Changes in response

• Clicks on explanatory links

• Use of prefer not to answer options

• Timing data

• Incidence of breakoff

• Privacy concerns

• Other specified responses

2023 SDR SOGI Item Batteries

The statement below will be shown to respondents before the start of the SOGI research questions.

The 2023 SDR questionnaire has ended. We are now asking questions that are being used for research purposes. Data collected after this point will not be used and disseminated in any aggregated form for official statistics.

The three batteries of SOGI questions to be asked in the 2023 SDR are as follows:

Battery 1	Battery 2	Battery 3
OMB Recommendations on Best Practices	SOGI Minority Y/N Option	Higher-level Specificity Option
Sex at Birth and Gender Identity Items
What sex were you assigned at birth, on your original birth certificate?	What sex were you assigned at birth, on your original birth certificate?	What sex were you assigned at birth, on your original birth certificate?
1. Male	1. Male	1. Male
2. Female	2. Female	2. Female
How do you currently describe yourself?	Do you consider yourself to be a gender minority (e.g., transgender, non-binary)?	How do you currently describe yourself?
Select all that apply.		Select all that apply.
1. Male	1. Yes	1. Man
2. Female	2. No	2. Woman
3. Transgender	3. I prefer not to answer	3. Transgender
4. I use a different term – Specify: [free-text]		4. Gender non-conforming
		5. Non-binary
		6. Genderfluid
		7. Genderqueer
		8. I use a different term - Specify: [free-text]
Click here to learn why these questions are asked and how data will be protected.	Click here to learn why these questions are asked and how data will be protected.	Click here to learn why these questions are asked and how data will be protected.
The "Click here" message will be shown at the bottom of the screen that includes both the sex at birth and gender identity items for each of the three batteries. The text shown will be as follows: The SDR is asking gender identity questions to be more inclusive and to study the differences in career or educational outcomes among minority groups. Your response will be kept confidential and aggregated in a format that does not lead to individual identification.
Sexual Orientation Item
Which of the following best represents how you think of yourself?	Do you consider yourself to be a sexual minority (e.g., gay, lesbian, bisexual)?	Which of the following best represents how you think of yourself?
		Select all that apply.
1. Gay or lesbian	1. Yes	1. Gay or lesbian
2. Straight, that is not gay or lesbian	2. No	2. Straight
3. Bisexual	3. I prefer not to answer	3. Bisexual
4. I use a different term – Specify: [free-text]		4. Asexual
5. I don’t know		5. Pansexual
		6. Fluid
		7. Queer
		8. I use a different term - Specify: [free-text]
Click here to learn why these questions are asked and how data will be protected.	Click here to learn why these questions are asked and how data will be protected.	Click here to learn why these questions are asked and how data will be protected.
The "Click here" message will be shown at the bottom of the screen with the sexual orientation item for each of the three batteries. The text shown will be as follows: The SDR is asking a sexual orientation question to be more inclusive and to study the differences in career or educational outcomes among minority groups. Your response will be kept confidential and aggregated in a format that does not lead to individual identification.

2023 SDR SOGI Experiment: Probe Questions

Gender Identity (GI) Probe Questions

To help us improve the questions we may include in future iterations of our survey, we would like your feedback on the gender identity question. Below is the question we asked you earlier:

[Display GI question with response options w/o R’s answer]

GP1. How comfortable or uncomfortable are you with providing your gender identity information in a U.S. federal government survey?

O Very comfortable

O Somewhat comfortable

O Somewhat uncomfortable

O Very uncomfortable

GP2. How easy or difficult was it for you to report your gender identity information in this question?

O Very easy

O Somewhat easy

O Somewhat difficult

O Very difficult

GP3. Which of the following statements are true for you when thinking about this question?

Select all that apply.

 This information is important for ensuring equity for gender minorities

 I consider my gender identity to be my own private information

 I have questions about how this information would be useful to the SDR

 I have concerns about how this information may be used

 The response options do not reflect how I think of myself

 I did not understand the question

 I have other concerns – Specify: ____________________________________

 I do have concerns about providing this information

[Randomize the sequence of response options except “I have other concerns – Specify” and “I do not have any concerns about sharing this information,” which will be anchored at the bottom.]

Sexual Orientation (SO) Probe Questions

Again, to help us improve the questions we may include in future iterations of our survey, we would like your feedback on the sexual orientation question. Below is the question we asked you earlier:

[Display SO question and the response options w/o R’s answer]

SP1. How comfortable or uncomfortable are you with providing your sexual orientation information in a U.S. federal government survey?

O Very comfortable

O Somewhat comfortable

O Somewhat uncomfortable

O Very uncomfortable

SP2. How easy or difficult was it for you to report your sexual orientation information in this question?

O Very easy

O Somewhat easy

O Somewhat difficult

O Very difficult

SP3. Which of the following statements are true for you when thinking about this question?

Select all that apply.

 This information is important for ensuring equity for sexual minorities

 I consider my sexual orientation to be my own private information

 I have questions about how this information would be useful to the SDR

 I have concerns about how this information may be used

 The response options do not reflect how I think of myself

 I did not understand the question

 I have other concerns – Specify: ____________________________________

 I do have concerns about providing this information

[Randomize the sequence of response options except “I have other concerns – Specify” and “I do not have any concerns about sharing this information,” which will be anchored at the bottom]

1 https://www.whitehouse.gov/wp-content/uploads/2023/01/SOGI-Best-Practices.pdf

  

D.5-13