Note to Reviewer - SOGI Online Testing

March 1, 2017

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Robin Kaplan Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Online Pretesting of SOGI Questions in the CPS

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study is 52 hours.

If there are any questions regarding this project, please contact Robin Kaplan at

202-691-7383.

1. Introduction and Purpose

There has been interest in collecting sexual orientation and gender identity (SOGI) information in the CPS to add analytic value to the labor force data already collected. Although research has been done on the wording and sensitivity of these questions, most of that work has been done in the context of health or demographic surveys. Additionally, respondents have always been asked to report this information for themselves. Since the CPS is both a labor force survey, as well as asks for a single respondent to report information for all household members, careful research must be done to ensure SOGI information can be collected reliably, accurately, and without any negative impacts on the existing CPS questions.

OSMR has been conducting work to test the feasibility of adding these questions to the CPS by conducting individual cognitive interviews, paired cognitive interviews, and focus groups in an ongoing research study. The purpose of this ongoing testing will be to determine initial feasibility of collecting sexual orientation and/or gender identity in the CPS. The end result of this research is expected to be a recommendation for next steps for the sexual orientation and gender identity questions, such as a field test or testing with additional subgroups. This research will not result in a recommendation that can be used to immediately implement the questions in the CPS.

To help complement this ongoing research, OSMR also proposes doing an online version of the cognitive interview protocol. Online cognitive interview probing studies ask participants to answer a combination of closed and open-ended survey questions with embedded probes where participants can enter their reactions to and interpretations of the questions. Online cognitive interview probing studies have gained in popularity in the survey methods research area recently, as this method can complement the results of in-person cognitive interviews with a greater number of participants with better geographic representation (Meitinger & Behr, 2016). Because in our ongoing research study the in-person interviews will be limited to a set of four cities, an online component of this research will add value by gaining the perspective of more participants from all over the country. In addition, online studies allow for participant anonymity, so we may obtain more honest responses from participants in this mode than we would in-person (e.g., Kreuter et al., 2008). Thus, we expect that adding an online component to this ongoing study will add additional perspectives we would not obtain from the in-person interviews alone, and will complement the results we obtain from the ongoing research study.

2. Research Design

In this research, we will use online data collection with participants recruited from Amazon.com’s Mechanical Turk. Although the Mechanical Turk population may not be representative of the entire U.S. population, studies using MTurk samples obtain similar results to surveys using population-based samples (e.g., Mullinix et al., 2016). Samples obtained from MTurk are more representative of the general population than other convenience samples, such as university students (e.g., Berinsky et al., 2012), using the OSMR participant database, which includes only people in the Washington, DC metro area, or the ongoing research study, which is limited to 4 cities in the United States. Even as such, the results of this study are more concerned with internal validity than the representativeness of any one population. All results will be interpreted with caution given the sample was pulled from MTurk and interpreted in conjunction with the results from the in-person interviews. Results from this online study alone will not be used to make recommendations for the CPS but rather will provide additional context for the other ongoing work, and also contribute to the survey research literature on measurement of sexual orientation and gender identify.

Cognitive interviews from the ongoing study will provide an in-depth understanding of the participant’s thought processes and reactions to the questions. The online cognitive interviews will seek to mimic as closely as possible the in-person cognitive interviews, but adapted for an online mode. Like in the ongoing research study, participants will be asked survey questions from the Current Population Survey, with the SOGI questions included, to provide context. After being asked the CPS and SOGI questions, participants will be asked debriefing questions (via open-ended prompts/probes) to better understand their responses and reactions to the questions, both when reporting for themselves as well as when reporting for other household members.

OSMR researchers will take these open-ended responses to the probing questions and systematically code them for patterns and themes observed in participants’ reactions and interpretations of the questions, when answering for both themselves as well as other household members. We will also assess how well participants understand SOGI terms based on their open-ended responses. In addition, a number of closed-ended responses will ask participants to rate their comfort level, ease or difficulty, and confidence in answering the questions for themselves versus other household members. We will be able to analyze these results across a number of important demographic factors of our participants, including age, region of the country, household size, education, income, and employment status. The full testing protocol is included in Appendix A.

3. Participants

Up to 150 Amazon Mechanical Turk participants will be recruited. This sample size was determined because similar online probing studies (e.g., Meitinger & Behr, 2016). This sample size also takes into account break-offs, incomplete data, and participants who do not follow the task instructions. In addition, we will pretest the survey instrument via TryMyUI.com with 5 participants. TryMyUI is an online testing website where respondents can complete a set of self-administered tasks while thinking “out loud” and respond to follow-up, scripted probes. TryMyUI provides a video recording of the output and each test can last up to 20 minutes. These pilot participants will be asked to think aloud while completing a subset of the survey questions. This pretesting will ensure that participants understand how to complete the roster for their household, that the task is clear, the questions are worded clearly, and the instrument works as intended. The pilot tests will be conducted iteratively so that any modifications can be tested with pilot participants before launching the full study.

4. Burden Hours

The maximum total burden hours for this study will be 52. This survey is expected to take a maximum of 20 minutes per participant.

Table 1. Estimated Burden Hours

	# of Participants Screened	Minutes per participant for Screening	Total Screening Burden	Maximum number of Participants	Minutes per participant for data collection	Total Collection Burden	Total Burden (Screening + Collection)
Mechanical Turk	150	0	0	150	20	50 hours	50
TryMyUI	5	0	0	5	20	1.67 hours	2
Total Burden							52 hours

5. Payment to Participants

Mechanical Turk participants will receive $1.50 for participating in this survey, a typical rate for similar tasks. A total of 150 participants will be screened. The costs for the main survey total $315.00 (1.50 x 150, plus a $90 commission fee to the company). The pretesting pilot participants recruited from TryMyUI will receive the standard TryMyUI fee of $10 each for their participation. The payment for these additional 5 pilot participants will total $50.

Recruiting of participants will be handled by Amazon Mechanical Turk. Once participants are recruited into the study, they will be given a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers.

6. Confidentiality

Participants will be informed of the OMB number and the voluntary nature of the study.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. This survey will take approximately 20 minutes to complete. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. Your participation is voluntary, and you have the right to stop at any time.

Attachments

Appendix A: Survey protocol

References

Berinsky, A.J., Huber, G.A. and Lenz, G.S. (2012) ‘Evaluating Online Labor Markets for Experimental Research: Amazon.com’s Mechanical Turk’, Political Analysis, 20(3), pp. 351–368. doi: 10.1093/pan/mpr057.

Kreuter, F., Presser, S., & Tourangeau, R. (2008). Social desirability bias in CATI, IVR, and Web surveys the effects of mode and question sensitivity. Public Opinion Quarterly, 72(5), 847-865.

Meitinger, K., & Behr, D. (2016). Comparing cognitive interviewing and online probing: Do they find similar results?. Field Methods, 28(4), 363-380.

Mullinix, K.J., Leeper, T.J., Druckman, J.N. and Freese, J. (2015) ‘The Generalizability of Survey Experiments’, Journal of Experimental Political Science, 2(2), pp. 109–138. doi: 10.1017/XPS.2015.19.

Appendix A: Survey protocol

Introduction

This is a survey for the US Bureau of Labor Statistics. This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. This survey will take approximately 20 minutes to complete. Your participation is voluntary and you have the right to stop at any time.

We are looking for information about how respondents answer our surveys. Please take your time as you answer these questions. The information you provide will contribute to valuable research at the BLS, one of the leading Federal statistical agencies.

This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

<page break>

First we have some questions about your household.

1. How many people, 15 years old or older, are living or staying with you, including yourself?

   • 1 (just yourself)

   • 2 people

   • 3 people

   • 4 people

   • 5 people

   • 6 or more people

2. Please enter the initials (first, middle, and last) of each of the people who are living or staying with you.

<insert appropriate number of text boxes based on response to Q1>

<page break>

1. (skip if only 1 person living in the household) You indicated that [TEXT FILL] other [person/people] are living or staying with you. In the space(s) below, please indicate how you are related to [that/each] person.

[TEXT FILL initials of Person 1]

[TEXT FILL initials of Person 2]

….and so forth as appropriate.

<Dropdown menu with the following selections for each person>

• Opposite sex spouse (husband/wife)

• Opposite sex unmarried partner

• Same sex spouse (husband/wife)

• Same sex unmarried partner

• Child

• Grandchild

• Parent

• Brother/ sister

• Other relative (aunt, uncle, cousin, in law)

• Foster child

• Housemate/roommate

• Roomer/boarder

• Other nonrelative

<page break>

2. What is your sex?

• Male

• Female

• Don’t know

• Prefer not to say

3. Was your sex recorded as male or female at birth?

• Male

• Female

• Don’t know

• Prefer not to say

4. Do you describe yourself as male, female, or transgender?

• Male

• Female

• Transgender

• Don’t know

• Prefer not to say

5. Which of the following best represents how you think of yourself?

• Gay or Lesbian

• Straight, that is not gay, lesbian, or bisexual

• Bisexual

• Something else

• Don’t know

• Prefer not to say

<page break>

[This section will randomly select 3 of the household members entered into the roster]

1. You indicated that Person 1, your [TEXT FILL PERSON 1’s RELATION], is living or staying with you. What is their sex?

• Male

• Female

• Don’t know

• Prefer not to say

2. Was their sex recorded as male or female at birth?

• Male

• Female

• Don’t know

• Prefer not to say

3. Do they describe themselves as male, female, or transgender?

• Male

• Female

• Transgender

• Don’t know

• Prefer not to say

4. Which of the following best represents how they think of themselves?

• Gay or Lesbian

• Straight, that is not gay, lesbian, or bisexual

• Bisexual

• Something else

• Don’t know

• Prefer not to say

<REPEAT LOOP FOR THE NUMBER OF PEOPLE IN THE HOUSEHOLD, UP TO 3>

<page break>

1. What is your current marital status?

• Married

• Widowed

• Divorced

• Separated

• Never married

• Don’t know

• Prefer not to say

2. What is your age?

• _____

• Prefer not to say

3. Are you of Hispanic, Latino or Spanish origin?

• Yes

• No

• Don’t know

• Prefer not to say

4. What is your race?

• White

• Black or African American

• American Indian or Alaska Native

• Asian

• Other

• Don’t know

• Prefer not to say

5. What is the highest level of education you’ve completed?

• Less than high school

• High school diploma or GED

• Some college

• Associate degree

• Bachelor’s degree

• Graduate school degree

• Prefer not to say

6. Are you currently employed:

• Full time (go to 7)

• Part time (go to 7)

• Unemployed (go to next page)

• Student (go to next page)

• Retired (go to next page)

• Other, specify (go to 7)

• Prefer not to say (go to 7)

7. What is your occupation?

• Administrator, manager

• Teacher

• Professional

• Administrative support, including clerical

• Sales

• Technician

• Protective Services

• Other service

• Machine or transportation operator

• Construction worker, mechanic

• Farming

• Forestry, fishing or grounds keeping

• Armed Forces

• Other, specify:______

• Don’t know

• Prefer not to say

8. What is your job title? ____

• Don’t know

• Prefer not to say

<page break>

1. What is your annual income? ____

• Don’t know

• Prefer not to say

2. [If 1 is blank], In which of these ranges does your annual income fall?

• Less than $25,000

• $25,000 - $50,000

• $50,001 – $75,000

• $75,001 - $100,000

• More than $100,000

• Don’t know

• Prefer not to say

3. What is your annual household income? ____

• Don’t know

• Prefer not to say

4. [If 3 is blank] In which of these ranges does your annual household income fall?

• Less than $25,000

• $25,000 - $50,000

• $50,001 – $75,000

• $75,001 - $100,000

• More than $100,000

• Don’t know

• Prefer not to say

<page break>

1. Because of a physical, mental, or emotional condition, do you or anyone in your household have serious difficulty concentrating, remembering, or making decisions?

• Yes

• No

• Don’t know

• Prefer not to say

2. Do you have difficulty dressing or bathing?

• Yes

• No

• Don’t know

• Prefer not to say

3. Because of a physical, mental, or emotional condition, do you have difficulty doing errands alone such as visiting a doctor’s office or shopping?

• Yes

• No

• Don’t know

• Prefer not to say

<page break>

In the survey we’re looking at, we ask one person to answer a set of questions for everyone in the household. Now we’d like to get your feedback on some of the questions you answered, and what it would be like to answer these questions about other household members 15 years old or older.

Thanks for answering the questions so far. Now we’d like to get your feedback on some of them, to understand how well they worked and what your reactions are. Please answer the next set of questions as completely as possible, as that will give us the most information to understand your experience.

For background, we’re testing a survey where one person answers the same set of questions for everyone in their household.

1. Did any of the questions stand out to you for any reason? Please explain the reasons they stood out to you.

<page break>

One of the questions asked:

“Was your sex recorded as male or female at birth?”

1. In your own words, what is this question asking? ____

2. What was your reaction to this question? ____

3. How easy or difficult was it to answer this question about yourself?

   • Very easy

   • Easy

   • Neither easy nor difficult

   • Difficult

   • Very difficult

Please explain (optional): ______

4. How easy or difficult was it to answer this question for [TEXT FILL PERSON 1]? <repeat question for up to 3 household members>

   • Very easy

   • Easy

   • Neither easy nor difficult

   • Difficult

   • Very difficult

Please explain (optional): ______

5. How confident are you about your response to this question for [TEXT FILL PERSON 1]? <repeat question for up to 3 household members>

   • Not at all confident

   • A little confident

   • Moderately confident

   • Very confident

   • Completely confident

Please explain (optional): ______

6. If someone in your household was answering this question about you, how confident are you that they would give the correct answer?

   • Not at all confident

   • A little confident

   • Moderately confident

   • Very confident

   • Completely confident

   • It depends (please explain): ______________

7. If other members of your household were nearby and could listen to your answer, how comfortable would you feel answering this question about yourself?

   • Not at all comfortable

   • A little comfortable

   • Moderately comfortable

   • Very comfortable

   • Completely comfortable

   • It depends (please explain): ______________

8. If other members of your household were nearby and could listen to your answer, how comfortable would you feel answering this question about them?

   • Not at all comfortable

   • A little comfortable

   • Moderately comfortable

   • Very comfortable

   • Completely comfortable

   • It depends (please explain): ______________

<page break>

We have another question we’d like your feedback on. It asked:

“Do you describe yourself as male, female, or transgender?”

1. In your own words, what is this question asking?

2. What does “transgender” mean to you?

3. How easy or difficult was it to answer this question about yourself?

   • Very easy

   • Easy

   • Neither easy nor difficult

   • Difficult

   • Very difficult

Please explain (optional): ______

4. How easy or difficult was it to answer this question for [TEXT FILL PERSON 1]? <repeat question for up to 3 household members>

   • Very easy

   • Easy

   • Neither easy nor difficult

   • Difficult

   • Very difficult

Please explain (optional): ______

5. How confident are you about your response to this question for [TEXT FILL PERSON 1]? <repeat question for up to 3 household members>

   • Not at all confident

   • A little confident

   • Moderately confident

   • Very confident

   • Completely confident

Please explain (optional): ______

6. If someone in your household was answering this question about you, how confident are you that they would give the correct answer?

• Not at all confident

• A little confident

• Moderately confident

• Very confident

• Completely confident

• It depends (please explain): ______________

Please explain (optional): ______

7. If other members of your household were nearby and could listen to your answer, how comfortable would you feel answering this question about yourself?

   • Not at all comfortable

   • A little comfortable

   • Moderately comfortable

   • Very comfortable

   • Completely comfortable

   • It depends (please explain): ______________

Please explain (optional): ______

8. If other members of your household were nearby and could listen to your answer, how comfortable would you feel answering this question about them?

   • Not at all comfortable

   • A little comfortable

   • Moderately comfortable

   • Very comfortable

   • Completely comfortable

   • It depends (please explain): ______________

Please explain (optional): ______

<page break>

We have another question we’d like your feedback on. It asked:

“Which of the following best represents how you think of yourself?”

• Gay or Lesbian

• Straight, that is not gay, lesion, or bisexual

• Something else

• Don’t know

• Prefer not to say

1. What was your overall reaction to this question? ____

2. In your own words, what is this question asking? ____

3. How easy or difficult was it to answer this question about yourself?

   • Very easy

   • Easy

   • Neither easy nor difficult

   • Difficult

   • Very difficult

Please explain (optional): ______

4. How easy or difficult was it to answer this question about [TEXT FILL PERSON 1]? <repeat question for up to 3 household members>

   • Very easy

   • Easy

   • Neither easy nor difficult

   • Difficult

   • Very difficult

Please explain (optional): ______

5. How confident are you about your response to this question about [TEXT FILL PERSON 1]? <repeat question for up to 3 household members>

   • Not at all confident

   • A little confident

   • Moderately confident

   • Very confident

   • Completely confident

Please explain (optional): ______

6. If someone in your household was answering this question about you, how confident are you that they would give the correct answer?

   • Not at all confident

   • A little confident

   • Moderately confident

   • Very confident

   • Completely confident

   • It depends (please explain): ______________

Please explain (optional): ______

7. If other members of your household were nearby and could listen to your answer, how comfortable would you feel answering this question about yourself?

   • Not at all comfortable

   • A little comfortable

   • Moderately comfortable

   • Very comfortable

   • Completely comfortable

   • It depends (please explain): ______________

Please explain (optional): ______

8. If other members of your household were nearby and could listen to your answer, how comfortable would you feel answering this question about them?

   • Not at all comfortable

   • A little comfortable

   • Moderately comfortable

   • Very comfortable

   • Completely comfortable

   • It depends (please explain): ______________

Please explain (optional): ______

<page break>

We are thinking about asking questions about sexual orientation and gender identity in a national government survey about employment.

By sexual orientation, we mean the question that asked if you were gay or lesbian, straight or something else. By gender identity, we mean the questions that asked if you were male, female or transgender.

These questions are not currently asked on our employment survey, so we’re looking for feedback on the idea of adding them.

1. What are your reactions to this idea?

2. Why do you think the government is interested in asking people about sexual orientation and gender identity in a survey about employment?

3. How negative or positive would you feel if you were asked to provide information about your sexual orientation and gender identity in a government survey about employment?

   • Very negative

   • Negative

   • Neither negative nor positive

   • Positive

   • Very positive

Please explain (optional): ______

4. How negative or positive would others in your household feel if they were asked provide information about your sexual orientation and gender identity in a government survey about employment?

   • Very negative

   • Negative

   • Neither negative nor positive

   • Positive

   • Very positive

   • Don’t know

Please explain (optional): ______

5. How negative or positive would you feel if you were asked to provide information about the sexual orientation and gender identity of everyone else in your household 15 years old or older?

   • Very negative

   • Negative

   • Neither negative nor positive

   • Positive

   • Very positive

Please explain (optional): ______

6. How likely would you be to participate in a government survey in the future that asked you to provide information about the sexual orientation and gender identity of everyone else in your household 15 years old or older?

   • Not at all likely

   • A little likely

   • Moderately likely

   • Very likely

   • Extremely likely

Please explain (optional): ______

7. Do you have anything else you would like to tell us about the collection of information about sexual orientation and gender identity?

Page 28 of 28