Form Attachment A and B Attachment A and B Survey Protocol

Attachment A: Survey 1 content

Welcome! Thanks for your interest in this study. On the following pages, you’ll be asked to respond to a survey that will ask you a variety of types of questions. Please consider the questions carefully. You will also be asked about your experience answering those questions.

This HIT is part of our research into how to improve surveys. We’re interested in your opinions and there are no right or wrong answers. We are not collecting any personally identifiable information in the survey questions. The study should take about 10 minutes.

Please do not use your browser's back button. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. This survey will take approximately 10 minutes to complete. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this study.

---page break---

1. When looking for work, people sometimes do not accept the first job that they are offered. What would you say is the more likely reason that someone would not accept the first job they are offered?

The job doesn’t offer enough hours

The security protocols are too strict

2. A study of native born residents in Newland found that two-thirds of the children developed considerable levels of nearsightedness after starting school, while their illiterate parents and grandparents, who had no opportunity for formal schooling, showed no signs of this disability.

If the above statements are true, which of the following conclusions is most strongly supported by them?

(A) Only people who have the opportunity for formal schooling develop nearsightedness.

(B) People who are illiterate do not suffer from nearsightedness.

(C) The nearsightedness in the children is caused by the visual stress required by reading and other class work.

(D) Only literate people are nearsighted.

(E) One-third of the children are illiterate.

3. Below are several statements that may or may not represent your own beliefs or attitudes. Please consider them carefully and choose the one that you think best represents you. If more than one seems to fit, then choose the one that best fits. If none seem to fit well, please still choose one that fits best.

I am very familiar with the arguments for increases in national security

I am somewhat familiar with the arguments for increases in national security

I am not familiar with the arguments for increases in national security

4. Choose the option that best represents you:

I am very interested in the economy

I am somewhat interested in the economy

I am not interested in the economy

---page break---

5. Please think about this situation. Two scientists want to know if a certain drug is effective against high blood pressure. The first scientist wants to give the drug to one thousand people with high blood pressure and see how many of them experience lower blood pressure levels. The second scientist wants to give the drug to five hundred people with high blood pressure, and not give the drug to another five hundred people with high blood pressure, and see how many in both groups experience lower blood pressure levels. Which is the better way to test this drug?

All 1000 get the drug

500 get the drug, 500 don't

6. Please think about this situation. A school teacher has 20 students in her class. Half of the students go out to the playground for recess. How many students are left in the class?

None

5

10

15

7. How do you prefer to answer surveys- on a desktop computer or a mobile smartphone?

Desktop computer

Mobile smartphone

8. Thinking of movies that were released in 2010, what was your favorite?

[open text entry]

---page break---

9. How many hours per day do you usually spend at a computer (not including your phone)?

[numeric text entry]

10. Yesterday, how many times did you check your phone?

[numeric text entry]

11. Thinking about the last seven days, how many hours total did you spend watching television?

[numeric text entry]

12. In what month did you purchase the computer that you are currently using?

[choice of months or not applicable]

---page break---

13. Think back to a large purchase that you made – a car, a TV, or a computer, perhaps. Now, please think of another expense that you made at around the same time. What was that expense?

[open text entry]

14. Think back to the most recent purchase that you made, of anything of any size – groceries, gas, or something online, perhaps. Was your expense more or less than $20?

The recent purchase was more than $20

The recent purchase was less than $20

15. How many people, including children, live in your neighborhood?

0-50

51-100

101-150

… [continue in groups of 50]

1001 or more

16. About what percentage of households in your neighborhood own at least one vehicle?

0-10%

11-90%

91-100%

---page break---

17. Lycopene, glutathione, and glutamine are powerful antioxidants that neutralize the free radicals that are produced in the body as a result of routine bodily processes. Do these antioxidants cause aging?

Yes

No

18. The liver, lungs, and heart are all organs of the human body. Are these organs all necessary for regular bodily functioning?

Yes

No

19. Do you think that people under the age of 14 suffer any ill effects from watching programs with violence in them? By ill effects I mean increased aggression in school or at home, increased nightmares, inability to concentrate on routine chores, and so on. By violence, I mean graphic depictions of individuals inflicting physical injuries on others or on themselves, depictions of individuals wantonly damaging property or possessions, abusive behavior and language to others, and so on.

Yes

No

20. Do you think that teenagers spend a lot of their time using social media?

Yes

No

---page break---

Thanks for answering those questions.

Now, please think back to how you felt as you were answering the questions you just finished answering before you came to this screen – during the main survey.

1. How burdensome was participating in this survey?

Extremely burdensome

Very burdensome

Somewhat burdensome

A little burdensome

Not at all burdensome

---page break---

2. We would like to understand how your experience participating in this survey relates to other experiences in real life.

1. Name an activity from real life that you would give a rating of "Extremely burdensome".

[open text entry]

And please explain what makes that activity extremely burdensome:

[open text entry]

2. Name an activity from real life that you would give a rating of “Not at all burdensome”.

[open text entry]

And please explain what makes that activity not at all burdensome:

[open text entry]

3. Name an activity from real life that you would give a middle rating of “Somewhat burdensome”.

[open text entry]

And please explain what makes that activity somewhat burdensome:

[open text entry]

---page break---

Please think back to how you felt as you answered questions during the main survey.

3. How short or long did you feel the survey was?

Very short

Somewhat short

Neither short nor long

Somewhat long

Very long

3. How easy or difficult did you feel it was to come up with your answers?

Very easy

Somewhat easy

Neither easy nor difficult

Somewhat difficult

Very difficult

3. How much mental effort did you put into answering the questions?

Very high effort

Somewhat high effort

Neither high nor low effort

Somewhat low effort

Very low effort

3. How long would you guess that it took to complete the survey? Consider the time starting from the instructions screen until the thank-you screen.

_____ minutes

3. How important to your community did you feel this survey was?

Extremely important

Very important

Somewhat important

Slightly important

Not at all important

3. How helpful for your community did you feel this survey was?

Extremely helpful

Very helpful

Somewhat helpful

Slightly helpful

Not at all helpful

3. How interesting did you feel the topics covered by this survey were?

Extremely interesting

Very interesting

Somewhat interesting

Slightly interesting

Not at all interesting

3. How sensitive did you feel the topics covered by this survey were?

Extremely sensitive

Very sensitive

Somewhat sensitive

Slightly sensitive

Not at all sensitive

3. How intrusive did you feel the topics covered by this survey were?

Extremely intrusive

Very intrusive

Somewhat intrusive

Slightly intrusive

Not at all intrusive

3. How private did you feel the topics covered by this survey were?

Extremely private

Very private

Somewhat private

Slightly private

Not at all private

---page break---

And now just a few final questions about yourself.

1. In what year were you born?

[open text box; numeric]

2. What is your sex?

Male

Female

3. Are you Hispanic or Latino?

Yes

No

4. What is your race? (Please select one or more)

American Indian or Alaska native

Asian

Black or African American

Native Hawaiian or Pacific Islander

White

5. What is the highest degree you have received?

No schooling completed

Elementary school diploma

High school diploma or the equivalent (GED)

Associate degree

Bachelor’s degree

Master’s degree

Professional or doctorate degree

---page break---

And one last question.

Group 1:

The survey that you are taking right now was specifically designed with you, the respondent, in mind. Sometimes we hear from respondents that our surveys should be easier to complete and so we have rewritten some questions to be simpler for you. We are studying ways to reduce the feelings of burden that we impose on our respondents, and we hope that by simplifying some questions, your experience was an improvement over what past respondents may have felt.

Please take a moment now to rate your experience taking this survey.

Very positive

Somewhat positive

A little bit positive

Neither positive nor negative

A little bit negative

Somewhat negative

Very negative

Group 2:

The survey that you are taking right now was designed to improve data quality. Sometimes the data we get from respondents is not very good and so we have rewritten some questions to be more demanding and require more effort from you. We are studying how best to write survey questions to ensure that we get accurate data from our respondents, and we hope that by writing some more difficult questions, the data we received from you is improved over what we got from respondents in the past.

Please take a moment now to rate your experience taking this survey.

Very positive

Somewhat positive

A little bit positive

Neither positive nor negative

A little bit negative

Somewhat negative

Very negative

Group 3:

Please take a moment now to rate your experience taking this survey.

Very positive

Somewhat positive

A little bit positive

Neither positive nor negative

A little bit negative

Somewhat negative

Very negative

---page break---

We have recorded your answer as [FILL: Very positive/Somewhat positive/A little bit positive/Neither positive nor negative/A little bit negative/Somewhat negative/Very negative]. Your responses will help us to investigate the impact on the survey experience of questions like those that you were asked today.

Here is your completion code. Please paste this code into the HIT to verify your participation.

[random numeric code]

Thank you for your participation!

Attachment B: Survey 2 Content

Welcome back!

On the following pages, you’ll be asked to tell us about what it’s like to participate in surveys. We are specifically interested in your experience participating in the last survey that you completed for us. There are no right or wrong answers. We are not collecting any personally identifiable information in the survey questions. Your responses will be used to understand the survey experience so that we can learn what it is like to answer questions like these. Our objective with this study is to improve the survey experience for respondents.

The study will take less than 10 minutes to complete. Please do not use your browser's back button. 

Please enter the survey invitation code that you received by email:
[open text entry]

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. This survey will take approximately 10 minutes to complete. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this study.

---page break---

First, three questions about the last survey:

1. What do you remember about the 1^st survey that you took for us?

[long open text entry]

2. How did you decide whether to participate in this survey right now?

[long open text entry]

3. Do you remember what rating you gave that 1^st survey? If you do remember, please indicate which rating you chose. If not, that’s ok! – just mark that option below.

Very positive

Somewhat positive

A little bit positive

Neither positive nor negative

A little bit negative

Somewhat negative

Very negative

I don’t remember what rating I chose

---page break---

Now, we’d like to ask you about your thoughts about participating in surveys. These questions are important because they will help us to learn how to design better surveys in the future. Please answer honestly; your responses will not affect your earnings in any way.

4. In your own words, what does it mean for participating in a survey to be “burdensome”?

[long open text entry]

5. In your own words, what does it mean for a survey question to be “burdensome”?

6. Write a survey question that you would consider “extremely burdensome”. Do not include your response.

[long open text entry]

---page break---

In the first survey that you took for us last week, we asked you a range of questions – you may have noticed that some were more burdensome than others to answer. We would like to learn which ones you thought were more burdensome to answer. On the following pages, you will see pairs of screenshots of questions that you saw. Please choose which one you thought was more burdensome to answer and describe what made you feel that way.

Do you think that people under the age of 14 suffer any ill effects from watching programs with violence in them? By ill effects I mean increased aggression in school or at home, increased nightmares, inability to concentrate on routine chores, and so on. By violence, I mean graphic depictions of individuals inflicting physical injuries on others or on themselves, depictions of individuals wantonly damaging property or possessions, abusive behavior and language to others, and so on. Yes No

How many people, including children, live in your neighborhood? 0-50 51-100 101-150 … [continue] 1001 or more

And please briefly explain why that question was more burdensome to answer:

[open text entry]

How do you prefer to answer surveys- on a desktop computer or a mobile smartphone? Desktop computer Mobile smartphone

Do you think that teenagers spend a lot of their time using social media? Yes No

And please briefly explain why that question was more burdensome to answer:

[open text entry]

How many hours per day do you usually spend at a computer (not including your phone)? [numeric text entry]

Yesterday, how many times did you check your phone? [numeric text entry]

And please briefly explain why that question was more burdensome to answer:

[open text entry]

When looking for work, people sometimes do not accept the first job that they are offered. What would you say is the more likely reason that someone would not accept the first job they are offered? The job doesn’t offer enough hours The security protocols are too strict

Think back to a large purchase that you made – a car, a TV, or a computer, perhaps. Now, please think of another expense that you made at around the same time. What was that expense? [open text entry]

And please briefly explain why that question was more burdensome to answer:

[open text entry]

---page break---

11. Which of the following would make a survey feel burdensome to you?
    (Select all that apply)

• Long

• Boring

• Unimportant

• Useless

• Annoying

• Intrusive

• Sensitive

• Embarrassing

• Tedious

• Difficult

• No benefit to me

• Disorganized

• Unethical

• Other ___________

[RANDOMIZE ORDER OF THE RESPONSE OPTIONS]

---page break---

That’s all the questions we have for you.

Thank you for participating in this research. The purpose of this study was to explore perceptions of burden of survey respondents, and how survey designers may be able to improve the survey experience. We apologize if participating in this survey caused you any confusion or frustration. It is our hope that this study will help to reduce the burden that surveys impose on respondents in the future by informing our understanding of how it feels to take burdensome surveys.

Please do not discuss the content or purpose of the study with anyone else for the next few days while we are still collecting data. Thanks for your help in keeping this study valid. If you have any other comments on this topic, please leave them here:

[open text entry]