Note to Reviewer - Contingent Worker Supplement

March 31, 2016

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Brandon Kopp Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Pretesting of Proposed Changes to the CPS Contingent Worker Supplement

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study is 87 hours.

If there are any questions regarding this project, please contact Brandon Kopp at

202-691-7514.

1. Introduction and Purpose

The Contingent Worker and Alternate Work Arrangement Supplement (referred to below as the Contingent Worker Supplement or CWS) to the Current Population Survey (CPS) was last conducted in 2005. Contingent workers are those workers that do not expect their job to last or reported that their job is temporary. Alternative work arrangements include on-call positions, independent contractors, and temporary help agency workers. Since 2005, the Internet and the emergence of smartphone applications has changed the way many workers find and complete their jobs. For this reason, CPS stakeholders have proposed the addition of questions to the next round of the CWS, to be administered in May 2017, to capture these electronically mediated, contingent work arrangements.

There are two types of electronic mediation that the newly proposed questions are meant to capture. The first are arrangements where a worker is matched with a customer through an Internet site or smartphone application, is paid through the company that runs that site or application, and performs an in-person service. Examples of this type of arrangement are Uber, where drivers are matched with riders through an application, or Washio, where workers pick up the customer’s laundry and deliver it to a laundry service. In both of these arrangements, the workers choose when and whether they work and perform the service for the customer in person.

In the second type of electronically mediated, contingent work, the worker finds short duration tasks, ranging from several minutes to several hours, through an online service and the tasks are performed entirely online. Workers may be asked to design a logo for a business given pre-defined specifications (e.g., 99designs.com) or to add descriptive keywords to photos or documents (e.g., Amazon Mechanical Turk). In these types of arrangements, the worker finds tasks listed on a website, chooses which to participate in, completes and/or submits their work through the website, and is then paid through the service based on a rate set by the person who initiated the task.

CPS stakeholders are interested in measuring how many workers are engaged in these types of arrangements as their main job, their secondary job, or as a source of secondary earnings. That is, questions regarding these work arrangements will only be asked of those who reported having worked for pay during the previous week.

CPS survey staff have allotted space for four questions in the CWS in order to measure these concepts. Given the complexity of the concepts and the small number of questions, cognitive testing is especially important. The purpose of this testing will be to:

1. Understand the measurement properties of the electronic mediation questions. That is, do the proposed questions maximize the number of true positives (reporting yes when engaged in one of the work arrangements described above) and minimize the number of false positives (reporting yes when not engaged in one of the work arrangements described above)?

2. Test minor wording changes to the questions that have been developed

3. Determine if introductory or transition language is necessary between the current set of contingent worker questions and the electronic mediation questions

4. Determine what interviewer instructions or help screens are necessary to explain the key concepts

2. Research Design

To accomplish these goals, question variations will be tested using a mixture of in-lab cognitive interviewing and online testing.

Cognitive Interviews. Cognitive interviews provide an in-depth understanding of the respondents’ thought processes and reactions to the questions. These interviews will be done one-on-one in the Office of Survey Methods Research (OSMR) laboratory at the Bureau of Labor Statistics. Interviews will be conducted by staff from the OSMR who are experienced in conducting these types of interviews. Participants will be asked survey questions from the Current Population Survey and the Contingent Worker Supplement to provide context for the electronic mediation questions. After being asked the electronic mediation questions, participants will be asked debriefing questions to better understand the accuracy of their reports and the thought process used to formulate their responses. The testing protocol is included in Appendix A, Part 1.

Online Testing. These questions will also be tested using a sample of participants recruited through Amazon’s Mechanical Turk (mTurk) service and from the TryMyUI service. These participants will complete an online form which asks them an abbreviated set of CPS and CWS questions, the electronic mediation questions, and several debriefing questions, including questions to probe the accuracy of their responses to the electronic mediation questions and about their experience answering those questions. This online testing is an unmoderated cognitive interview. Both of these platforms are especially useful in this context because (1) mTurk and TryMyUI workers are the types of workers we are attempting to capture with the electronic mediation questions and (2) they allow us to achieve greater geographic diversity in our testing sample. The reason for testing on these two separate platforms is because we wish to capture slightly different groups of online task workers. TryMyUI sessions are screen recorded and participants provide their data verbally, as opposed to mTurk workers who will interact only with the online questionnaire. These varying requirements may affect how these workers think about the electronic mediation questions.

The testing protocol for mTurk participants is included in Appendix A, Part 2 and the protocol for TryMyUI participants is in Appendix A, Part 3. These protocols are very similar. The main difference is that, since TryMyUI records audio, those participants will be encouraged to talk aloud while answering the questions.

The findings from all modes of testing will be evaluated qualitatively. The debriefing feedback from both modes is the primary outcome to be used to revise question wordings.

As this is pretesting, we expect that modifications may be made during the course of the study based on initial results. Although the goals of the testing, and overall design, will remain the same, findings from preliminary results may be used to improve the interview questions. Modifications are likely to range from slight changes to question wording to the inclusion of additional instructions.

3. Participants

Cognitive Interviews. Thirty participants will be recruited for cognitive interviews using targeted advertisements in newspapers and online bulletin boards (see Appendix C). Participants will be screened to ensure either they or a member of their household is engaged in a work arrangement that meets the definitions described in Section 1 (see Recruitment Script in Appendix B). A small group of five participants will be selected from the existing OSMR database, without any screening. This will allow testing of the questions among people who do not have these types of work arrangements and may not be familiar with the terms, as well as exploration of proxy reporting.

Online Testing. One hundred and forty-five participants will be recruited using a convenience sample from Amazon Mechanical Turk of adult U.S. citizens (18 years and older). These participants will be screened (see Appendix A, Part 2) to ensure they have experience relevant to the questions being asked. An additional five participants will be recruited using the TryMyUI testing service. Similar recruitment criteria will be used for TryMyUI participants.

4. Burden Hours

The total burden hours for this study will be 87.

Cognitive Interviews. The number of participants needed for testing a given question is dependent upon the results and the point at which saturation is reached and no further information is being gained from additional interviews. Given that, our goal is to interview no more than 30 participants in the OSMR lab. We anticipate that each session will last approximately 60 minutes (30 total burden hours). Participants will be asked several screening questions about their work arrangements as part of recruitment (see Appendix B). Screening questions and scheduling an appointment is expected to take no longer than 10 minutes and participants will not be compensated for time spent on scheduling. Given the targeted nature of the advertisement, we anticipate a high qualification rate for the screening questions (approximately 75%), so estimate we will need to contact approximately 40 people in order to get 30 interviews. This will add an additional 6.7 burden hours (40 participants X .1667 hours) to the study, for a total of 36.7 burden hours for cognitive interviews.

Online Testing. Our goal is to obtain feedback from 150 online participants; 145 from mTurk and 5 from TryMyUI. We anticipate that, in both cases, the task will take no longer than 20 minutes, including screening, CPS and CWS questions, electronic mediation questions, and debriefing questions for a total of 50 burden hours (150 participants X 0.33 hours).

5. Payment to Participants

A total of $1540 will be spent on payments to participants.

Cognitive Testing. The 30 participants will each be paid $40 for their participation in testing for a total of $1200.

Online Testing. Each of the 145 mTurk participants will be paid $2.00 for participating in the study, a typical rate provided by Mechanical Turk for similar tasks. TryMyUI participants will be paid $10 for their participation; this is a standard fee required by TryMyUI. A total of $340 will be paid directly to Amazon Mechanical Turk and TryMyUI for participant fees.

6. Data Confidentiality

Cognitive Interviews. Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of a national employment survey. Participants will be given a consent form to read and sign (see Appendix D). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

Online Testing. Recruiting of participants will be handled by Amazon Mechanical Turk and TryMyUI. Once participants are recruited into the study, they will be provided a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics’ servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Appendix A, Part 1:	Cognitive Interviewing Protocol
Appendix A, Part 2:	Online Testing Protocol (mTurk)
Appendix A, Part 3:	Online Testing Protocol (TryMyUI)
Appendix B:	Recruitment Script (Cognitive Interviews Only)
Appendix C:	Recruitment Advertisement
Appendix D:	Consent Agreement Form and Privacy Act Statement

Appendix A, Part 1: Cognitive Testing Protocol

The protocol includes the questions that will be tested, additional questions that provide context for the tested question, and debriefing questions. The goal is to identify the most effective wording for questions and response choices by understanding the associations participants have with terms used in the question wording and how they interpret question meanings. The interviewer may ask unscripted follow-up questions (e.g., “Can you tell me more about that?”) if a participant’s answer suggests additional, pertinent information can be gained.

Introductory Material

• Hi! Thank you for coming in today.

• I am …... [This is my colleague ( ) who will be taking notes for us today]

• Have you participated in any of our studies before?

• Consent Form/Permission to audiotape

• Explanation:

  • We are going to be working with some questions from the Current Population Survey, an ongoing survey that provides a continuous flow of information about the characteristics of the American workforce.

  • What we are going to do today is go through a few questions from the survey and get your reactions to them.

  • The purpose of today’s session is to help us find out more about how people respond to these questions. Basically, we’re trying to find out what you think a question is asking and how hard it is to answer. We are not here to evaluate you, we are looking to improve the questions, so there are no wrong answers. All the information you give us will be kept completely confidential, and will be used to improve the survey questions.

• Any questions before we begin?

Question Number	Questions	Response Options	Who Receives Question
Roster1	Please tell me the names and ages of everyone who lives in your household	Open ended	Everyone
Roster 2	Is (name) employed?	Yes No	Everyone
Demographic Questions, ask as a block for each HH member
EDUCA	What is the highest level of school (name/you) (has/have) completed or the highest degree (name/you) (has/have) received?	(31) Less than 1st grade (32) 1st, 2nd, 3rd or 4th grade (For example: BA, AB, BS) (33) 5th or 6th grade (34) 7th or 8th grade (35) 9th grade (36) 10th grade (37) 11th grade (38) 12th grade NO DIPLOMA (39) HIGH SCHOOL GRADUATE- high school DIPLOMA or the equivalent (For example: GED) (40) Some college but no degree (41) Associate degree in college - Occupational/vocational program (42) Associate degree in college -- Academic program (43) Bachelor's degree (44) Master's degree (For example: MA, MS, MEng, MEd, MSW, MBA) (45) Professional School Degree (For example: MD,DDS,DVM,LLB,JD) (46) Doctorate degree (For example: PhD, EdD)	Everyone
HSPNON	(Are / Is) (name/you) of Hispanic, Latino, or Spanish origin?	(1) Yes (2) No	Everyone
RACE	I am going to read you a list of race categories. You may choose one or more races. For this survey, Hispanic origin is not a race. Are you: White; Black or African American; American Indian or Alaska Native; Asian; OR Native Hawaiian or Other Pacific Islander? Do not probe unless response is Hispanic or a Hispanic origin. Enter all that apply, separate with commas	White Black or African American American Indian or Alaska Native Asian Native Hawaiian or Other Pacific Islander Other - DO NOT READ	Everyone
Labor Force Questions, for up to 3 HH employed members, identified during roster
LABOR	Next, I am going to ask a few questions about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___		Everyone
PUWK	LAST WEEK, did you do ANY work for pay?	Yes, go to PUMJ No, go to PUABS Don't Know Refused	Everyone
PUABS	LAST WEEK, did you have a job either full or part time? Include any job from which you were temporarily absent.	Yes No Don't Know Refused	Employed, PUWK = 1
PUMJ	LAST WEEK, did you have more than one job (or business), including part-time, evening or weekend work?	Yes, No, Don't Know Refused	Employed, PUABS = 1 PUMJ = 1
PUHRUSL1	How many hours per week do you USUALLY work at your (main) job? (If PUMJ is "yes") By main job we mean the one at which you USUALLY work the most hours.	If pumj=1 ask puhrusl2 else go to puio1int	All employed, fill based on more than one job, PUMJ=1
PUHRUSL2	How many hours per week do you USUALLY work at your other job(s)?		All employed, fill based on more than one job, PUMJ=1
PUIO1INT	Now I have a few questions about the (main) job - (a) (at which you worked LAST WEEK.) (b) (from which you were absent LAST WEEK.) (Was.../Were you) employed by government, by a private company, a non-profit organization, or (was.../were you) self-employed (or working in a family business)?	Government Private company Non-profit organization incl. tax exempt and charitable organizations Self-employed Working in family business	All employed, fill based on more than one job, PUMJ=1
PUIO1INC	What is the name of the (company) (non-profit organization) (business) for which ... (works) (at your/his/her Main job)?	Open-ended	All employed, fill based on more than one job, PUMJ=1
PUIO2INC	"What kind of business or industry is this?" READ IF NECESSARY What do they make or do where you work?	Open-ended	All employed
PUIO1OCC	What kind of work (do) you do, that is, what (is) your occupation? (For example: plumber, typist, farmer)	Open-ended	All employed
PUIO1DT	What (are) your usual activities or duties at this job? (For example: typing, keeping account books, filing, selling cars, operating printing press, laying brick)	Open-ended	All employed
	Contingent Work Supplement, for up to 3 employed HH members, identified during Labor Force Questions
S1	Some people are in temporary jobs that only list for a limited time or until the completion of a project. Is your (main) job temporary?	Yes No Don't Know Refused	All employed
S2	Provided the economy does not change and your job performance is adequate can you continue to work for your current employer as long as you wish?	Yes No Don't Know Refused	All employed
S3	Are you working only until a specific project is completed?	Yes No Don't Know Refused	S2=2
S4	Were you hired to temporarily replace another worker?	Yes No Don't Know Refused	S3=2
S5	Were you hired for a fixed period of time?	Yes No Don't Know Refused	S3=2
S6	Is your job a year round job or is it only available during certain times of the year?	Yes No Don't Know Refused	S3=2
S7	How much longer do you expect to work in your current job?	Less than or equal to 1 year More than 1 year Don't Know Refused	S3,4,5=2 S6=1
S8	What is the main reason you expect to work at your current job for less than a year?	Open-ended	S7=1
S9	If it were not for this reason, could you have kept working at the job you had last week.	Yes No Don't Know Refused	S7=1
S10	(For your main job), are you paid by a temporary help agency?	Yes No Don't Know Refused	S2=2
S11	Even though you told me your job is not temporary, are you paid by a temporary help agency?	Yes No Don't Know Refused	S2 =1
S12	Note the wording in parentheticals in the questions below are only used for the first person in the household. If an inquiry is being made for additional people in the household the wording is omitted) (Some people are in a pool of workers who are ONLY called to work as needed, although they can be scheduled to work for several days or weeks in a row, for example substitute teachers, and construction workers supplied by a union hiring hall. These people are sometimes referred to as ON-CALL workers.) Were you an ON-CALL worker last week (in your main job)?	Yes No Don't Know Refused	All employed
S13	(Some people get work by waiting at a place where employers pick up people to work for a day. These people are sometimes called DAY LABORERS). Were you a DAY laborer last week?	Yes No Don't Know Refused
S14	(Some people provide employees or their services to others under contract. A few examples of services that can be contracted out include security, landscaping or computer programming). Did you work for a company that contracts out your services last week?	Yes No Don't Know Refused
S15	Are you usually assigned to more than one customer?	Yes No Don't Know Refused
S16	Do you usually work at the customer’s worksite?	Yes No Don't Know Refused
S17	Last week, were you working as an independent contractor, an independent consultant, or a freelancer. (that is someone who obtains customers on their own to provide a product or service)	Yes No Don't Know Refused
	Electronic Mediation Questions, for up to 3 employed HH members, identified during Labor Force Questions
EMQIntro	I now have a few questions about how the Internet and mobile apps have changed the way some people work.
S18	Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app. For example, driving a person from one point to another using your own car, delivering people’s laundry to a dry cleaner, or helping a person assemble furniture. Does this describe any work you did last week?	Yes No Don't Know Refused	All employed
S19	Was that for your (main) job, (second job), or additional work for pay:	Main Job Second Job Additional Work For Pay Don’t Know Refused	S18=1
S20	Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online. For example, data entry, labeling photos, translating text, or other micro-tasks. Does this describe any work you did last week?	Yes No Don't Know Refused	All employed
S21	Was that for your (main) job, (second job), or additional work for pay:	Main Job Second Job Additional Work For Pay Don’t Know Refused	S20=1

Debriefing

OK, those are the survey questions I wanted to get through. Now, I will go back and ask you some follow-up questions about what we just went through.

S18 Block

1. I am going to read back one of the questions and I would like for you to tell me, in your own words, what the question is asking:

S18	Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app. Does this describe any work you did last week?

2. [if S18=1] You said [PARTICIPANTS ANSWER] to that question. What company or companies did (you/NAME) work for?

3. [if S18=1] (Do you/Does NAME) consider working for [Company 1 through N] to be a job or something else? Please explain.

4. [if S18=1] Can you tell me more about the work (you do/NAME does) for [Company 1 through N]?

5. [if S18=1 for main respondent] Please describe what you do at the job so an average person could understand.

6. [if S18=1 for proxy respondent] How would NAME describe (that job/those jobs) to you?

7. [if S18=1] How many hours (do you/does NAME) typically work at (that job/those jobs) each week?

8. [if S18= 1] How many hours did you work at [Company 1 through N] last week?

9. [if S18=1] How long does it take to complete a task at [Company 1 through N]?

10. [if S18= 1] Do you decide when you work for [Company 1 through N]?

11. [if S18 = 2] Have you ever been paid for work that you found through these websites or apps?

Uber	Lyft
Grubhub	Taskrabbit
Instacart	Washio
Luxe	Postmates
Heal

• Yes

• No

12. [if 11 = “Yes”] When was the last time you were paid for work found through these websites or apps?

13. [if 12 = “Last week”] Earlier, when I asked about “short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app” you didn’t report this work. Can you tell me more about that?

S20 Block

14. I am going to read back another question and I would like for you to tell me, in your own words, what the question is asking:

S20	Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online. Does this describe any work you did last week?

15. [if S20=1] You said [PARTICIPANTS ANSWER] to that question. What company or companies did (you/NAME) work for?

16. [if S18=1] (Do you/Does NAME) consider working for [Company 1 through N] to be a job or something else? Please explain.

17. [if S20=1] Can you tell me more about (that job/those jobs)?

18. [if S20=1 for main respondent] Please describe what you do at the job so an average person could understand.

19. [if S20=1 for proxy respondent] How would NAME describe (that job/those jobs) to you?

20. [if S20=1] How many hours (do you/does NAME) typically work at (that job/those jobs) each week?

21. [if S20= 1] How many hours did you work at [Company 1 through N] last week?

22. [if S20= 1] How long does it take to complete a task at [Company 1 through N]?

23. [if S20= 1] Do you decide when you work for [Company 1 through N]?

24. [if S20= 1] What is a typical task like on [Company 1 through N]?

25. [if S20 = 2] Have you ever been paid for work done through any of the following sites/companies?

Amazon Mechanical Turk	CrowdFlower
99Designs	crowdSPRING
Freelancer	Hiretheworld
OneSpace	Crowdsource.com
Clickworker	Shorttask
Microworkers

• Yes

• No

26. [if 25 = “Yes”] When was the last time you completed work on that site?

27. [if 26 = “Last week”] Earlier, when I asked about “short, paid tasks through companies that maintain online lists of tasks” you didn’t report this work. Can you tell me more about that?

28. Last week, did you earn money from any other source?

    1. [if 28= “yes”] Can you tell me more about that?

29. What does the term “crowdsourcing” mean to you?

30. Would you describe any of the work you do as crowdsourcing?

31. Are you familiar with the term “gig economy”?

    1. [if 31 = “yes”] What is the “gig economy”?

    2. [if 31 = “yes”] Would you describe any of the work you do as part of the “gig economy”?

32. [if Uber not mentioned] Have you heard of the company Uber?

    1. Please describe that company to me

33. [if Mechanical Turk not mentioned] Have you heard of Amazon Mechanical Turk?

    1. Please describe that company to me

34. [if Grubhub not mentioned] Have you heard of Grubhub?

    1. Please describe that company to me

35. [if Taskrabbit not mentioned] Have you heard of Taskrabbit?

    1. Please describe that company to me

Appendix A, Part 2: Online Testing Protocol (mTurk)

For online testing, debriefing questions will focus on assessing the accuracy of responses to the target questions. The goal is to monitor the responses from a large number of online participants to detect irregular responses. This study will be listed on mTurk as a single task with an initial screening question. The screening criteria for the study will be listed in the study invitation. Potential participants will see the following information on the mTurk site.

OMB Control Number: 1220-0141

Expiration Date: 04-30-2018

Description:	We'll ask you questions related to work you have done for pay within the last week and then ask for your feedback on those questions.
Keywords:	memory, survey, research
Qualifications Required:	Participants in this study must have found paid tasks to complete through a service that links them to customers using a website or app not including Mechanical Turk. Examples include Uber, Lyft, Grubhub, Taskrabbit, Washio, CrowdFlower, etc. OR Participants in this study must have worked last week for pay.

If they click the link to the study, they will be asked the initial screening question.

If the participant’s answer fits the screening criteria for that module, they will be forwarded to a screen with the introductory text on the next page. If they choose to proceed, they will be asked several demographic questions, the focal questions for that module, and then the debriefing questions.

If the participant’s answer does not fit the screening criteria, they will see the following text.

“Thank you for your time, but you do not meet the qualifications for this study.”

Welcome! Thanks for your interest in our survey. You’re here because we have asked you to participate in our research. We are asking you and hundreds of other people to tell us about what they think.

Unlike some surveys or online tasks you may be familiar with, we ask that you complete this survey all at one time and that you only start once you are in a quiet place where you won't be disturbed. The survey will take no longer than 20 minutes to complete. Only share information you're comfortable with - nothing too personal - but please be honest and follow the instructions.

 Please do not use your browser's back button. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this study.

---page break---

On the following pages, you’ll be asked about work you have done for pay during the last week. We will then ask you some additional questions to better understand your answers.

Let’s get started!

---page break---

We’ll start by asking a few questions about you.

What is your age in years?

What is your gender?

• Male

• Female

What is the highest level of school you have completed or the highest degree you have received?

• Grades 1-12/No Diploma

• High School Diploma or Equivalent

• Some College

• Associate’s Degree

• Bachelor’s Degree

• Master’s Degree

• Doctorate or Professional Degree

Are you Hispanic or Latino?

• Yes

• No

What is your race?

• American Indian or Alaska Native

• Asian

• Black or African-American

• Native Hawaiian or Other Pacific Islander

• White

In which state do you live?

---page break---

1. The next few questions will ask about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___
   
   LAST WEEK, did you do ANY work for pay?

• Yes

• No

2. LAST WEEK, did you have a job either full or part time? Include any job from which you were temporarily absent.

• Yes

• No

3. LAST WEEK, did you have more than one job, including part-time, evening or weekend work?

• Yes

• No

4. How many hours per week do you USUALLY work at your main job?
   
   By main job we mean the one at which you USUALLY work the most hours.

5. How many hours per week do you USUALLY work at your other job(s)?

6. Were you employed by government, by a private company, a non-profit organization, or were you self-employed or working in a family business?

• Government

• Private company

• Non-profit organization incl. tax exempt and charitable organizations

• Self-employed

• Working in family business

7. What is the name of the (company)(non-profit organization) (business) for which you work?

8. What kind of business or industry is this?

9. What are your usual activities or duties at this job?
   (For example: typing, keeping account books, filing, selling cars, operating printing press, laying brick)

Supplement

1. Some people are in temporary jobs that only list for a limited time or until the completion of a project. Is your main job temporary?

• Yes

• No

2. Provided the economy does not change and your job performance is adequate can you continue to work for your current employer as long as you wish?

• Yes

• No

3. Are you working only until a specific project is completed?

• Yes

• No

4. Even though you told me your job is not temporary, are you paid by a temporary help agency?

• Yes

• No

5. Some people are in a pool of workers who are ONLY called to work as needed, although they can be scheduled to work for several days or weeks in a row, for example substitute teachers, and construction workers supplied by a union hiring hall. These people are sometimes referred to as on-call workers.
   
   Were you an on-call worker last week in your main job?

• Yes

• No

6. Some people get work by waiting at a place where employers pick up people to work for a day. These people are sometimes called day laborers.
   
   Were you a day laborer last week?

• Yes

• No

7. Some people provide employees or their services to others under contract. A few examples of services that can be contracted out include security, landscaping or computer programming.
   
   Did you work for a company that contracts out your services last week?

• Yes

• No

8. Are you usually assigned to more than one customer?

• Yes

• No

9. Do you usually work at the customer’s worksite?

• Yes

• No

10. Last week, were you working as an independent contractor, an independent consultant, or a freelancer (that is someone who obtains customers on their own to provide a product or service)?

• Yes

• No

I now have a few questions about how the Internet and mobile apps have changed the way some people work.

11. Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app.
    
    For example, driving a person from one point to another using your own car, delivering people’s laundry to a dry cleaner, or helping a person assemble furniture.
    
    Does this describe any work you did last week?

• Yes

• No

12. Was that for your main job, second job, or additional work for pay?

• Main Job

• Second Job

• Additional Work For Pay

13. Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online.
    
    For example, data entry, labeling photos, translating text, or other micro-tasks.
    
    Does this describe any work you did last week?

• Yes

• No

14. Was that for your main job, second job, or additional work for pay?

• Main Job

• Second Job

• Additional Work For Pay

Debriefing

Next, we will ask some questions about your experiences taking this survey.

S11 Block

One of the questions we asked was worded as follows

Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app.

Does this describe any work you did last week?

1. [if S11= “Yes”] You said yes to this question. What company or companies did you work for?

2. [if S11= “Yes”] Do you consider working for [Company 1 through N] to be a job or something else? Please explain.

3. [if S11= “Yes”] [Company 1 through N ] Please describe what you do at the job so an average person could understand.

4. [if S11= “Yes”] How many hours do you typically work at (that job/those jobs) each week?

5. [if S11= “Yes”] How many hours did you work at [Company 1 through N] last week?

6. [if S11= “Yes”] How long does it take to complete a task at [Company 1 through N]?

7. [if S11 = “Yes”] Do you decide when you work for [Company 1 through N]?

8. [if S11 = “No”] Have you ever been paid for work that you found through these websites or apps?

Uber	Lyft
Grubhub	Taskrabbit
Instacart	Washio
Luxe	Postmates
Heal

• Yes

• No

9. [if 8 = “Yes”] When was the last time you were paid for work found through these websites or apps?

S13 Block

Another question we asked was worded as follows:

Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online.

Does this describe any work you did last week?

10. [if S13= “Yes”] You said yes to this question. What company or companies did you work for?

11. [if S13= “Yes”] Do you consider working for [Company 1 through N] to be a job or something else? Please explain.

12. [if S13= “Yes”] [Company 1 through N] Please describe what you do at the job so an average person could understand.

13. [if S13= “Yes”] How many hours do you typically work at [Company 1 through N] each week?

14. [if S13= “Yes”] How many hours did you work at [Company 1 through N] last week?

15. [if S13= “Yes”] How long does it take to complete a task at [Company 1 through N]?

16. [if S13= “Yes”] Do you decide when you work for [Company 1 through N]?

17. [if S13= “Yes”] What is a typical task like on [Company 1 through N]?

18. [if S13 = “No”] Have you ever been paid for work done through any of the following sites/companies?

Amazon Mechanical Turk	CrowdFlower
99Designs	crowdSPRING
Freelancer	Hiretheworld
OneSpace	Crowdsource.com
Clickworker	Shorttask
Microworkers

• Yes

• No

19. [if 18 = “Yes”] When was the last time you completed work on that site?

General Questions

20. Last week, did you earn money from any other source?

2. [if 20= “Yes”] From what other sources did you earn money last week?

Appendix A, Part 3: Online Testing Protocol (TryMyUI)

For online testing, debriefing questions will focus on assessing the accuracy of responses to the target questions. The goal is to monitor the responses from a large number of online participants to detect irregular responses. As with mTurk, the screening criteria for the study will be listed in the study invitation. Potential participants will see the following information on the TryMyUI site.

NOTE: This protocol is very similar to the mTurk protocol (Appendix A, Part 2). The main difference is in instructions that encourage the participant to think aloud as they are answering the questions. TryMyUI records audio from the participants’ computers.

OMB Control Number: 1220-0141

Expiration Date: 04-30-2018

Description:	We'll ask you questions related to work you have done for pay within the last week and then ask for your feedback on those questions.
Qualifications Required:	Participants in this study must have found paid tasks to complete through a service that links them to customers using a website or app not including TryMyUI. Examples include Uber, Lyft, Grubhub, Taskrabbit, Washio, Amazon Mechanical Turk, CrowdFlower, etc.

All participants who believe they meet the outlined qualifications will be allowed to participate in the study.

Welcome! Thanks for your interest in our survey. You’re here because we have asked you to participate in our research.

These questions might be used in a future survey, so we want them to be clear and easy

to answer.

Please remember to talk out loud as you answer each question, and mention anything that

confuses you or which you find difficult to answer.

The survey will take no longer than 20 minutes to complete. Only share information you're comfortable with - nothing too personal - but please be honest and follow the instructions.

After you have answered the survey questions, a section will appear with some Follow-up Questions. These questions are not survey questions. Instead, they ask for additional information to help us better understand how you arrived at your answer. You answer these questions by talking out loud. If you already provided this information when you were answering the survey question, you don't have to provide it again. Just skip to the next follow-up question.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this study.

---page break---

On the following pages, you’ll be asked about work you have done for pay during the last week. We will then ask you some additional questions to better understand your answers.

Let’s get started!

---page break---

1. The next few questions will ask about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___
   
   LAST WEEK, did you do ANY work for pay?

• Yes

• No

2. LAST WEEK, did you have a job either full or part time? Include any job from which you were temporarily absent.

• Yes

• No

3. LAST WEEK, did you have more than one job, including part-time, evening or weekend work?

• Yes

• No

4. How many hours per week do you USUALLY work at your main job?
   
   By main job we mean the one at which you USUALLY work the most hours.

5. How many hours per week do you USUALLY work at your other job(s)?

6. Were you employed by government, by a private company, a non-profit organization, or were you self-employed or working in a family business?

• Government

• Private company

• Non-profit organization incl. tax exempt and charitable organizations

• Self-employed

• Working in family business

7. What is the name of the (company)(non-profit organization) (business) for which you work?

8. What kind of business or industry is this?

9. What are your usual activities or duties at this job?
   (For example: typing, keeping account books, filing, selling cars, operating printing press, laying brick)

Supplement

1. Some people are in temporary jobs that only list for a limited time or until the completion of a project. Is your main job temporary?

• Yes

• No

2. Provided the economy does not change and your job performance is adequate can you continue to work for your current employer as long as you wish?

• Yes

• No

3. Are you working only until a specific project is completed?

• Yes

• No

4. Even though you told me your job is not temporary, are you paid by a temporary help agency?

• Yes

• No

5. Some people are in a pool of workers who are ONLY called to work as needed, although they can be scheduled to work for several days or weeks in a row, for example substitute teachers, and construction workers supplied by a union hiring hall. These people are sometimes referred to as on-call workers.
   
   Were you an on-call worker last week in your main job?

• Yes

• No

6. Some people get work by waiting at a place where employers pick up people to work for a day. These people are sometimes called day laborers.
   
   Were you a day laborer last week?

• Yes

• No

7. Some people provide employees or their services to others under contract. A few examples of services that can be contracted out include security, landscaping or computer programming.
   
   Did you work for a company that contracts out your services last week?

• Yes

• No

8. Are you usually assigned to more than one customer?

• Yes

• No

9. Do you usually work at the customer’s worksite?

• Yes

• No

10. Last week, were you working as an independent contractor, an independent consultant, or a freelancer (that is someone who obtains customers on their own to provide a product or service)?

• Yes

• No

I now have a few questions about how the Internet and mobile apps have changed the way some people work.

11. Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app.
    
    For example, driving a person from one point to another using your own car, delivering people’s laundry to a dry cleaner, or helping a person assemble furniture.
    
    Does this describe any work you did last week?

• Yes

• No

12. Was that for your main job, second job, or additional work for pay?

• Main Job

• Second Job

• Additional Work For Pay

13. Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online.
    
    For example, data entry, labeling photos, translating text, or other micro-tasks.
    
    Does this describe any work you did last week?

• Yes

• No

14. Was that for your main job, second job, or additional work for pay?

• Main Job

• Second Job

• Additional Work For Pay

Follow-Up Questions

Next, we will ask some follow-up questions about your experiences taking this survey. Please remember to speak aloud.

S11 Block

One of the questions we asked was worded as follows

Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app.

Does this describe any work you did last week?

1. [if S11= “Yes”] You said yes to this question. What company or companies did you work for?

2. [if S11= “Yes”] Do you consider working for [Company 1 through N] to be a job or something else? Please explain.

3. [if S11= “Yes”] [Company 1 through N ] Please describe what you do at the job so an average person could understand.

4. [if S11= “Yes”] How many hours do you typically work at (that job/those jobs) each week?

5. [if S11= “Yes”] How many hours did you work at [Company 1 through N] last week?

6. [if S11= “Yes”] How long does it take to complete a task at [Company 1 through N]?

7. [if S11 = “Yes”] Do you decide when you work for [Company 1 through N]?

8. [if S11 = “No”] Have you ever been paid for work that you found through these websites or apps?

Uber	Lyft
Grubhub	Taskrabbit
Instacart	Washio
Luxe	Postmates
Heal

• Yes

• No

9. [if 8 = “Yes”] When was the last time you were paid for work found through these websites or apps?

S13 Block

10. Another question we asked was worded as follows:

Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online.

Does this describe any work you did last week?

11. [if S13= “Yes”] You said yes to this question. What company or companies did you work for?

12. [if S13= “Yes”] Do you consider working for [Company 1 through N] to be a job or something else? Please explain.

13. [if S13= “Yes”] [Company 1 through N] Please describe what you do at the job so an average person could understand.

14. [if S13= “Yes”] How many hours do you typically work at [Company 1 through N] each week?

15. [if S13= “Yes”] How many hours did you work at [Company 1 through N] last week?

16. [if S13= “Yes”] How long does it take to complete a task at [Company 1 through N]?

17. [if S13= “Yes”] Do you decide when you work for [Company 1 through N]?

18. [if S13= “Yes”] What is a typical task like on [Company 1 through N]?

19. [if S13 = “No”] Have you ever been paid for work done through any of the following sites/companies?

Amazon Mechanical Turk	CrowdFlower
99Designs	crowdSPRING
Freelancer	Hiretheworld
OneSpace	Crowdsource.com
Clickworker	Shorttask
Microworkers

• Yes

• No

20. [if 19 = “Yes”] When was the last time you completed work on that site?

General Questions

21. Last week, did you earn money from any other source?

1. [if 21= “Yes”] Can you tell me more about that?

22. What does the term “crowdsourcing” mean to you?

23. Would you describe any of the work you do as crowdsourcing?

Appendix B: Recruitment Script and Screening

Participants for this study will be recruited from those who call the OSMR offices in response to an advertisement (see Appendix C). Several questions asked to ensure that at least some will be of the participants who come in to the lab will have experience with topics asked about in the study. With the questions below, we are interested in finding participants who have some experience doing work related to these electronic mediation sources. We will also recruit 5 participants from a database maintained by BLS. These participants will not be asked screening questions.

OMB Control Number: 1220-0141

Expiration Date: 04-30-2018

Thank you for calling. I am recruiting for a study by the Bureau of Labor Statistics that asks people to. We will ask you questions about your employment situation and then ask some follow-up questions about your answers. We need to talk to people who have certain types of work arrangements. I have a few questions that will help us see if you are eligible to participate in this study. OK?

1. How many people are in your household, including yourself? _____

2. Are you currently employed? YES NO

   1. (if no) Are you currently looking for work? YES NO

3. In the last week, have you or anyone in your household found short, paid jobs or tasks through an Internet site or mobile app? YES NO

   1. (if yes) What is the name of the website or app?

Uber	Lyft
Grubhub	Taskrabbit
Washio	Instacart
Amazon Mechanical Turk	CrowdFlower
Crowdsource.com	Clickworker
Microworkers	Crowdsource.com
Upwork	OneSpace

If participant meets eligibility requirements:

You are eligible to participate in this study. Let’s get you scheduled.

If participant does not meet eligibility requirements:

I’m sorry but you are not eligible to participate in this study. If you would like, I can get your contact information and we will keep you in mind for future studies.

Thank you for your time.

If the person requests to know why they were not eligible. The recruiter will explain the eligibility requirements.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 10 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to the U.S. Department of Labor, OASAM/OCIO 200 C Street NW Washington DC 20210 and reference the OMB Control Number 1220-0141.

Appendix C: Recruiting Advertisement

Research Participants Needed

The U.S. Bureau of Labor Statistics is looking for:

Individuals that currently find work through any of the following websites or mobile apps:

Uber	Lyft
Grubhub	Taskrabbit
Washio	Instacart
Amazon Mechanical Turk	CrowdFlower
Crowdsource.com	Clickworker
Microworkers	Crowdsource.com
Upwork	OneSpace

to help develop questions that will be used in a national survey.

$40 for a 1 hour session

Call (202) 691-7524

For more information

Sessions will be conducted at the Bureau of Labor Statistics office near Union Station (2 Massachusetts Ave., Washington, DC 20212).

Appendix D: Consent Form and Privacy Act Statement

CONSENT FORM

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you up to 60 minutes to participate in this research.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires 04-30-2018.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.