Cognitive Testing Report

Current Population Survey Program

Electronically Mediated Work Question Cognitive Testing

Brandon Kopp & Jennifer Edgar

Bureau of Labor Statistics

September 30, 2016

Executive Summary

This report outlines the results of a study to test four questions related to two types of electronically mediated work that will be added to the May 2017 Contingent Worker Supplement to the Current Population Survey. These two types of work include arrangements (1) where a worker is matched with a customer through an Internet site or smartphone application, is paid through the company that runs that site or application, and performs an in-person service (e.g., Uber) and (2) where a worker finds short duration tasks, ranging from several minutes to several hours, through an online service and the tasks are performed entirely online (e.g., Upwork).

The draft questions, which were developed by BLS staff in consultation with outside stakeholders, were tested using a combination of 24 in-lab cognitive interview sessions and online data collection, through Amazon Mechanical Turk (mTurk), from 138 survey participants. Although both types of testing indicated that the questions generally measured what they were intended to measure, the testing also uncovered several issues with the questions:

1. Participants included sites like Craigslist as an applicable example of both In-Person and Online Tasks.

2. Participants in both modes included jobs that have a high degree of interaction with technology, but not necessarily a job platform, as being included in the In-Person Task question. Specifically, participants who find clients through social media and participants who work for businesses where customers can place their orders through apps or websites thought the In-Person Task question applied to their situation.

3. Several participants who are employed by a company to do data entry believed that the Online question applied to them.

4. Many participants in the mTurk testing reported Online Task platforms (e.g., mTurk) as part of their answer to the In-Person Task question.

5. Many mTurk participants did not include their experience with mTurk as part of the Online Task question though this may be due to the fact that they were tested on the mTurk platform and assumed that that experience should be excluded.

6. Participants had some difficulty differentiating between second job and additional work for pay.

To address these concerns we proposed the following changes to the questions. It should be noted that these recommendations have not been tested so it is not clear whether they will resolve any of the issues mentioned above.

1. There should be a clearer distinction between the In-Person and Online Task questions. Much of the rest of the wording in the question is the same, the key distinction between the two is that in one participants perform tasks outside of their home and usually at the location of the person for whom they are performing the service and in the other the tasks are accomplished entirely online. This can be accomplished by emphasizing these words in the question.

2. In order to prevent people from reporting Online Tasks during the In-Person Task question, participants should be informed that they will have the option to report for both. This can be accomplished through introductory language to the questions.

3. To reduce the underreporting of paid activities that participants think of as “not a job,” emphasis should be given to the fact that we are interested in ANY work that they did during the reference period. It should be noted that the basic CPS questions inquiring about work already includes this emphasis.

4. To help people better differentiate between traditional delivery and transportation jobs and the electronically mediated work we are attempting to measure with the In-Person Task question, the examples should emphasize the use of one’s own car by moving it to the beginning of the example sentence.

5. The examples should be reworded so that they better illustrate the types of jobs each question is attempting to capture.

Final question recommendations are provided in Section 4.1.

1. Introduction

The Contingent Worker Supplement (CWS) to the Current Population Survey (CPS) was last conducted in 2005 but is scheduled to be refielded in May 2017. Contingent workers are those workers that do not expect their job to last or reported that their job is temporary. The supplement also asks about a number of alternative employment arrangements, including on-call workers, independent contractors, and temporary help agency workers. In order to maintain data comparability over time, the questionnaire will be largely the same as that used in 2005. However, new types of work have emerged since 2005, such as individuals obtaining customers or online tasks through companies that match them, often through a mobile app. For this reason, BLS, with input from CPS stakeholders, has proposed the addition of four questions to the May 2017 CWS to capture these electronically mediated work arrangements.

There are two types of electronic mediation that the newly proposed questions are meant to capture. The first are arrangements where a worker is matched with a customer through an Internet site or smartphone application, is paid through the company that runs that site or application, and performs an in-person service. Examples of this type of arrangement are people who use their own cars drive customers from one place to another, after being matched with riders through an application (such as Uber), or workers who pick up the customer’s laundry and deliver it to a laundry service (matched through a company such as Washio). In both of these arrangements, the worker is electronically matched to a customer, chooses when and whether they work, and performs the service for the customer in-person.

In the second type of electronically mediated work, the worker finds short duration tasks, ranging from several minutes to several hours, through an online service, and the tasks are performed entirely online. Workers may be asked to design a logo for a business given pre-defined specifications (through a company such as 99designs.com) or to add descriptive keywords to photos or documents (through a company such as Amazon Mechanical Turk). In these types of arrangements, the worker finds tasks listed on a website, chooses which to participate in, completes and/or submits their work through the website, and is then paid through the service based on a rate set by the person who initiated the task.

CPS stakeholders and BLS staff are interested in measuring how many workers are engaged in these types of arrangements as their main job, their secondary job, or as a source of secondary earnings. It should be noted, however that questions regarding these work arrangements will only be asked of those who reported having worked for pay during the previous week.

Because of the need to keep the survey questionnaire largely the same as that used in 2005 and the need to limit respondent burden, the number of questions that can be added to the CWS to measure these concepts is limited to four. With input from CPS stakeholders, industry experts, and academics, BLS staff drafted four possible questions to measure the prevalence of electronically mediated work. Given the complexity of the concepts and the small number of questions, cognitive testing is especially important. The purpose of this testing was to:

1. Understand the measurement properties of the electronic mediation questions. That is, do the proposed questions maximize the number of true positives (reporting “yes” when engaged in one of the work arrangements described above) and minimize the number of false positives (reporting “yes” when not engaged in one of the work arrangements described above)?

2. Test minor wording changes to questions that BLS initially developed and obtained feedback on from stakeholders, industry experts, and academics.

3. Determine if introductory or transition language is necessary between the current set of CWS questions and the electronic mediation questions.

4. Determine what interviewer instructions or help screens are necessary to explain the key concepts.

Testing the original set of CWS questions was out of scope for this project as those questions were to be implemented without change from the 2005 instrument.

2. Research Design

To accomplish these goals, question variations were tested using a mixture of in-lab cognitive interviewing and online testing. The in-lab cognitive interviews will more closely match the CPS interview format (interviewer directed and often in-person) and the online testing will allow us to expand recruiting outside the Washington, DC area and recruit a larger number of individuals involved in electronically mediated work. All of the protocols are in Appendix A.

1. Cognitive Interviews

   1. Procedures. Cognitive interviews provide an in-depth understanding of the respondents’ thought processes and reactions to the questions. These interviews were done one-on-one in the Office of Survey Methods Research (OSMR) laboratory at the Bureau of Labor Statistics (BLS). Participants were asked survey questions from the Current Population Survey and the Contingent Worker Supplement to provide context for the electronic mediation questions. After being asked the electronic mediation questions, participants were asked debriefing questions to better understand the accuracy of their reports and the thought process used to formulate their responses. Participants were paid $40 for their participation.

   2. Participants. Twenty-four participants were recruited to participate in this study. To recruit for the in-lab cognitive tests, advertisements were placed on Craigslist. Each ad specified either a list of companies the participants must work for (e.g., Uber or Lyft) or certain professions (e.g., taxi drivers). These ads were supplemented by fliers handed out at a local taxi stop and a pizza restaurant. Those who answered the fliers and ads were asked several screening questions to determine whether they worked for the industries we were interested in capturing. Table 1 shows the areas we recruited for and the number of participants who identified as working for them.

Table 1: The number of participants who reported electronically mediated work, or not, by industry.

	Food Delivery	Transportation	Freelance
	(e.g., Grubhub or Postmates)	(e.g., Uber or Lyft)	(e.g., Taskrabbit)
Electronically Mediated	3	7	3
	(e.g., pizza or Chinese food delivery)	(e.g., taxi drivers)	(e.g., people who find temporary work on sites like Craigslist)
Not Electronically Mediated	3	4	4

Participants in the cognitive interviews were between the ages of 25 and 45. Figure 1 shows the distribution of participants’ ages by whether they reported an electronically mediated job or not (i.e., reported “Yes” for themselves on either of the electronic mediation question). Due to the small sample sizes, there is no clear differentiation between these groups in terms of age.

Figure 1: Distribution of participants’ ages by whether they reported electronically mediated work.

Participants in the cognitive testing also had a range of educational attainment. Those who reported having found work through an electronically mediated platform during the previous week tended to have a higher education than those who did not report this time of work.

Table 2: Educational attainment by whether a participant reported finding work through an electronically mediated platform.

Educational Attainment	Not Electronically Mediated (n = 10)	Electronically Mediated (n=14)
Grades 1-12/No Diploma	20%	0%
High School Diploma or Equivalent	20%	29%
Some College	40%	29%
Associate Degree	0%	7%
Bachelor's Degree	20%	21%
Master's Degree	0%	7%
Doctorate or Professional Degree	0%	7%

2. Mechanical Turk Testing¹

   1. Procedures. These questions were also tested using a sample of participants recruited through Amazon Mechanical Turk (mTurk) service. These participants completed an online form which asked them an abbreviated set of CPS and CWS questions, the electronic mediation questions, and several debriefing questions, including questions to probe the accuracy of their responses to the electronic mediation questions and about their experience answering those questions. This online testing is an unmoderated cognitive interview. The mTurk platform was especially useful in this context because (1) mTurk workers are the type of workers we are attempting to capture with the electronic mediation questions and (2) it allow us to achieve greater geographic diversity in our testing sample. Participants were paid $2 for an estimated 20 minute study. This is a commonly used pay rate for mTurk studies.

2. Participants. All participants were asked an initial screener question (see below) that was used to keep track of our quotas for different categories of work (online/offline, in-person/online tasks). Seven hundred twenty people completed the screener for this study.

“LAST WEEK, Did you do any work for pay at these places? Please mark all that apply.²

• Grubhub, Postmates, Instacart, or Favor

• Uber or Lyft

• Taskrabbit, Handy, or Washio

• Upwork, Crowdflower, Clickworker, Crowdsource.com, Onespace, or Microworkers

• Any restaurant or grocery store delivery food

• Any taxi company OTHER than Uber or Lyft

• Angieslist.com, Care.com, or Craigslist

• No, I did not do any work for pay last week at these places”

If the participant selected one of the following categories they were considered to have answered “Yes” to an “In-Person Task” Screener:

• Grubhub, Postmates, Instacart, or Favor

• Uber or Lyft

• Taskrabbit, Handy, or Washio

If the participant selected the following category, they were considered to have answered “Yes” to an “Online Task” Screener. NOTE: mTurk was not included in this list.

• Upwork, Crowdflower, Clickworker, Crowdsource.com, Onespace, or Microworkers

Of the 720 participants who completed the screener, 175 were selected to participate in the study³. Of those selected to participate, 17 did not complete the target electronic mediation questions.⁴ Finally, 20 participants were located outside of the United States and were excluded.⁵ The analyses below are based on the 138 participants from the United States who completed those questions.

The numbers in Table 2 do not sum to 138 because many participants recorded more than one answer for the screener question. The 113 participants who reported working for pay in at least one of the listed categories the week prior to the study reported work in, on average, 2.2 categories. The remaining 25 participants reported not working at any of the listed categories the previous week. These participants were purposefully selected into the study in order to see if they would report their experience with mTurk without a prompt.

Table 3: The number of participants who reported electronically mediated work, or not, by industry in the mTurk Study.

	Food Delivery	Transportation	Freelance	Online
	(e.g., Grubhub or Postmates)	(e.g., Uber or Lyft)	(e.g., Taskrabbit)	(e.g., Upwork)
Electronically Mediated	31	47	26	55
	(e.g., pizza or Chinese food delivery)	(e.g., taxi drivers)	(e.g., people who find temporary work on sites like Craigslist)	NA
Not Electronically Mediated	36	18	35

More than half of the participants in the mTurk sample were between 25 and 34. The age distributions for those who participated in electronically mediated work and those who did not are similar. This is likely due to the fact that all of the participants have experience with mTurk and are thus a select group of people already.

Figure 2: Distribution of participants’ ages by whether they reported electronically mediated work.

Similar to the cognitive testing sample, participants who reported some type of electronically mediated work in the mTurk sample tended to have a higher education level than those who did not report such work.

Table 4: Educational attainment by whether a participant reported finding work through an electronically mediated platform.

Educational Attainment	Not Electronically Mediated (n = 47)	Electronically Mediated (n = 92)
High School Diploma or Equivalent	19%	7%
Some College	30%	24%
Associate Degree	13%	11%
Bachelor's Degree	32%	45%
Master's Degree	6%	13%
Doctorate or Professional Degree	0%	1%

One of the advantages of using a platform like mTurk is obtaining greater geographic diversity than we can get through a study conducted in the Washington, DC area. Figure 3 shows that participants were spread throughout most of the country (with the noticeable exception of the Plains states). It also shows that participants tended to be from larger cities like Detroit, Chicago, and Dallas.

Figure 3: Geographic distribution of mTurk participants by whether they reported engaging in electronically mediated work during the previous week or not.

3. Results

This section will discuss the results of the testing for the four target questions organized by question.

1. In-Person Tasks

The first question (displayed below) in the new series on electronic mediation of work focuses on work for companies that have developed a computerized platform (either through a smartphone application or a website) that brings together people who need an in-person service (e.g., food delivery, a ride from one place to another) with people who want to perform that service. The company that owns the platform also handles the payment from the customer and disburses money to the worker.

A number of such services exist today. The quintessential example of this type of company is Uber, a ride sharing service, but other arrangements exist as well. These types of electronic platforms exist for food delivery, household maintenance and cleaning, dry cleaning, even in-home medical care.

Introduction	I now have a few questions about how the Internet and mobile apps have changed the way some people work.
In-Person	Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app. For example, driving a person from one point to another using your own car, delivering people’s laundry to a dry cleaner, or helping a person assemble furniture. Does this describe any work you did last week?	Yes No

1. Cognitive Testing. Overall, the In-Person Task question performed well during the cognitive interviews. Tables 3 and 4 present the results for this question for participants’ self-response and their proxy responses, respectively. The rows in these table denote whether the participant replied “Yes” or “No” when asked the In-Person Task question and the columns denote whether or not they reported qualifying (in-scope) work during the debriefing. All but four participants accurately reported having found work through an In-Person platform during the past week or not.⁶

There were four false positives in self-reported in person, electronically mediated work, out of a total of 14 instances reported. Two of these participants reported “Yes” to this question because they believed Craigslist qualified as an in-scope platform. It does not qualify because participants are not paid through the website. Rather they are paid in person. One participant has several home businesses including cosmetics, hair styling, and photography. She reported “Yes” to this question because she occasionally finds clients through online platforms like Craigslist or social media. Finally, one participant reported “Yes” to this question because he finds research studies through various online and offline sources and attends them in-person.

Table 5: Cross-tabulation table showing participants’ answer to the In-Person Task question for him or herself and whether they reported work that meets the definition of the question.

	Reported in-scope “In-Person” work
Answer to In-Person Task Question	Yes	No
Yes	10	4
No	0	10

For proxy reporting there were two false positive reports, out of a total of 3 instances reported. One participant when reporting for himself reported in-scope work (Uber) and out-of-scope work (a home business). He reported “Yes” to the In-Person Task question for his son because his son occasionally worked with him at the home business. The other participant was the same person who, as a self-report, said that he finds research studies and attends in-person. He has a roommate that does this also.

Table 6: Cross-tabulation table showing participants’ answer to the In-Person Task question for other members of his or her household and whether they reported work that meets the definition of the question.

	Reported in-scope “In-Person” work
Answer to In-Person Task Question	Yes	No
Yes	1	2
No	0	21

During the debriefing, the question was reread to participants and they were asked to describe what kind of work that question is asking about. All but 4 of the participants were able to give an adequate definition or name one company that meets the definition (often Uber). Many of these same participants also included in their definition sites like Craigslist or businesses like Giant’s Peapod service, mainly because customer interactions with the business are through an app or website. None of the participants worked for a grocery delivery service like Peapod.

2. mTurk Testing. The In-Person Task question did not perform as well in mTurk Testing. Of the 57 people who selected “Yes” on the In-Person Task question, 18 (32 percent) did not provide an in-scope work description during the debriefing.

There were 18 false positive reports in the mTurk Testing, out of 49 instances reported. Seven of these reports were due to participants reporting their experience with Mechanical Turk.⁷ Similar to the cognitive interviews, two participants listed Craigslist as in-scope work. Five participants listed traditional jobs at taxi companies, restaurants, or an in-home computer repair business. These participants described their jobs as involving some sort of online means of being connected to their customers (e.g., customers could make orders via an app or website). The remaining four participants reported a home business, volunteer activities, mistakenly answering “Yes” to the question, and that the question about where they work was not applicable.

There were also 13 false negative reports, out of 81 reports of having not done any In-Person Tasks. This is based on a follow-up question that provided participants with a list of In-Person task platforms and asked them if they had ever worked for any of those companies. If they responded “Yes,” they were asked when they last worked at that company. Eight listed exact dates that were within the reference period of the survey. Five others listed only a month and year that overlapped with the reference period. Those five are included to provide a conservative estimate of question performance.

Table 7: Cross-tabulation table showing participants’ answer to the In-Person Task question for him or herself and whether they reported work that meets the definition of the question.

	Reported in-scope “In-Person” work
Answered to In-Person Task Question	Yes	No	Not Sure8
Yes	31	18	8
No	13	68	0

2. Online Tasks

Another one of the new electronic mediation questions (displayed below) focused on work that is done entirely online. These are often referred to as micro-tasks because they are usually presented to workers in short chunks lasting only a short time. Similar to the In-Person Task platforms, the Online Task platforms connect people or businesses who need a task done (e.g., data transcription) with workers who will perform those tasks. Workers are usually paid on a per-task basis and the Online Task platform is responsible for collecting the payments from those ordering the task and dispersing them to workers. Examples of this type of platform are Amazon Mechanical Turk, Upwork, and Clickworker.

Online

Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online. For example, data entry, labeling photos, translating text, or other micro-tasks. Does this describe any work you did last week?

Yes No

1. Cognitive Testing. There were very few affirmative responses to the Online Task questions during the in-lab cognitive testing. We did not specifically recruit people who found work through Online platforms because, with limited interviews, we felt we would gain more insight about Online platforms from the mTurk Testing. Tables 6 and 7 show the results for the Online Task question for self-reported responses and proxy responses, respectively. Nearly all participants reported true negative responses as determined through the debriefing (see Footnote 5).

The only two participants who reported personally doing Online Tasks made false positive responses. Both of these participants also made false positive reports for the In-Person Task question. One participant reported “Yes” for the Online Task question because he found clients for his home business online. Another reported “Yes” because some of the studies he finds through Craigslist or other means require him to fill out an online form. During the debriefing it was determined that he does not work for any of the platforms considered in-scope for this question.

Table 8: Cross-tabulation table showing participants’ answer to the Online Task question for him or herself and whether they reported work that meets the definition of the question.

	Reported in-scope “Online” work
Answer to Online Task Question	Yes	No
Yes	0	2
No	0	22

For the proxy reports, the same two participants gave non-optimal answers. One participant said his roommate also completes online surveys as part of studies he finds through Craigslist or other means. The other reported “Don’t Know” when asked whether his son has found work through an Online Task platform.

Table 9: Cross-tabulation table between participants’ answer to the Online Task question for other members of his or her household and whether they reported work that meets the definition of the question.

	Reported in-scope “Online” work
Answer to Online Task Question	Yes	No
Yes	0	1
No	0	22
Don’t Know	0	1

2. mTurk Testing. The Online Task question had mixed results amongst the mTurk participants. Sixty-four percent of participants accurately classified their work experience during the reference period. There were very few false positives. The false positives that were found were based on Craigslist, traditional data entry jobs, and one wrote in “None.”

This question had a significant problem with false negative reporting, that is participants who said “No” to the Online Task question but had in fact worked for an Online Task platform during the reference period. In particular, these participants did not include their experience working for Amazon Mechanical Turk during the previous week. As mentioned earlier, five of these participants had listed Mechanical Turk for the In-Person Task question but did not report it again for the Online Task question (two reported it for both). Participants’ responses as to why they did not include this work are in Table 9. Mostly, participants did not think it was included. This is possibly a testing effect; that is, since we conducted the test on Mechanical Turk, participants may have thought that we would not want them to include it. Several participants also reported they do not think of it as work or a job.

Table 10: Cross-tabulation table showing participants’ answer to the In-Person Task question for him or herself and whether they reported work that meets the definition of the question.

	Reported in-scope “Online” work
Answered to Online Task Question	Yes	No	Not Sure9
Yes	77	4	15
No	31	11

Table 11: Participants responses when asked why they had not included Mechanical Turk when answering the Online Task question.

• I don't see this as a job or primary form of income

• I didn't realize that Mechanical Turk was included

• I think I misunderstood the question originally

• I made a mistake in the clicking of the bubbles

• I thought this was only if that was your primary job. I did not think the survey was asking in general and thought they were asking if you did any short tasks as a primary job, not in addition too your primary job.

• misunderstood the question

• I misunderstood what the question was asking. When I reviewed it further I realized that it was referring to the type of work offered through Mechanical Turk. I apologize for the discrepancy.

• I did not know mTurk was included.

• because I dont really count my turking as a source of income, I do it when I have free time, and I have not used the income from it on anything yet.

• I didn't read it correctly.

• I did not think of Mturk when I originally read the question. Only when I saw it listed on the list the 2nd time the question was asked did it come to mind for me.

• I could have sworn I marked I checked Yes. Perhaps I was confused.

• I assumed the question referred to a company which facilitated payment between customer and business.

• I did not understand that question at first.

• I did not know that Mechanical Turk was included on the list of jobs.

• I did not completely understand the wording of the question I suppose. I don't recall answering no.

• I misread the question

3. Which Job

If a respondents responds “Yes” to either the In-Person or Online Task questions, they are asked whether they consider the work they do on that platform their main job, their second job, or some other work they do for pay (question displayed below).

In the context of the CPS interview, “main job” and “second job” have a distinct meaning that is made clear through the questions. In both the cognitive interviews and mTurk testing, a series of questions were asked about the participant’s main job, but no questions were asked about their second job.

S19

Was that for your (main) job, (second job), or additional work for pay?

Main Job Second Job Additional Work For Pay

1. Cognitive Testing. Participants’ responses were evenly spread across the response options (see Tables 10 and 11). The cognitive interviews did not specifically probe about participants’ answers to this question to determine their accuracy or the thought process that was used to arrive at an answer. Interviewers did probe when participants displayed obvious difficulty answering this question. This happened for numerous jobs. These participants were unsure whether to count their, or their household member’s, work for a platform as a second job or additional work for pay because they did not necessarily view that work as a formal job.

Three participants reported finding work through an In-Person platform as their main job. This is verifiable through their answers to the CPS. Two of the three did report their jobs (one at Uber, one at Postmates) as their main job during the CPS interview. The third participant reported working for a taxi service as his main job, but also drives his personal car for Uber and Lyft. It is not clear whether he considered “driving” to be his main job or whether he was confused about the meaning of the response options to the question.

Table 12¹⁰: Frequency of participants’ responses to the Which Job question following the question on In-Person Tasks.

	All11	Worked In-Person Job12
Main Job	5	3
Second Job	4	4
Additional Work For Pay	5	4

Table 13: Frequency of participants’ responses to the Which Job question following the question on Online Tasks. ¹³

	All	Worked Online Job
Main Job	1	0
Second Job	0	0
Additional Work For Pay	1	0

2. mTurk Testing. Unlike the cognitive interview participants, most of the mTurk participants considered electronically mediated work to be additional work for pay. This was especially prevalent amongst those who found work through an Online platform where 70 percent of participants selected “Additional Work for Pay.” Table 12 and 13 show the distribution of participants’ answers to these questions.

There was very low item response to the CPS question asking participants the name of the company they work for so, unlike in the cognitive interviews, it is not possible to verify whether participants who reported the electronically mediated work as their main job did in fact report it as their job during the CPS.

Table 14: Frequency of participants’ responses to the Which Job question following the question on In-Person Tasks.

	All	Worked In-Person Job
Main Job	18	7
Second Job	14	8
Additional Work For Pay	25	16

Table 15: Frequency of participants’ responses to the Which Job question following the question on Online Tasks.

	All	Worked Online Job
Main Job	12	7
Second Job	22	16
Additional Work For Pay	62	54

4. Conclusion

Both the cognitive testing and the mTurk testing uncovered several issues with the questions. The main findings are that:

1. Participants include sites like Craigslist as an applicable example of both In-Person and Online Tasks.

2. Participants in both modes included jobs that have a high degree of interaction with technology, but not necessarily a job platform, as being included in the In-Person Task question. Specifically, participants who find clients through social media and participants who work for businesses where customers can place their orders through apps or websites thought the In-Person Task question applied to their situation.

3. Several participants who are employed by a company to do data entry believed that the Online question applied to them.

4. Participants in the mTurk testing reported Online Task platforms as part of their answer to the In-Person Task question.

5. Many mTurk participants did not include their experience with mTurk as part of the Online Task question though this may be due to the fact that they were tested on the mTurk platform and assumed that that experience should be excluded.

6. Participants had some difficulty differentiating between second job and additional work for pay.

1. Recommendations

Overall, despite the above issues, the questions generally captured the intended information. To address the concerns identified, we propose the following changes to the questions. It should be noted that these recommendations have not been tested so it is not clear whether they will resolve any of the issues mentioned above.

1. There should be a clearer distinction between the In-Person and Online Task questions. Much of the rest of the wording in the question is the same; the key distinction between the two is that in one participants perform tasks outside of their home and usually at the location of the person for whom they are performing the service and in the other the tasks are accomplished entirely online. This can be accomplished by emphasizing these words in the question.

2. In order to prevent people from reporting Online Tasks during the In-Person Task question, participants should be informed that they will have the option to report for both. This can be accomplished through introductory language to the questions.

3. To reduce the underreporting of paid activities that participants think of as “not a job,” emphasis should be given to the fact that we are interested in ANY work that they did during the reference period. It should be noted that the basic CPS questions inquiring about work already includes this emphasis.

4. To help people better differentiate between traditional delivery and transportation jobs and the electronically mediated work we are attempting to measure with the In-Person Task question, the examples should emphasize the use of one’s own car by moving it to the beginning of the example sentence.

5. The examples should be reworded so that they better illustrate the types of jobs each question is attempting to capture.

The recommended wording based on discussions with BLS staff about the above results is:

Intro	I now have a few questions related to how the Internet and mobile apps have led to new types of work arrangements. I will ask first about tasks that are done in-person and then about tasks that are done entirely online.
1	Some people find short, IN-PERSON tasks or jobs through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app or website. For example, using your own car to drive people from one place to another, delivering something, or doing someone’s household tasks or errands. Does this describe ANY work (you/NAME) did LAST WEEK? Yes No
2	Was that for (your/NAME’s) (main) job, (your/NAME’s second job), or (other) additional work for pay? Main job Second job Additional work for pay NOTE: Wording about “main” job, second job, and “other” work for pay will only appear if the respondent indicates having a second job during the main CPS interview.
3	Some people select short, ONLINE tasks or projects through companies that maintain lists that are accessed through an app or a website. These tasks are done entirely online and the companies coordinate payment for the work. For example, data entry, translating text, web or software development, or graphic design. Does this describe ANY work (you/NAME) did LAST WEEK? Yes No
4	Was that for (your/NAME’s) (main) job, (your/NAME’s second job), or (other) additional work for pay? Main job Second job Additional work for pay

2. Caveats

Overall, the questions performed adequately. We have to contend with the issue of having a limited number of questions to ask about a complicated, nuanced topic. The target concepts have several identifying features, all of which had to be included in the question wording. While another format, or number of questions, would have allowed for more explanation of the key concepts and screening, there was only space for four questions and we want to create the most ideal four questions to cover the concepts of interest.

We believe that some of the misreporting can be attributed to properties of the testing rather than to the questions themselves. For example, we had to screen participants in order to find a relatively small population (i.e., people who find work through these electronically mediated platforms). The process of this screening gives some indication to participants what the study is about and may affect the way they approach the testing situation.

Also, conducting the interviews online allowed us greater geographic diversity, but led to an artificial testing situation because the survey was self-administered. Participants couldn’t ask questions and there were no interviewers to probe if a participant provided an insufficient or uncertain response. If the participant reported an answer and then wanted to change it based on the reading of later questions, they couldn’t do so.

Finally, in order to fit the interview and debriefing into an adequate block of time we had to pick and choose what questions to include from the CPS to add context. As previously mentioned, our decision to not add questions related to participants’ second jobs may have affected the way they answered the “which job” question.

Appendix A: Interview Protocols

In-Lab Cognitive Interviews

Question Number	Questions	Response Options	Who Receives Question
Roster1	Please tell me the names and ages of everyone who lives in your household	Open ended	Everyone
Roster 2	Is (name) employed?	Yes No	Everyone
Demographic Questions, ask as a block for each HH member
EDUCA	What is the highest level of school (name/you) (has/have) completed or the highest degree (name/you) (has/have) received?	(31) Less than 1st grade (32) 1st, 2nd, 3rd or 4th grade (33) 5th or 6th grade (34) 7th or 8th grade (35) 9th grade (36) 10th grade (37) 11th grade (38) 12th grade NO DIPLOMA (39) HIGH SCHOOL GRADUATE - high school DIPLOMA or the equivalent (For example: GED) (40) Some college but no degree (41) Associate degree in college - Occupational/vocational program (42) Associate degree in college - Academic program (43) Bachelor's degree (For example: BA, AB, BS) (44) Master's degree (For example: MA, MS, MEng, MEd, MSW, MBA) (45) Professional School Degree (For example: MD,DDS,DVM,LLB,JD) (46) Doctorate degree (For example: PhD, EdD)	Everyone
HSPNON	(Are / Is) (name/you) of Hispanic, Latino, or Spanish origin?	(1) Yes (2) No	Everyone
RACE	I am going to read you a list of race categories. You may choose one or more races. For this survey, Hispanic origin is not a race. Are you: White; Black or African American; American Indian or Alaska Native; Asian; OR Native Hawaiian or Other Pacific Islander? Do not probe unless response is Hispanic or a Hispanic origin. Enter all that apply, separate with commas	White Black or African American American Indian or Alaska Native Asian Native Hawaiian or Other Pacific Islander Other - DO NOT READ	Everyone
Labor Force Questions, for up to 3 HH employed members, identified during roster
LABOR	Next, I am going to ask a few questions about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___		Everyone
PUWK	LAST WEEK, did you do ANY work for pay?	Yes, go to PUMJ No, go to PUABS Don't Know Refused	Everyone
PUABS	LAST WEEK, did you have a job either full or part time? Include any job from which you were temporarily absent.	Yes No Don't Know Refused	Employed, PUWK = 1
PUMJ	LAST WEEK, did you have more than one job (or business), including part-time, evening or weekend work?	Yes No Don't Know Refused	Employed, PUABS = 1 PUMJ = 1
PUHRUSL1	How many hours per week do you USUALLY work at your (main) job? (If PUMJ is "yes") By main job we mean the one at which you USUALLY work the most hours.	If PUMJ=1 ask PUHRUSL2 else go to PUIO1INT	All employed, fill based on more than one job, PUMJ=1
PUHRUSL2	How many hours per week do you USUALLY work at your other job(s)?		All employed, fill based on more than one job, PUMJ=1
PUIO1INT	Now I have a few questions about the (main) job - (a) (at which you worked LAST WEEK.) (b) (from which you were absent LAST WEEK.) (Was.../Were you) employed by government, by a private company, a non-profit organization, or (was.../were you) self-employed (or working in a family business)?	Government Private company Non-profit organization incl. tax exempt and charitable organizations Self-employed Working in family business	All employed, fill based on more than one job, PUMJ=1
PUIO1INC	What is the name of the (company) (non-profit organization) (business) for which ... (works) (at your/his/her Main job)?	Open-ended	All employed, fill based on more than one job, PUMJ=1
PUIO2INC	What kind of business or industry is this? READ IF NECESSARY: What do they make or do where you work?	Open-ended	All employed
PUIO1OCC	What kind of work (do) you do, that is, what (is) your occupation? (For example: plumber, typist, farmer)	Open-ended	All employed
PUIO1DT	What (are) your usual activities or duties at this job? (For example: typing, keeping account books, filing, selling cars, operating printing press, laying brick)	Open-ended	All employed
	Contingent Worker Supplement, for up to 3 employed HH members, identified during Labor Force Questions
S1	Some people are in temporary jobs that only list for a limited time or until the completion of a project. Is your (main) job temporary?	Yes No Don't Know Refused	All employed
S2	Provided the economy does not change and your job performance is adequate can you continue to work for your current employer as long as you wish?	Yes No Don't Know Refused	All employed
S3	Are you working only until a specific project is completed?	Yes No Don't Know Refused	S2=2
S4	Were you hired to temporarily replace another worker?	Yes No Don't Know Refused	S3=2
S5	Were you hired for a fixed period of time?	Yes No Don't Know Refused	S3=2
S6	Is your job a year round job or is it only available during certain times of the year?	Yes No Don't Know Refused	S3=2
S7	How much longer do you expect to work in your current job?	Less than or equal to 1 year More than 1 year Don't Know Refused	S3,4,5=2 S6=1
S8	What is the main reason you expect to work at your current job for less than a year?	Open-ended	S7=1
S9	If it were not for this reason, could you have kept working at the job you had last week.	Yes No Don't Know Refused	S7=1
S10	(For your main job), are you paid by a temporary help agency?	Yes No Don't Know Refused	S2=2
S11	Even though you told me your job is not temporary, are you paid by a temporary help agency?	Yes No Don't Know Refused	S2 =1
S12	Note the wording in parentheticals in the questions below are only used for the first person in the household. If an inquiry is being made for additional people in the household the wording is omitted. (Some people are in a pool of workers who are ONLY called to work as needed, although they can be scheduled to work for several days or weeks in a row, for example substitute teachers, and construction workers supplied by a union hiring hall. These people are sometimes referred to as ON-CALL workers.) Were you an ON-CALL worker last week (in your main job)?	Yes No Don't Know Refused	All employed
S13	(Some people get work by waiting at a place where employers pick up people to work for a day. These people are sometimes called DAY LABORERS). Were you a DAY laborer last week?	Yes No Don't Know Refused
S14	(Some people provide employees or their services to others under contract. A few examples of services that can be contracted out include security, landscaping or computer programming). Did you work for a company that contracts out your services last week?	Yes No Don't Know Refused
S15	Are you usually assigned to more than one customer?	Yes No Don't Know Refused
S16	Do you usually work at the customer’s worksite?	Yes No Don't Know Refused
S17	Last week, were you working as an independent contractor, an independent consultant, or a freelancer? (that is, someone who obtains customers on their own to provide a product or service)	Yes No Don't Know Refused
	Electronic Mediation Questions, for up to 3 employed HH members, identified during Labor Force Questions
EMQIntro	I now have a few questions about how the Internet and mobile apps have changed the way some people work.
S18	Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app. For example, driving a person from one point to another using your own car, delivering people’s laundry to a dry cleaner, or helping a person assemble furniture. Does this describe any work you did last week?	Yes No Don't Know Refused	All employed
S19	Was that for your (main) job, (second job), or additional work for pay?	Main Job Second Job Additional Work For Pay Don’t Know Refused	S18=1
S20	Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online. For example, data entry, labeling photos, translating text, or other micro-tasks. Does this describe any work you did last week?	Yes No Don't Know Refused	All employed
S21	Was that for your (main) job, (second job), or additional work for pay?	Main Job Second Job Additional Work For Pay Don’t Know Refused	S20=1

Debriefing

OK, those are the survey questions I wanted to get through. Now, I will go back and ask you some follow-up questions about what we just went through.

S18 Block

1. I am going to read back one of the questions and I would like for you to tell me, in your own words, what the question is asking:

S18	Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app. Does this describe any work you did last week?

2. [if S18=1] You said [PARTICIPANTS ANSWER] to that question. What company or companies did (you/NAME) work for?

3. [if S18=1] (Do you/Does NAME) consider working for [Company 1 through N] to be a job or something else? Please explain.

4. [if S18=1] Can you tell me more about the work (you do/NAME does) for [Company 1 through N]?

5. [if S18=1 for main respondent] Please describe what you do at the job so an average person could understand.

6. [if S18=1 for proxy respondent] How would NAME describe (that job/those jobs) to you?

7. [if S18=1] How many hours (do you/does NAME) typically work at (that job/those jobs) each week?

8. [if S18= 1] How many hours did you work at [Company 1 through N] last week?

9. [if S18=1] How long does it take to complete a task at [Company 1 through N]?

10. [if S18= 1] Do you decide when you work for [Company 1 through N]?

11. [if S18 = 2] Have you ever been paid for work that you found through these websites or apps?

Uber	Lyft
Grubhub	Taskrabbit
Instacart	Washio
Luxe	Postmates
Heal

• Yes

• No

12. [if 11 = “Yes”] When was the last time you were paid for work found through these websites or apps?

13. [if 12 = “Last week”] Earlier, when I asked about “short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app” you didn’t report this work. Can you tell me more about that?

S20 Block

14. I am going to read back another question and I would like for you to tell me, in your own words, what the question is asking:

S20	Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online. Does this describe any work you did last week?

15. [if S20=1] You said [PARTICIPANTS ANSWER] to that question. What company or companies did (you/NAME) work for?

16. [if S18=1] (Do you/Does NAME) consider working for [Company 1 through N] to be a job or something else? Please explain.

17. [if S20=1] Can you tell me more about (that job/those jobs)?

18. [if S20=1 for main respondent] Please describe what you do at the job so an average person could understand.

19. [if S20=1 for proxy respondent] How would NAME describe (that job/those jobs) to you?

20. [if S20=1] How many hours (do you/does NAME) typically work at (that job/those jobs) each week?

21. [if S20= 1] How many hours did you work at [Company 1 through N] last week?

22. [if S20= 1] How long does it take to complete a task at [Company 1 through N]?

23. [if S20= 1] Do you decide when you work for [Company 1 through N]?

24. [if S20= 1] What is a typical task like on [Company 1 through N]?

25. [if S20 = 2] Have you ever been paid for work done through any of the following sites/companies?

Amazon Mechanical Turk	CrowdFlower
99Designs	crowdSPRING
Freelancer	Hiretheworld
OneSpace	Crowdsource.com
Clickworker	Shorttask
Microworkers

• Yes

• No

26. [if 25 = “Yes”] When was the last time you completed work on that site?

27. [if 26 = “Last week”] Earlier, when I asked about “short, paid tasks through companies that maintain online lists of tasks” you didn’t report this work. Can you tell me more about that?

28. Last week, did you earn money from any other source?

    1. [if 28= “yes”] Can you tell me more about that?

29. What does the term “crowdsourcing” mean to you?

30. Would you describe any of the work you do as crowdsourcing?

31. Are you familiar with the term “gig economy”?

    1. [if 31 = “yes”] What is the “gig economy”?

    2. [if 31 = “yes”] Would you describe any of the work you do as part of the “gig economy”?

32. [if Uber not mentioned] Have you heard of the company Uber?

    1. Please describe that company to me

33. [if Mechanical Turk not mentioned] Have you heard of Amazon Mechanical Turk?

    1. Please describe that company to me

34. [if Grubhub not mentioned] Have you heard of Grubhub?

    1. Please describe that company to me

35. [if Taskrabbit not mentioned] Have you heard of Taskrabbit?

    1. Please describe that company to me

mTurk Online Interviews

This study was listed on mTurk as a single task with an initial screening question. The screening criteria for the study were listed in the study invitation. Potential participants saw the following information on the mTurk site.

OMB Control Number: 1220-0141

Expiration Date: 04-30-2018

Description:	We'll ask you questions related to work you have done for pay within the last week and then ask for your feedback on those questions.
Keywords:	memory, survey, research
Qualifications Required:	Participants in this study must have found paid tasks to complete through a service that links them to customers using a website or app not including Mechanical Turk. Examples include Uber, Lyft, Grubhub, Taskrabbit, Washio, CrowdFlower, etc. OR Participants in this study must have worked last week for pay.

If they clicked the link to the study, they were asked the initial screening question.

If the potential participant reported working for a type of company that we were interested in testing, they were sent to the full survey. If they did not work for one of the types we were interested in, they were thanked and exited the survey.

Welcome! Thanks for your interest in our survey. You’re here because we have asked you to participate in our research. We are asking you and hundreds of other people to tell us about what they think.

Unlike some surveys or online tasks you may be familiar with, we ask that you complete this survey all at one time and that you only start once you are in a quiet place where you won't be disturbed. The survey will take no longer than 20 minutes to complete. Only share information you're comfortable with - nothing too personal - but please be honest and follow the instructions.

 Please do not use your browser's back button. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this study.

---page break---

On the following pages, you’ll be asked about work you have done for pay during the last week. We will then ask you some additional questions to better understand your answers.

Let’s get started!

---page break---

We’ll start by asking a few questions about you.

What is your age in years?

What is your gender?

• Male

• Female

What is the highest level of school you have completed or the highest degree you have received?

• Grades 1-12/No Diploma

• High School Diploma or Equivalent

• Some College

• Associate’s Degree

• Bachelor’s Degree

• Master’s Degree

• Doctorate or Professional Degree

Are you Hispanic or Latino?

• Yes

• No

What is your race?

• American Indian or Alaska Native

• Asian

• Black or African-American

• Native Hawaiian or Other Pacific Islander

• White

In which state do you live?

---page break---

1. The next few questions will ask about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___
   
   LAST WEEK, did you do ANY work for pay?

• Yes

• No

2. LAST WEEK, did you have a job either full or part time? Include any job from which you were temporarily absent.

• Yes

• No

3. LAST WEEK, did you have more than one job, including part-time, evening or weekend work?

• Yes

• No

4. How many hours per week do you USUALLY work at your main job?
   
   By main job we mean the one at which you USUALLY work the most hours.

5. How many hours per week do you USUALLY work at your other job(s)?

6. Were you employed by government, by a private company, a non-profit organization, or were you self-employed or working in a family business?

• Government

• Private company

• Non-profit organization incl. tax exempt and charitable organizations

• Self-employed

• Working in family business

7. What is the name of the (company)(non-profit organization) (business) for which you work?

8. What kind of business or industry is this?

9. What are your usual activities or duties at this job?
   (For example: typing, keeping account books, filing, selling cars, operating printing press, laying brick)

Supplement

1. Some people are in temporary jobs that only list for a limited time or until the completion of a project. Is your main job temporary?

• Yes

• No

2. Provided the economy does not change and your job performance is adequate can you continue to work for your current employer as long as you wish?

• Yes

• No

3. Are you working only until a specific project is completed?

• Yes

• No

4. Even though you told me your job is not temporary, are you paid by a temporary help agency?

• Yes

• No

5. Some people are in a pool of workers who are ONLY called to work as needed, although they can be scheduled to work for several days or weeks in a row, for example substitute teachers, and construction workers supplied by a union hiring hall. These people are sometimes referred to as on-call workers.
   
   Were you an on-call worker last week in your main job?

• Yes

• No

6. Some people get work by waiting at a place where employers pick up people to work for a day. These people are sometimes called day laborers.
   
   Were you a day laborer last week?

• Yes

• No

7. Some people provide employees or their services to others under contract. A few examples of services that can be contracted out include security, landscaping or computer programming.
   
   Did you work for a company that contracts out your services last week?

• Yes

• No

8. Are you usually assigned to more than one customer?

• Yes

• No

9. Do you usually work at the customer’s worksite?

• Yes

• No

10. Last week, were you working as an independent contractor, an independent consultant, or a freelancer (that is someone who obtains customers on their own to provide a product or service)?

• Yes

• No

I now have a few questions about how the Internet and mobile apps have changed the way some people work.

11. Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app.
    
    For example, driving a person from one point to another using your own car, delivering people’s laundry to a dry cleaner, or helping a person assemble furniture.
    
    Does this describe any work you did last week?

• Yes

• No

12. Was that for your main job, second job, or additional work for pay?

• Main Job

• Second Job

• Additional Work For Pay

13. Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online.
    
    For example, data entry, labeling photos, translating text, or other micro-tasks.
    
    Does this describe any work you did last week?

• Yes

• No

14. Was that for your main job, second job, or additional work for pay?

• Main Job

• Second Job

• Additional Work For Pay

Debriefing

Next, we will ask some questions about your experiences taking this survey.

S11 Block

One of the questions we asked was worded as follows

Some workers find short, in-person jobs or tasks through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app.

Does this describe any work you did last week?

1. [if S11= “Yes”] You said yes to this question. What company or companies did you work for?

2. [if S11= “Yes”] Do you consider working for [Company 1 through N] to be a job or something else? Please explain.

3. [if S11= “Yes”] [Company 1 through N ] Please describe what you do at the job so an average person could understand.

4. [if S11= “Yes”] How many hours do you typically work at (that job/those jobs) each week?

5. [if S11= “Yes”] How many hours did you work at [Company 1 through N] last week?

6. [if S11= “Yes”] How long does it take to complete a task at [Company 1 through N]?

7. [if S11 = “Yes”] Do you decide when you work for [Company 1 through N]?

8. [if S11 = “No”] Have you ever been paid for work that you found through these websites or apps?

Uber	Lyft
Grubhub	Taskrabbit
Instacart	Washio
Luxe	Postmates
Heal

• Yes

• No

9. [if 8 = “Yes”] When was the last time you were paid for work found through these websites or apps?

S13 Block

Another question we asked was worded as follows:

Some workers select short, paid tasks through companies that maintain online lists of tasks. These tasks typically take between a few minutes and a few hours to complete and are done entirely online.

Does this describe any work you did last week?

10. [if S13= “Yes”] You said yes to this question. What company or companies did you work for?

11. [if S13= “Yes”] Do you consider working for [Company 1 through N] to be a job or something else? Please explain.

12. [if S13= “Yes”] [Company 1 through N] Please describe what you do at the job so an average person could understand.

13. [if S13= “Yes”] How many hours do you typically work at [Company 1 through N] each week?

14. [if S13= “Yes”] How many hours did you work at [Company 1 through N] last week?

15. [if S13= “Yes”] How long does it take to complete a task at [Company 1 through N]?

16. [if S13= “Yes”] Do you decide when you work for [Company 1 through N]?

17. [if S13= “Yes”] What is a typical task like on [Company 1 through N]?

18. [if S13 = “No”] Have you ever been paid for work done through any of the following sites/companies?

Amazon Mechanical Turk	CrowdFlower
99Designs	crowdSPRING
Freelancer	Hiretheworld
OneSpace	Crowdsource.com
Clickworker	Shorttask
Microworkers

• Yes

• No

19. [if 18 = “Yes”] When was the last time you completed work on that site?

General Questions

20. Last week, did you earn money from any other source?

2. [if 20= “Yes”] From what other sources did you earn money last week?

1 Four participants were also tested using the TryMyUI testing service. However, because of significant technical issues and results that overlap with the cognitive interviews and mTurk testing, those results are not presented here.

2 NOTE: The order of the response options was randomized for each respondent.

3 The sample size was set for 175 for this study due to budget reasons. Those who answered the screener question were selected for the study if they reported work related to one of the categories whose quota had not yet been met. If a participant selected multiple categories, he/she was counted as having met the quota for the category that the experimenters believed would be the most difficult to find in the mTurk sample of participants (this determination was made subjectively). The selection process was not perfect and resulted in an uneven distribution of participants across the categories in Table 3.

4 Two of the participants who broke off exited the survey at the time of the introduction to the electronic mediation questions. One participant began clicking through the interview at that point without answering questions. The remaining 13 exited the survey before the electronic mediation questions.

5 The study invitation said that it was only open to U.S. residents. This does not always work, however. While these participants did appear to answer the questions in a reasonable way, they were excluded to ensure that language issues do not influence the results.

6 True negative responses, responding “No” to the In-Person Task question when they had in fact not found work through an In-Person Task platform, was determined through the debriefing. Participants were shown an extensive but non-exhaustive list of companies and asked if they had ever performed work for those companies and, if so, when they had last worked for them.

7 Of these seven participants, two responded “No” to the Online Task question, two listed mTurk again as an example for that question (indicating that they may have rethought their answer to the previous question), one listed a different in-scope job for the Online Task question, and two reported “None” or skipped the Online Task question.

8 Eight participants either skipped the question resulting in null responses or they entered something like “I’d rather not say.”

9 Due to a programming error, 10 participants who responded “Yes” to the Online Task question said that never received the question asking them to list companies that they’ve worked for. The classification of these participants was added to the “Not Sure” category.

10 This table combines self and proxy responses.

11 Responses from all participants, whether or not they accurately reported having found work through an In-Person platform.

12 Responses only for those who accurately reported having found work through an In-Person platform.

13 This table contains only two, incorrectly reported proxy responses.