Note to Reviewer

June 2, 2015

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Erica Yu Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the ‘Testing of Impact of Outlet Questions on Accuracy of Expenditure Recall’ Study

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 332.

If there are any questions regarding this project, please contact Erica Yu at 202-691-7924.

1. Introduction and Purpose

The Consumer Expenditure Survey (CE) and Consumer Price Index (CPI) are considering integrating outlet questions into the redesigned CE recall and records interviews and individual diaries. These outlet data are currently collected by the Telephone Point of Purchase Survey (TPOPS). Integrating outlets into the CE would involve modifying the CE interview (CEQ) to collect outlet information. The impact of the inclusion of outlet questions in the CEQ is still unknown: there is potential for the impact to range from an increase in respondent burden and decrease in data quality by the addition of questions, to a decrease in respondent burden and increase in data quality by taking advantage of the natural hierarchy of expenses and outlets; or an outcome in between. In order to make an informed decision as to whether to include outlet collection in the CEQ, more information is needed about the impact of inclusion. This study focuses on implementation of the outlet questions for the recall interview.

2. Research Design

Experimental Design

The purpose of this test is to provide evidence as to whether the reporting of outlet information aids recall of expenditure information. This online, self-administered study will provide evidence regarding the impact of collecting outlet information on recall accuracy, which can be generalized to the face-to-face mode of the CEQ. The interview questions have been translated from the conventional face-to-face mode to an online self-administered mode.

Given that the impact of collecting outlet information may vary depending on how outlets are collected during the interview, this study tests three possible recall interview formats. These formats have been designed to provide information that will be compelling to the CE program office regarding whether outlets aid recall of expenditures. The formats are described below and screenshots are included in Appendix A. Each participant will be randomly assigned to one of these recall format versions; the groups will then be compared quantitatively.

• Control format: This version mimics the current CEQ, without outlet questions, whereby participants will be prompted to recall expenditures with questions including detailed expenditure items. These data provide a reference point for understanding data quality for this task without any outlet questions.

Example: “Let’s talk about Home Furnishings and related items. Have you purchased any sofas?” If yes, the interview would then collect a brief description of the item, expenditure amount, and month of purchase. Next, the interview would move on to “…Any living room chairs?” and so on.

• Plus format: This version mimics the current CEQ, plus additional outlets questions as item-specific follow-up questions.

Example: “Let’s talk about Home Furnishings and Related Items. Have you purchased any sofas?” If yes, the interview would then collect, as in the Control format, a brief description of the item, expenditure amount, and month of purchase. Alongside this information, the interview would also collect outlet information: “And at what merchant did you have that expense?”

• Section-driven Outlets format: This version follows the section-level structure of the CE interview but does use an item-level structure. This format prompts participants with broad CE expenditure sections (Home furnishings and related items) and then a question to identify outlets related to that broad category. The reported outlets are then used to prompt the participants to report the expenditures from that outlet. Whereas the other interview formats use CE expenditure items to prompt recall, this format uses self-reported outlets to prompt recall. This version enables analysis of whether outlet prompts improve recall of items compared to expenditure item prompts.

Example: “Now let’s talk about Home Furnishings and Related Items. Have you had any expenses at stores that sell Home Furnishings and Related Items?” and then “You mentioned that you shopped at Sofas R Us. What did you get there?”

Differences between the three versions have been limited as much as possible to only the key differences described above regarding the collection of outlet information. All other design features and procedures will be held constant across the three groups.

Methodologically, the present study deviates from typical OSMR lab studies in two important ways:

• Online data collection: To facilitate quantitative comparisons between the different recall format versions, large samples sizes are needed. Online data collection affords large samples quickly and at low cost. The change in mode from the CEQ in-person interview to an online survey limits the generalizability of the study findings, but we believe that the primary research question about the effect of outlets on data quality of recalled expenditures should not be affected by mode. And it is also important to note that this study aims only to identify the presence of effects but not to infer effect sizes to the CEQ survey sample; the online convenience sample will not be representative of the CE survey sample. Online data collection has been used successfully by OSMR for a wide range of studies similar to this one, including recall studies and respondent burden studies and with questions including open-ended text responses and multiple choice selections.

• Fictional expenditures: In conventional recall studies, for which participants report their own expenditures, it is not possible to compare a participant’s reported expenditures to their actual expenditures, which precludes any analysis of data quality and accuracy. To address this limitation, the present study utilizes an initial “learning phase” at the start of the task during which participants will be instructed to learn about a series of presented items and remember them as their own expenditures. The same expenditures will be shown to all participants, which enables analysis of data quality and comparisons across participants – including for item price, item type, month of expenditure, and outlet name. Paradigms similar to this are regularly used in experiment-based studies of recall from memory in the psychology literature. To accelerate the transfer of the learned items into long-term memory and clear the items from working memory, participants will complete a “distractor task” between learning the items and recalling the items (see Appendix B).

Learning Phase Design

All participants will complete a “learning phase.” During this portion of the task, the to-be-remembered information will be presented as a narrative spanning a three-month reference period, embedded with narrative choice points designed to increase the autobiographical nature of the memories. All participants will undergo the same learning phase before being randomly assigned to one of the three interview formats being tested.

The narratives include outlet-item-expenditure scenarios that the CE and CPI programs are interested in learning about. These scenarios represent conditions under which different recall formats may increase or decrease the burden of reporting and the quality of reporting. These conditions are below:

• The outlet is salient but the item is not

The participant is highly likely to remember the outlet but may not remember detail about the item.

• The item is salient but the outlet is not

The participant is highly likely to remember the item but may not remember the outlet.

• The outlet is visited repeatedly buying different items

Reporting a purchase from an outlet may cue another purchase from the same outlet, even if the item was purchased at a different time and is of another expenditure category.

• An item is purchased repeatedly from different outlets

The participant may not report or be able to recall different outlets when reporting identical items.

• The outlet sells items from multiple expenditure categories

Reporting an item may cue the participant to recall the other items from the same transaction, even if the other items are from different expenditure categories.

• Multiple expenditures are driven by a goal but are purchased at different times, from different outlets, and across expenditure categories

Reporting one item may cue the participant to recall the related items.

Each scenario will include information about the date of purchase, the outlet, the expenditure amount, and a description and image of the item. Items purchased together in a single transaction will be presented together. Each scenario will be independent from the other scenarios.

Items typically sourced from the CE diary rather than the CEQ interview will also be included as purchased items in the learning phase. The inclusion of these items will provide evidence as to how diary items will be reported by respondents during an interview.

To minimize order effects, the presentation order of the scenarios will be randomized where possible; for example, for conditions that require multiple expenditures be made across months, the expenditures will be fixed to appear in separate months but the order within month will be randomized.

Protocol and Materials

The protocol for the study is described in Appendix C. The recall portion of the task will use a selection of expenditure categories relevant to the expenses made in the narrative based on the production CAPI instrument (Version 14.43, 05-24-2013; OMB information collection instrument information stored at http://www.reginfo.gov/public/do/PRAViewIC?ref_nbr=201304-1220-004&icID=46041): Appliances, Household Equipment, and Other Selected Items; Home Furnishings and Related Household Items; Clothing and Clothing Services; Subscriptions, Memberships, Books, and Entertainment Expenses. The learning phase scenarios are described in Appendix D. Debriefing questions to be asked at the end of the task are included in Appendix E.

3. Measures of Interest

Measures of data quality are the main focus of this study. Participants will report expenditure amount, month of purchase, and outlet name; these will be compared to the information presented during the learning phase. Analyses will include comparisons across groups of deviations from actual information and under- and over-reporting. Time taken to complete the task (as measured by the online survey instrument) as well as debriefing ratings of respondent burden will also be compared across groups.

This study will focus on the effect of outlets on recall in general; this is not a test of the usability of an instrument or the wording of questions.

4. Participants

Participants will be recruited using a convenience sample from Amazon Mechanical Turk¹ of adult U.S. citizens (18 years and older); this study is focused on internal validity rather than representativeness of any population. This research design requires a sample of 600 participants in order to sufficiently explore the variables of interest. These participants will be randomly assigned to the 3 survey formats described (200 participants per group).

The primary goal of the proposed research is to explore the effects of recall format on reported expenditure data quality. Given basic assumptions about the experimental design (effect size = 0.3, power = 0.8, and significance level = 0.05 two-tailed), a minimum sample size of 200 participants in each group is expected to be sufficient. This study collects an additional 40 participants to account for potential break-offs and other data quality problems.

Additionally, there will be 2 rounds of pilot testing. An initial pilot test will recruit 4 participants from TryMyUI². These participants will primarily provide feedback about the saliency and clarity of the narratives. An additional 20 participants will be included in a final pilot testing round, testing the full task. These pilot participants will be assigned to the 2 main experimental protocols (excluding the control condition), resulting in 10 participants in each condition. These data will be examined to check that manipulations are having their intended effect.

5. Burden Hours

Our goal is to obtain feedback from 660 participants (including 20 pilot participants). We anticipate that each of these participants will spend up to 30 minutes on the task. Additionally, we anticipate that the initial pilot testing of the narratives using 4 TryMyUI participants will require 20 minutes for each test, resulting in 80 additional minutes of burden total. Combined, we anticipate that this study will result in no more than 332 burden hours.

6. Payment to participants

We will recruit 660 participants from the Amazon Mechanical Turk database (including 20 pilot participants). Participants will be compensated $3.50 for participating in the study, a typical rate provided by Mechanical Turk for similar tasks; a total of $2,310 will be paid directly to Amazon Mechanical Turk for participant fees. We will recruit an additional 4 participants from TryMyUI who will each be paid $10 compensation. A total of $40 will be paid to TryMyUI for participant fees.

7. Data confidentiality

Recruiting of participants will be handled by Amazon Mechanical Turk and TryMyUI. Once participants are recruited into the study, they will be sent a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Attachments

Appendix A – Recall Formats

Appendix B – Distractor Task

Appendix C – Protocol

Appendix D – Learning Scenarios

Appendix E – Debriefing Questions

Appendix A: Three recall formats

Participants will be randomly assigned to one of the three recall formats. The assigned format will be used for the participant throughout the entire recall portion of the task.

Control Format

This format is a simplified, self-administered version of the current CEQ. Participants will be prompted with section-level information (household furnishings and related items) and then detailed questions (sofas, living room chairs, living room tables). Each item will be presented on a screen by itself. The participant will move on to the next screen to receive the next question.

Plus Format

This format duplicates the Control format with the addition of a field for reporting merchant.

Outlets Format

This format provides section-level structure but does not ask about specific items. Participants are asked to report merchants first, and then the expenses at those merchants. The merchant name is pre-filled by default but participants may edit the field.

Appendix B: Distractor Task

A distraction in between learning items and recalling those items facilitates the clearing of working memory and the encoding of the learned items into long-term memory. In this task, participants will be shown images and asked to make judgments unrelated to the main task. These images will include items learned in the main task alongside images of items that were not presented during learning. This process of distraction increases the effectiveness of the break between learning and recall, which may otherwise be too short for effective testing of recall from memory.

Below is an illustration of the instructions and response scale. Participants will be shown 15 images, ordered randomly. These are images that were not presented during the learning phase.

---page break---

You’re done shopping!

And now for your next task, please consider the items presented in the following images. Please rate each item according to its level of quality.

[insert image]

Very high quality

Somewhat high quality

Neither high nor low quality

Somewhat low quality

Very low quality

---page break---

Images to be used:

Appendix C: Protocol

1. Introduction

---page break---

Welcome!

Thanks for your interest in our study. You’re here because we have asked you to participate in our study on recall of past purchases from memory. In this task, you’ll go through a game where you’ll make choices and shop for items, and then later tell us about what you bought. You won’t be asked to share any information about your own shopping.

Unlike some surveys or online tasks you may be familiar with, we ask that you complete this survey all at one time and that you only start once you are in a quiet place where you won't be disturbed. The survey takes about 30 minutes, on average. Only share information you're comfortable with - nothing too personal - but please be honest and follow the instructions.

 Please do not use your browser's back button. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this study.

---page break---

2. Instructions

---page break---

Instructions

We’ll first ask you to read a first-person story in which you’ll make choices about what you want to do next. It may be different from your real life but try to immerse yourself in the character and imagine that it is you.

Please read through each screen completely before continuing on to the next screen. Please don’t write anything down or otherwise record what’s on the screen – just focus on the story!

Afterwards, we’ll ask you about your life in the game and a few questions about your life in the real world.

First, let’s learn about your character. Continue on to the next screen to get started.

---page break---

Your Profile

Here is the backstory on your character:

Where do you live?

You live in the suburbs of a city called Springfield.

You have family that lives nearby and you see them often.

What do you do?

You have a steady office job at a small company in the city. You’ve been working on updating your home and trying your hand at some DIY repairs. In your spare time, you like to watch movies and hang out with friends.

Even if this doesn’t sound like your life, try to imagine that this is you and that the events you’ll read about are part of your life.

Ok, let’s get started.

---page break---

3. Learning Phase (see Appendix D)

4. Distractor Task (see Appendix B)

5. Recall phase, random assignment to one of three formats (see Appendix A)

6. Debriefing Questions (see Appendix E)

7. Thank you

---page break---

Thank you for participating in this study.

A survey like the one that you just completed is conducted regularly nationwide to create a picture of how much money US households are spending so that we can calculate inflation and the cost of living.

We need people to answer the survey accurately and give us a complete picture of their spending, but we also need to make the survey easy to answer and as short as possible.

If you have any thoughts about this study or how we can improve the way that we ask about household expenditures, please leave them in the box below. Thank you!

[open ended text entry box]

Appendix D: Learning Phase Narrative

All scenarios will follow a standard format that comprises an expenditure narrative, a choice related to the expense, and a summary of the transaction that includes all critical expenditure information. The instrument will force participants to spend at least three seconds on each of the screens, to encourage attentive reading of the narrative.

Condition	Screen 1: Expenditure Narrative	Screen 2: Choice Related to Expense	Screen 3: Summary of Transaction
Outlet is salient and item is not: Store name is memorable and described at length Item is not described	While waiting for friends to meet you for dinner, you find yourself standing outside of a few shops. One store catches your eye: it has a sign in the shape of a boat that reads “Tie-Tanic.” You wander in to find the store is full of tchotchkes. You pick out a gift for your brother’s birthday.	At the cash register, they offer you a choice of wrapping paper. Which do you choose?	That was a great store – you plan to come back to Tie-Tanic for a present for your brother next year, too. Here’s a summary of your purchases: Merchant: Tie-Tanic Total spent (not including tax): $12.99 [Fill selection]
Item is salient and outlet is not: Item is memorable Multiple similar stores that may lower recall accuracy for outlet	You come home to find out your dog has ripped up the bottom of your living room curtains. What’s worse, your parents are coming to visit next week and you don’t want your mom saying “I told you not to get that dog.” You need to get new curtains today!	Where do you go?	You end up finding some curtains that look like the ones your mom bought for you and that the dog destroyed. Here’s a summary of your purchases: Merchant: [Fill selection name] [Fill selection logo] Total spent (not including tax): $54.99
Outlet is visited repeatedly: -A single outlet is visited multiple times -The purchased items are unrelated and from different expenditure categories	You want to decorate your house in preparation for the next time you have guests over. A nice, but cheap, picture frame would be a good start – maybe you’ll frame that photo of your dog that you took in the park. You head to Walmart to pick up a frame.	Which one do you choose?	Once you’ve made your selection, you check out at the front of the store. You pick up a few extra items and add them to your shopping basket, too. Here’s a summary of your purchases: Merchant: Walmart Total spent (not including tax): $32.97 [Fill selection]
	Friends and family are coming over this weekend for a cookout. When you invited people over, the date seemed so far away… but now it’s only a few days until the party. Looks like you need to get yourself a grill! You stop by Walmart to buy one.	Which one do you choose?	Getting the grill is enough for the day – you’ll get the rest of the party supplies later. Here’s a summary of your purchases: Merchant: Walmart Total spent (not including tax): $85.99 [Fill selection]
An item is purchased repeatedly from different outlets: -Scenario is similar to routine or habitual purchases -Outlet information may not be strongly associated with the items	The new movie in the theaters this week is “Furious 7”. It’s playing at the Uptown Cinema - your favorite place to see a movie. Your friend takes a little bit of persuading to agree to come with you, but when you offer to buy the tickets – it’s a deal.	Which show time would you prefer? FURIOUS 7 Showtime: Fri 9:15p FURIOUS 7 Showtime: Fri 5:45p FURIOUS 7 Showtime: Sun 6:15p FURIOUS 7 Showtime: Sun 4:15p	You’ve got the tickets before the movie sold out. Now you just have to wait in line to get good seats… better stock up on some snacks. Here’s a summary of your purchases: Merchant: Uptown Cinema Total spent (not including tax): $22.00 [Fill selection] [Fill selection]
	You still haven’t had a chance to see the latest “Avengers” movie, even though you’ve heard a lot about it. It got decent reviews, so you’ll give it a shot.	When do you want to go? AVNGRSAGEO Showtime: Tue 5:45p AVNGRSAGEO Showtime: Thu 5:45p AVNGRSAGEO Showtime: Fri 8:15p AVNGRSAGEO Showtime: Fri 9:45p	It’s definitely a movie for getting popcorn at the concession stand! Here’s a summary of your purchases: Merchant: Cinema City Total spent (not including tax): $15.50 [Fill selection]
	Looking over the movie listings, you see that “Jurassic World” is going to be playing at The Screen on the Green. You get a couple of friends together and you plan to see it this week.	Which version do you want to see? Standard	Hope it’s a good movie! Here’s a summary of your purchases: Merchant: The Screen on the Green Total spent (not including tax): $12.00 [Fill selection]
	Cinema City is showing some classic movies this week. There’s a double feature of classic Alfred Hitchcock movies – Rear Window and Vertigo.	When do you want to see it? HITCHDOUB Showtime: Sat 12:45p HITCHDOUB Showtime: Sat 3:30p HITCHDOUB Showtime: Sun 3:30p	If these are good, maybe you’ll have a chance to check out some of the other classics being shown? Here’s a summary of your purchases: Merchant: Cinema City Total spent (not including taxes): $16.00 [Fill selection]
Outlet crosses expenditure categories: -The purchased items are unrelated to each other -Some items may not be memorable (storage bin) but the association with the other items from the outlet may improve recall accuracy	On your way home, you stop by Target to pick up a few things you need. The toaster broke and you have not had waffles for over a week now. Speaking of waffles - you remember that you need to get a birthday present for her.	Sweaters are on sale- she’s always wearing sweaters. Which do you pick?	She may or may not like it, but it’s the thought that counts, right? Here’s a summary of your purchases: Merchant: Target Total spent (not including taxes) $49.98 [Fill selection]
Goal-driven purchases: -Items are from different expenditure categories and different outlets but are all related to a single goal	The do-it-yourself approach to updating your bathroom is taking over your life. In order to complete the part of the project you’re working on, you realize that you have to stop what you’re doing and run to Ace Hardware around the corner to pick up a socket wrench.	The store has 4 different kinds. Which do you choose?	You’ll need a bigger tool box soon… but that’s a purchase for another day. Here’s a summary of your purchases: Merchant: Ace Hardware Total spent (not including taxes): $26.49 [Fill selection]
	If you never have to see your bathroom again, it will be too soon. The long process of fixing it up continues. Today, you’re picking out lighting. Home Depot is in a shopping center with other stores you need to run errands at, so you head there.	You pick out four that are in your price range. Which one do you choose?	Luckily, it comes with the hardware you need for installation. Here’s a summary of your purchases: Merchant: The Home Depot Total spent (not including taxes) $39.99 [Fill selection]
	Another weekend coming up? Another day to spend fixing up your bathroom. This time, you’re installing new towel rods. You’ve heard that Lowe’s has a good selection, so you head there.	You narrow it down to just four options. Which do you choose?	Good – it will match the rest of the bathroom fixtures. Here’s a summary of your purchases: Merchant: Lowe’s Total spent (not including taxes): $36.99 [Fill selection]

The presentation of these scenarios will be randomized to the extent possible, in order to minimize order effects. For conditions with multiple expenditures that should be distributed across months (e.g., going to the movies), the expenditures will be fixed to appear in separate months but the order within month will be randomized.

In addition to the expenditure information, each scenario will include the month and date of the expenditure, to be assigned after randomization so that the scenarios will appear to occur in chronological order. For example, whichever scenario is presented first will be labeled for all participants as occurring on June 5^th; the second scenario will be labelled as occurring on June 17^th; the third on June 23^rd; and the fourth on June 29^th.

Appendix E: Debriefing Questions

Were there any other items that you remember but have not yet entered?

[empty table in the same format as used throughout the survey]

---page break---

For each item reported above, ask the following:

What was the main reason for why you had not entered [Fill Description of the Expense]?

It did not fit into any of the categories that were asked about.

I remembered a little bit about the item but not enough to enter it.

I forgot about the item and only remembered after I should have entered it.

Other, please specify [text entry]

---page break---

And now we’d like to hear your thoughts about the task you just did. There will be a chance to write in your general comments at the end, but first we’d like to ask a few questions.

1. Were the items that you bought as part of the story easy or difficult to remember?

Very easy

Somewhat easy

Neither easy nor difficult

Somewhat difficult

Very difficult

2. Were the merchants and places that you shopped at as part of the story easy or difficult to remember?

Very easy

Somewhat easy

Neither easy nor difficult

Somewhat difficult

Very difficult

3. Were the expense amounts ($) easy or difficult to remember?

Very easy

Somewhat easy

Neither easy nor difficult

Somewhat difficult

Very difficult

4. Were the months of the expenses easy or difficult to remember?

Very easy

Somewhat easy

Neither easy nor difficult

Somewhat difficult

Very difficult

5. If you have anything to add about what it was like to remember the expenses, please write those here.

[open ended text entry box]

---page break---

6. Thinking about the part of the task where we asked you to tell us about your expenses – how burdensome would you say answering the questions felt?

Not at all burdensome

Slightly burdensome

Somewhat burdensome

Very burdensome

Extremely burdensome

7. Compared to other HITs that you’ve done on mTurk, which would you say was more burdensome?

It was less burdensome than others HITs

It was more burdensome than other HITs

It was about the same amount of burden

8. When we asked you to tell us about the expenses from the story, how clear did you think the questions were?

Not at all clear

Slightly clear

Somewhat clear

Very clear

Extremely clear

9. When we asked you to tell us about the expenses from the story, did the questions we asked make it easy or difficult to report the expenses?

Very easy

Somewhat easy

Neither easy nor difficult

Somewhat difficult

Very difficult

10. Think about your own life – if you were asked to recall your own expenditures from the last three months, what would help you to remember everything that you bought? Please rank the options below from Most Helpful (1) to Least Helpful (4).

(randomized order)

• A list of specific examples of things that people buy – like TVs, sports equipment, sofas, and mugs.

• A list of general categories of things that people buy – like home furnishings and clothing.

• A place where I could list the stores that I shopped at.

• A calendar where I could make notes about events and things that I did.

11. If there is something else that would help that we haven’t listed above, please write it in below.

[open text entry box]

---page break---

1. During the first half of the task when you read the story and made choices, did you do any of the following? Please mark all that apply.

(randomized order)

• Think about the similarities and differences between the story and my life

• Make choices as if they were real life choices

• Take notes or otherwise kept a record of the story

• Memorize as much detail as I could

• Stop paying attention after a while

2. How focused did you feel while you read the story and made choices? Please be truthful - your response will not be used against you.

Not at all focused

Somewhat focused

Slightly focused

Very focused

Extremely focused

---page break---

And now a few final questions about yourself.

1. What is your sex?

Male

Female

2. In what year were you born?

[open text box; numeric]

3. Are you Hispanic or Latino?
   Yes
   No

4. What is your race? (Please select one or more)

American Indian or Alaska native

Asian

Black or African American

Native Hawaiian or Pacific Islander

White

5. What is the highest degree you received?

No schooling completed

Elementary school diploma

High school diploma or the equivalent (GED)

Associate degree

Bachelor’s degree

Master’s degree

Professional or doctorate degree

1 Mechanical Turk is an online platform that offers a crowdsourcing “marketplace”. Individuals can volunteer to participate in short tasks and be paid small amounts of money ($4-6/hour pro-rated, on average). “Workers” may not be demographically representative of the United States but social science researchers are increasingly using Mechanical Turk for large-scale studies. Eligibility criteria can be set to restrict participation to only individuals over 18 years old and located in the United States.

2 TryMyUI is an online platform that offers audio and screen recordings from research participants trained to think aloud while completing tasks. Tasks may last no longer than 20 minutes. Research participants typically provide feedback on the clarity of the task and their thought processes while completing the task. OSMR has successfully used TryMyUI for remote cognitive interviewing and pilot testing of experiments in the past.

22