Note to Review - CE Diary Outlet Study

October 29, 2014

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Erica Yu Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the ‘Testing of Outlet Questions in CE Redesign Diary’ Study

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 71.5.

If there are any questions regarding this project, please contact Erica Yu at 202-691-7924.

1. Introduction and Purpose

The Consumer Expenditure Survey (CE) and Consumer Price Index (CPI) are considering integrating outlet questions into the redesigned CE recall and records interviews and individual diaries. These outlet data are currently collected by the Telephone Point of Purchase Survey (TPOPS). Much of the outlet data required by CPI are for expenses that will be collected using individual diaries under the planned CE redesign. The impact of the inclusion of outlet questions in the CE is still unknown: there is potential for the impact to range from an increase in respondent burden and decrease in data quality by the addition of questions, to a decrease in respondent burden and increase in data quality by taking advantage of the natural hierarchy of expenses and outlets; or an outcome in between. In order to make an informed decision as to whether to include outlet collection in the CE, more information is needed about the impact of inclusion. This study is one of two lab studies to be conducted; this study focuses on diary implementation.

This study will focus on understanding whether collecting outlet information in the diary affects reported burden, ease of data entry, and data quality.

2. Research Design

Three versions of the diary have been designed and developed as wireframe mock-ups in Microsoft Access. These mock-ups simulate home PC use of the CE diaries (simulating real-world mobile diary use is not in scope for this study). Each participant will be randomly assigned to one of these diary versions; the groups will then be compared qualitatively.

Each version of the diary is briefly summarized below, and the key differences between versions on the home screen and expense entry screen are described. Screenshots are included in Appendix A.

• Item-based (control condition): This diary matches the current diary, which does not collect outlet data. The diary is structured around items, such that the summary screen shows each item entered. Within the instrument, information is captured one expense on a screen at a time. The diary home screen summarizes the participant’s itemized expense history. These participants’ burden ratings and indications of data quality will serve as a benchmark for the experimental conditions.

• Item-based with outlet follow-up: This diary appends outlet entry as a follow-up question after expense entries; in other words, this diary follows the same design as the control diary but with an additional follow-up question for each item asking about the outlet at which that expense was made.

• Outlet-based: This diary is organized around outlets. The participant first enters outlet-level information: purchase date and outlet name. Then, the participant enters the expenses for that outlet, at the item-level. As with the other versions of the diary, the participant enters one expense on a screen at a time. After entering an expense, the participant has the option to enter another expense from that same outlet. After indicating that he or she is finished entering expenses from that outlet, the participant is returned to a home page summarizing the participant’s outlet history. By clicking on an outlet name, a summary of item-level expenses from that outlet is shown.

Differences between the three versions have been limited as much as possible to only the key differences described above regarding the entry of outlet information. All other design features and procedures will be held constant across the three groups.

All participants will be given a series of scenarios describing fictional expenses, and be asked to record those expenses in a diary. The participants will also be asked to enter one of their own recent, real-world expenses in the diary. The fictional expense tasks are designed to understand the following issues that may indicate the impact of inclusion of outlets on the CE, related to data quality and burden:

• An outlet-based structure may be confusing to participants if outlets are not clearly defined (e.g., a Starbucks within a grocery store, which operates as a separate outlet), outlets are unknown (e.g., he or she does not know the name for the outlet, such as at a parking meter), or outlet mode is not clear (e.g., online purchase with ship-to-store).

• An outlet-based structure may make it more obvious that combined event expenditures (e.g., a night out including dinner and a movie) should be entered separately since they occurred at different outlets.

• An outlet-based structure may help to reduce the expense summary screen if the diary contains many items (e.g., an item-based summary screen for a grocery shopping trip with 30 individual expenditure items compared to an outlet-based summary screen with 1 entry for the grocery store).

• An outlets-based summary screen may cause difficulty for participants if the same outlet is shopped at more than once during the reference period (e.g., two entries for “Target”).

• An outlets-based summary screen may help participants distinguish between similar expenses (e.g., two different items, both described as “shirts”, from different stores).

• An outlets-based summary screen may cause difficulty for participants if they must search for an item that is not typically associated with that outlet (e.g., milk from Target).

• When outlets are collected as follow-up questions, entering the same outlet information repeatedly for multiple items from one store may feel burdensome.

The protocol is described in Appendix B and accompanying receipts in Appendix C. Debriefing questions are included in Appendix D.

3. Measures of interest

At least two observers will either attend every interview session or watch a screen recording of each interview. The observers will rate whether each participant was successful in completing each task. Task success requires completing the task without assistance from the interviewer; the participant may make errors as long as he or she corrects them before finishing the task. Task failure may be due to misunderstanding of CE data requirements, usability challenges, or any other reason. These ratings will be tallied and reported for qualitative comparison across diary versions. Observer notes will also be considered.

The time taken for each task will be collected using the task timers within the Morae software.

3. Participants

Participants will be recruited from the OSMR participant database. Efforts will be made to select participants with varying levels of age, gender, education, and race based on self-reported information collected during an initial recruitment process and stored in the database.

4. Burden Hours

Our goal is to obtain feedback from 65 participants; 5 pilot participants and 60 participants, to be randomly assigned to each of the 3 diary instruments. We anticipate that the interview session will last no longer than one hour. Recruiting and screening prospective participants is estimated to take 5 minutes per participant and expect to contact 75 respondents in order to reach our goal of 65 for total recruiting burden of 6.25 hours. Burden hours participation are expected to total no more than 60 minutes per participant, for a total of 65 burden hours. Total burden for recruitment and participation is 71.5.

5. Payment to Respondents

A $40 incentive will be provided to all participants.

6. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of the Consumer Expenditure survey. Participants will be given a consent form to read and sign (Appendix E). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

Appendix A: Screenshots of the 3 Diaries

Screenshots have been cropped for space. Only blank screen space has been cropped out.

Item-based (control)

Summary screen

Expense entry screen

Appendix A: Screenshots of the 3 Diaries

Item-based with outlet follow-up

Summary screen

Expense entry

Appendix A: Screenshots of the 3 Diaries

Outlet-based
Summary screen

Expense entry

Appendix B: Protocol

Introduction

• Hi! Thank you for coming in today.

• I have a couple colleagues in the next room that will be observing and taking notes.

• Explanation of the study purpose:

  • Today we’re going to be testing a part of the Consumer Expenditure Survey. This survey collects information about how US households spend their money. We currently ask people to complete a diary to keep track of expenses during a week-long period, but now we want to try out a few different designs to see what works best.

  • It’s important to note that this is not a test of your ability. Our purpose today is to find out how easy or difficult it is to use the diary and to identify ways to improve it. Your feedback now can help make our future work more productive.

• Consent Form and permission to record screen and audio.

• Any questions before we begin?

Background about the CE Diary

• The Consumer Expenditure Survey is about understanding how US households spend their money. We provide our respondents with a diary, like this paper one, so they can keep track of daily expenses –eating out, groceries, household shopping, that kind of thing. In our session today, I’ll ask you to imagine that you are one of our respondents and you are entering your daily expenses into a diary for us. We ask thousands of people to do this to understand how much of certain types of items people are buying.

• We recently developed an electronic version of the diary, and that’s what we’ll be working with today. I’d like to start our session by walking through the diary – first, how to enter your expenses. Here are some screenshots of the diary to look at while we talk about it.

  • For participants in the “Item-based” and “Item-based with outlet follow-up” diary conditions:
    You’ll enter each of your expenses separately into the diary. Say I bought a gallon of milk and a loaf of bread at the grocery store yesterday. I would enter those in one at a time. First, I would enter yesterday’s date. Then, for “Description” I would enter “Milk” and then how much I paid “3.99”. Then, the diary prompts me to label this expense with one of these pre-assigned labels – it’s Food for Home Consumption. Then, I answer the follow-up questions. For this one, I would say “Fresh”. And, since I bought it for myself I would leave this checkbox empty.

    • For the “Item-based with outlet follow-up” diary condition:
      Then I would type in that I got it from Safeway in Washington, DC. Here, for “vendor”, we’re looking for the name of the place where you spent your money – be it Safeway for your groceries or Pepco for your electricity bill.

Done. Save it, and then enter my second item – the bread. So I do this again.

• For participants in the “Outlet-based” diary condition:
  You’ll enter each of your shopping trips into the diary. Say I bought a gallon of milk and a loaf of bread at the grocery store yesterday. First, I would enter yesterday’s date. Then, where I bought the items: Safeway in Washington, DC. Here, for “vendor”, we’re looking for the name of the place where you spent your money – be it Safeway for your groceries or Pepco for your electricity bill.
  Then, it prompts me to enter all of the stuff that I bought there that day. First it prompts me to label this expense with one of these pre-assigned labels – it’s milk so it is Food for Home Consumption. Then I’ll enter “Milk” for $3.99. Then, I answer the follow-up questions. For this one, I would say “Fresh”. Done. Save it, and then enter my second item. Since it is from the same store, I can hit “Add another item” and continue on. See, the store information is still there.

• Does that make sense?

• Now you can always come back to your Home screen to see a summary of your expenses.

  • For participants in the “Item-based” and “Item-based with outlet follow-up” diary conditions:
    See, here are all of the expenses you entered – the milk and the bread. If you’re ever looking for an item – to check that you entered it or to maybe go back and delete it if you returned the item – then you can look for it here.

  • For participants in the “Outlet-based” diary condition:
    See, here it shows us that I entered in a shopping trip to Safeway. If I click on Edit, it then shows me all of the items that I entered for this Safeway shopping trip – it’s like a receipt. If you’re ever looking for an item – to check that you entered it or to maybe go back and delete it if you returned the item – then you can look for it here.

• Does that make sense? We can go through another example if you’d like.

• Now, that’s how it will work for most things you enter into the diary. Expenses for eating out are a little different. We care about the entire meal altogether, not each dish you ordered.

  • For participants in the “Item-based” and “Item-based with outlet follow-up” diary conditions:
    You start off the same way – entering the date. Then you type in a description for the meal – maybe “Dinner with Joe”. Then you enter the total expense for the meal and answer the follow-up questions.

  • For participants in the “Outlet-based” diary condition:
    You enter the vendor information like normal. And then you answer the follow-up questions about the meal, including any expenditures on alcohol.

In that way, it’s different from groceries where we’re interested in the items that you bought. Here, we just need to know answers to these basic follow-up questions.

Tasks

Now I’m going to ask you to enter expenses into the diary. Most of them will be made-up expenses. Imagine that you have agreed to take part in our survey and you’re task is to enter the information into the diary. Do you have any questions before we begin?

Ok, I’ll describe each expense to you and present it to you in the form of a receipt or bill. Please enter that expense into the diary. While you’re entering the expense, imagine that you’re at home on the computer and that I’m not here. When you finish entering something into the diary, let me know. I’ll ask you a quick question about how easy or difficult that task was, and then we’ll move on to the next thing.

1. Enter own food expense

Let’s start with something from real life. Think back to the last purchase you made. Please add that item to the diary.

2. Enter an expense from Jmart

You went shopping at Jmart on December 10^th. You got some spray cleaner, some Tupperware, and some milk. Here’s your receipt. Please enter these into the diary.

3. Enter an expense from Percy’s

The next day, you stopped by a department store called Percy’s on your way home from work. You picked up a shirt. Please enter that into the diary.

4. Enter an expense from Jmart

You started having second thoughts about the shirt you just bought. You went back to Jmart and ended up buying a shirt that you had seen the last time you were there. Here’s your receipt.

5. Return shirt (from Percy’s)

Now you want to return the first shirt you got. Please go into the diary and delete that shirt from the diary.

6. Enter an expense from Sahara.com

That night, you went online to pick out a book for a friend. Here’s your receipt. Please enter this book into the diary.

7. Enter multiple expenses from ValUMart.

You went shopping at ValUMart and picked up several things for your household. Here is your receipt. Please enter these expenses.

Great, we’re about halfway through now.

Open up a new diary, with 15 grocery expenses from ValuMart grocery store already filled in. Items include apples, oranges, crackers, cookies, frozen pizza, etc.

We don’t want to make you do too much typing so we’ve gone ahead and entered some expenses for you – just some groceries. Pretend that these are yours, you’ve already entered them.

8. Enter expenses from Sureway and café within Sureway

Imagine that you went in to Sureway, your neighborhood grocery store, in the morning on your way to work. It’s easy for you because you like to get a coffee in the morning at Fourbucks, and there’s a Fourbucks located inside of Sureway. So, you go in to get a coffee to drink on your way to the office, and a loaf of bread for home. Here are your receipts.

9. Enter expenses from GoodBuy

Imagine that you bought an accessory to go with your TV at home. You bought it online and had it shipped to the store so you could pick it up. You had it shipped to your local GoodBuy location. Here’s your order information.

10. Enter expenses from dinner and a movie

Now, it’s your friend’s birthday so you treat them to a night out. Dinner at a restaurant called Harvest Table and a movie at J Street Cinema, and you paid for both. Here’s your receipt from dinner and the two movie ticket stubs.

11. Enter parking meter fees

While you’re out that night, you had to pay for parking. You don’t get a receipt so you enter the information as soon as you get home - the meter cost $2.25.

12. Enter payment for car insurance bill

You get your car insurance bill in the mail and send off the payment by check. Here’s the bill. Please enter this expense into the diary.

13. Edit parking expense

You remembered that actually you had to feed the meter twice when you were out with your friend. Go back to that entry for parking and change the total expense to be $5.50.

Post-task follow-up questions

How easy or difficult was that task you just finished?
Present a show card with the following options:
Very Easy, Easy, Somewhat Easy, Neither Easy nor Difficult, Somewhat Difficult, Difficult, Very Difficult
Probe for reason, if any.

Appendix C: Receipts

T ask 2 receipt Task 3 receipt

Appendix C: Receipts

T ask 4 receipt

Appendix C: Receipts

Task 6 receipt

Appendix C: Receipts

Task 7 receipt

Appendix C: Receipts

T ask 8 receipts

Appendix C: Receipts

Task 9 receipt

Appendix C: Receipts

T ask 10 receipts

Appendix C: Receipts

T ask 11 receipt

Task 12 receipt

Appendix D: Debriefing Questions

I’d like to ask you some questions about your experience with the diary just now.

1. What did you think of the diary?

2. Overall, was it easy or difficult to use? Very Easy, Easy, Somewhat Easy, Neither Easy nor Difficult, Somewhat Difficult, Difficult, Very Difficult

3. How confident did you feel in filling out the entries in the diary? Not at all Confident, A Little Confident, Somewhat Confident, Very Confident, Extremely Confident

4. Were there any tasks that you found particularly difficult?

5. When I asked you to edit or delete an expense, was it easy or difficult to find that expense? Very Easy, Somewhat Easy, Neither Easy nor Difficult, Somewhat Difficult, Very Difficult

6. Thinking about the way the diary was organized – was it easy to find what you were looking for?

7. Did the organization of the diary (refer to items or outlets, depending on the condition) make it easy or difficult to enter your expenses?

8. Say you had agreed to participate in this study in real life. How long could you see yourself using this diary to enter your expenses every day?

9. Is there anything you would change to improve it?

   1. Are there any functions or features that would make the diary better or easier to use?

And now I’d like to switch gears and ask you some questions about your shopping habits, in general, in real life.

1. Thinking about your shopping for the last week - think to yourself, you don’t have to tell me – can you remember whether you spent any money, on anything big or small?

   1. If No – ok, maybe think about any shopping from the last few months or so? Can you remember whether you spent any money?

Ok, thinking about that now, what is meaningful to you? Can you remember the individual items that you bought? Can you remember the stores that you shopped at? Can you remember who you bought it for or who you were with when you bought it?

Appendix E: Consent form

CONSENT FORM

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: February 28, 2015

Appendix E: Consent form

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

4