Note to Reviewer - CE Mobile Diary Testing

June 4, 2018

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Brandon Kopp Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the ‘Testing of the Redesigned Consumer Expenditure Mobile Diary 2018’ Study

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 23.

If there are any questions regarding this project, please contact Brandon Kopp at 202-691-7514.

1. Introduction and Purpose

In 2017, the Division of Consumer Expenditure Surveys contracted with Westat to build a prototype online web instrument for the Consumer Expenditure Diary Survey (CED). The newly designed instrument was usability tested with a small number of participants¹. Since that time, a number of design changes have been requested, including the removal of an option for individual household members to fill out their own diaries and the addition of fields related to the collection of businesses where expenditures were made (outlets).

This study will assess the usability of an updated prototype web instrument accessed through mobile, looking specifically at whether the inclusion of outlet questions leads to data quality issues. The mobile layout was chosen for testing because the earlier testing had focused on the full-sized web layout. Given limited resources and the additional burden of developing wireframes for the web version, it was decided to focus solely on the mobile layout.

2. Research Design

A wireframe that simulates the mobile web version of the diary was developed in Microsoft Access (See Appendix A). A simulated version was developed to enable staff to test the requested design revisions before finalizing instrument specifications. Although the wireframe does not have the full functioning of the web instrument, lab testing with the wireframe can inform instrument design and training materials. The mock-up of a mobile website will be presented to participants on a computer monitor in the Office of Survey Methods Research usability lab. Camtasia software will be used to record the session. Participants will interact with the wireframe using the computer’s mouse and keyboard.

As this is pretesting, we expect that modifications may be made during the course of the study based on initial interviews/sessions/groups. Although the goals of the testing, and overall design, will remain the same, findings from preliminary results may be used to improve the interview guide, the debriefing questions, or the instrument.

All participants will be given a series of scenarios describing fictional expenses, and be asked to record those expenses in the diary. The participants will also be asked to enter one of their own recent, real-world expenses in the diary. The fictional expense tasks are designed to understand the following issues that may indicate the impact of inclusion of outlets on the CE, related to data quality and burden:

• The collection of outlets may be confusing to participants if outlets are not clearly defined (e.g., a Starbucks within a grocery store, which operates as a separate outlet) or outlets are unknown (e.g., he or she does not know the name for the outlet, such as at a parking meter).

• The collection of outlets may make it more obvious that combined event expenditures (e.g., a night out including dinner and a movie) should be entered separately since they occurred at different outlets.

• For certain types of expenses, the instrument preloads the date of expense, outlet name, and item category, which may confuse participants and lead to lower data quality if they do not edit these fields when needed. However, it is hoped that feature will reduce burden and limit the amount of data entry participants have to do when entering a string of expenses (e.g., entering several grocery purchases).

The protocol is described in Appendix B and accompanying receipts in Appendix C. Debriefing questions are included in Appendix D.

3. Measures of interest

At least two observers will either attend every interview session or watch a screen recording of each interview. The observers will rate whether each participant was successful in completing each task. Task success requires completing the task without assistance from the interviewer; the participant may make errors as long as he or she corrects them before finishing the task. Task failure may be due to misunderstanding of CE data requirements, usability challenges, or any other reason. These ratings will be tallied and reported for qualitative comparison across diary versions. Observer notes will also be considered.

The time taken for each task will be collected using the task timers within the Morae software.

3. Participants

Participants will be recruited from the OSMR participant database. Efforts will be made to select participants with varying levels of age, gender, education, and race based on self-reported information collected during an initial recruitment process and stored in the database. The pool of participants will also be limited to those who have not participated in any previous tests of the CE Diary.

4. Burden Hours

Our goal is to obtain feedback from 20 participants. We anticipate that the interview session will last no longer than one hour. Recruiting and scheduling of up to 30 prospective participants is estimated to take an additional 5 minutes per participant. The study is anticipated to require a total of 23 burden hours. A $40 incentive will be provided to all participants who take part in the study.

	N	Burden Hours/ Participant	Total Burden Hours
Initial Recruitment	30	0.083	2.5
In-Lab Session	20	1	20
	TOTAL	1 hr 5 min	23 hours

5. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of the Consumer Expenditure survey. Participants will be given a consent form to read and sign (Appendix E). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

See the following attachments:

Appendix A – Screenshots

Appendix B – Interview Protocol

Appendix C – Receipts

Appendix D – Debriefing Questions

Appendix E – Consent Form

Appendix A: Screenshots

Screenshots have been cropped for space. Only blank screen space has been cropped out. The screenshots below, including the mock-iPhone border will be presented on a computer monitor. Participants will interact with the device using a mouse and keyboard.

Home Screen	Add Expense – Choose Date	Add Expense – Choose Category

Add Expense – Food Away Questions	Add Expense – Food at Home Questions	Add Expense – Clothing Questions

Add Expense – Other Category Questions	Edit Expense		Description/Cost Information
Type Information

Appendix B: Protocol

Introduction

• Hi! Thank you for coming in today.

• I have a couple colleagues in the next room that will be observing and taking notes.

• Explanation of the study purpose:

  • Today we’re going to be testing a part of the Consumer Expenditure Survey. This survey collects information about how US households spend their money. We currently ask people to complete a diary to keep track of expenses during a week-long period, but now we are trying a few different designs to see what works best.

  • It’s important to note that this is not a test of your ability. Our purpose today is to find out how easy or difficult it is to use the diary and to identify ways to improve it. Your feedback now can help make our future work more productive.

• Consent Form and permission to record screen and audio.

• Any questions before we begin?

Background about the CE Diary

• The Consumer Expenditure Survey is about understanding how US households spend their money. We provide our respondents with a diary, like this paper one, so they can keep track of daily expenses –eating out, groceries, household shopping, that kind of thing. In our session today, I’ll ask you to imagine that you are one of our respondents and you are entering your daily expenses into a diary for us. We ask thousands of people to do this to understand what people are buying.

• We recently developed a test version of the diary, and that’s what we’ll be working with today. Everything will be presented on the computer and you will be using the keyboard and mouse, but this is meant to represent a mobile website.

• I’d like to start our session by walking through the diary – first, how to enter your expenses. Here are some screenshots of the diary to look at while we talk about it.

• You’ll enter each of your expenses separately into the diary. Say I bought a gallon of milk and a loaf of bread at the grocery store yesterday. I would enter those in one at a time. First, I would select yesterday’s date. Then I would type in that I got it from Safeway using the Business field. Then, the diary prompts me to label this expense with one of these pre-assigned labels – it’s Food for Home Consumption. I would then enter the “Description;” “Milk” and then how much I paid; “3.99”. Finally, I answer the follow-up questions. For this one, I would say “Fresh”. And, since I bought it for myself I would answer no to whether I purchased it for someone outside my household.
  
  When I press “Save,” it records everything I entered and then resets the fields, but it leaves the date, the business name, and the item category filled in. This is done so you can enter a list of items from a shopping trip together. If you want to enter items for a different store or a different date, you can press the “clear” button. If you want to enter more items from that same store and date, you can also make changes to the category if necessary. For example, if you also bought toothpaste at Safeway, you could select “All other..” from the category list and enter the remaining information.

• Does that make sense?

• Now you can always come back to your Home screen to see a summary of your expenses. See, here are all of the expenses you entered – the milk and the bread. If you’re ever looking for an item – to check that you entered it or to maybe go back and delete it if you returned the item – then you can look for it here.

• Does that make sense? We can go through another example if you’d like.

• Now, that’s how it will work for most things you enter into the diary. Expenses for eating out are a little different. We care about the entire meal altogether, not each dish you ordered.
  
  You start off the same way – entering the date and the name of the restaurant. You then select “Food and Drinks Away From Home” as the category. Then you type in a description for the meal – maybe “Dinner with Joe”. For this category, we are interested in the total expense for the meal. There are a couple additional questions about what meal it was purchased for and whether alcohol was included.

In that way, it’s different from groceries where we’re interested in the items that you bought. Here, we need to know about the meal as a whole. When you press “Save” you will be brought back to the Home screen.

• Do you have any questions before we get started? You can ask questions as we’re going through but I may give you vague answers because we are interested in finding out how you would use a diary like this if someone wasn’t around. You can use the “i” icons to get more information.

Tasks

Now I’m going to ask you to enter expenses into the diary. Most of them will be made-up expenses. Imagine that you have agreed to take part in our survey and you’re task is to enter the information into the diary. Do you have any questions before we begin?

Ok, I’ll describe each expense to you and present it to you in the form of a receipt or bill. Please enter that expense into the diary. While you’re entering the expense, imagine that you’re at home on your phone and that I’m not here. When you finish entering something into the diary, let me know. I’ll ask you a quick question about how easy or difficult that task was, and then we’ll move on to the next thing.

[NOTE TO INTERVIEWER] After each task ask the following question.

How easy or difficult was it to complete this task?

• Very easy

• Somewhat easy

• Neither easy nor difficult

• Somewhat difficult

• Very difficult

Task	Task Name	Text Read to Respondents
1.	Enter own expenses	Let’s start with something from real life. Think back to the last food purchase you made. Please add that to the diary. Think back to the last non-food purchase you made and enter that into the diary.
2.	Mixed Item Types	You went shopping at Jmart on [DATE]. You got some spray cleaner, some Tupperware, and some milk. Here’s your receipt. Please enter these into the diary.
4.	Multiple of same type	The next day, you stopped by a department store called Percy’s on your way home from work. You picked up two shirts. Please enter that into the diary.
5.	Same Type, Different Outlet	You went back to Jmart and ended up buying a shirt that you had seen the last time you were there. Here’s your receipt.
6.	Delete Item	Now you want to return the button down shirt you bought at Percy’s. Please go into the diary and delete that shirt from the diary.
7.	Online Purchase	That night, you went online to pick out a book for a friend. Here’s your receipt. Please enter this book into the diary.
7.	Long List Same Item Category	You went shopping at ValUMart and picked up several things for your household. Here is your receipt. Please enter these expenses.
8.	Multiple Embedded Outlets	Imagine that you went in to Sureway, your neighborhood grocery store, in the morning on your way to work. It’s easy for you because you like to get a coffee in the morning at Fourbucks, and there’s a Fourbucks located inside of Sureway. So, you go in to get a coffee to drink on your way to the office, and a loaf of bread for home. Here are your receipts.
9.	Verify Expenses	You are worried that you didn’t enter all the purchases from your trip to ValUMart. Check to make sure that you entered everything.
10.	Multiple Expenses, Different Categories	Now, it’s your friend’s birthday so you treat them to a night out. Dinner at a restaurant called Harvest Table and a movie at J Street Cinema, and you paid for both. Here’s your receipt from dinner and the two movie ticket stubs.
11.	Outlet Unknown	The next day when you get to work, the lot is full so you have to park on the street at a meter. You don’t get a receipt so you enter the expense when you get to your desk - the meter cost $2.25.
12.	Bill	You get your life insurance bill in the mail and send off the payment by check. Here’s the bill. Please enter this expense into the diary.

Appendix D: Debriefing Questions

I’d like to ask you some questions about your experience with the diary just now.

1. What did you think of the diary?

2. Overall, was it easy or difficult to use? Very Easy, Easy, Somewhat Easy, Neither Easy nor Difficult, Somewhat Difficult, Difficult, Very Difficult

3. How confident did you feel in filling out the entries in the diary? Not at all Confident, A Little Confident, Somewhat Confident, Very Confident, Extremely Confident

4. Were there any tasks that you found particularly difficult?

5. When I asked you to edit or delete an expense, was it easy or difficult to find that expense? Very Easy, Somewhat Easy, Neither Easy nor Difficult, Somewhat Difficult, Very Difficult

6. Say you had agreed to participate in this survey in real life. How long could you see yourself using this diary to enter your expenses every day?

7. Is there anything you would change to improve it?

8. Are there any functions or features that would make the diary better or easier to use?

Appendix E: Consent form

Consent Form

OMB Control Number: 1220-0141

Expiration Date: March 31, 2021

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 60 minutes to participate in this research.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141 and expires March 31, 2021.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974 as amended (5 U.S.C. 552a), this study is being conducted by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under the authority of 29 U.S.C. 2.  Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections.  The information will only be used by and disclosed to BLS personnel and contractors who need the information for activities related to improving BLS information collection.  Information on routine uses can be found in the system of records notice,  DOL/BLS – 14, BLS Behavioral Science Research Laboratory Project Files (81 FR 47418). 

1 See the following link for OMB Package explaining previous procedures: https://www.reginfo.gov/public/do/PRAViewIC?ref_nbr=201408-1220-002&icID=220548

4