Note to Reviewer - Mobile Diary Web Application

June 13, 2013

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Brandon Kopp Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Usability Testing of the Consumer Expenditure Mobile Diary Web Application

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 10.

If there are any questions regarding this project, please contact Brandon Kopp at

202-691-7514 or Jean Fox at 202-691-7370.

1. Introduction and Purpose

The Consumer Expenditure Survey (CE) Program currently uses a paper diary to collect household expenditures. As part of ongoing improvements to the survey, the Bureau of Labor Statistics (BLS) and the Census Bureau (Census) have begun field testing a web-based diary instrument. The web diary may help some respondents with data entry and it has the potential to lead to more complete responses through periodic data checks by field representatives during the collection period. It does not solve one recurring data collection problem however; collecting accurate data on those purchases that do not yield a receipt and/or are forgotten before one returns home to enter items into the diary. To help solve this issue, BLS and Census are designing a version of the web-diary specifically for use on a smartphone. The usability testing described below will provide feedback on the optimal design of the mobile web diary.

2. Research Design

This testing will use a functioning wireframe prototype of the instrument (see Appendix C). Although the final product is a mobile-optimized web application, we are early in development, so we do not have a version that will run on mobile devices yet. Therefore, we will conduct this study with a prototype we built in PowerPoint, which runs on a laptop/desktop. We believe that doing this testing early, before much coding has been done, will give us some good insights about the instrument’s usability in time to incorporate changes. We plan to do a second round of testing on mobile devices when we have a functioning website to ensure that the instrument is usable when used as intended.

Participants will come to the OSMR research lab at BLS. They will be seated at a computer running Morae software. This software will record a video of the computer screen as well as the participant and facilitator’s voices during the session. Several observers will monitor the session from an adjacent room.

The session facilitator will first cover the introductory material in Appendix A; collect written, informed consent; and receive verbal permission from the participant to record the testing session. The facilitator will then ask respondents to complete fourteen tasks using the wireframe diary (see Appendix B). These tasks will center around the key functions of the diary; expense entry, editing, and deletion. For twelve of the tasks, participants will be given a scenario or fake receipts for data entry. For two, non-directed tasks, participants will be asked to enter information they remember from a recent purchase. Following each task, participants will be asked to rate the ease or difficulty of the task using a seven-point scale ranging from “Very Difficult” to “Very Easy” using the Morae software. Participants’ success or failure at the task and any problems they had while completing the task will be noted by the observers. Blocks of the task (see Appendix B) will be counterbalanced so that some participants will receive Block A first and some will receive Block B first.

Once the fourteen tasks are complete, participants will be asked a series of follow-up questions about their overall experience (Appendix B). If a participant had a noteworthy issue with one of the tasks, the facilitator may ask extemporaneous questions to garner feedback on why the issue occurred and potential ways to fix it.

3. Participants

We will recruit 10 participants from the OSMR participant database. During the recruiting process, we will screen potential participants to find participants who have experience using smartphones, since that is the target audience for the instrument (see Appendix D for all the screening questions). Screening Question 1 will be used to identify participants with the desired experience. Questions 2 and 3 will be used to help stratify the group of participants. We will aim to balance participants across these groups.

4. Burden Hours

Our goal is to obtain feedback from 10 participants in the OSMR lab. We anticipate that each session will average 45 minutes and last no longer than one hour, for a total of 10 burden hours. Participants will be compensated $40 for their time.

5. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of the Consumer Expenditure mobile diary survey. Participants will be given a consent form to read and sign (See Appendix E). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

Appendix A: Introduction

• Hi! Thank you for coming in today.

• Have you participated in any of our studies before? (if yes, Which ones?)

• I have a couple colleagues in the next room that will be able to see this computer screen and they are able to see and hear us. They will be taking notes.

• Consent Form/Permission to audiotape

• Explanation:

  • We currently ask people to complete a paper diary where they enter what they bought, the cost of each item, and some other information. With the increased use of smartphones, we want to give people the option of entering this information into their phones.

  • Today, I’ll ask you to enter a variety of expenses into the smartphone diary and then ask you about your experience. In some cases, I’ll provide you with a scenario or receipt that has information to enter. In others, I’ll ask you to enter information about something you’ve purchased recently.

  • It’s important to note that this is not a test of your ability. Our purpose today is to find out how easy or difficult it is to use the diary and to identify ways to improve it. As you will see in a second, we are at a very early stage in the design process, so your feedback now can help make our future work more productive.

• Any questions before we begin?

Appendix B: Cognitive Testing Protocol

NOTE: Text in italics will be read to participants

Block A

1. Log In

Let’s get started. First, I’d like you to use this UserID and password to login to the diary…

2. Enter start date and email address on start screen

You will see a screen asking you to select your start date. Please select June 1^st as your start date

Below the start date, you will see that we ask for your e-mail address so that we can send you reminders during the period that you are assigned to enter your expenses. You can skip that box. Please select the “Continue” button

3. Review information screen

Looking at this screen, if you were interested in finding information about the survey, where would you go?

4. Entering expenses

1. Think back to the last purchase you made, other than food. Please add that item to the diary as if the purchase was made on June 1^st.

Now, imagine that today is Sunday, June 2^nd…

2. You’re at the mall shopping and buy a pair of jeans for yourself for $61.95. Please add that item to the diary.

3. On your way home, you stop at the cleaners to pick-up some dry cleaning. You paid $32.50. Please enter that expense.

4. That evening, you meet up with a friend and buy dinner for you and your friend. Here is the receipt. For meals out, we only need the total that you spent on the meal including tax and tip, not the cost of the individual items.

5. After dinner, you remember that earlier while shopping you had also purchased a DVD – you can’t remember exactly how much you paid but think it was approximately $15. Please enter that item.

5. Editing the Cost

Later at home, you find the receipt for the DVD you bought earlier that day. See the receipt below. Please change the entry to reflect the correct price of the DVD.

6. Deleting an item

The next day you decide that the jeans you bought were really too expensive and take it back to the store to return it. Please go back and delete that item.

Block B

1. Log In

Now, I am going to have you log out of the diary and log back in. You can use the same UserID and Password

2. Entering expenses

1. Think back to the last food purchase you made. Please add that item to the diary as if the purchase was made on June 1^st.

Now, imagine that today is Monday, June 3^nd …

2. You just went to the grocery store to pick up a few items – enter the items you purchased from the receipt below: Please keep in mind that for multiple items purchased at a store, we would like you to enter the individual items separately. For each item, you can enter the item cost before tax but do include any coupons or discounts that are applied.

3. You then remember that you had written a check to pay your electric bill yesterday on Sunday, June 2^nd but had forgotten to enter the expense. Add the $126.00 that you paid on Sunday, June 2^nd for the electric bill.

3. Editing the Date

Later on, you remember that the electric bill you paid was actually on Saturday, June 1^st, not Sunday, June 2^nd. Please change that entry to reflect the correct date.

Follow-up Questions

1. What did you think of the mobile diary?

2. Was it easy or difficult to use? Very Easy, Somewhat Easy, Neither Easy nor Difficult, Somewhat Difficult, Very Difficult

3. How confident did you feel in filling out the entries in the diary? Not at all Confident, A Little Confident, Somewhat Confident, Very Confident, Extremely Confident

4. How much training do you think the average person would need to get started on using the diary? None, A Little, A Moderate Amount, A Lot

5. Were there any tasks that you found particularly difficult?

6. Is there anything you would change to improve it?

   1. Are there any functions or features that would make the mobile diary better or easier to use?

7. Imagine I came to your door and asked you to take part in a survey about what you spend money on. Would you be willing to use an instrument like this to keep track of your expenses?

   1. How long would you be willing to use this mobile diary to keep track of your expenses?

   2. Do you think you would need technical assistance?

   3. Would you rather fill out a diary on a phone like this or on a computer? Why?

   4. Would you rather fill out a diary on a phone or a paper diary? Why?

Appendix C: Mobile Diary Screenshots

Login	Initial Setup	Information
Expense Summary	Common Entry	Food Away From Home

Food At Home	Clothing	Other Expense
Edit
	NOTE: Edit screens look like the corresponding data entry screen, with the exceptions of the “Delete” button, the font color, and the fact that previously entered data is pre-filled. To save file size, the “Food Away From Home” Edit screen on the left is shown as an example.

Appendix D: Screening Questions

1. Do you own and use a smartphone?

IF UNSURE - A smartphone is a phone on which you can easily access emails, download files and applications, as well as view websites and generally surf the internet. Popular brands of smartphone include iPhone, Blackberry, and Android phones such as the Samsung Galaxy.

• Yes (Go to Q2)

• No (Not eligible to participate)

• Don’t know (Not eligible to participate)

2. (If Yes to Q1) How much experience do you have with using a smartphone?

• None (Not eligible to participate)

• A Little (Not eligible to participate)

• A Moderate Amount

• A Lot

3. (If Yes to Q1) How much experience do you have with using applications or apps on a smartphone?

• None (Not eligible to participate)

• A Little

• A Moderate Amount

• A Lot

Appendix E: Consent Form

CONSENT FORM

The Bureau of Labor Statistics (BLS) is conducting research to improve the way we collect information on the things people buy. In this study, we will ask you to complete several tasks using a mock-up of a mobile website. These tasks involve entering information about various purchases; some of which will be provided to you in the form of receipts or scenarios and some we will ask you to think of yourself.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate that it will take you no longer than an hour to participate in this research (ranging from 45 to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: February 28, 2015

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

13