Note to Reviewer - Usability Testing of the CE Mobilie Diary Web Application - Phase 2

September 10, 2013

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Brandon Kopp Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Usability Testing of the Consumer Expenditure Mobile Diary Web Application – Phase 2

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 32.5.

If there are any questions regarding this project, please contact Brandon Kopp at

202-691-7514 or Jean Fox at 202-691-7370.

1. Introduction and Purpose

The proposed study is a follow-up to the recently completed study: Usability Testing of the Consumer Expenditure Mobile Diary Web Application.

The Consumer Expenditure Survey (CE) Program currently uses a paper diary to collect household expenditures but is redesigning the diary as part of ongoing improvements to the survey. Changes to the survey include the addition of web and mobile modes of data collection. The primary goal of the mobile web version of the diary is to solve a recurring data collection challenge: collecting accurate data on purchases that do not yield a receipt or are forgotten before one returns home to enter items into the diary.

To accomplish this goal, the Bureau of Labor Statistics (BLS) and the U.S. Census Bureau (Census) designed a web diary instrument for use on smartphones and tablets (hereafter, referred to as smartphones). A wireframe prototype of this instrument was tested during the usability study “Usability Testing of the Consumer Expenditure Mobile Diary Web Application.” That initial study provided feedback on possible problems with the diary as well as recommendations for the optimal design of the diary. The study concluded that participants were able to easily use the diary and quickly complete tasks typical of diary data entry, though some modifications to the instrument and help materials are necessary.

The current usability test will address the following objectives:

1. Usability on respondents’ own devices and compatibility across platforms: The wireframe prototype approximated the screen size of typical smartphones but used a desktop keyboard for data entry. A usability test of the instrument prototype as it would be used by a respondent on his or her own device is needed.

2. Diary placement instructions: Some participants did not understand the level of information that was needed for different types of expenses. Testing is needed to understand how placement instructions and respondent help materials affect participants’ ability to complete tasks accurately and efficiently with no help from the experimenter.

3. Navigational symbols and screen organization: Some participants did not understand the function of several buttons. Also, some participants raised concerns about the organization of information in the instrument. The navigation and organization have been revised and the changes need to be tested.

4. Long item lists: The wireframe prototype was not designed to accept more than five items. Furthermore, some participants expressed an unwillingness to enter many items during probing about a hypothetical scenario. A task that requires the entry of many items was added to the current study to ensure the web diary facilitates the entry of many items and to obtain input from participants about entering many items.

2. Research Design

Participants will come to the Office of Survey Methods Research (OSMR) research lab at BLS. The testing will take place in a usability testing room with a facilitator and a video camera set up to record the screen of a mobile device. This recording will capture the smartphone screen, the participants’ actions (e.g., swipes and taps), the participants’ voice, and the facilitator’s voice. Observers will monitor the session from an adjacent room.

The session facilitator will begin by introducing the purpose of the study (Appendix A). Thereafter, the instructions given to participants will be determined by a split-ballot design, to understand the effectiveness of the proposed placement instructions. Half of the participants will receive a full placement protocol, as if from a field representative, while the other half of the participants will receive only the pamphlet as a reference resource without any additional guidance (Appendix B). Tasks are divided into two blocks and the order of the blocks will be counterbalanced across participants to minimize order effects.

The session facilitator will ask respondents to complete 13 tasks using the diary (See Appendix C for diary screenshots and Appendix D for protocol). Following each task, participants will be asked to rate the ease or difficulty of the task using a seven-point scale ranging from “Very Difficult” to “Very Easy.” The facilitator will verbally ask for the rating after each task and record each participant’s responses. Once the tasks are complete, the facilitator will ask a series of follow-up questions about participants’ overall experience with the instrument and placement (Appendix E). If, at any time during the testing session, a participant has a noteworthy issue with one of the tasks, the facilitator may spontaneously probe to understand the issue.

Participants’ success or failure at each task and any problems they had while completing the tasks will be noted by observers according to the following two ratings: fully successful or not successful. A participant is ‘fully successful’ in completing a task when he or she completes the stated task as intended with no help from the facilitator. For tasks that require the entry of expenses into the diary (including multiple expense tasks), this rating should only be given if all information is entered correctly. That is, the correct date, price, and category. The description should give enough information to place it into a CE-relevant category and category-specific questions should also be answered correctly. A participant is ‘not successful’ when he or she does not complete the task as intended or can only do so after asking the facilitator for assistance (though the participant can consult help materials). For tasks that require the entry of expenses into the diary, the rating of ‘not successful’ should be given if the participant enters unusable data. For example, if a participant enters a full receipt as a single entry (when item level detail is required) or if the participant enters the description of an item that is insufficient to classify it into a CE-relevant category. The entry of incorrect information should also be rated as ‘not successful.’ For tasks that require the editing of expenses, the rating should be given if the participant alters the data in a way other than intended. For example, if a participant completes the “Delete Jeans” task by removing the cost and resaving the expense. Compatibility of the diary with participants’ own devices across a range of platforms will be monitored throughout the study across all tasks.

3. Participants

Census will recruit 30 participants from a Census-maintained participant database. During the recruiting process, recruiters will screen individuals to identify those who are part of our target population of individuals with experience using smartphones (Appendix F: Screening Questions). Selected individuals will be directed to come to the OSMR lab at BLS. Screening Question 1, 3, and 4 will be used to identify participants with the desired characteristics of owning and having varying levels of experience with smartphones. Questions 2 will be used to ensure a range of platforms is represented in the sample. We will aim to balance participants across these groups. We will also seek to balance the number of participants with characteristics associated with level of household spending; education, income, and household size. This information is already in the Census database and will not require additional screening questions.

4. Burden Hours

Our goal is to obtain feedback from 30 participants in the OSMR lab. We anticipate that each session will average 45 minutes and last no longer than one hour. Screening potential participants from the Census database is estimated to take an additional 5 minutes per participant. Burden hours for screening and participation are expected to total no more than 65 minutes per participant, for a total of 32.5 burden hours. Participants will receive $40 for their time.

5. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of the Consumer Expenditure mobile diary survey. Participants will be given a consent form to read and sign (Appendix G). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

Appendix A: Introduction

• Hi! Thank you for coming in today.

• I have a couple colleagues in the next room that will be observing and taking notes.

• Explanation of the study purpose:

  • Today we’re going to be testing a part of the Consumer Expenditure survey. This survey collects information about how US households spend their money. We currently ask people to complete a paper diary to keep track of expenses, but with the increased use of smartphones, we want to give people the option of entering this information into their phones.

  • It’s important to note that this is not a test of your ability. Our purpose today is to find out how easy or difficult it is to use the diary and to identify ways to improve it. Your feedback now can help make our future work more productive.

• Consent Form and permission to audiotape

• Any questions before we begin?

Appendix B: Placement Protocol and Pamphlet

Half of the participants will receive only the pamphlet. The other half will receive the pamphlet plus an explanation of the contents of the pamphlet and the basic data requirements of the diary.

Appendix C: Screenshots of Mobile Diary

Login	Initial Setup	Expense Summary
Common Entry	Food Away From Home	Food At Home

Clothing	Other Expense

Appendix D: Interview Protocol

Diary Task Instructions

Now I’m going to ask you to complete a series of tasks using the diary. We are trying to simulate what it would be like for you to complete the diary. If you have any questions while we’re going through, remember that there are materials here and help menus within the diary.

Do you have any questions before we begin?

Block A

1. Log In

Let’s get started. First, I’d like you to use this Username and password to log into the diary…

2. Set Personalized Password

Next you will be given the option to set a personalized password. You will be logging into the diary several times, so you will want to use something you can remember. Please do not use a password that you use somewhere else, like your email. It’s important that you don’t forget the password since we can’t quickly reset it, so do whatever you would normally do to keep track of a password.

3. Set Start Date

Next, you will see a screen asking you to select your start date. Please select October 21st as your start date. Below the start date, you will see that we ask for your e-mail address. You can skip that box. Please select the “Continue” button.

4. Enter Own Food Expense

Think back to the last food purchase you made. Please add that item to the diary as if the purchase was made on October 21^st.

Log Out – Once you’ve entered your expense, please log out.

5. Enter Book for Friend, Enter Jeans for Self

On October 22^nd, you go shopping and buy a book for a friend and a pair of jeans for yourself. Here are the receipts. Please enter these expenses into the diary.

Log Out – Once you’ve entered these expenses, please log out.

6. Enter Car Insurance Bill

Later on the 22^nd, you pay your car insurance bill online. This is the billing statement. Please enter this expense into the diary.

Log Out – Once you’ve entered your expense, please log out.

7. Delete Pants

The next day, on the 23^rd, you decide to return the pants that you had bought. Please go back and delete that item.

8. Change Book Details

You also decide that, rather than give the book to your friend, you are going to keep it for yourself. Please update that item to reflect that the book was purchased for you.

Log Out – Once you’ve changed the information for that item, please log out.

Block B

1. Enter Own Non-Food Purchase

Think back to the last purchase you made, other than food. Please add that item to the diary as if the purchase was made on October 23^rd.

Log Out – Once you’ve entered your expense, please log out.

2. Enter Dinner, Enter Movie

On October 24^th, you treat a friend to dinner and a movie and you pay for both. You decide to enter the purchases into the diary as you’re waiting for the movie to begin. Here is your ticket stub and the receipt from dinner.

3. Enter Drinks

On the way home from the movie, you and your friend stop to get a couple drinks. Here is the receipt. Please enter this expense into the diary.

Log Out – Once you’ve entered this expense, please log out.

4. Edit Price of Drinks Purchase

The next day you realize that you had left a $5 bill as a tip for the drinks you purchased, but forgot to enter that as part of the expense. Please change the entry to reflect the full price paid for the drinks.

5. Enter Long Receipt

On the 25^th, you go to the Val-U-Mart superstore to buy a few things for your house – enter your expenses from this receipt.

Block C (if participant did not use buttons during testing)

I have a few additional questions for you about the diary.

1. Ask about INFO button

[Point to INFO button] What do you think that this button does?

2. Ask about HOME button

Please press the button to add an expense.

[Point to the HOME button] What do you think that this button does?

3. Ask about HELP button

[Point to the HELP button] What do you think that this button does?

(if help) What would you expect to find on the help screen?

Appendix E: Follow-up Questions

1. What did you think of the mobile diary?

2. Was it easy or difficult to use? Very Easy, Somewhat Easy, Neither Easy nor Difficult, Somewhat Difficult, Very Difficult

3. How confident did you feel in filling out the entries in the diary? Not at all Confident, A Little Confident, Somewhat Confident, Very Confident, Extremely Confident

4. How much training do you think the average person would need to get started on using the diary? None, A Little, A Moderate Amount, A Lot

5. Were there any tasks that you found particularly difficult?

6. Is there anything you would change to improve it?

   1. Are there any functions or features that would make the mobile diary better or easier to use?

7. Imagine I came to your door and asked you to take part in a survey about what you spend money on. Would you be willing to use a mobile diary like this to keep track of your expenses?

1. How long would you be willing to use this mobile diary to keep track of your expenses?

2. Do you think you would need technical assistance?

3. Would you rather fill out a diary on a phone like this or on a computer? Why?

4. Would you rather fill out a diary on a phone or a paper diary? Why?

IF YES to 7 – indicates willingness to use the app

8. Imagine that you have agreed to keep track of your expenses for a week and you have the web diary on your phone.

1. When would you record your expenses? Check all that apply.

• Right away, at the cash register

• Right after I leave the store or make the payment

• Whenever I have downtime throughout the day

• At the end of the day

• Whenever I have downtime throughout the week

• At the end of the week

2. How would you record your expenses? Check all that apply.

• Keep the receipts

• Use the app when I don’t have a receipt

• Use the app when I only have one expense to enter

• Make notes on paper to refer to later

• Make notes in another budget app to refer to later

• Use my memory to remember expenses

9. You had some experience with entering multiple items from a single receipt. How many items would you be willing to enter from a single receipt? ________

10. You logged into the web diary several times. What did you think of that experience?

Appendix F: Screening Questions

1. Do you own a smartphone or tablet?

IF UNSURE –

A smartphone is a phone on which you can easily access emails, download files and applications, as well as view websites and generally surf the internet. Popular brands of smartphone include iPhone, Blackberry, and Android phones such as the Samsung Galaxy.

A tablet is a device on which you can easily access emails, download files and applications, as well as view websites and generally surf the internet. The typical screen size of a tablet is larger than a smartphone. Popular brands of tablets include iPad, Kindle Fire, and Google Nexus.

• Yes (Go to Q2)

• No (Not eligible to participate)

• Don’t know (Not eligible to participate)

2. Which operating system (OS) is on your device?

   • iOS/Apple/iPhone/iPad

   • Blackberry/RIM

   • Android/Google

   • Windows

   • Palm

   • Other

   • Don’t Know/Not Sure

3. (If Yes to Q1) How much experience do you have with using smartphones or tablets?

• None (Not eligible to participate)

• A Little

• A Moderate Amount

• A Lot

4. (If Yes to Q1) How much experience do you have with using applications or apps on smartphones or tablets?

• None (Not eligible to participate)

• A Little

• A Moderate Amount

• A Lot

Appendix G: Informed Consent

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 02-28-2015

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

4