Note to Reviewer - CE Web and Mobile Diary

December 17, 2015

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Brandon Kopp Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Usability Testing of the Consumer Expenditure Web and Mobile Diaries

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 33.2.

If there are any questions regarding this project, please contact Brandon Kopp at

202-691-7514.

1. Introduction and Purpose

The Consumer Expenditure Survey (CE) Program currently uses a paper diary to collect household expenditures but is redesigning the diary as part of ongoing improvements to the survey. Changes to the survey include the addition of desktop Internet and mobile Internet modes of data collection. These modes of collection for the CE Diary Survey are referred to as the Web Diary and Mobile Diary, respectively.

Both the Web and Mobile Diaries went through several rounds of usability testing prior to more extensive field testing. Following this testing, it was determined that (1) the Web and Mobile Diaries should interact with one another so that a respondent can enter data through either or both and (2) that the overall look of the Web and Mobile sites should be similar to make transitions between them more seamless for respondents. These decisions resulted in extensive changes being made to the Web Diary.

The current usability test will address the following objectives:

1. Usability of the redesigned Web Diary: Is the redesigned Web Diary usable for research participants? Are there any changes that can be made to improve the experience of future CE respondents?

2. Interoperability of the Web and Mobile Diaries: Can participants learn how to login and use both systems as intended? Does switching from one mode to the other cause problems? Do participants prefer one mode over the other? What expectations do participants have regarding the interoperability of the two modes?

3. Diary placement instructions: What instructions will be most helpful in teaching participants how to use the diary?

2. Research Design

Participants will come to the Office of Survey Methods Research (OSMR) research lab at BLS. The testing will take place in a usability testing room with a facilitator present. Participants will be given access to the Web Diary on a desktop computer and they will be asked to use their own smartphone to access the Mobile Diary. A video camera will be set up to record the screen of a mobile device and software will record the screen of the desktop computer. Both recordings will capture the device screen, the participants’ actions (e.g., swipes, taps, or clicks), the participants’ voice, and the facilitator’s voice. Observers will monitor the session from an adjacent room.

The session facilitator will begin by introducing the purpose of the study (Appendix A). The facilitator will then instruct the participant on what information should be entered into the diary and how to use both the Mobile and Web versions of the diary. These instructions will be similar to those provided by Census Bureau field representatives when they place the diary in respondents’ homes.

The session facilitator will ask respondents to complete 15 tasks using the diary (See Appendix B for diary screenshots and Appendix C for protocol). Participants will be asked to complete some of the tasks using the Web Diary and others using the Mobile diary. During later tasks, participants will be given the choice to use either. The tasks that are specific to a device will be counterbalanced so that half of the participants will complete those tasks on one device (e.g., the desktop computer) and the other half will complete those tasks on the other device (e.g., the smartphone). This counterbalancing is spelled out in more detail in Appendix C.

Following each task, participants will be asked to rate the ease or difficulty of the task using a seven-point scale ranging from “Extremely Difficult” to “Extremely Easy.” The facilitator will verbally ask for the rating after each task and record each participant’s responses. Once the tasks are complete, the facilitator will ask a series of follow-up questions about participants’ overall experience with the instrument (Appendix D). If, at any time during the testing session, a participant has a noteworthy issue with one of the tasks, the facilitator may spontaneously probe to understand the issue.

Participants’ success or failure at each task and any problems they had while completing the tasks will be noted by observers according to the following two ratings: fully successful or not successful. A participant is ‘fully successful’ in completing a task when he or she completes the stated task as intended with no help from the facilitator. For tasks that require the entry of expenses into the diary (including multiple expense tasks), this rating will only be given if all information is entered correctly. That is, the correct date, price, and category. The description should give enough information for human coders to place the item into a CE-relevant category and category-specific questions should also be answered correctly. A participant is ‘not successful’ when he or she does not complete the task as intended or can only do so after asking the facilitator for assistance (though the participant can consult help materials). For tasks that require the entry of expenses into the diary, the rating of ‘not successful’ will be given if the participant enters unusable data. For example, if a participant enters a full receipt as a single entry (when item level detail is required) or if the participant enters the description of an item that is insufficient to classify it into a CE-relevant category. The entry of incorrect information should also be rated as ‘not successful.’ For tasks that require the editing of expenses, the rating will be given if the participant alters the data in a way other than intended. For example, if a participant completes the “Delete Jeans” task by removing the cost and resaving the expense. Compatibility of the diary with participants’ own devices across a range of platforms will be monitored throughout the study across all tasks.

3. Participants

Thirty participants will be recruited from a database maintained by the Office of Survey Methods Research (OSMR). During the recruiting process, recruiters will screen individuals to identify those who are part of our target population of individuals with experience using smartphones (Appendix E: Screening Questions). Selected individuals will be directed to come to the OSMR lab at BLS. Screening Question 1, 3, and 4 will be used to identify participants with the desired characteristics of owning and having varying levels of experience with smartphones. Questions 2 will be used to ensure a range of platforms is represented in the sample. We will aim to balance participants across these groups. We will also seek to balance the number of participants with characteristics associated with level of household spending; education, income, and household size. This information is already in the OSMR database and will not require additional screening questions.

4. Burden Hours

Our goal is to obtain feedback from 30 participants in the OSMR lab. We anticipate that each session will last no longer than one hour. Screening potential participants from the OSMR database is estimated to take an additional 5 minutes per participant. Since these will be participants who have worked with us before and because smartphones are common, we believe we will have a high success rate (i.e., participants who are eligible and willing to participate); approximately 80%. This means we will have to screen 38 people in order to find 30 eligible participants. Burden hours for screening and participation are expected to total 33.2.

5. Payment to Respondents

Participants will receive $40 for their time.

6. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of the Consumer Expenditure mobile diary survey. Participants will be given a consent form to read and sign (Appendix F). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

Appendix A: Introduction

• Hi! Thank you for coming in today.

• I have a couple colleagues in the next room that will be observing and taking notes.

• Explanation of the study purpose:

  • Today we’re going to be testing a part of the Consumer Expenditure survey. This survey collects information about how US households spend their money. We currently ask people to complete a paper diary to keep track of expenses, but with the increased use of the Internet, we want to give people the option of entering this information on their computers and smartphones. Today we’ll be working with a desktop Internet version and a mobile Internet version of the website.

  • It’s important to note that this is not a test of your ability. Our purpose today is to find out how easy or difficult it is to use the diary and to identify ways to improve it. Your feedback now can help make our future work more productive.

• Consent Form and permission to audiotape

• Any questions before we begin?

Appendix B: Screenshots of Mobile and Web Diaries

Login	Initial Setup	Expense Summary
Common Entry	Food Away From Home	Food At Home

Clothing	Other Expense	Edit Expense

Login

Initial Setup

Expense Summary

Common Entry

Food Away From Home

Food At Home

Clothing

Other Expense

Appendix C: Interview Protocol

NOTES: Dates highlighted in yellow will change depending on timing of the testing and the date range set in the diary.

Rows shaded in green note where there is a change between the Web and Mobile diary.

Diary Task Instructions

Now I’m going to ask you to complete a series of tasks using both the mobile and computer diary. We are trying to simulate what it would be like for you to complete the diary. If you have any questions while we’re going through, remember that there are materials here and help menus within the diary.

Do you have any questions before we begin?

	Task Title and Instructions	Group 1	Group 2
1.	Navigate to Website and Log In	Mobile	Web
	Let’s get started. First, I’d like you to use [this computer/your smartphone] to navigate to this website and log into the diary using this Username and Password… CE Mobile Diary URL: http://respond.census.gov/poc Username: [FILL USERNAME] Password: [FILL PASSWORD]
2.	Set Start Date	Mobile	Web
	Next, you will see a screen asking you to select your start date. Please select January 20th as your start date. Below the start date, you will see that we ask for your e-mail address. You can skip that box. Please select the “Continue” button.
3.	Enter Own Food Expense	Mobile	Web
	Think back to the last purchase you made, other than food. Please add that item to the diary as if the purchase was made on January 21st.
4.	Navigate to Website and Log In	Web	Mobile
	Now, I would like you to navigate to the website using [the computer/your smartphone] and login. You can use the login information I gave you earlier.
5.	Enter Own Non-Food Purchase	Web	Mobile
	Think back to the last food purchase you made. Please add that item to the diary as if the purchase was made on January 21st.
6.	Enter Book for Friend, Enter Jeans for Self	Web	Mobile
	On January 22nd, you go shopping and buy a book for a friend and a pair of jeans for yourself. Here are the receipts. Please enter these expenses into the diary.
7.	Enter Dinner, Enter Movie	Web	Mobile
	After shopping, you go to dinner and a movie with a friend and you pay for both. You decide to enter the purchases into the diary [as you’re waiting for the movie to begin/once you get home]. Here is your ticket stub and the receipt from dinner.
8.	Enter Car Insurance Bill	Web	Mobile
	On January 23rd, you pay your car insurance bill online. This is the billing statement. Please enter this expense into the diary.
9.	Change Book Details	Web	Mobile
	You also decide that, rather than give the book to your friend, you are going to keep it for yourself. Please update that item to reflect that the book was purchased for you.
10.	Switch Modes	Mobile	Web
	We will now switch to the [computer/smartphone]. Please open the diary and login on that device.
11.	Delete Pants	Mobile	Web
	On the 24th, you decide to return the pants that you had bought. Please go back and delete that item.
12.	Enter Shoes	Mobile	Web
	While you’re out returning the pants, you stop at Percy’s department store and buy a pair of running shoes. Here is the receipt. Please enter the shoes into the diary.
13.	Enter Multiple Items
	On January 25th, you go shopping for groceries. Here is the receipt.
14.	Enter Food Away Expense	Mobile	Web
	On January 26th, you stop at a coffee shop on your way to work and pick up breakfast; coffee and a bagel. Here is your receipt.
15.	Enter Long Receipt	Choice	Choice
	For the next task, you can enter the information on either the computer or mobile version. Whichever you think will be easier for you. On the 25th, you go to the Val-U-Mart superstore to buy a few things for your house – enter your expenses from this receipt

Appendix D: Follow-up Questions

1. What did you think of the mobile diary?

1. Was it easy or difficult to use? Very Easy, Somewhat Easy, A Little Easy, Neither Easy nor Difficult, A Little Difficult, Somewhat Difficult, Very Difficult

2. How confident did you feel in filling out the entries in the diary? Not at all Confident, A Little Confident, Somewhat Confident, Very Confident, Extremely Confident

2. What did you think of the computer diary?

1. Was it easy or difficult to use? Very Easy, Somewhat Easy, A Little Easy, Neither Easy nor Difficult, A Little Difficult, Somewhat Difficult, Very Difficult

2. How confident did you feel in filling out the entries in the diary? Not at all Confident, A Little Confident, Somewhat Confident, Very Confident, Extremely Confident

3. Did you prefer one version over the other?

   1. Which One?

   2. Why?

4. If you had a choice between the versions you just used and a paper diary, which would you prefer? Why?

5. Imagine that you agreed to keep track of your expenses for a week. Do you think you would use one of these versions more than the other?

1. About what percentage of the time would you use the Mobile version? How about the computer version?

2. Do you think you would use one or the other in certain situations? Tell me more about that.

6. Is there anything you would change about either version of the diary?

   1. Are there any functions or features that would make either of them better or easier to use?

Appendix E: Screening Questions

1. Do you own a smartphone or tablet?

IF UNSURE –

A smartphone is a phone on which you can easily access emails, download files and applications, as well as view websites and generally surf the internet. Popular brands of smartphone include iPhone, Blackberry, and Android phones such as the Samsung Galaxy.

A tablet is a device on which you can easily access emails, download files and applications, as well as view websites and generally surf the internet. The typical screen size of a tablet is larger than a smartphone. Popular brands of tablets include iPad, Kindle Fire, and Google Nexus.

• Yes (Go to Q2)

• No (Not eligible to participate)

• Don’t know (Not eligible to participate)

2. Which operating system (OS) is on your device?

   • iOS/Apple/iPhone/iPad

   • Blackberry/RIM

   • Android/Google

   • Windows

   • Palm

   • Other

   • Don’t Know/Not Sure

3. (If Yes to Q1) How much experience do you have with using smartphones or tablets?

• None (Not eligible to participate)

• A Little

• A Moderate Amount

• A Lot

4. (If Yes to Q1) How much experience do you have with using applications or apps on smartphones or tablets?

• None (Not eligible to participate)

• A Little

• A Moderate Amount

• A Lot

Appendix F: Informed Consent

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141 and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 02-28-2015

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

4