Note to Reviewer - CE Electronic Diary Test

March 9, 2016

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Brandon Kopp Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Usability Testing of Improvements to the CE Electronic Diary.

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 33 hours.

If there are any questions regarding this project, please contact Brandon Kopp at

202-691-7514.

1. Introduction and Purpose

In the Consumer Expenditure Diary Survey (CED), respondents are asked to complete two, one-week diaries in which they record purchases made by their household during that week. The CED and its companion survey the Consumer Expenditure Interview Survey have experienced declining response rates and concerns about measurement error. One recommendation that has been made for addressing these issues in the CED is to provide respondents with the option to record their purchases through a web-based instrument, which presents a number of benefits over paper diaries. An electronic diary was developed for this purpose.

Through several recent usability and field tests, the electronic diary has demonstrated potential, but also some limitations. Participants in usability tests often enter purchases at a higher level of aggregation than is desired for classification (e.g., they enter “groceries” when we need them to enter “milk,” “eggs,” “bread,” etc.). In field tests, response rates for the electronic diary are consistently lower than they are for the paper diary. Also, respondents enter fewer items in the electronic diary than the paper diary.

Due to time and resource constraints prior to a recent field test, some ideas that were suggested for the desktop version of the electronic diary were not explored and implemented. In order to progress towards the improvement and exploration of more options for the electronic diary, the Division of Consumer Expenditure Surveys (DCES) at the Bureau of Labor Statistics obtained the services of a contractor to develop and usability test electronic diary improvements for the desktop version for use in an upcoming field test.

The objectives of this study are to:

1. Conduct usability testing on two, newly designed CE electronic diary instruments, to ensure that respondents are able to properly use the diaries.

2. Provide a recommendation for which design to implement and changes to make in the future.

2. Research Design

Participants will come to the usability research lab at the Westat facility in Rockville, MD. The testing will take place in a usability testing room with a facilitator and a computer that has software to record the actions in the internet browser. This recording will capture the participants’ interaction with the two prototype electronic diaries as well as the participants’ and facilitator’s voices. Observers will monitor the session from an adjacent room.

The session facilitator will begin by introducing the purpose of the study (Appendix A) and obtaining informed consent (see Appendix E for informed consent document). The facilitator will then describe the basic functions of the diary as well as the information we would like the participant to record. The facilitator will then ask respondents to complete 6 tasks using one version of the diary (see Appendix C for diary screenshots). The tasks will consist of asking the participant to enter, edit, and delete expenses using a mixture of expenses from their own experience and fake expenses created for this study (see Appendix B for the protocol). Participants will be encouraged to think aloud while completing the tasks. Following each task, participants will be asked to rate the ease or difficulty of the task using a seven-point scale ranging from “Very Difficult” to “Very Easy.” The facilitator will verbally ask for the rating after each task and record each participant’s responses. After working through the 6 tasks, the facilitator will ask several follow-up questions about that version of the diary (see Appendix B).

The facilitator will then walk the participant through the second version of the diary and ask them to complete 6 tasks with this version (see Appendix C). These tasks will be similar to those used with the first version, but covering different expenditures. The facilitator will then ask follow-up questions specific to that diary (see Appendix B). The order of the two diaries will be counterbalanced so that 15 participants will receive Version 1 first and then 2 and the other half will receive Version 2 first and then 1.

After the participant has completed tasks with both diaries, the facilitator will ask questions about their overall experience with the diaries, including the participants’ preference for one or the other. Finally, the facilitator will follow-up with the participant about any data entry problems he/she had with either diary. Due to the exploratory nature of this research, the facilitator may ask questions that are not listed in Appendix B. These questions will generally be directed at asking the participant to explain their thought processes (e.g., “I see that you ___. Can you tell me more about that.”) or to expand upon an answer given to a debriefing question.

Participants’ success or failure at each task and any problems they had while completing the tasks will be noted by observers according to the following two ratings: fully successful or not successful. A participant is ‘fully successful’ in completing a task when he or she completes the stated task as intended with no help from the facilitator. For tasks that require the entry of expenses into the diary (including multiple expense tasks), this rating should only be given if all information is entered correctly. That is, the correct date, price, and category. The description should give enough information to place it into a CE-relevant category and category-specific questions should also be answered correctly. A participant is ‘not successful’ when he or she does not complete the task as intended or can only do so after asking the facilitator for assistance (though the participant can consult help materials). For tasks that require the entry of expenses into the diary, the rating of ‘not successful’ should be given if the participant enters unusable data. For example, if a participant enters a full receipt as a single entry (when item level detail is required) or if the participant enters the description of an item that is insufficient to classify it into a CE-relevant category. The entry of incorrect information should also be rated as ‘not successful.’ For tasks that require the editing of expenses, the rating should be given if the participant alters the data in a way other than intended. For example, if a participant completes the “Delete Shirt” task by removing the cost and resaving the expense.

3. Participants

Westat will recruit 30 participants from a Westat-maintained participant database and responses to an advertisement posted on Craigslist (see Appendix F). Westat will seek to balance the number of participants with characteristics associated with level of household spending; education, income, and household size. Potential participants will be screened using the script in Appendix D.

4. Burden Hours

The total estimated burden hours for this study is 33. Our goal is to obtain feedback from 30 participants. We anticipate that each session will approximately one hour for a total of 30 burden hours. Screening potential participants from the Westat database is estimated to take an additional 5 minutes per participant (see Appendix D for recruitment script). Given that participants are being recruited from the Westat database, we expect a high acceptance rate. We estimate the total burden of recruitment to be 3 hours (40 potential participants X 5 min.).

	N	Burden Hours/ Participant	Total Burden Hours
Recruitment	40	0.083	3
Usability Testing	30	1	30
	TOTAL	1 hr. 5 min.	33 hours

5. Payment to Participants

Each participant will be compensated $40 for their participation in this study.

6. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used to improve the design of the Consumer Expenditure diary survey. Participants will be given a consent form to read and sign (Appendix E). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

Appendices

Appendix A – Introduction

Appendix B – Interview Protocol

Appendix C – Screenshots of Electronic Diary

Appendix D – Recruitment Script

Appendix E – Consent Form

Appendix D - Advertisement

Appendix A: Introduction

Thank you for coming in today. Let me tell you a little about Westat and what we’re going to do today.

Westat is a company that conducts research on many topics, for different organizations. This particular project is for the Bureau of Labor Statistics. The Bureau of Labor Statistics conducts a study every year that asks people to record all of the purchases that all members of their household get for an entire week. Westat has been asked to test some different procedures to make it easier for people to report what they purchase. For this task today we will be asking you to record some expenses on the computer. Some of the expenses you will be entering will be on receipts that we have created for you and other times you will enter expenses from the receipts that you brought with you today. Your participation today will help us design a better system for people to enter their purchases in this study.

At this stage we are still developing the study procedures and figuring out how to make it as easy as possible for people to record their purchases. So that is what you’ll be helping us with here today. Do you have any questions about that?

Before we get started, I want to let you know that everything we cover in the interview will be kept private - only the few people actually working on the project will have access to the information you share with us. If there are any questions you don’t want to answer that is perfectly fine - We’re really more interested in how the questions we ask and tools we are using work.

If it’s OK with you, I would like to video record our conversation, just so that I can review it later when we’re analyzing all the interview data. (And finally, some of our colleagues from the Bureau of Labor Statistics are here today – behind the mirror - to observe and see for themselves how the study is going.)

CONSENT FORM: Here is a form I must ask you to look over and sign – it basically covers the points I’ve just gone over with you and indicates you have agreed to take part (ONE COPY FOR RESPONDENT, ONE COPY FOR US)

BEGIN INTERVIEW:

I’m going to be showing you two versions of a diary where you can enter your expenses. Here is what the first version looks like. [DEMONSTRATE BY ENTERING ONE EXPENSE.]

Any questions? Let’s begin.

I’m going to give you some tasks that I would like for you to do on the computer.

Appendix B: Interview Protocol

[NOTE TO INTERVIEWER] After each task ask the following question.

How easy or difficult was it to complete this task?

• Very easy

• Somewhat easy

• Neither easy nor difficult

• Somewhat difficult

• Very difficult

Task	Task Name	Text Read to Respondents
1.	Login 1	First, I’d like you to use this user name and password to log in.
2.	Lunch	Next, you will see a screen instructing you to enter an expense below. On [DATE] you go out to eat lunch with your child and order one meal for yourself and one meal for your child. Enter the expense into the diary.
3.	Groceries	On the way home you stop at the grocery store and purchase some food. Enter this expense in the diary.
4.	Shopping trip 1	On [DATE] you go shopping at a local discount store and purchase a chair, a video game for your child, and a book to read. You also stop to purchase gas and get your car washed on the way home. Enter the expenses in the diary.
5.	Return book	You decide to return the book that you bought. Please go back and delete that item.
6.	Medicine	On [DATE] you stop at the pharmacy to purchase some medicine to treat your sore throat. You also pick up some personal care products. Enter these expenses in the diary.
		Now I would like to show you another version of the diary. [DEMONSTRATE BY ENTERING ONE EXPENSE.]
7.	Login 2	Now, I’d like you to use this user name and password to log in to another version of the diary.
8.	Dinner	Next, you will see a screen instructing you to select the day that some expenses occurred. On [DATE] you and your spouse go out to eat dinner at a restaurant. Enter the expense into the diary.
9.	Movie	You also go out to a movie that night after dinner. Enter this expense in the diary.
10.	Shopping trip 2	On [DATE] you take the bus downtown to shop for some jewelry, cosmetics, and clothes. Enter the expenses in the diary.
11.	Return shirt	You decide to return shirt that you bought. Please go back and delete that item.
12.	Own expense	Enter the first three purchases on the receipt that you brought with you to the interview.

Debriefing Questions

[ASK Q1 TO Q4 AFTER EACH VERSION OF THE DIARY]

Q1. What do you think about this version of the diary?

Q2. Was the computer diary easy or difficult to use?

• Extremely difficult

• Very difficult

• Somewhat difficult

• Neither easy nor difficult

• Somewhat easy

• Very easy

• Extremely easy

Q3. How confident did you feel in filling out the entries in the diary?

• Not at all confident

• A little confident

• Somewhat confident

• Very confident

• Extremely confident

Q4. How much training do you think the average person would need to get started using the diary?

• None

• A little

• A moderate amount

• A lot

[ASK Q5 AFTER BOTH VERSIONS OF THE DIARY HAVE BEEN COMPLETED]

Q5. Which version of the diary do you prefer – the first one or the second one? Why? (remind respondent what each one looked like if needed)

Q6. What was the most difficult when entering these expenses?

Q7. What was the easiest part of entering expenses?

Q8. Do you have any thoughts for how to make this easier? If yes, what are your thoughts?

INTERVIEWERS: If there is any time left, check with observers to see if they have any additional questions for this participant. If not, thank the participant and dismiss (give them their incentive and have them sign the receipt).

Appendix C: Screenshots of Electronic Diaries

VERSION 1

Sign-in screen.

Main data collection screen

Example expense data collection fields (Food & Drinks Away From Home)

Example expense data collection fields (Food & Drinks For Home Consumption)

Example expense data collection fields (Clothing, Shoes, Jewelry, & Accessories)

Example expense data collection fields (All Other Categories)

Example expense enter and new expense data collection fields

Example edit of entered expense

VERSION2

Sign in screen

Expense day selection and summary expense screen

Expense type selection

Example expense entry (Food and Drinks Away From Home)

Example expense entry (Food and Drinks for Home Consumption)

Example expense entry (Clothing, Shoes, Jewelry, & Accessories)

Example expense entry (All Other Categories)

Example additional expense entry

Review of entered expenses

Expense date entry with summary of entered expenses shown

Example of other expense type question items

Appendix D: Recruitment Script

OMB Number 1220-0141

Expires: April 30, 2018

Recruiting Script

Thank you for your interest in the study about daily expenses. This study will take about 60 minutes. You will answer questions about some daily expenses that you may have. The study will take place at Westat in Rockville, MD. In order to make sure that you qualify for this study we need to ask a few brief questions about you.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 5 minutes per response. The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to the U.S. Department of Labor, OASAM/OCIO 200 C Street NW Washington DC 20210 and reference the OMB Control Number 1220-0141.

1. What is your age?

___________

IF UNDER 18 TELL PERSON THAT THEY ARE INELIGIBLE.

2. Do you have any difficulty reading in English?

Yes – TELL PERSON THAT THEY ARE INELIGIBLE

No

3. Are you male or female? (if needed)

Yes

No

4. What is the highest level of education you have completed?

LESS THAN HIGH SCHOOL

COMPLETED 12TH GRADE/HIGH SCHOOL GRADUATE

SOME COLLEGE/2-YEAR COLLEGE/TECHNICAL SCHOOL

COLLEGE DEGREE

GRADUATE OR PROFESSIONAL SCHOOL

5. Are you of Hispanic or Latino Origin?

YES, HISPANIC OR LATINO

NO, NOT HISPANIC OR LATINO

6. What race do you consider yourself? MARK ONE OR MORE.

AMERICAN INDIAN OR ALASKA NATIVE

(SPECIFY TRIBAL AFFILIATION(S) _____________)

NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER

ASIAN

BLACK OR AFRICAN AMERICAN

WHITE

7. At home, do you have any of the following? Select one answer choice on each row.

	Yes	No
Wi-Fi or some other Internet Connection
A desktop or laptop computer
A tablet, such as Apple iPad or Samsung Galaxy Tab
A smartphone

8. How often do you use each of the following? Select one answer choice on each row.

	Never	Rarely	Sometimes	Very Often	Always
A desktop or laptop computer
A tablet, such as Apple iPad or Samsung Galaxy Tab
A smartphone

9. Could you bring with you a receipt from a recent purchase that you have made?

YES

NO

IF INELIGIBLE

Thank you very much for your interest, but you are not eligible for this study. Thank you very much for your time. We will destroy the information you have provided.

IF ELIGIBLE

Thank you for answering all my questions. We will contact you if you are selected to participate. Can I have your name, phone number and/or email so that we can get in touch with you?

COLLECT RESPONDENT NAME, ADDRESS, AND PHONE NUMBER.

Name: ________________________________________________________________

Address: ______________________________________________________________

City: _________________________________ State: ______ Zip Code: ___________

Phone: ____________________

IF NEEDED: If you have any questions about the legitimacy of the study or need more information, you can contact Aaron Maitland, at (240) 314-2595 or [email protected]

Appendix E: Informed Consent Document

Consumer Expenditure Survey Informed Consent Form

Westat, a private research firm, has been hired by the U.S. Bureau of Labor Statistics (BLS) to test survey questions about the purchases made by people in the United States. The BLS is conducting this study to make it easier for people to report the purchases that they make. The survey questions and procedures that you will be testing today will eventually be used on a survey of household across the United States to learn about the products and services that they purchase.

Your task during this interview will be to take the survey on a laptop computer and tell us if you have any problems doing so. We will ask you about your understanding of the survey and the user-friendliness of the questions and the computer used to complete the survey.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

• Your participation is completely voluntary. You may stop at any time, and you can skip any questions you do not wish to answer.

• All information obtained during this study will be treated as secure and will only be used to develop and improve the questionnaire. We will not share your answers with anyone outside of the Consumer Expenditure Survey project research team.

• The interviewer will video record the discussion and take notes. In addition, project researchers may observe the interview. The researchers will destroy the video recording as soon as they complete the questionnaire development process.

• The interview should take about an hour.

• The report summarizing the findings will not contain any names or identifying information.

• You will receive $40 cash as a token of our appreciation for completing the session.

• If you have questions about this research, please contact Aaron Maitland, the Project Director at (240-314-2595; [email protected])

• If you have questions about your rights and welfare as a research participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, mention that you are calling about the consumer expenditure study, and leave a phone number beginning with the area code. Someone will return your call as soon as possible.

• A copy of this consent form has been provided for your records.

• Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires on April 30, 2018.

If you agree to participate in this interview, please sign the following statement.

I have read this form (or this form was read to me).

□ I agree to participate in this study.

□ I agree to be video recorded.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Appendix F Advertisement

Help us test survey questions about daily expenses for the Bureau of Labor Statistics

Earn $40

To participate, you must be:

• Available for 60 minutes

• At least 18 years old

• Able to read and speak in English

• Own a desktop or laptop computer

To see if you are eligible, please call {NAME} {NUMBER}

9