Note to Reviewer - Usability Testing CE Diary

February 24, 2016

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Brandon Kopp Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for usability testing of the redesigned CE paper diary

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study is 96 hours.

If there are any questions regarding this project, please contact Brandon Kopp at

202-691-7514.

1. Introduction and Purpose

The Bureau of Labor Statistics has conducted the Consumer Expenditure Diary Survey (CED) since 1980. Respondents to the CED report all expenditures they make during the diary period by writing details about their purchases, in as close to real time as possible, into a booklet provided to them by a Census Bureau field interviewer. For “Food Away from Home” purchases, respondents enter details about the overall meal including the total cost. For all other purchases (groceries, clothes, entertainment, etc.), respondents are asked to enter item-level detail. For example, if the respondent spends $20 at a grocery store, they would enter that they spent $4 on milk, $2 on bread, $3.50 on yogurt, and so on. In addition, they provide further detail about the type of packaging for each item.

In an effort to alleviate burden, slow or reverse the decline in response rates, and respond to comments from noted survey design researchers like Don Dillman, CED staff have developed an alternative version of the paper diary form. The new version consolidates the four main diary categories onto two, facing, diary pages so that all expenses for a single day can be entered without flipping pages. An effort was also made to reduce the amount of instructions and examples so that respondents are not confused or intimidated. A complete list of changes to the diary is included in Appendix A. A full, PDF version of the diary form is included as Appendix D.

2. Research Design

The purpose of this study is to test whether the proposed changes to the CED work as the designers intended. Specifically, it is necessary to investigate whether the new diary form is easy for respondents to understand, that data quality is not diminished by the elimination of the instructions and examples, and that the overall user experience is positive. Unlike a personal or telephone interview, filling out the diary is a self-administered activity. Respondent errors cannot be corrected by the interviewer in real-time; rather, errors may cause unnecessary non-response, harm to data quality, or increases in burden. Understanding how changes to the diary may affect the way respondents interact with the diary in their homes is also of great importance to CE stakeholders.

To accomplish these goals, the new diary form will be tested using a mixture of in-lab usability testing and at-home diary completion. Participants will be asked to visit the lab two times, between six and ten days apart. In between visits, they will be asked to enter their household’s expenses into the redesigned CE Diary. A brief description of the methodology is below and the full protocol is available in Appendix B.

Visit 1. At the first visit, participants will be introduced to the diary form and the interviewer will explain what should be entered in the different sections of the diary and how to enter it. This is similar to diary placement that is currently conducted by field interviewers.

After “placement” is complete, half of the participants will be given three receipts with fictitious expenses along with scenarios and asked to make entries for the items on those receipts into the diary. The interviewer will observe the entry of these expenses and follow-up with the participant about any issues. All participants will be given the opportunity to ask questions. This is being done to examine the benefit of providing illustrative examples to participants during placement.

Finally, the interviewer will describe the week-long, at-home diary completion task to the participant and answer any questions they have. If a second visit has not been scheduled, one will be made at that time.

At-Home Diary Completion. Participants will be asked to take the diary home with them and enter every expense made by them or by other members of their household for one week. The interviewer will call the participant approximately 3 days after Visit 1 to see if the respondent has any questions about filling out the diary or if they have had any issues with filling it out.

Visit 2. The participant will return to the lab between 6 and 10 days after Visit 1, bringing the diary with them. At the beginning of the second interview, participants will be debriefed about their diary-keeping experience. Specifically, they will be asked how they went about filling out the diary (e.g., did they complete it each evening, every few days, or at the end of the week; where did the diary sit during the week), whether they encountered any problems while making entries in the diary, and whether they made use of various elements of the diary (e.g., example and instruction pages). They will also be asked about the ease of use of the diary and about their recommendations for improving the diary.

Following the debriefing about their experience, all participants will be given a new, blank diary. The interviewer will provide participants with three scenarios and receipts for expenses and ask them to enter the items on the receipts into the diary (these scenarios will be different from the scenarios used during Visit 1). The interviewer will observe how the participants makes the entries and then, after all three scenarios are complete, ask follow-up questions about the process.

Finally, the interviewer will review the participant’s home diary with them and ask about any noticeable data entry issues (e.g., entering “groceries” instead of individual grocery items, erasing or crossing out expenses). Participants will also be asked about any expenses they may have forgotten to enter or chose not to enter. The participant will be given another chance to comment on the diary and then the visit will be ended.

After the participant has left, a coder will review the participant’s diary and record (1) the total number of entries by category, (2) the total dollar value of expenditures entered, (3) the number of diary questions with missing data, and (4) any data entry errors (e.g., entering items in the wrong category, not providing enough detail to categorize item). Form issues will also be noted (e.g., whether participants were not provided with enough space to record all of their expenses).

3. Participants

Thirty participants will be recruited from a participant database maintained by OSMR. Efforts will be made to select participants with varying levels of education, income, occupation, and household size. Participants will be told during the initial recruitment that they must be available for two sessions, approximately one week apart, and be willing to complete the diary at home for a week. They will also be told about the total estimated amount of time we are asking for (i.e., 3 hours). The recruiter will attempt to schedule both sessions at that time. If the participant can only schedule the first meeting at the time of initial recruitment, the second interview will be set up at the end of the first meeting.

4. Burden Hours

The total burden hours for this study will be 96. Thirty participants will be recruited to participate in this study. We anticipate a moderate acceptance rate for the initial recruitment contact with potential participants (approximately 75%), so estimate we will need to contact approximately 40 people in order to get 30 interviews. We expect these initial recruitment calls to last 5 minutes. The screening protocol is included in Appendix E. This will add 3 burden hours (40 participants X .083 hours) to the study.

The first visit will be short; 20 minutes at most. This will add 10 burden hours (30 participants X 0.33 hours/interview) to the study. Additionally, we expect that participants in this study will spend as long as 1.75 hours completing the diary during the diary week (~15 minutes per day) for a total of 53 burden hours (30 participants X 1.75 hours). Finally, we expect the second interview to take up to one hour to complete, adding an additional 30 burden hours (30 participants X 1 hour/interview).

	N	Burden Hours/ Participant	Total Burden Hours
Initial Recruitment	40	0.083	3
Visit 1	30	0.33	10
Diary Completion At Home	30	1.75	53
Visit 2	30	1	30
	TOTAL	3 hr. 10 min.	96 hours

5. Payment to Participants

Each participant will be compensated a total of $80 for their participation in this study. Because it is vital that the participant returns for the second interview, we will compensate the participant $30 for their first visit and $50 at the time of their second visit. We believe that offering a greater incentive for the second visit (while still maintaining an average of $40 per visit) will be helpful in limiting the number of breakoffs between the first and second study. Again, if the participant does not come back for the second interview, we do not get their diary back and we do not get their feedback; both key to the success of this study. A total of $2400 will be spent on payments to participants.

6. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of a national Consumer Expenditure survey. Participants will be given a consent form to read and sign (See Appendix C). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

See the following attachments:

Appendix A – List of Changes to the CE Diary

Appendix B – Interview Protocol

Appendix C – Consent Form

Appendix D – Diary Form

Appendix E – Screening Protocol

Appendix A: List of Changes to the CE Diary

A full version the instrument as PDF files will be sent along with this protocol. This Appendix describes the changes that were made to the diary form.

Increase paper size to legal and fit all four diary sections on two pages

This is the main overall design change. If you look at the initial paper diaries in the ‘70s, the form was taller – essentially legal size (8.5 x 14). This allowed for an entire diary day to be covered by two open pages.

One of the criticisms that Don Dillman made about the current diary is that it requires a lot of page flipping:

The request to record expenditures by day and into broad categories requires respondents to flip pages back and forth as they move between instruction and recording pages. The diary lacks a clear navigational path, and the visual layout makes completing the diary difficult.

Having all of the diary sections covered on two pages will return the diary form to its original intent of allowing respondents to easily document their expenses on an open two pages. When a respondent turns a page, they are doing so because they’re moving to the next diary day.

Modify the number of rows to fit four sections into two pages

Fitting all four sections (parts) of the diary onto two pages instead of four requires adjusting the number of rows for each section. Currently each section has about the same number of rows per page (Meals has a few less than the other sections): Meals, 22; Food, 25; Clothing, 25; and Other, 25.

It appears that 35 rows easily fit on the Food page. The Meals, Clothing, and Other sections will need professional layout to determine the exact number of rows, but it appears that 6, 8, and 12 rows, respectively, fit well on the right page.

Reduce/Reorganize Instructions and Examples

One of the points Don Dillman made in an FCSM presentation on the Diary is that there are too many instructions/examples.

• Eliminate the examples on the front/left flap.

• Eliminate the examples at the top of each section.

• Move the ‘General Instructions’ and ‘How to Fill Out Your Diary’ to the Front Flap

Replace the two Age Range and Sex columns with one set of Age-Sex columns

Currently, we have a Sex column for Male/Female, and we have an Age Range column for Under 2, 2-15, and 16 & Over. This requires checking two boxes per item.

In the past we have asked for just noting one age-sex category, and returning to that would be one less step per item reported for the respondent to mark.

The total number of columns would remain the same at five: Was it for: Child under 2 years; Male 2-15; Female 2-15; Male 16 & Over; Female 16 & Over.

Appendix B: Testing Protocol

The protocol includes the script that will be used for all three parts of this test (i.e., Visit 1, the mid-week phone call, and Visit 2). The interviewer may ask unscripted follow-up questions (e.g., “Can you tell me more about that?”) if a participant’s answer suggests additional, pertinent information can be gained.

Visit 1

• Hi! Thank you for coming in today.

• I have a couple colleagues in the next room that will be observing and taking notes.

• Our recruiter told you that this study involves two visits, one today and one in about a week, right?

• Explanation of the study purpose:

  • In this study, we’re going to be looking at the Consumer Expenditure Diary Survey. This survey collects information about how US households spend their money. We are interested in updating our diary and want to get your feedback on the design, how easy or difficult it is to use, and whether there’s anything we can do to improve it.

  • It’s important to note that this is not a test of your ability. If the diary is difficult to use, that’s something that we need to work on and your feedback can help us do that.

• Today I’m going to describe how to use the diary [and have you enter a few fake expenses into the diary]. I’m then going to ask you to take it home for a week and enter all of your actual household expenses. In a week, I’ll ask you to come back and I’ll ask you some questions about what it was like to fill out the diary. How does that sound?

• [Consent Form and permission to audiotape]

• Any questions before we begin?

• The Consumer Expenditure Diary Survey is a household survey. That is, we ask the person who’s filling it out to write in all of the expenses they and their household have during the diary week. To get started, I’d like to get some information about you and the people in your household.

• How many people are in your household, including yourself? ______

• How old are you?

• [if more than 2 in HH] OK, and who is the next person? I don’t need their name, just how they are related to you.

Name	Relationship	Age	Gender
	Participant		M F
			M F
			M F
			M F
			M F

• Now I will describe the diary how to use the diary; what information we want you to enter and where it goes.

  • The diary is divided up by day and by type of item. Each day of the week will have its own set of pages. These days refer to the day the purchase was made, not the day you are entering it. So if you buy some groceries on Monday, you would flip to this page and enter them.

  • In addition to days of the week, the diary is also broken into categories; Food and Drink Away from Home (or restaurant-type meals); Food and Drink for Home Consumptions (things that you eat or prepare at home); Clothing, Shoes, Jewelry, and Accessories; and All other Products and Services (which is everything that doesn’t fit into the other three categories.
    
    The reason we divide the world of purchases up this way is because we’re interested in slightly different information depending on what category an items is in.

  • Let’s start with Food and Drink Away From Home. For this category, we want you to enter:

    • What meal it was purchased for

    • A general description of the overall meal. For example, Chinese takeout.

    • We also want you to check one of these boxes describing what type of place you purchased it from; fast food, a full service restaurant, a vending machine, or a cafeteria.

    • You would then enter the total cost of the meal, including tax and tip.

    • If alcohol was purchased at the meal, you would check what types of alcohol were purchased and enter the price of the alcohol in this box here

  • The other three categories are a bit different. For Food and Drink Away From Home, you entered the whole meal as one line. For everything else (groceries, clothes, bills, paper towels) we need each item entered separately.

  • In Food and Drinks for Home Consumption, for example, if you spent $20 at the grocery store, we don’t want to know that you spent $20 on groceries, we want to know that you spent $4 on milk, $2 on bread, $3 on frozen peas, and so on.

    • Also for grocery items we need to know what type of packaging the item came in…whether it was fresh, frozen, bottled or canned, or something else.

    • We need the price for that individual item without tax, and

    • We need to know if you purchased it for someone outside of your household as a gift or a donation.

  • The next category is Clothing, Shoes, Jewelry, and Accessories. Accessories, here, means clothing accessories; things like hats, belts, wallets, or purses. Like with groceries, we need each item entered separately. So if you buy a shirt and a pair of pants, each of those should be on a separate line.

    • We need a description of what you bought

    • The cost of that item, without tax

    • The gender and age category of the person it was purchased for

    • And whether it was purchased for someone outside your household

  • Our last category is All Other Products, Services, and Expenses. Again, this is a catchall category for anything that doesn’t fit into the other three categories. This will include things like dry cleaning, movie tickets, gasoline. These too need to be itemized.

    • For these items, we just need a description of what was purchased

    • The cost, without tax, and

    • Whether it was purchased for someone outside your household

  • If you have any questions about what to enter into the diary or where to enter it, there are examples at the top of the page as well as in the front.

  • Do you have any questions about what to enter in the diary or how to enter it?

Diary Entry Task [for half of participants]

Now I’m going to ask you to complete a series of tasks using the diary. If you have any questions while we’re going through, remember that there are help materials in the diary. If you get really stuck, you can ask me for help.

1. Imagine that today is Monday, April 4^th and you go to the store to pick up a few things. This is your receipt.

2. The next day, you stopped by a department store called Percy’s on your way home from work and pick up two shirts. Please enter them into the diary.

3. After shopping, you meet up with a friend and have dinner at a restaurant. You pay for both of you. This is the receipt.

Task 1	Task 2	Task 3

Debriefing Questions

Now I’d like to ask you a few questions about the diary.

1. What do you think of the diary so far?

2. How easy or difficult do you think it will be to enter items into the diary this week?

• Extremely easy

• Very easy

• Somewhat easy

• Neither easy nor difficult

• Somewhat difficult

• Very difficult

• Extremely difficult

3. How confident do you feel about filling out the diary this week?

• Not at all confident

• A little confident

• Somewhat confident

• Very confident

• Extremely confident

4. Do you need any further training before taking it home?

• Yes

• No

1. [if Yes] What would you like to know more about?

End of Visit 1

• That is all for this time.

• Have you scheduled your visit for next week?

  • [if No] OK, can we schedule you now?

• If you have any questions during the week, my phone number and email are written on the diary. If I’m not there, please leave a message and I’ll get back to you as soon as possible.

• Is it alright if I call you on [THREE DAYS AFTER MEETING] to see how things are going?

  • [if Yes] Will the number that our recruiter has for you work, or is there another number I should call?

  • [if No] That’s OK. I’ll see you at our meeting next week and see how things went.

• Thank you for coming in.

Mid-Week Telephone Call

• Hi. This is ______ from BLS. I’m calling to see how things are going with the expenditure diary.

• Do you have any questions? Yes, No

• Have you entered any expenses into the diary? Yes, No

  • [if Yes] Did you have any issues with what you recorded in the diary?

• Thank you for your participation so far. I will see you on ______.

Visit 2

• Thank you for coming back. Today, I’m going to ask you about what it was like to complete the diary, enter a few more expenses, and we’ll take a quick look at what you entered this week.

• [Consent Form and permission to audiotape]

• Any questions before we begin?

Debriefing Questions

1. What was it like to fill out the diary this week?

I am interested in finding out about what the experience was like. So I have a couple questions that get at that.

2. On which days did you enter expenses?

   • Monday

   • Tuesday

   • Wednesday

   • Thursday

   • Friday

   • Saturday

   • Sunday

3. On the days that you entered items into the diary, when did you enter them?

• Shortly after having each expense

• After making several expenses

• At the end of the day

• Other ________

4. Where did you complete the diary?

5. Where was the diary when you weren’t filling it out?

6. How would you rate the level of effort needed to fill out the diary for a week?

• Not at all effortful

• A little effortful

• Somewhat effortful

• Very effortful

• Extremely effortful

7. How easy or difficult was the diary to use?

• Extremely easy

• Very easy

• Somewhat easy

• Neither easy nor difficult

• Somewhat difficult

• Very difficult

• Extremely difficult

8. How confident are you that you entered everything the way I described it last week?

• Not at all confident

• A little confident

• Somewhat confident

• Very confident

• Extremely confident

9. How burdensome would you say it was to fill out the diary this week?

• Not at all burdensome

• A little burdensome

• Somewhat burdensome

• Very burdensome

• Extremely burdensome

10. Did you have any issues while entering items into the diary?

• Yes

• No

10a. [if Yes] Can you tell me more about that?

11. Did you use the instructions?

• Yes

• No

11a. [if Yes] Can you tell me more about that?

11b. [if Yes] How frequently did you use the instructions?

12. Did you look at the examples [Show participant examples in the diary]?

• Yes

• No

12a. [if Yes] Can you tell me more about that?

12b. [if Yes] How frequently did you look at the examples?

13. Do you have any suggestions for how we can improve the diary?

Diary Entry Task [for all participants]

Now I’m going to ask you to complete a series of tasks using the diary. I would like you to do that again. If you have any questions while we’re going through, remember that there are help materials in the diary. If you get really stuck, you can ask me for help.

1. Imagine that today is Monday, April 11^th and you go to the store to pick up a few things. This is your receipt.

2. The next day, you stopped by a recreational store and picked up some pants and socks for yourself for a weekend hike. Please enter them into the diary.

3. On your way to work the next day, you stop by a coffee shop to pick up some breakfast. This is the receipt.

Task 1	Task 2	Task 3

Diary Entry Task Review

Interviewer will review entries made by participant and follow-up in an open-ended fashion if there are any issues (e.g., if items are combined “groceries” or “clothes”). For example, “I see that you entered the pants and the socks together. Can you tell me more about that?”

At Home Diary Review

Let’s take a look at what you entered this week.

Interviewer will review entries with the participant and follow-up in an open-ended fashion if there are any issues (e.g., if items are combined “groceries”). For example, “I see that you have a $20 entry for groceries. Can you tell me more about that?”

1. Were there any expenses that you did not enter into the diary?

• Yes

• No

1a. [if Yes] Please describe those expenses and why you didn’t enter them.

2. Did you enter items for other people in your household?

• Yes

• No

3. Do you have bills set up with automatic payment that may have been charged this week?

• Yes

• No

3a. Did you enter those bills in the diary?

• Yes

• No

4. Were there any expenses that you didn’t enter because it would have felt burdensome to do so (for example, a long grocery receipt)?

• Yes

• No

4a. Can you tell me more about that?

5. Were there any small expenses, at vending machines, coffee shops, convenience stores, that you didn’t enter?

• Yes

• No

End of Visit 2

Do you have any other questions or comments about the diary?

Thank you for your participation.

Appendix C: Consent Form

CONSENT FORM

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

(Visit 1) We estimate it will take an average of 20 minutes to participate in this research today and another 1 hour and 45 minutes to record the entries in the Diary at home for the next week.

(Visit 2) We estimate it will take an average of 60 minutes to participate in this research today.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires on April 30, 2018.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Appendix E: Recruitment Protocol

Participants in this study will be recruited from a database maintained by OSMR. The database contains demographic information about the potential participants, so participants of varying income, education, occupation, and household size groups will be drawn and asked to participate via a phone call. The phone call will be roughly scripted, as shown below, but is impossible to anticipate all questions that potential participants may ask or what explanation may be required.

OMB Number 1220-0141

Expires: April 30, 2018

Recruitment script:

Hello, this is _____ from the Bureau of Labor Statistics. I am calling to see if you are interested in participating in an upcoming study. In this study, we will ask you to come to our offices at BLS two times…about a week apart. In between those meetings, we want you to record all of the purchases you, and those in your household, make during that week in a spending diary. At the first meeting we will explain how to fill out the diary and at the second meeting we’ll ask you some questions about what it was like to use the diary. The first meeting will be about 20 minutes long, the second meeting will be about an hour, and we estimate that you will spend a total of about an hour and a three-quarters filling out the diary during the week for a total of about 3 hours of time commitment. To compensate you for your time, we are offering $80 for this study. Thirty dollars will be paid at the first meeting and $50 will be paid at the second meeting.

Are you interested in participating?

[if Yes] OK, let’s get you scheduled. If you know your availability, I would like to schedule both meetings today. If you don’t, I’ll get you scheduled for the first meeting and you can work out a time for the second one with your interviewer when you come in.

[if No] OK, thank you for your time. Have a good day.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 5 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to the U.S. Department of Labor, OASAM/OCIO 200 C Street NW Washington DC 20210 and reference the OMB Control Number 1220-0141.

Page 21 of 21