Usability Testing of redesigned CE paper Diary

Appendix B: Testing Protocol

The protocol includes the script that will be used for all three parts of this test (i.e., Visit 1, the mid-week phone call, and Visit 2). The interviewer may ask unscripted follow-up questions (e.g., “Can you tell me more about that?”) if a participant’s answer suggests additional, pertinent information can be gained.

Visit 1

• Hi! Thank you for coming in today.

• I have a couple colleagues in the next room that will be observing and taking notes.

• Our recruiter told you that this study involves two visits, one today and one in about a week, right?

• Explanation of the study purpose:

  • In this study, we’re going to be looking at the Consumer Expenditure Diary Survey. This survey collects information about how US households spend their money. We are interested in updating our diary and want to get your feedback on the design, how easy or difficult it is to use, and whether there’s anything we can do to improve it.

  • It’s important to note that this is not a test of your ability. If the diary is difficult to use, that’s something that we need to work on and your feedback can help us do that.

• Today I’m going to describe how to use the diary [and have you enter a few fake expenses into the diary]. I’m then going to ask you to take it home for a week and enter all of your actual household expenses. In a week, I’ll ask you to come back and I’ll ask you some questions about what it was like to fill out the diary. How does that sound?

• [Consent Form and permission to audiotape]

• Any questions before we begin?

• The Consumer Expenditure Diary Survey is a household survey. That is, we ask the person who’s filling it out to write in all of the expenses they and their household have during the diary week. To get started, I’d like to get some information about you and the people in your household.

• How many people are in your household, including yourself? ______

• How old are you?

• [if more than 2 in HH] OK, and who is the next person? I don’t need their name, just how they are related to you.

Name	Relationship	Age	Gender
	Participant		M F
			M F
			M F
			M F
			M F

• Now I will describe the diary how to use the diary; what information we want you to enter and where it goes.

  • The diary is divided up by day and by type of item. Each day of the week will have its own set of pages. These days refer to the day the purchase was made, not the day you are entering it. So if you buy some groceries on Monday, you would flip to this page and enter them.

  • In addition to days of the week, the diary is also broken into categories; Food and Drink Away from Home (or restaurant-type meals); Food and Drink for Home Consumptions (things that you eat or prepare at home); Clothing, Shoes, Jewelry, and Accessories; and All other Products and Services (which is everything that doesn’t fit into the other three categories.
    
    The reason we divide the world of purchases up this way is because we’re interested in slightly different information depending on what category an items is in.

  • Let’s start with Food and Drink Away From Home. For this category, we want you to enter:

    • What meal it was purchased for

    • A general description of the overall meal. For example, Chinese takeout.

    • We also want you to check one of these boxes describing what type of place you purchased it from; fast food, a full service restaurant, a vending machine, or a cafeteria.

    • You would then enter the total cost of the meal, including tax and tip.

    • If alcohol was purchased at the meal, you would check what types of alcohol were purchased and enter the price of the alcohol in this box here

  • The other three categories are a bit different. For Food and Drink Away From Home, you entered the whole meal as one line. For everything else (groceries, clothes, bills, paper towels) we need each item entered separately.

  • In Food and Drinks for Home Consumption, for example, if you spent $20 at the grocery store, we don’t want to know that you spent $20 on groceries, we want to know that you spent $4 on milk, $2 on bread, $3 on frozen peas, and so on.

    • Also for grocery items we need to know what type of packaging the item came in…whether it was fresh, frozen, bottled or canned, or something else.

    • We need the price for that individual item without tax, and

    • We need to know if you purchased it for someone outside of your household as a gift or a donation.

  • The next category is Clothing, Shoes, Jewelry, and Accessories. Accessories, here, means clothing accessories; things like hats, belts, wallets, or purses. Like with groceries, we need each item entered separately. So if you buy a shirt and a pair of pants, each of those should be on a separate line.

    • We need a description of what you bought

    • The cost of that item, without tax

    • The gender and age category of the person it was purchased for

    • And whether it was purchased for someone outside your household

  • Our last category is All Other Products, Services, and Expenses. Again, this is a catchall category for anything that doesn’t fit into the other three categories. This will include things like dry cleaning, movie tickets, gasoline. These too need to be itemized.

    • For these items, we just need a description of what was purchased

    • The cost, without tax, and

    • Whether it was purchased for someone outside your household

  • If you have any questions about what to enter into the diary or where to enter it, there are examples at the top of the page as well as in the front.

  • Do you have any questions about what to enter in the diary or how to enter it?

Diary Entry Task [for half of participants]

Now I’m going to ask you to complete a series of tasks using the diary. If you have any questions while we’re going through, remember that there are help materials in the diary. If you get really stuck, you can ask me for help.

1. Imagine that today is Monday, April 4^th and you go to the store to pick up a few things. This is your receipt.

2. The next day, you stopped by a department store called Percy’s on your way home from work and pick up two shirts. Please enter them into the diary.

3. After shopping, you meet up with a friend and have dinner at a restaurant. You pay for both of you. This is the receipt.

Task 1	Task 2	Task 3

Debriefing Questions

Now I’d like to ask you a few questions about the diary.

1. What do you think of the diary so far?

2. How easy or difficult do you think it will be to enter items into the diary this week?

• Extremely easy

• Very easy

• Somewhat easy

• Neither easy nor difficult

• Somewhat difficult

• Very difficult

• Extremely difficult

3. How confident do you feel about filling out the diary this week?

• Not at all confident

• A little confident

• Somewhat confident

• Very confident

• Extremely confident

4. Do you need any further training before taking it home?

• Yes

• No

1. [if Yes] What would you like to know more about?

End of Visit 1

• That is all for this time.

• Have you scheduled your visit for next week?

  • [if No] OK, can we schedule you now?

• If you have any questions during the week, my phone number and email are written on the diary. If I’m not there, please leave a message and I’ll get back to you as soon as possible.

• Is it alright if I call you on [THREE DAYS AFTER MEETING] to see how things are going?

  • [if Yes] Will the number that our recruiter has for you work, or is there another number I should call?

  • [if No] That’s OK. I’ll see you at our meeting next week and see how things went.

• Thank you for coming in.

Mid-Week Telephone Call

• Hi. This is ______ from BLS. I’m calling to see how things are going with the expenditure diary.

• Do you have any questions? Yes, No

• Have you entered any expenses into the diary? Yes, No

  • [if Yes] Did you have any issues with what you recorded in the diary?

• Thank you for your participation so far. I will see you on ______.

Visit 2

• Thank you for coming back. Today, I’m going to ask you about what it was like to complete the diary, enter a few more expenses, and we’ll take a quick look at what you entered this week.

• [Consent Form and permission to audiotape]

• Any questions before we begin?

Debriefing Questions

1. What was it like to fill out the diary this week?

I am interested in finding out about what the experience was like. So I have a couple questions that get at that.

2. On which days did you enter expenses?

   • Monday

   • Tuesday

   • Wednesday

   • Thursday

   • Friday

   • Saturday

   • Sunday

3. On the days that you entered items into the diary, when did you enter them?

• Shortly after having each expense

• After making several expenses

• At the end of the day

• Other ________

4. Where did you complete the diary?

5. Where was the diary when you weren’t filling it out?

6. How would you rate the level of effort needed to fill out the diary for a week?

• Not at all effortful

• A little effortful

• Somewhat effortful

• Very effortful

• Extremely effortful

7. How easy or difficult was the diary to use?

• Extremely easy

• Very easy

• Somewhat easy

• Neither easy nor difficult

• Somewhat difficult

• Very difficult

• Extremely difficult

8. How confident are you that you entered everything the way I described it last week?

• Not at all confident

• A little confident

• Somewhat confident

• Very confident

• Extremely confident

9. How burdensome would you say it was to fill out the diary this week?

• Not at all burdensome

• A little burdensome

• Somewhat burdensome

• Very burdensome

• Extremely burdensome

10. Did you have any issues while entering items into the diary?

• Yes

• No

10a. [if Yes] Can you tell me more about that?

11. Did you use the instructions?

• Yes

• No

11a. [if Yes] Can you tell me more about that?

11b. [if Yes] How frequently did you use the instructions?

12. Did you look at the examples [Show participant examples in the diary]?

• Yes

• No

12a. [if Yes] Can you tell me more about that?

12b. [if Yes] How frequently did you look at the examples?

13. Do you have any suggestions for how we can improve the diary?

Diary Entry Task [for all participants]

Now I’m going to ask you to complete a series of tasks using the diary. I would like you to do that again. If you have any questions while we’re going through, remember that there are help materials in the diary. If you get really stuck, you can ask me for help.

1. Imagine that today is Monday, April 11^th and you go to the store to pick up a few things. This is your receipt.

2. The next day, you stopped by a recreational store and picked up some pants and socks for yourself for a weekend hike. Please enter them into the diary.

3. On your way to work the next day, you stop by a coffee shop to pick up some breakfast. This is the receipt.

Task 1	Task 2	Task 3

Diary Entry Task Review

Interviewer will review entries made by participant and follow-up in an open-ended fashion if there are any issues (e.g., if items are combined “groceries” or “clothes”). For example, “I see that you entered the pants and the socks together. Can you tell me more about that?”

At Home Diary Review

Let’s take a look at what you entered this week.

Interviewer will review entries with the participant and follow-up in an open-ended fashion if there are any issues (e.g., if items are combined “groceries”). For example, “I see that you have a $20 entry for groceries. Can you tell me more about that?”

1. Were there any expenses that you did not enter into the diary?

• Yes

• No

1a. [if Yes] Please describe those expenses and why you didn’t enter them.

2. Did you enter items for other people in your household?

• Yes

• No

3. Do you have bills set up with automatic payment that may have been charged this week?

• Yes

• No

3a. Did you enter those bills in the diary?

• Yes

• No

4. Were there any expenses that you didn’t enter because it would have felt burdensome to do so (for example, a long grocery receipt)?

• Yes

• No

4a. Can you tell me more about that?

5. Were there any small expenses, at vending machines, coffee shops, convenience stores, that you didn’t enter?

• Yes

• No

End of Visit 2

Do you have any other questions or comments about the diary?

Thank you for your participation.