Various Demographic Area Pretesting Activities

Generic Clearence for Questionnaire Pretesting Research

omb1203CEDiaryenc2

Various Demographic Area Pretesting Activities

OMB: 0607-0725

Document [docx]
Download: docx | pdf



Date_______________________; Participant #________; Researcher: _______________


General Introduction: Consumer Expenditure Diary Instrument



Thank you for your time today. My name is XX and I work here with the Human Factors and Usability group. I will be working with you today. If you have a cell phone, please turn it off or put it on vibrate. We will be evaluating the design of the online Consumer Expenditure Diary Survey by having you interact with it. The diary helps us learn about the buying habits of people in the United States. People who complete the diary helps us understand more about the products and services that are bought by people in this country. Your experience with the survey is an essential part of our work. I did not create the diary, so please feel free to share both your positive and negative reactions to it. We are not evaluating you or your skills. Rather, you are helping us see how well the survey works. The entire session should last about an hour. Your comments and feedback will be used to improve the online diary.

First, I would like to ask you to read and sign this consent form (Attachment B).

Hand the participant the consent form; give time to read and sign; sign your own name and date it if you have not already done so.

It explains the purpose of today’s session and informs you of your rights as a participant. It also tells you that we would like to videotape the session, with your permission. Only those of us connected with the project will review the tape and any other data collected during the session, and the information will be used solely for research purposes. We may use clips from the tape to illustrate key points about the survey to the online diary design team. In addition, observers from the project team may observe this session in another room.

I would like you to tell me your impressions and thoughts about the diary screens as you look at them. In other words, I would like you to ``think aloud'' and talk to me about your impressions. If anything was confusing on the screen, please let me know. If you expect to see some item of information let me know if you see it or not.

Pull up www.wtop.com in Firefox.

Before we get started, let's practice thinking aloud, since it's not something that you would normally do while working online. Pretend that you have a minute or two to kill at your desk at work or at home and talk me through your thought process as you try to find something interesting to read on this news site.



Ok, that’s exactly what I would like for you to do throughout the session. If at any time during the session you get quiet, I may remind you to talk to me. This is not to interrupt your thought process, but simply to remind you to keep talking to me. Please focus on verbalizing what you are thinking as you complete the survey.

Start the tape.

While you are completing the diary, we will record the movements of your eyes with our eye-tracking monitor to get a record of where you are looking on the screen.

Now I am going to calibrate your eyes for the eye-tracking.

Do Calibration



Now that we have your eyes calibrated, we are ready to begin. Please respond to the survey online as you would at home.

Please imagine that you received this packet from me, a Census Bureau field representative, at your home. Please use the training and the packet materials to access the online Consumer Expenditure diary. I will tell you when to begin the tasks, which are printed on the sheets of paper on the left-hand side of your desk. Please be sure to take a moment to read the training handout (Attachment C) on how to use the diary.

First, you will need to log into the system using the information provided in the handout. You will need access to an email account to complete the login process. You may use your own email account, or you may use an account created by our team for this study. Please let me know which option you prefer.

Pull up Internet Explorer.

If they choose the lab account, it is [email protected][email protected] and the password is census.

Please complete the login process and remember to think aloud as you proceed.



I am going to go around to the other room to do a sound check. While I am doing that, please take a moment to complete this questionnaire. [Hand Participant questionnaire on Computer experience and demographics (Attachment D)]

I’m going to leave but we will still be able to communicate through a series of microphones and speakers. Do you have any questions before we begin?

Leave room. Once in control room do a sound check and start the eye-tracking software: Tobii Studio.

Allow the participant to proceed through the entire instrument without probing. You may answer their questions if they have trouble, but hold all probes until debriefing.



Things to watch for to ask about in debriefing:

  • Expressions of confusion or frustration

  • Expressions of satisfaction or being pleased with the outcome of interacting with the diary

  • Issues with including or excluding items

  • Awareness or use of help

  • Attempts to change entries

  • Overall ability to navigate within the web diary

  • Verbal requests for help

_____________________________________________________________________________________





Tasks These have been partially adapted from Shields (2004). Each participant will only receive one set of tasks. The tasks involve asking respondent to complete the diary with differently-organized receipt information.



Group 1: Receipts organized by Day and Person

Task 1:

  • The members of your household are Dan Smith, Amanda Smith, Ben Smith, and Catherine Smith. The participant will be either Dan or Amanda Smith and will be referred to as YOU during the session and not “Dan” or “Amanda.”

  • Each member of the family gave you their receipts and lists of purchases for the past two weeks. Enter all the following purchases into the Consumer Expenditure online diary. Please remember to ‘think aloud’ as you work through this task.



On [INSERT DATE], Dan Smith purchased:

  1. Newspaper $1.50

  2. Fishing Magazine $3.75

  3. Egg and cheese McMuffin from McDonald’s for breakfast for $2.99 (including tax)

  4. Coffee from Starbucks for $4.00 (including tax)

  5. Taxi Fare and tip $20.00

  6. Sneakers for his 6 year old niece $29.99

  7. Sweatshirt for 18 year old nephew $25.99

On [INSERT DATE], Dan and Amanda Smith went out to dinner. They each spent:

  1. $35 on lobsters, baked potatoes and green beans. (including tax and tip)

  2. $20 on a bottle of wine. (including tax)

  3. $7 on dessert. (including tax)

  4. $65.43 on gas for their car (including tax)

On [INSERT DATE], Ben Smith purchased:

  1. Computer game for friend’s birthday present, $50.85

  2. Roast beef sandwich and a soda from a deli for lunch for $9.54 (including tax)

  3. Candy bar from vending machine for $0.75 (including tax)

  4. A slice of pizza at the mall for $2.50 (including tax)

  5. A pack of cigarettes $6.00 (excluding tax)

On [INSERT DATE], Catherine Smith purchased:

  1. Stuffed animal for $15.99

  2. Ice cream sandwich from ice cream truck $3.00

  3. A DVD from Amazon.com for $15.76 (including tax)

Amanda Smith went grocery shopping on [INSERT DATE], and purchased:

  1. White bread $12.50 (excluding tax)

  2. Frozen orange juice $1.25 (excluding tax)

  3. 3 apples $1.99 (excluding tax)

  4. Skim milk $3.34 (excluding tax)

  5. 5 TV dinners $10.50 (excluding tax)

  6. Pepsi six-pack $3.99 (excluding tax)

  7. Chicken wings, frozen $5.99

  8. Men’s deodorant $2.99 (excluding tax)



Dan Smith paid the bills on [INSERT DATE] and made the following payments:

  1. Mortgage payment $4,000

  2. Health Insurance bill $375

  3. Wrote a check for a church contribution $50.00

Task2.

  • After you finished entering all the purchases the Smith household made for the week, you review the entries to make sure you entered every entry listed, and the correct price.

As you are doing this, you notice that Catherine made a mistake on her list. The stuffed animal she purchased did not cost $15.99, it cost $25.99. Please make this correction now.

Remember to think aloud as you work.



  • Ben Smith remembers that his friend actually purchased the computer game (and he did not). Please make this correction now.

Remember to think aloud as you work.

Dan Smith notes that the coffee from Starbucks on [INSERT DATE] actually cost $4.57. Please make this correction now.

  • Task 3:

Please submit your data for the week.



Group 2: Receipt Information Organized by Type of Purchase Task 1: Either Computer or iPad first

  • The members of your household are Dan Smith, Amanda Smith, Ben Smith, and Catherine Smith. The participant will be either Dan or Amanda Smith and will be referred to as YOU during the session and not “Dan” or “Amanda.”

  • Each member of the family gave you their receipts and lists of purchases for the past two weeks. Enter all the following purchases into the Consumer Expenditure online diary. Please remember to ‘think aloud’ as you work through this task.





Food and Drink Away from Home



  1. Ice cream sandwich from ice cream truck $3.00 purchased by Catherine Smith on [INSERT DATE].

  2. Roast beef sandwich and a soda from a deli for lunch for $9.54 (including tax) purchased by Ben Smith on [INSERT DATE].

  3. Candy bar from vending machine for $0.75 (including tax) purchased by Ben Smith on [INSERT DATE].

  4. A slice of pizza at the mall for $2.50 (including tax) purchased by Ben Smith on [INSERT DATE].

  5. $35 on lobsters, baked potatoes and green beans. (including tax and tip) purchased by Dan Smith on [INSERT DATE].

  6. $35 on lobsters, baked potatoes and green beans (including tax and tip) purchased by Amanda Smith on [INSERT DATE].

  7. $20 on a bottle of wine. (including tax) purchased by Dan and Amanda Smith on [INSERT DATE].

  8. $7 on dessert. (including tax) purchased by Dan Smith on [INSERT DATE].

  9. $7 on dessert (including tax) purchased by Amanda Smith on [INSERT DATE].

  10. Egg and cheese McMuffin from McDonald’s for breakfast for $2.99 (including tax) purchased by Dan Smith on [INSERT DATE].

  11. Coffee from Starbucks for $4.00 (including tax) purchased by Dan Smith on [INSERT DATE].



Food and Drink for Home Consumption

  1. White bread purchased by Amanda Smith on [INSERT DATE] for $12.50 (excluding tax)

  2. Frozen orange juice purchased by Amanda Smith on [INSERT DATE] for $1.25 (excluding tax)

  3. 3 apples purchased by Amanda Smith on [INSERT DATE] for $1.99 (excluding tax)

  4. Skim milk purchased by Amanda Smith on [INSERT DATE] for $3.34 (excluding tax)

  5. 5 TV dinners purchased by Amanda Smith on [INSERT DATE] for $10.50



Clothing, Shoes, Jewelry, and Accessories

  1. Sneakers purchased by Dan Smith on [INSERT DATE] for his 6 year old niece for $29.99

  2. Sweatshirt purchased by Dan Smith on [INSERT DATE] for 18 year old nephew for $25.99



All Other Products/Services

  1. Newspaper purchased by Dan Smith on [INSERT DATE] for $1.50

  2. Fishing magazine purchased by Dan Smith on [INSERT DATE] for $3.75.

  3. Computer game for friend’s birthday present, purchased by Dan Smith on [INSERT DATE] for $50.85

  4. Stuffed animal purchased by Dan Smith on [INSERT DATE] for $15.99

  5. Taxi Fare and tip spent by Dan Smith on [INSERT DATE] for $20.00

  6. Gas for the family car purchased by Dan Smith on [INSERT DATE] for $65.43

  7. A pack of cigarettes purchased by Ben smith for $6.00 (excluding tax) on [INSERT DATE].

  8. A DVD from Amazon.com purchased by Amanda Smith for $15.76 (including tax) on [INSERT DATE].

  9. Men’s deodorant purchased by Catherine Smith for $2.99 (excluding tax) on [INSERT DATE]

  10. A mortgage payment made by Dan Smith for $4,000.00 on [INSERT DATE]

  11. A Health Insurance bill payment made by Dan Smith on [INSERT DATE] for $375.00

  12. A church contribution payment made by Dan Smith on [INSERT DATE] for $50.00

Task 2

  • After you finished entering all the purchases the Smith household made for the week, you review the entries to make sure you entered every entry listed, and the correct price.

As you are doing this, you notice that Catherine made a mistake on her list. The stuffed animal she purchased did not cost $15.99, it cost $25.99. Please make this correction now.

Remember to think aloud as you work.

Task 3

  • Ben Smith remembers that his friend actually purchased the computer game (and Ben did not). Please make this correction now.

Remember to think aloud as you work.

Dan Smith notes that the coffee from Starbucks on [INSERT DATE] actually cost $4.57. Please make this correction now.



Task 4

Please submit your data for the week.

Task 5 (This will be limited to the subgroup of Group 2 who are screened as appropriate and will not be presented to all group 2 participants.)

Now please re-enter the information using this computer/iPad. We are testing the usability of the CE Diary Survey on both devices.



Task 3: Receipt Information is Disorganized



  • The members of your household are Dan Smith, Amanda Smith, Ben Smith, and Catherine Smith. The participant will be either Dan or Amanda Smith and will be referred to as YOU during the session and not “Dan” or “Amanda.”

  • Each member of the family gave you their receipts and lists of purchases for the past two weeks. Enter all the following purchases into the Consumer Expenditure online diary. Please remember to ‘think aloud’ as you work through this task.

  1. Stuffed animal for $15.99 purchased by Catherine Smith on [INSERT DATE].

  2. A church contribution payment made by Dan Smith on [INSERT DATE] for $50.00



  1. Coffee from Starbucks for $4.00 (including tax) purchased by Dan Smith on [INSERT DATE].

  2. Frozen orange juice $1.25 (excluding tax) purchased at grocery store by Amanda Smith on [INSERT DATE].

  3. 5 TV dinners $10.50 purchased at grocery store by Amanda Smith on [INSERT DATE].

  4. $35 on lobsters, baked potatoes and green beans. (including tax and tip) purchased at restaurant by Dan Smith on [INSERT DATE].

  5. $35 on lobsters, backed potatoes and green beans (including tax and tip) purchased at restaurant by Amanda Smith on [INSERT DATE].

  6. Pepsi six-pack $3.99 (excluding tax) purchased by Catherine Smith on [INSERT DATE]

  7. Chicken wings, frozen $5.99 (excluding tax) purchased by Catherine Smith on [INSERT DATE]



  2. $1.50 for newspaper purchased by Dan Smith on [INSERT DATE].

  3. Sneakers for Dan Smith’s 6 year old niece for $29.99 purchased by Dan on [INSERT DATE].

  4. Men’s deodorant $2.99 (excluding tax) (excluding tax) purchased by Catherine Smith on [INSERT DATE]



  1. A slice of pizza at the mall for $2.50 (including tax).

  2. Ice cream sandwich for $3.00 purchased by Catherine Smith from ice cream truck on [INSERT DATE].

  3. Fishing magazine purchased by Dan Smith on [INSERT DATE] for $3.75.

  5. 3 apple for $1.99 (excluding tax) purchased by Amanda Smith on [INSERT DATE].

  6. Skim milk for $3.34 (excluding tax) purchased by Amanda Smith on [INSERT DATE].

  7. Roast beef sandwich and a soda for $9.54 (including tax) purchased from a deli for lunch by Ben Smith on [INSERT DATE].

  8. Computer game, $50.85 purchased for friend’s birthday present by Ben Smith on [INSERT DATE].

  9. $7 on dessert. (including tax) purchased by Dan Smith at restaurant on [INSERT DATE].

  10. $7 on dessert (including tax) purchased at restaurant by Amanda Smith on [INSERT DATE].

  11. A mortgage payment made by Dan Smith for $4,000.00 on [INSERT DATE]



  1. $20 on a bottle of wine (including tax) purchased at restaurant by Amanda Smith on [INSERT DATE].

  2. $20 on a bottle of wine (including tax) purchased by Dan Smith at restaurant on [INSERT DATE].

  3. Sweatshirt purchased by Dan Smith for $25.99 for an 18-year-old nephew on [INSERT DATE].

  4. A Health Insurance bill payment made by Dan Smith on [INSERT DATE] for $375.00



  1. Taxi Fare and tip for $20.00 spent by Dan Smith on [INSERT DATE].

  2. Egg and cheese McMuffin from McDonald’s for breakfast for $2.99 (including tax) purchased by Dan Smith on [INSERT DATE].

  3. A DVD from Amazon.com purchased by Amanda Smith for $15.76 (including tax) on [INSERT DATE].

  4. Candy bar from vending machine for $0.75 (including tax) purchased by Ben Smith on [INSERT DATE].

  5. Gas for the family car purchased by Dan Smith on [INSERT DATE] for $65.43

  6. A pack of cigarettes purchased by Ben smith for $6.00 (excluding tax) on [INSERT DATE].



  1. White bread purchased by Amanda Smith on [INSERT DATE] for $12.50 (excluding tax).

  1. A church contribution payment made by Dan Smith on [INSERT DATE] for $50.00

Task 2

  • After you finished entering all the purchases the Smith household made for the week, you review the entries to make sure you entered every entry listed, and the correct price.

As you are doing this, you notice that Catherine made a mistake on her list. The stuffed animal she purchased did not cost $15.99, it cost $25.99. Please make this correction now.

Remember to think aloud as you work.

Task 3

  • Ben Smith remembers that his friend actually purchased the computer game (and Ben did not). Please make this correction now.

Remember to think aloud as you work.

Dan Smith notes that the coffee from Starbucks on [INSERT DATE] actually cost $4.57. Please make this correction now.



Task 4

Please submit your data for the week.



Post-Session Segment

After the session is complete, tell the participant:



Thank you for completing the Consumer Expenditure Diary. I will be back to ask you some more questions about your experience shortly. In the meantime, please complete the paper satisfaction questionnaire at the upper right-hand corner of your desk (Attachment E).



Debriefing Questions:

  1. How easy or difficult did you think the login process was?

1

2

3

4

5

Very easy

Somewhat Easy

Neither easy or difficult

Not Very Easy

Very Difficult



  1. How secure did you think the login procedure was?



1

2

3

4

5

Very secure

Somewhat secure

Neither secure or insecure

Not Very Secure

Very Insecure





  1. How easy or hard was it for you to log in and create your password?

1

2

3

4

5

Very easy

Somewhat Easy

Neither easy or hard

Not Very Hard

Very Hard



  1. How easy or hard was it for you to find the email from the Census Bureau?

1

2

3

4

5

Very easy

Somewhat Easy

Neither easy or hard

Not Very Hard

Very Hard



  1. Do you have comments regarding the email from the Census Bureau? (text)

Shape1











  1. How easy or hard was it for you to log on to the diary once you had your account and password?

1

2

3

4

5

Very easy

Somewhat Easy

Neither easy or hard

Somewhat Hard

Very Hard



  1. Do you remember the email address that sent your login information?



No

Yes

  1. What was your reaction to that email address, if any?

Shape2















9. How easy or hard was it for you to select the correct dates for each purchase?”



1

2

3

4

5

Very easy

Somewhat Easy

Neither easy or hard

Somewhat Hard

Very Hard



  1. Did you understand the difference between the sections of the diary?

No

Yes



  1. How easy or hard was it for you to decide what section to put purchases in?

1

2

3

4

5

Very easy

Somewhat Easy

Neither easy or hard

Somewhat Hard

Very Hard







  1. Once you decided which section to enter a purchase in, how easy or hard was it for you to enter each purchase?

1

2

3

4

5

Very easy

Somewhat Easy

Neither easy or hard

Somewhat Hard

Very Hard



  1. How easy or hard was it for you to enter the dollars and cents into the purchase price?

1

2

3

4

5

Very easy

Somewhat Easy

Neither easy or hard

Somewhat Hard

Very Hard




14. If you had entered information about purchases that might be embarrassing to you, how comfortable or uncomfortable would you be to learn that your spouse or another household member, would also be able to access the information in the instrument?

1

2

3

4

5

Very Comfortable

Somewhat Comfortable

Neither comfortable or Uncomfortable

Somewhat Uncomfortable

Very Uncomfortable





15. Were there some purchases that were easier to enter than others? Which?

No

Yes, specify: ___________________________________________





16. Were there some purchases that were more difficult to enter than others? Which?

No

Yes, specify: ___________________________________________





17. Was it clear when you should submit data?

No

Yes

18. Was it clear when you were finished entering information for the Consumer Expenditure Diary Survey?

No

Yes









19. How clear was it that you had submitted your data to the Census Bureau?



1

2

3

4

5

Very Clear

Somewhat Clear

Neither clear or Unclear

Somewhat Unclear

Very Unclear



20. Is there any information that you would have like to have known while entering information into the online survey?

No

Yes, specify: ___________________________________________





Shape3 21. How would you find out more about what to include or exclude in certain expense categories? (allow them to open up the page for this)





Open the summary screen.

21. Did you utilize the 'Summary' tab, which allows the user to see a total amount of their expenses by category?

No

Yes



21b. If YES, Is this a feature that you found useful?

No

Yes











21c. If YES, How likely would you be use this information if you were completing this survey at home?

1

2

3

4

5

Very Likely

Somewhat Likely

Neither Likely nor Unlikely

Somewhat Unlikely

Very Unlikely





22. Looking at the pie chart, how interesting is this to you?

1

2

3

4

5

Very Interesting

Somewhat Interesting

Neither Interesting or Uninteresting

Somewhat Uninteresting

Very Uninteresting



23. Would you use this to change any of your responses?

No

Yes



24. Do you think you would change your spending habits based on this information?

No

Yes



Open the Print Expenses Tab

25. Did you utilize the 'Print Expenses' tab, which allows the user to see and print a table showing all of the expenses they reported?

No

Yes

25b. If YES: Is this ‘Print Expenses’ tab a feature that you found useful?

No

Yes



25c. If YES: How likely would you be to print and use this information if you were completing this survey at home?

1

2

3

4

5

Very Likely

Somewhat Likely

Neither Likely nor Unlikely

Somewhat Unlikely

Very Unlikely



26. How comfortable would you be with sharing this information with other members in your household if you were completing this with your own expenses?

1

2

3

4

5

Very Comfortable

Somewhat Comfortable

Neither comfortable or Uncomfortable

Somewhat Uncomfortable

Very Uncomfortable

Shape4 27. If you have other people in your household, how would you share the login and password information or would you have one person doing the entering?





Open the Welcome Screen.

28. How easy or difficult was it locating the “Go to Login Page” link?

1

2

3

4

5

Very easy

Somewhat Easy

Neither easy or hard

Somewhat Hard

Very Hard

For the iPad participants in Group 2:

29. Was it easier to enter the information for the CE Diary on the computer or the iPad?

Computer

iPad



30. If you were completing this survey at home, would you be more likely to complete it using a computer or a tablet like an iPad?

Computer

iPad

31. Please describe overall experience with completing the survey on the iPad.

Shape5

18 | Page


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorKathleen T. Ashenfelter
File Modified0000-00-00
File Created2021-02-02
© 2024 OMB.report | Privacy Policy