Note to Reviewer - Info Book Testing

June 20, 2017

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Erica Yu Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for Testing of CEQ Information Booklet contents

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 234.

If there are any questions regarding this project, please contact Erica Yu at

202-691-7924.

1. Introduction and Purpose

The Consumer Expenditure Survey (CE) provides data on the buying habits of American consumers, including both expenditures and income for the consumer unit. The survey’s goal is to capture a complete picture of the household’s spending. A major component of the CE is the in-person interview, during which the field representative (FR) leads the respondent through a series of questions designed to prompt recall of recent expenditures. In addition to these question prompts read aloud by the FR, the survey program also developed an Information Booklet (OMB information collection information stored at https://www.reginfo.gov/public/do/PRAViewIC?ref_nbr=201607-1220-002&icID=46041) designed to be read by the respondent as the FR reads aloud the survey questions. In regards to expenditures collection, the Information Booklet supplements the survey question wording with examples of eligible items that are considered by the survey to be part of the expense category. For example, the category “Shirts, sweaters, blouses, or tops” – although the category itself is already a list of items – includes an Information Booklet list of: cardigan, sweater vest, dress shirts, pullover, ski sweater, v-neck sweater, sport shirts, knit blouses, maternity tops, and t-shirts. The purpose of providing this supplementary list of examples is to further prompt the respondent to recall items that may not immediately come to mind based on the category name or question wording alone.

The Information Booklet is regularly revised to account for changes in products available to consumers (eg, adding “FitBit” to “General Sports Equipment”). However, testing of the examples provided may indicate recommendations that increase data quality by improving respondent comprehension of the survey questions. Survey methods research has shown that the examples given in a survey can alter a respondent’s understanding of the category. For example, Tourangeau, Conrad, Couper, and Ye (2014) showed that reporting of consumption of atypical foods was increased when the atypical foods were included as examples. These findings suggest that the CEQ Information Booklet examples may be influencing respondent recall, with possible biases if the examples provided do not provide an accurate definition of the category. The two purposes of this study are 1) to understand whether the current Information Booklet examples influence respondent comprehension of the expenditure categories, and 2) to develop recommendations for examples that improve respondent comprehension (any recommendations would undergo expert review or pre-testing separately before being implemented in production).

2. Research Design

To first understand the effect of the current Information Booklet examples on expenditure reporting, this study will compare comprehension with and without the current Information Booklet examples. Additionally, this comparison provides insight into the differences between in-person interviewing and telephone interviewing, which FRs sometimes do with households in later waves of the survey, because interviews conducted by telephone cannot use the Information Booklet.

1. Current examples provided (n=200 each group)
   Comparison: the survey question only vs. the survey question and the examples

Concurrently, an additional four experimental comparisons will be made to understand the types of comprehension errors being made. Each of these studies will be administered separately, with participants eligible to participate in only one.

2. Number of examples provided AND Inclusion instructions (n=100 each subgroup)
   A) Long lists of examples (e.g., the current Information Booklet gives 30 examples for “Small electrical kitchen appliances”) may communicate to respondents a sense of exhaustiveness, such that the list is considered a complete definition of the category.
   B) The current Information booklet uses the language “including” to introduce each list of examples. Participants may be interpreting this term as restricting the category to those items only.
   These two factors may interact. We use a 2x2 design that crosses these two factors:
   Comparison: a list of 20 items vs. a list of 6 items
   Comparison: “including” vs. “including but not limited to”

3. Specificity of examples provided (n=100 each group)
   Specific items, as compared to broad clusters of items, may provide examples of the category without providing information about the definition or boundaries of the category.
   Comparison: 6 specific items vs. 6 broad clusters for category member items

4. Typicality of examples provided (n=100 each group)
   As Tourangeau, et al. (2014) showed, providing atypical examples may improve respondent understanding of the definition or boundaries of the category.
   Comparison: 6 typical items vs. 6 atypical items

5. Exclusion statements (n=100 each group)
   Particularly with broad categories, the CEQ sometimes uses exclusion statements to specify when certain expenses should not be included. These statements appear in italics at the bottom of a list of examples. Exclusions statements may have unintended consequences on respondent comprehension of the definition or boundaries of the category.
   Comparison: A “do not include” statement vs. no statement

To the greatest extent possible, all other factors beyond the specific target factor (e.g., typicality) will be held equal within a pair of comparison groups.

Each comparison will be administered for the following eight expenditure categories. These categories were chosen for being standalone categories (i.e., they are not catch-all categories like “other vehicle or engine repairs”), frequently purchased item categories that participants are likely to be familiar with even if they have not had the expense during the reference period of the last three months. The relevant parts of the Information Booklet are shown in Appendix A.

• Small electrical kitchen appliances

• Electrical personal care appliances

• Computers, computer systems, or related hardware

• Musical instruments, supplies, or accessories

• Lawn mowing machinery or other yard equipment

• General sports equipment

• Closet and storage items

• Recreational lessons or other instructions

The materials for each comparison will be taken from the current Information Booklet. Where the Information Booklet does not have examples that meet the study specifications (e.g., fewer than 20 examples), additional examples will be created that are similar to the current examples.

Tasks

The protocol structure is shown in Appendix B. Participants will first be shown the existing expenditure question (e.g., Since the first of March, have you or any members of your household had any expenses for small electrical kitchen appliances?). After the participant answers the expenditure question and reports an expenditure amount, the participant will be asked debriefing questions to probe their comprehension of the question wording. Debriefing includes enumeration of the reported expense, an open-ended question asking what expenses the participant had that they considered reporting but did not include (probe for under-reporting of expenses); an open-ended “name five” task in which participants are asked to list the first five items that come to mind and would be eligible for the target category; and an “include/exclude” task in which participants are asked to mark whether they consider an item in or out of the category, or not sure; and a typicality rating task whereby participants rate the typicality of the Information Booklet examples that were shown to them. The include/exclude task will use items from the Information Booklet for the target expenditure category as well as related expenditure categories (to act as lures), and where the Information Booklet does not have enough examples, additional examples will be created that are similar to the current examples.

Analysis

Comprehension will be measured quantitatively by the relative accuracy of participant judgments on the include/exclude task, as well as qualitatively by the severity and types of comprehension errors made on the name five task. Comparisons will be made within the specified comparison groups only, not across specified comparisons.

3. Participants

For testing, 1,400 participants will be recruited using a convenience sample from Amazon Mechanical Turk of adult U.S. citizens (18 years and older).

Participants volunteering to take part in these studies will be randomly assigned to one of the experimental comparisons and expenditure categories. Our goal is to collect data from 100 participants for each comparison group. Additional participants (total of 200 participants for each comparison group) will be collected for the comparison between the survey question only vs. the survey question and the examples, to provide greater detail on how the current Information Booklet affects responses. There will be no eligibility criterion.

4. Burden Hours

We anticipate that each survey will average 10 minutes, including target questions and debriefing questions, for a total of 234 burden hours (1,400 participants x 0.167 burden hours).

5. Payment to Respondents

For online testing, each of the 1,400 participants will be paid $1.00 for participating in the study, a typical rate provided by Mechanical Turk for similar tasks; a total of $1,400.00 will be paid directly to Amazon Mechanical Turk for participant fees.

6. Data Confidentiality

Recruiting of participants will be handled by Amazon Mechanical Turk. Once participants are recruited into the study, they will be provided a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: April 30, 2018). You are not required to respond to this collection unless it displays a currently valid OMB control number. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Attachments:

Appendix A – Information Booklet excerpts

Appendix B – Protocol

Appendix A: Current Information Booklet excerpts

Small electrical kitchen appliances, including –

blender bread maker coffee grinder coffee maker crockpot deep fryer electric barbecue electric can opener electric fondue set electric frying pan

electric grill electric iron electric knife electric timer electric wine chiller electric wok food processor hot plate ice cream maker juicer

mixer pizza oven popcorn maker rice cooker sandwich grill slow cooker smoothie maker toaster toaster oven waffle iron

Electrical personal care appliances, including –

curling iron denture cleaner digital scale electric hair trimmer electric razor

electric toothbrush facial sauna flat iron foot bath hair dryer

heating pad make-up mirror massager Waterpik

Computers, computer systems, or related hardware, including

cables CD/DVD drive computer printers external hard drive keyboards laptops

memory modems monitors mouse netbooks scanner

Musical instruments, supplies, or accessories, including –

brass instruments carrying case clarinet drums guitar keyboards music books music stand piano picks reeds	rosin saxophone sheet music stringed instruments strings for musical instruments trombone trumpet valve oil woodwinds any other musical accessories
Do not include repairs, music lessons, or band uniforms.

Lawn mowing machinery or other yard equipment, including –

edger electric lawn trimmer garden hose lawn mower rake shovel

snow blower spreader tiller tractor (farm, garden, etc.) weed digger wheelbarrow

General sports equipment, including –

badminton set baseball bat baseball glove basketball basketball hoop bowling ball boxing equipment football Frisbee golf bag golf clubs helmets karate equipment	lawn games pads racquetball racquetball racket roller blades skateboard soccer ball sports protective equipment sports uniform table tennis equipment tennis racket volleyball
Do not include any shoes for sports. Do not include any equipment for water sports.

Closet and storage items, including –

drawer organizers drawers garment bag garment racks

hampers hangers hanging bags shelf organizers

shoe bag shoe rack storage bins underbed storage

Recreational lessons or other instructions, including –

cooking dancing driving golf horseback riding instructional day camps

music needlepoint painting photography sailing self defense

sewing skiing skydiving swimming tennis

Appendix A: Protocol

Welcome! Thanks for your interest in our survey. You’re here because we have asked you to participate in our research to improve survey questions. We are asking you and hundreds of other people to tell us about what they think.

Unlike some surveys or online tasks you may be familiar with, we ask that you complete this survey all at one time and that you only start once you are in a quiet place where you won't be disturbed. The survey takes about 10 minutes, on average. Only share information you're comfortable with - nothing too personal - but please be honest and follow the instructions.

 Please do not use your browser's back button. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: April 30, 2018). You are not required to respond to this collection unless it displays a currently valid OMB control number. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this study.

---page break---

We are asking for your help to improve our survey questions. We are considering changing one of our surveys and we need to learn whether the new version will make sense to respondents.

On the following pages, you’ll be asked to answer questions about expenditures your household may or may not have had over the last three months. We will then ask you some additional questions to better understand your answers.

Let’s get started!

---page break---

We’ll start by asking a few questions about you.

What is your age in years?

What is your gender?

• Male

• Female

• Prefer not to say

What is the highest level of school you have completed or the highest degree you have received?

• Grades 1-12/No Diploma

• High School Diploma or Equivalent

• Some College

• Associate’s Degree

• Bachelor’s Degree

• Master’s Degree

• Doctorate or Professional Degree

Are you Hispanic or Latino?

• Yes

• No

What is your race? (you may select more than one)

• American Indian or Alaska Native

• Asian

• Black or African-American

• Native Hawaiian or Other Pacific Islander

• White

---page break---

Now we’ll ask you about your recent expenses:

Since the first of April, have you or any members of your household had any expenses for [FILL]?

[Here are some examples of the kinds of expenses that would fit in this category:]

[Item Item Item Item]

[Item Item Item Item]

[Item Item Item Item]

[Item Item Item Item]

[Do not include XX]

Yes

No

---page break---

If Yes to the expense question:

What was the total expense?

[numeric text entry]

---page break---

If Yes to the expense question:

Please tell us what individual expenses went into that total expense that you just reported.

[table with 5 rows and columns for Item and Amount Paid]

---page break---

If Yes to the expense question:

Were there any items or services that your household had expenses for that you considered might fit into this category (FILL), but that you decided to exclude from your initial response?

[5 text-entry boxes]

---page break---

We are trying to understand what kinds of expenses the phrase “(FILL)” makes people think of. What are the FIRST five expenses that would fit into this category that come to mind? Even if you have never had that expense yourself, please include it.

[5 text-entry boxes]

---page break---

Please look through the items listed below and mark whether you would consider the item as part of “(FILL)”. If you are not sure, mark “not sure”.

	Definitely YES	Definitely NO	Not sure
[Item]
[Item]
[Item]

---page break---

If saw any Information Booklet examples:

At the beginning of this survey, we showed you a list of examples of items that were included in the category “FILL”. We want to learn if the examples we showed are “typical” of this category – by typical, we mean common or usual.

Here’s an example of what we mean:

How typical is each of these items of the category “Fruit”?

Apple = Very typical

Pomelo = Not at all typical

Please tell us below your opinion about how typical of the category “FILL” each item is.

	Very typical	Somewhat typical	Not at all typical
[Item]
[Item]
[Item]

---page break---

Thanks so much for your participation! Your unique completion code for mTurk is: [FILL].

11