Note to Reviewer - CE Globals

June 20, 2017

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Erica Yu Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for Supplemental Testing of Global Questions for the CE Recall and Records Interviews

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 240.

If there are any questions regarding this project, please contact Erica Yu at

202-691-7924.

1. Introduction and Purpose

This proposed study is a follow-up to the study titled “Submission of Materials for the Global Questions Testing for the CE Recall and Records Interviews”, which was submitted by Brandon Kopp in July 2016 and approved by OMB.

The Consumer Expenditure Survey (CE) provides data on the buying habits of American consumers, including both expenditures and income for the consumer unit. BLS is currently redesigning the CE with the goals of reducing both measurement error and response burden. In the redesigned CE, respondents will provide a wide range of expenditures, from monthly utility bills to irregular entertainment expenses, through the Recall interview, the Diary, and the Records interview.

The in-person Records and Recall interviews will require respondents to report their household’s expenditures for a selected set of expenditure categories for the preceding three month period. To reduce the burden associated with the current CE interview, BLS is developing global questions that collect data at a higher level of aggregation (e.g., large appliances instead of washers, dryers, refrigerators, freezers, etc.) than data traditionally collected in the CE. Previous research (Crafts, et al., 2015; Edgar, 2011; Kopp, 2012; Kopp & Yu, 2013) on the global questions has revealed mixed results. The focus of this study is to develop and test global questions for the Records and Recall interviews with the objective of administering fewer questions than the current CE interview without sacrificing accuracy.

In 2014-2017, work was conducted by Westat on behalf of BLS to cognitively test global questions (two data collection periods, both approved by OMB; the most recent one cited above). At the end of the final task order in 2017, Westat recommended that question wordings for several item categories be re-tested before being implemented. The two primary reasons for this recommendation are: an insufficient number of participants in Westat’s testing actually had the targeted expense or question wording was revised at the end of testing and there has not yet been an opportunity to test the proposed wording with participants. Both of these issues can be addressed by further testing of the questions.

2. Research Design

While the previously OMB-approved study conducted by Westat used iterative cognitive interviewing techniques, the current proposed study will make two substantive methodological changes. First, the questions will be tested individually (rather than clustered together in protocols with 10-20 questions) in order to enable recruitment of participants with the relevant expenses. Second, the questions will be tested online (rather than in the lab) in order to collect a large sample efficiently. Furthermore, testing will be conducted all at once without a design for iterative rounds of data collection.

The expenditure questions that will be tested are listed in a table format in Appendix A. The wording of the questions shown reflects Westat’s final recommendation for the question wording after their testing.

Before the target question, participants will be informed that the purpose of the task is to help BLS improve question wording. Before seeing the target global question, the participants will be shown a statement that the question they will be asked is only one out of a long survey, and that they should: (1) exclude certain expenses because they would already have been reported earlier in the survey; (2) consider certain expenses because they would have already been reported earlier in the survey (e.g. a list of electronics that CE respondents would normally be asked about before seeing a question asking about repairs of electronics); or (3) no additional information; the context information shown follows the protocol used by Westat during their 2017 testing. Although these statements add complexity to the task, this component is necessary in order to test question comprehension as if the individual was participating in the full survey. These context items are included in Appendix A.

After the participant answers the expenditure question and reports an expenditure amount, the participant will be asked debriefing questions to probe their comprehension of the question wording. Debriefing includes enumeration of the global expense, an open-ended question asking what expenses the participant had that they considered reporting but did not include (probe for under-reporting of expenses); an open-ended “name five” task in which participants are asked to list the first five items that come to mind and would be eligible for the target category; and an “include/exclude” task in which participants are shown items from the CE InfoBook (OMB information collection information stored at https://www.reginfo.gov/public/do/PRAViewIC?ref_nbr=201607-1220-002&icID=46041; relevant excerpts in Attachment1) and asked to mark whether they consider the item in or out of the category, or not sure (Appendix B). Below is an example of how the InfoBook is used in the include/exclude task (order of items to be randomized):

Purchase of any RV, boat, trailer, or plane

	Definitely IN	Definitely OUT	Not sure
boat with outboard motor cabin cruiser jet skis launch
canoes kayaks rowboats
ATV trailer boat trailer cargo trailer
gliders helicopters
horse trailer motorcycle trailer utility trailer
planes
rowing shells sailboats
pontoon boat power boat yacht

3. Participants

For testing, 1,440 participants will be recruited using a convenience sample from Amazon Mechanical Turk of adult U.S. citizens (18 years and older).

Each set of questions will be advertised for recruitment independently and individuals may participate in more than 1 of the 32 modules, due to the structure of the Mechanical Turk platform. Our goal is to collect data from 30 participants for each module (960 participants) who have the targeted expenses. To recruit these participants, the global category name will be included in the description of the study as an eligibility criterion. In addition, 15 participants will be recruited for each module, without the use of any eligibility criteria, to collect data on whether the global question does not erroneously screen out people who have the global category expenses (480 participants). These 15 participants will see the initial target global question and the two debriefing comprehension tasks; if they do not have the expenditure then they will not be asked the expenditure or enumeration questions.

4. Burden Hours

We anticipate that each survey will average 10 minutes, including screening, target questions, and debriefing questions, for a total of 240 burden hours (1,440 participants x 0.1667 burden hours).

5. Payment to Respondents

For online testing, each of the 1,440 participants will be paid $1.00 for participating in the study, a typical rate provided by Mechanical Turk for similar tasks; a total of $1,440.00 will be paid directly to Amazon Mechanical Turk for participant fees.

6. Data Confidentiality

Recruiting of participants will be handled by Amazon Mechanical Turk. Once participants are recruited into the study, they will be provided a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: April 30, 2018). You are not required to respond to this collection unless it displays a currently valid OMB control number. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Attachments:

Appendix A – Expenditure questions

Appendix B – Protocol

Attachment 1 – excerpts from 2017 CE InfoBook

Appendix A: Expenditure questions

In the table below, the second column shows the question wording for each target global category being tested. Each item follows after the question stem: “Since the first of April, have you or any members of your household had any expenses for …”, unless otherwise noted. The third column shows any context information that will be shown to the participant. The context information attempts to compensate for the single-question nature of the task, whereby participants do not experience the full survey and all of the survey questions that appear before the target question. Many items are listed with exclusion instructions due to there being questions collecting expenditures for those items appearing earlier in the full survey (e.g., #5 below targets disposal of recreational vehicles; the context information will instruct participants to exclude disposal of automobiles). Other items are listed with consideration instructions for a similar but different reason: the context information will instruct participants to consider related expenses that appear earlier in the survey and may cue them to remember target expenses (e.g., #17 below targets computer repairs; the context information will instruct participants to think about any computers they may have purchased).

Item #	Question Wording	Context Items
1	Rental or leasing of any car, truck, motorcycle, or plane excluding any employer paid vehicles
2	Rental or leasing of any RV, camper, trailer, ATV or other recreational vehicle	Rental or leasing of any car, truck, motorcycle, or plane excluding any employer paid vehicles
3	Rental or leasing of any boat with or without a motor	Rental or leasing of any RV, camper, trailer, ATV or other recreational vehicle
4	Purchase of any RV, boat, trailer, or plane	Rental or leasing of any RV, camper, trailer, ATV or other recreational vehicle
5	Have you or anyone in your household disposed of any recreational vehicles such as motor homes or boats? How much did you receive from selling the vehicle or getting reimbursed for it?	Disposal of any automobiles, trucks, minivans, vans or SUVs
6	Vehicle maintenance or repair that includes labor, such as oil change, tune-up, brake work, repair, or replacement services	Tire purchases or mounting
7	Vehicle registration fees	Driver’s licenses, Vehicle inspections
8	Premiums for any commercial health insurance plan such as a PPO, HMO, dental or vision plan	Medicare, Medicare Part D
9	Daycare, nursery, or preschool tuition and fees;	Recreational lessons or other instruction
10	Any window coverings, including curtains, drapes, blinds, or shades? Do NOT include cost of hardware purchased separately.
11	Indoor lamps, ceiling or wall lighting, or ceiling fans? Do NOT include light bulbs
12	Household decorative items such as artificial flowers or plants, clocks, art, mirrors, or seasonal decorations	Fresh flowers or potted plants
13	Living, family, or recreation room furniture? Do not include TV or sound equipment.	Kitchen or dining room furniture; infant furniture; office furniture for home use; renting or leasing of any furniture
14	Any bedroom furniture, mattresses or box springs;	Bedroom linens
15	Rental or repair of outdoor equipment or tools, including motorized, electric, or manual	Gardening or lawn care services
16	Power or non-power tools, their accessories, or hardware	Barbeque grills; lawn or yard tools
17	Repairs or warranties for computers, phones, tablets, or digital book readers. Do NOT include repairs or warranties for TVs.	Consider: Computer, computer systems, or related hardware; Computer software including computer games; Computer accessories; Digital book readers or tablets; Apps, games, or ringtones for devices
18	Rental or repair of any audio or video electronic equipment	Consider: Televisions, all types including those installed in vehicles; DVD players, VCRs, DVRs, or video cameras; Handheld personal music players; Stereos, radios, speakers, and sound components, including those installed in vehicles; Other sound or video equipment, including accessories
19	Rental of computer, telephone, or office machines for non-business use
20	Rental of small household equipment such as a toaster, space heater, or vacuum cleaner	Purchase or installation of household appliances such as refrigerator or home freezer, clothes washer or dryer, cooking stove, range, or oven, or dishwasher; Purchase or installation of small kitchen appliances, window air conditioners, or portable heating or cooling equipment
21	Men’s suits, sportcoats, or tailored jackets	men’s outerwear
22	Women’s suits, sportcoats, or tailored jackets	women’s outerwear
23	Boys’ suits, sportcoats, or vests	boy’s outerwear
24	Girls’ dresses, or suits	girl’s outerwear
25	Jewelry or watches, for men, women, or children
26	Purchase of exercise equipment, sport, and recreational goods excluding clothing and athletic shoes
27	Maintenance or repairs for any exercise equipment, sport, or recreational goods including warranties or service contracts
28	Photography equipment, professional services, or products including printed photos. Do not include phones or picture frames.
29	Pet medical care or pet services	Pet food; Purchase of pets’ Pet supplies and medicines
30	Public transportation including taxis, limousine service, and car rides book through an app	taxis, cabs, limousine service, or car rides booked through an app
31	[Did you go on an out of town trip?] …alcoholic beverages at restaurants, bars, or fast food places while on the out of town trip
32	[Did you go on an out of town trip?] …alcoholic beverages at grocery stores, convenience stores, or liquor stores while on the out of town trip

Appendix B: Protocol

Welcome! Thanks for your interest in our survey. You’re here because we have asked you to participate in our research to improve survey questions. We are asking you and hundreds of other people to tell us about what they think.

Unlike some surveys or online tasks you may be familiar with, we ask that you complete this survey all at one time and that you only start once you are in a quiet place where you won't be disturbed. The survey takes about 5 minutes, on average. Only share information you're comfortable with - nothing too personal - but please be honest and follow the instructions.

 Please do not use your browser's back button. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: April 30, 2018).  You are not required to respond to this collection unless it displays a currently valid OMB control number. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this study.

---page break---

We are asking for your help to improve our survey questions. We are adding new questions to our survey and we need to learn whether they make sense to respondents.

On the following pages, you’ll be asked to answer questions about expenditures your household may or may not have had over the last three months. What you will see is just one question out of a long survey so you may see some additional context information as well. We will then ask you some additional questions to better understand your answers.

Let’s get started!

---page break---

We’ll start by asking a few questions about you.

What is your age in years?

What is your gender?

• Male

• Female

• Prefer not to say

What is the highest level of school you have completed or the highest degree you have received?

• Grades 1-12/No Diploma

• High School Diploma or Equivalent

• Some College

• Associate’s Degree

• Bachelor’s Degree

• Master’s Degree

• Doctorate or Professional Degree

Are you Hispanic or Latino?

• Yes

• No

What is your race? (Can select more than one race)

• American Indian or Alaska Native

• Asian

• Black or African-American

• Native Hawaiian or Other Pacific Islander

• White

---page break---

Now we’ll ask you about your recent expenses:

[When answering the question below, please [consider/exclude] any [FILL]].

Since the first of April, have you or any members of your household had any expenses for [FILL]?

Yes

No

---page break---

If Yes to the expense question:

What was the total expense?

[numeric text entry]

---page break---

If Yes to the expense question:

Please tell us what individual expenses went into that total expense that you just reported.

[table with 5 rows and columns for Item/Service and Amount Paid]

---page break---

If Yes to the expense question:

Were there any items or services that your household had expenses for that you considered might fit into this category (FILL), but that you decided to exclude from your initial response?

[5 text-entry boxes]

---page break---

We are trying to understand what kinds of expenses the phrase “(FILL)” makes people think of. What are the FIRST five expenses that would fit into this category that come to mind? Even if you have never had that expense yourself, please include it.

[5 text-entry boxes]

---page break---

Please look through the items listed below and mark whether you would consider the item as part of the category “(FILL)”. If you are not sure, mark “not sure”.

	Definitely YES	Definitely NO	Not sure
[Item]
[Item]
[Item]

---page break---

Thanks so much for your participation! Your unique completion code for mTurk is: [FILL].

16