CE Global Questions Cognitive Study

August 2, 2011

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Jennifer Edgar Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the CE Global Questions Cognitive Study

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for the study is 37.

If there are any questions regarding this project, please contact Jennifer Edgar at

202-691-7528.

Note: This package reflects a change in the order of tasks in the self-administered portion of the study. No other details have been changed.

Introduction and Purpose

The use of global questions has been long discussed in the context of a redesigned CE to allow for collection of complete household spending in a time-efficient way. By replacing multiple detailed questions (e.g., shirts, pants, shorts, shoes) with a few global questions (e.g., clothing, footwear, accessories), it is possible that useful expenditure data can be collected in a fraction of the time. There are multiple possible uses of global questions, including in a split questionnaire design, as part of an allocation model to incorporate CED data in the CEQ, or to replace detailed questions in a shortened bounding interview.

The quality of the data collected by global questions has been addressed in past research such as Goldenberg & Steinberg, 2010; Weber, 2002; and RTI 2000. The current CE Measurement Issues study is also exploring the topic. These studies found that overall the level of expenditures collected from global questions tend to be higher than from the associated detailed questions, but without a ‘gold standard’, it has not been possible to identify which question type yields better quality data.

Additionally, these studies have not addressed the cognitive processes behind global questions: how respondents answer these questions. Without understanding the response strategies respondents use when answering global questions, the reasons for the difference between global and detailed questions are unknown. Exploring the response process for global questions is a key step in evaluating the quality of the data they collect, and will inform other comparisons between global and detailed questions.

As noted in the 2010 CE Methods Workshop, laboratory research can be used to evaluate global questions: “Some relatively straightforward laboratory investigations could provide a great deal of information about memory and response strategies, which could be informative about the relative advantages of global and specific questions (Beatty, 2010).” Similarly, Peychetv (2010) noted that “the decision to use global questions, and for which categories, needs to be informed by an understanding not only of the differences in estimates and their properties, but also by the causes for the differences.” Both experts suggested the use of cognitive research to evaluate global questions.

The purpose of this study is to address the following research questions:

1. When respondents answer global questions, what specific items do they include, and how does their definition of the global category align with the intended meaning of the question (and the associated detailed questions)? For example do they include coats in clothing? Socks? Headbands?

2. How do respondents arrive at their answer to global questions (e.g., do they estimate, identify specific purchases and add)?

2. Research Design

To address the research questions, two methods will be used, a self-administered web survey and an interviewer-administered cognitive interview.

Self-administered Web Survey

Participants will complete a web survey administered by TryMyUI (questions provided as Attachment A). They will answer three global clothing questions, three open-ended questions asking them to report detailed clothing purchases made in the last month, three questions asking them to list as many examples as they can for each of the global clothing categories, and then complete a categorization exercise where they indicate to which global category a given item belongs. Following those questions, they will be asked the standard CEQ clothing questions.

TryMyUI

TryMyUI (www.TryMyUI.com) is an online usability testing company which allows participants to participate in usability studies from their homes. Once they have joined the TryMyUI participant group, they are screened based on various demographic characteristics and sent links to usability tasks they qualify for. Each usability task is limited to twenty minutes and one URL. During the testing session, participants follow online instructions and use a think-aloud approach to complete the task described. A video showing the participant completing the task is then provided.

For this study, this usability testing site is being used as a way to collect a large amount of data in an efficient (both in terms of financial and staff resources) way. TryMYUI is responsible for recruitment, screening, administration of the survey, and data collection. The company claims to be able to provide data within hours of receiving the study from the sponsor. The same data would take weeks for OSMR to collect in the usability lab.

Participants will receive a link to an online survey (hosted by SurveyMonkey) as noted in Attachment A. For most of the tasks, they will be instructed to simply answer the questions, typing or clicking their response as on a standard web survey. To take advantage of the video and talk aloud capabilities of TryMYUI, a few questions will have instructions for participants to describe their thought processes as they answered (retrospective) or answer (concurrent) the questions.

Interviewer-administered Interview

A separate group of participants will complete a standard cognitive interview in the OSMR lab, using the protocol included as Attachment B. Each participant will be asked the three global clothing questions, and then will be asked a series of debriefing questions to identify response strategies and calculations. Questions about question interpretation, category definition and item inclusion/exclusion will also be asked during the debriefing. Following the debriefing, the standard CEQ clothing questions will be asked. Finally, the participant will be asked to evaluate how well they feel the global clothing questions and CEQ clothing questions align.

If the above interview topics are completed in less than 60 minutes, we will repeat the process for another CE category. The category will be one of the following: household items, appliances, furniture, education or medical, assigned based on their participant number (with the goal of having an equal number of participants complete each category). Outlines for the interview protocol for these categories are shown in Attachment C. As time allows within the testing session, the process will be repeated for additional categories.

3. Participants

Self-administered Survey

Participants are recruited by TryMyUI. TryMyUI qualifies candidate users based on their ability to stay on task and to clearly voice everything they, see, read, think, or do as they are interacting with a website. They are volunteers who respond to a recruitment ad on their website. Participants will be selected with varying levels of age, gender, and education.

Interviewer-administered Interview

Participants will be recruited from the OSMR participant database. Efforts will be made to select participants with varying levels of age, gender, and education based on self-reported information provided during the initial recruitment process. Participants will be screened to ensure they’ve purchased clothing within the past month.

4. Payment

Self-administered Survey

In the TryMyUI framework, participants are paid $10 per completed survey. This is a fixed amount set by the company.

Interviewer-administered Interview

Participants will receive $40.

5. Burden Hours

The self-administered survey is limited to twenty minutes, with up to 50 participants completing it. The interviewer-administered interview is expected to take up to 60 minutes, with up to 20 participants completing it, for a total of 37 burden hours.

5. Data Confidentiality

Participants will be informed as to the voluntary nature of the study, and that the study will be used for internal purposes to improve the design of a national consumer expenditure interview survey. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice. OMB Control Number: 1220-0141

Self-administered Survey

The following notice will be placed on the first page of the survey: “Note: This survey is being administered by surveymonkey.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.”

Interviewer-administered Interview

Prior to the interview, participants will be given a consent form to read and sign (Attachment D).

6. References

Beatty, P. (2010). Considerations Regarding the Use of Global Survey Questions Prepared for the Consumer Expenditures Survey Methods Workshop, December 8-9, 2010.

Goldenberg, K. & Steinberg, B. (2010). Global Questions for the Consumer Expenditure Quarterly Interview Survey: Do they Achieve Equivalent Results? Internal BLS Report.

Peytchev, A. (2010). Global versus Specific Questions for the Consumer Expenditure Survey. Prepared for the Consumer Expenditures Survey Methods Workshop, December 8-9, 2010.

RTI (2000). CE Interview Global Questions Research: BLS Conference Presentation Report. Internal BLS report.

Weber, W. (2002). CE Diary food-at-home “global-detailed” Comparisons. Interim report. Internal BLS Report.

Attachment A: Self-administered Web Survey Questions

Welcome!

Unlike some other TryMyUI studies, you will not be using a website to complete a task for this study. Instead you’ll be answering survey questions.

You’ll only be asked to talk aloud as you complete some of the sections, but feel free to do so at any point. The instructions will tell you when you need to talk aloud.

The Bureau of Labor Statistics is exploring ways to better measure how Americans spend their money. We will be asking you about recent clothing purchases. Some questions may sound similar, but they are different, so please read each carefully.

Part 1: Global Clothing

Instructions: Please provide a response to each question. You do not need to think aloud as you answer these questions, only type in your response.

1. How much have you or any member of your household spent on clothing since the first of [refmonth]? $____________

2. How much have you or any member of your household spent on footwear since the first of [refmonth]? $____________

3. How much have you or any member of your household spent on jewelry and accessories since the first of [refmonth]? $____________

4. Now, please talk about how you arrived at your answer for question 1. Describe what you thought about or what you remembered as you answered the question.

Part 2: Detailed Clothing

Instructions: Please provide a response to each question. You do not need to think aloud as you answer these questions, only type in your response.

1. What clothing items have you or any member of your household bought since the first of [refmonth] and how much did each cost?

What clothing did you buy?	How much did it cost?

2. What footwear have you or any member of your household bought since the first of [refmonth] and how much did each cost?

What footwear did you buy?	How much did it cost?

3. What jewelry or accessories items have you or any member of your household bought since the first of [refmonth] and how much did each cost?

What jewelry or accessories did you buy?	How much did it cost?

Part 3. CEQ Clothing Questions

Instructions. For this last set of questions, please think aloud, describing what you’re thinking about as you answer each one.

Since the first of the [refmonth], have you or has any member of your household purchased any of the following items, either for members of your household or for someone outside your household?

NOTE: For each yes response, participant will be asked to specify item and cost

1. Coats, jackets or furs,

2. Sport coats or tailored jackets

3. Suits

4. Vests

5. Sweaters or sweater sets

6. Pants, jeans, or shorts

7. Dresses

8. Skirts

9. Shirts, blouses, or tops

10. Undergarments

11. Hosiery

13. Nightwear or loungewear

14. Accessories

15. Swimsuits or warm-up or ski suits

16. Uniforms, for which the cost is not reimbursed

17. Costumes

18. Footwear, including athletic footwear

19. Diapers

20. Layettes

21. Watches

22. Jewelry

If yes: What did you buy? Describe briefly the item purchased and cost (e.g. sweater, $20) _____________

Part 4: Examples

Instructions: Please list as many examples as you can for each category. You do not need to think aloud as you answer these questions, only type in your response.

These do not have to be things you have purchased in the past, just all examples you can think of.

Please list as many examples of clothing as you can

Please list as many examples of footwear as you can

Please list as many examples of jewelry and accessories as you can

Part 5: Categorization

Instructions: For each of the following items, please mark a column to indicate if the item is clothing, footwear, jewelry and accessories or something else. If you think the item falls into another category, please write it down.

You do not need to think aloud as you answer these questions, only type in your response.

	Categorize
Item	Clothing		Footwear	Jewelry and Accessories	Other (specify)
Apron
Athletic shirt
Athletic shorts
Bandannas
Bedroom slippers
Belts
Biking shoes
Blazer
Boots
Bra
Bridal headpiece
Casual shoes
Dance costume
Down vest
Dress shirt
Dress shoes
Dress slacks
Dresses
Ear muffs
Football cleats
Fur coat
Gloves
Golf skirts
Hair accessories
Halloween costume
Handkerchiefs
Housecoat
Hunting wear
Jacket
Jogging suit
Jump suit
Knee-highs
Knit blouses
Leotard
Maternity Pants
Men’s suit
Mittens
Night gown
Night shirt
Non-prescription sunglasses
Overalls
Pajamas
Panty-hose
Pullover
Purse
Raincoat
Robe
Sandals
Scarves
School uniforms
Shape wear
Shawl
Short sets
Ski suit
Ski sweater
Skirts
Shorts
Sneakers
Snow suit
Soccer cleats
Socks
Sport coat
Sport shirt
Sports jersey
Sports uniforms
Sweat shirt
Swimwear
Swim accessories
Tailored Jacket
Tennis outfit
Tennis shoes
Tennis skirts
Thermal underwear
Ties
Tights
t-shirt
Umbrellas
Undershirt
Underwear
Vests
V-neck sweater
Wallet
Wedding gown
Windbreaker
Winter coat
Woman’s suit
Work uniforms

Attachment B: Cognitive Interviewing Protocol

Global Clothing Questions

GC1. How much have you spent on clothing since the first of [refmonth]?

GC2. How much have you spent on footwear since the first of [refmonth]?

GC3. How much have you spent on jewelry and accessories since the first of [refmonth]?

Global Clothing Debriefing

1. You said you’ve spent ____ on clothing in the last month. How did you arrive at that answer? Probe for recall strategies as possible, enumeration, estimation, etc.

   1. Ask if thought of specific stores, events, timeframe, etc.

2. What specific clothing items have you, or someone in your household, purchased since the first of [refmonth]?

3. Other than the clothing items you just named, please give me some examples of items that are in the category of clothing

4. On a scale from one to ten, with ten being absolutely certain and 1 being totally uncertain, how sure are you that your answer of __ is accurate?

   1. If less than ten: Please think about the question again, but this time give me a range from the lowest amount to the highest amount that you think definitely includes how much you and members of your household spent on clothing since the first of [refmonth].

5. You said you’ve spent ____ on footwear in the last month. How did you arrive at that answer? Probe foras much detail as possible to identify the recall strategies as possible, enumeration, estimation, etc.

   1. Ask if thought of specific stores, events, timeframe, etc.

6. What specific footwear items have you, or someone in your household, purchased since the first of [refmonth]?

7. Other than the footwear items you just named, please give me some examples of items that are in the category of footwear.

8. On a scale from one to ten, with ten being absolutely certain and 1 being totally uncertain, how sure are you that your answer of __ is accurate?

   1. If less than ten: Please think about the question again, but this time give me a range from the lowest amount to the highest amount that you think definitely includes how much you and members of your household spent on footwear since the first of [refmonth]

9. You said you’ve spent ____ on jewelry and accessories in the last month. how did you arrive at that answer? Probe for recall strategies as possible, enumeration, estimation, etc.

   1. Ask if thought of specific stores, events, timeframe, etc.

10. What specific jewelry and accessories items have you, or someone in your household, purchased since the first of [refmonth]?

11. Other than the jewelry and accessories items you just named, please give me some examples of items that are in the category of jewelry and accessories

12. On a scale from one to ten, with ten being absolutely certain and 1 being totally uncertain, how sure are you that your answer of __ is accurate?

    1. If less than ten, Please think about the question again, but this time give me a range from the lowest amount to the highest amount that you think definitely includes how much you and members of your household spent on jewelry and accessories since the first of [refmonth]

Standard CEQ Clothing Questions

Now I’m going to ask you some other questions about purchases you or your household have made since the first of [refmonth]. Please tell me about everything you’ve bought, even if we’ve already talked about it.

Since the first of [refmonth], have you or has any member of your household purchased any of the following items, either for members of your household or for someone outside your household?

1. Coats, jackets or furs

2. Sport coats or tailored jackets

3. Suits

4. Vests

5. Sweaters or sweater sets

6. Pants, jeans, or shorts

7. Dresses

8. Skirts

9. Shirts, blouses, or tops

10. Undergarments

11. Hosiery

12. Nightwear or loungewear

13. Accessories

13. Swimsuits or warm-up or ski suits

14. Uniforms, for which the cost is not reimbursed

15. Costumes

16. Footwear, including athletic footwear

17. Diapers

18. Layettes

19. Watches

20. Jewelry

If yes: What did you buy? Describe briefly the item purchased _____________

Standard CEQ Clothing Debriefing

1. At first, I asked three questions about clothing, footwear and jewelry and accessories. Just now I asked 20 questions about the same topics. The first set were much more general and the second set were more detailed. After hearing the detailed set of questions, would you change your answers to the set of three general questions?

   1. If yes. How so? What would you change your answers to?

2. Which set of questions do you think gave the most accurate answers?

3. Were there any questions in the detailed set (give show card with list¹) that surprised you? Any that you didn’t think of before when we were talking?

4. Are there any types of clothing, footwear or accessories that you think should be added to the list (show card)?

Attachment C: Consent Form

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 02/29/12

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

1 Show card will list expenditure questions