June 13, 2013
NOTE TO THE REVIEWER OF: |
OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
|
FROM: |
Brandon Kopp Research Psychologist Office of Survey Methods Research
|
SUBJECT: |
Submission of Materials for testing the Effectiveness of Global Questions and Different Length Reference Periods in the CEQ |
Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.
The total estimated respondent burden hours for this study is 34 hours.
If there are any questions regarding this project, please contact Brandon Kopp at
202-691-7514 or Jennifer Edgar at 202-691-7528.
Introduction and Purpose
The Consumer Expenditure Survey (CE) is currently going through a redesign process. Two of the stated goals of this process are to reduce respondent burden and measurement error. These goals are not always compatible with one another. One may wish to reduce burden by asking more general questions that each cover a range of topic areas (referred to below as global questions), but this may make the respondent’s task of answering a question more difficult, leading to greater measurement error. This, counter-intuitively, may lead to an increase in overall burden if the respondent perceives the additional memory search and answer formulation requirements of the question as more burdensome.
The purpose of the current study is to examine one method that has been proposed to reduce respondent burden in a redesigned Consumer Expenditure Quarterly Interview Survey (CEQ); asking about expenditure categories at a higher level of aggregation. Asking a global question about a high-level expenditure category (e.g., furniture) rather than a series of questions about the items that make up the category (e.g., sofas, living room chairs, living room tables, etc.) would reduce the number of questions a respondent has to answer and potentially decrease burden. The challenge is determining the proper level of aggregation for a given category or item that allows for a decrease in burden while minimizing any associated increase in measurement error.
The current study is meant to provide preliminary information on this issue by asking participants about purchases they’ve made while varying the level of aggregation, and determining the measurement error properties of the their answers using cognitive interviewing. If this testing shows that the implementation of global questions is successful (i.e., burden is decreased while measurement error is held constant or reduced), then additional expenditure categories would be considered for more intensive, future testing in which measurement error is more thoroughly assessed through financial records collection or some other means.
2. Research Design
These data will be collected through a combination of in-person, cognitive interviews conducted in the laboratory at the Bureau of Labor Statistics and a self-directed, online interview process using an online usability testing service.
In-Lab Testing
Cognitive interviews provide an in-depth understanding of a respondent’s thought processes and reactions to a question. The cognitive interviews for this study will be done one-on-one, in the Office of Survey Methods Research (OSMR) laboratory, and will be audiotaped. Interviews will be conducted by staff from the OSMR who are experienced in conducting these types of interviews. The testing protocol is included in Appendix A.
Web Testing
TryMyUI (www.TryMyUI.com) is an online usability testing company which allows participants to participate in usability studies from their homes. Once they have joined the TryMyUI participant group, they are screened based on various demographic characteristics and sent links to usability tests they qualify for. Each usability test is limited to twenty minutes and one URL. During the testing session, participants follow online instructions and use a think-aloud approach to complete the task described. A video showing the participant completing the task is then provided.
For this study, this usability testing site is being used as a way to collect a large amount of data in an efficient (both in terms of financial and staff resources) way. TryMyUI is responsible for recruitment, screening, administration of the survey, and data collection. The company has been found to be able to provide high quality, qualitative data within hours of receiving a request from a study sponsor. The same data would take weeks for OSMR to collect in the usability lab. Participants are used to providing qualitative feedback as they work on online tasks, and have been found to make very insightful, in-depth comments about their response process as they work on the self-administered survey. This allows for both quantitative and qualitative data to be collected in a very time efficient manner.
In both testing groups, participants will be asked to report their overall expenditure for three global expenditure categories (i.e., furniture, computer-related expenses, and household linens) for the past 3 months. Participants will be asked to think aloud while coming up with their answer. They will then be asked to list items that they included in their answer in an open-ended way. They will then be shown a list of items (some of which BLS would consider part of the category, some of which would not be included) and be asked to select which items definitely belong to the category, then separately select which items definitely do not belong to the category (see Appendix C for item lists). Finally, participants will be asked several questions about how easy or difficult it was to report for a category and whether it would be easier for them to report for individual items.
TryMyUI participants will receive a link to an online survey (hosted by SurveyMonkey). The protocol will differ slightly from the in-lab testing due to limitations in the skip functions of the online questionnaire (see Appendix B). For most of the tasks, they will be instructed to simply answer the questions, typing or clicking their response as on a standard web survey. To take advantage of the video and talk aloud capabilities of TryMyUI, a few questions will have instructions for participants to describe their thought processes as they answered (retrospective) or answer (concurrent) the questions.
3. Participants
In-Lab Participants
Participants will be recruited from the OSMR participant database. During the recruiting process, we will screen potential respondents to find participants who have recently made particular purchases (see Appendix D for screening questions), because those experiences are the topic of the study. We will aim to balance participants across levels of education, income, and occupation, based on self-reported information that is stored in the OSMR participant database.
Web Participants
Participants are recruited by TryMyUI. TryMyUI qualifies candidate users based on their ability to stay on task and to clearly voice everything they, see, read, think, or do as they are interacting with a website. They are volunteers who respond to a recruitment ad on the TryMyUI website. Participants will be selected with varying levels of age, gender, and education. As with the lab participants, web participants will be screened to ensure they have purchased something from at least three of the expenditure categories being studied (screening questions shown in Appendix D).
4. Burden Hours
The number of participants is dependent upon the results and the point at which diminishing returns is reached and no further information is being gained from additional subjects. Given that, our goal is to obtain no more than 20 lab participants and 40 web participants. We anticipate that each lab session will last no longer than one hour for a total of 20 lab session burden hours. The TryMyUI interface limits interviews to 20 minutes, for a total of 14 TryMyUI burden hours. The grand total of burden hours requested for this study is 34.
In-lab participants will be compensated $40 for their time. In the TryMyUI framework, participants are paid $10 per completed survey. This is a fixed amount set by the company.
5. Data Confidentiality
In-Lab Participants
Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of a national Consumer Expenditure Interview Survey. Participants will be given a consent form to read and sign (see Appendix E). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.
Web Participants
Participants obtained through TryMyUI sign separate agreements with TryMyUI, and although BLS follows steps to ensure that their confidentiality is maintained, no pledge of confidentiality is provided to these participants.
The following notice will be placed on the first page of the survey: “Note: This survey is being administered by surveymonkey.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.”
Appendix A: In-Lab Testing Protocol
Hi! Thank you for coming in today.
I am …... [This is my colleague ( ) who will be taking notes for us today]
Have you participated in any of our studies before? (if yes, Which ones?)
Consent Form/Permission to audiotape
Explanation:
We are going to be working with some questions from the Consumer Expenditure Survey, an ongoing survey that provides information about the buying habits of American consumers. This information is used for a lot of things but its primary purpose is to help calculate inflation. Are you familiar with the inflation rate?
What we are going to do today is go through a few questions from the survey and get your reactions to them. The purpose of this session is to help us find out more about how people respond to these questions. Basically, we’re trying to find out what you think a question is asking and how hard it is to answer. We are not here to evaluate you, we are looking to improve the questions, so there are no wrong answers. All the information you give us will be kept completely confidential, and will be used to improve the survey questions.
Any questions before we begin?
Since the first of [the reference month], have you or any members of your household purchased, for you or your household or for someone outside of your household, any furniture?
(if Yes) What was the total cost of these items?
(if Yes) In what month did you purchase them?
Since the first of [the reference month], have you or any members of your household purchased or rented any computer hardware, software, accessories, and service?
(if Yes) What was the total cost of these items?
(if Yes) In what month did you purchase them?
Since the first of [the reference month], have you or any members of your household purchased any household linens or window coverings?
(if Yes) What was the total cost of these items?
(if Yes) In what month did you purchase them?
Furniture
(if Purchased) What items did you purchase? What was the cost of each of those items?
What comes to mind when I say “furniture?”
[Provide list of expenditure items] Which of these items do you think definitely belongs in the category “furniture?”
Which of the items do you think definitely does not belong in the category “furniture?”
Did looking at that list remind you of any other things you’ve purchased since the first of [the reference month]?
(if Yes) What else did you purchase?
(if Yes) What was the cost of those items?
(if Yes) Why didn’t you include these items in your earlier answer about furniture? [Forgot about purchase, Didn’t think item would be considered furniture, etc.]
Did you think this question referred to the rental of furniture? Why or why not?
Have you rented any furniture since the first of [the reference month]?
Was the question about how much furniture you’ve purchased since the first of [the reference month] easy or difficult to answer?
What made it [easy/difficult] to answer?
[Show current question item list] Would you rather be asked if you’ve purchased furniture, in general, or about these different types of furniture separately? Which would be easier for you to talk about? Why?
Computer hardware, software, and accessories
[if Purchased] What items did you purchase? What was the cost of those items?
What comes to mind when I say “computer hardware, software, accessories, and service?”
[Provide list of expenditure items] Which of these items do you think definitely belongs in the category “computer hardware, software, accessories, and service?”
Which of the items do you think definitely does not belong in the category “computer hardware, software, accessories, and service?”
Did looking at that list remind you of any other things you’ve purchased since the first of [the reference month]?
(if Yes) What else did you purchase?
(if Yes) What was the cost of those items?
(if Yes) Why didn’t you include these items in your earlier answer about computer hardware, software, accessories, and service? [Forgot about purchase, Didn’t think item would be considered part of this category, etc.]
Was the question about how much computer hardware, software, accessories, and service you’ve purchased since the first of [the reference month] easy or difficult to answer?
What made it [easy/difficult] to answer?
[Show current question item list] Would you rather be asked if you’ve purchased computer hardware, software, accessories, and service, in general, or about these different categories of computer items separately? Which would be easier for you to talk about? Why?
Household Linens and Window Coverings
[if Purchased] What items did you purchase? What was the cost of those items?
What comes to mind when I say “household linens or window coverings?”
[Provide list of expenditure items] Which of these items do you think definitely belongs in the category “household linens or window coverings?”
Which of the items do you think definitely does not belong in the category “household linens or window coverings?”
Did looking at that list remind you of any other things you’ve purchased since the first of [the reference month]?
(if Yes) What else did you purchase?
(if Yes) What was the cost of those items?
(if Yes) Why didn’t you include these items in your earlier answer about household linens or window coverings? [Forgot about purchase, Didn’t think item would be considered part of this category, etc.]
Was the question about how much household linens or window coverings you’ve purchased since the first of [the reference month] easy or difficult to answer?
What made it [easy/difficult] to answer?
[Show current question item list] Would you rather be asked if you’ve purchased household linens or window coverings, in general, or about these different categories of computer items separately? Which would be easier for you to talk about? Why?
Appendix B: Web Testing Protocol
Welcome!
Unlike some other TryMyUI studies, you will not be using a website to complete a task for this study. Instead you’ll be answering survey questions.
You’ll only be asked to talk aloud as you complete some of the sections, but feel free to do so at any point. The instructions will tell you when you need to talk aloud.
This voluntary study is being collected under OMB No. 1220-0141 and it is estimated it will take you no longer than 20 minutes to complete.
The Bureau of Labor Statistics is exploring ways to better measure how Americans spend their money. We will be asking you about recent purchases. Please read instructions for each question carefully, explaining your answer when instructed to.
Note: This survey is being administered by TryMyUI and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.
_________________________________________
Since the first of [the reference month], have you or any members of your household purchased, for you or your household or for someone outside of your household, any furniture?
(if Yes) What was the total cost of these items?
(if Yes) In what month did you purchase them?
(if Yes) What items did you purchase and what was the cost of each of those items?
Item Description |
Cost |
|
|
|
|
|
|
|
|
|
|
Can’t Remember Any Items
What comes to mind when you hear the word “furniture?” Please discuss your answer aloud.
Please indicate whether you think each item definitely belongs or definitely does not belong in the category “furniture?” [See Item Lists in Appendix C]
Did seeing the list in the previous question remind you of any other items you’ve purchased since the first of [the reference month]?
(if Yes) What else did you purchase and what was the cost of those items?
Item Description |
Cost |
|
|
|
|
|
|
|
|
|
|
No Additional Items
(if Yes) Why didn’t you include these items in your earlier answer about furniture Please discuss your answer aloud
Was the question about how much furniture you’ve purchased since the first of [the reference month] easy or difficult to answer? What made it [easy/difficult] to answer? Please discuss your answer aloud
Since the first of [the reference month], have you or any members of your household purchased or rented any computer hardware, software, accessories, and service?
(if Yes) What was the total cost of these items?
(if Yes) In what month did you purchase them?
(if Yes) What items did you purchase and what was the cost of each of those items?
Item Description |
Cost |
|
|
|
|
|
|
|
|
|
|
Can’t Remember Any Items
What comes to mind when you hear the words “computer hardware, software, accessories, and service?” Please discuss your answer aloud
Please indicate whether you think each item definitely belongs or definitely does not belong in the category “computer hardware, software, accessories, and service?” [See Item Lists in Appendix C]
Did seeing the list in the previous question remind you of any other items you’ve purchased since the first of [the reference month]?
(if Yes) What else did you purchase and what was the cost of those items?
Item Description |
Cost |
|
|
|
|
|
|
|
|
|
|
No Additional Items
(if Yes) Why didn’t you include these items in your earlier answer about computer hardware, software, accessories, and service? Please discuss your answer aloud
Was the question about how much computer hardware, software, accessories, and service you’ve purchased since the first of [the reference month] easy or difficult to answer? What made it [easy/difficult] to answer? Please discuss your answer aloud
Since the first of [the reference month], have you or any members of your household purchased any household linens or window coverings?
(if Yes) What was the total cost of these items?
(if Yes) In what month did you purchase them?
(if Yes) What items did you purchase and what was the cost of each of those items?
Item Description |
Cost |
|
|
|
|
|
|
|
|
|
|
Can’t Remember Any Items
What comes to mind when you hear the words “household linens or window coverings?” Please discuss your answer aloud
Please indicate whether you think each item definitely belongs or definitely does not belong in the category “household linens or window coverings?” [See Item Lists in Appendix C]
Did seeing the list in the previous question remind you of any other items you’ve purchased since the first of [the reference month]?
(if Yes) What else did you purchase and what was the cost of those items?
Item Description |
Cost |
|
|
|
|
|
|
|
|
|
|
No Additional Items
(if Yes) Why didn’t you include these items in your earlier answer about household linens or window coverings? Please discuss your answer aloud
Was the question about how much household linens or window coverings you’ve purchased since the first of [the reference month] easy or difficult to answer? What made it [easy/difficult] to answer? Please discuss your answer aloud
Appendix C: Item Lists
NOTE: Items with grey backgrounds are “lures” or do not belong in the global category. The background color will be removed for testing.
Furniture |
||||
|
|
|
|
|
Sofa |
Stove |
Sectional |
Office Chair |
Rocking Chair |
|
|
|
|
|
Bean Bag Chair |
End Table |
TV Stand |
Lamp |
Shoe Rack |
|
|
|
|
|
Mirror |
Coat Rack |
Television |
Ottoman |
Dinette Set |
|
|
|
|
|
Bookcase |
China Cabinet |
Dining Table |
Dishwasher |
Box Spring |
|
|
|
|
|
Headboard |
Mattress |
Bunk bed |
Bed Frame |
Crib |
|
|
|
|
|
Coffee Maker |
High Chair |
Patio Umbrella |
Deck Chairs |
File Cabinet |
|
|
|
|
|
Loveseat |
Room Divider |
Vase |
Pool Table |
Clock |
|
|
|
|
|
Refrigerator |
Recliner |
Drafting Table |
Gun Cabinet |
Garbage Disposal |
|
|
|
|
|
Play Pen |
Desk |
Entertainment Center |
Dresser |
Card Table |
|
|
|
|
|
Furniture |
||||
|
|
|
|
|
Sofas |
Living Room Chairs |
Living Room Tables |
Wall Units, Shelves Or Cabinets |
Infant’s Furniture |
|
|
|
|
|
Other Living Room Furniture |
Living Room Furniture Combinations |
Dining Room Or Kitchen Furniture |
Mattresses Or Box Springs |
Bedroom Furniture Other Than Mattresses Or Box Springs |
|
|
|
|
|
Patio, Porch, Or Outdoor Furniture |
Office Furniture For Home Use |
|
|
|
\ |
|
|
|
|
Computer Hardware, Software, Accessories, and Service |
||||
|
|
|
|
|
Mouse Pad |
Home Computer |
Sony eReader |
Printer Cartridge |
Computer Games |
|
|
|
|
|
Answering Machine |
Modem |
Computer Installation |
Digital Camera |
iPad |
|
|
|
|
|
Laptop |
Mouse |
GPS Device |
Memory (RAM) |
Typewriter |
|
|
|
|
|
External Hard Drive |
Cell Phone Charger |
Monitor |
Thumb Drive |
Blank DVD |
|
|
|
|
|
USB Flash Drive |
Computer Repair |
Scanner |
CD/DVD Drive |
iPhone |
|
|
|
|
|
Nook |
Printer |
Playstation Console |
Kindle |
Netbook |
|
|
|
|
|
Computer Hardware, Software, Accessories, and Service |
||||
|
|
|
|
|
Digital Book Readers |
Computers, Computer Systems, or Related Hardware |
Computer Software, Including Games |
Portable Memory |
Repair and Service Contract for Computer, Computer Systems, and Related Equipment |
|
|
|
|
|
Household Linens and Window Coverings |
||||
|
|
|
|
|
Sheets |
Shirt |
Furniture Protector |
Shower Curtain |
Pajamas |
|
|
|
|
|
Pillow |
Mattress Pad |
Cloth Diaper |
Bedspread |
Quilt |
|
|
|
|
|
Face Cloth |
Throw Rug |
Bath Mat |
Toilet Cover |
Scarf |
|
|
|
|
|
Undergarments |
Electric Blanket |
Cloth Napkins |
Placemats |
Air Conditioner |
|
|
|
|
|
Pillowcase |
Chair Pad |
Towels |
Washing Machine |
Decorative Pillow |
|
|
|
|
|
Blinds |
Shades |
Curtains |
Drapes |
Baby Blanket |
|
|
|
|
|
Duvet |
Slip Cover |
Comforter |
Doilies |
Sofa |
|
|
|
|
|
Sweater |
Blanket |
Vest |
Tablecloth |
Dish Towel |
|
|
|
|
|
Household Linens and Window Coverings |
||||
|
|
|
|
|
Bedroom Linens |
Bathroom Linens |
Kitchen or Dining Room Linens |
Other Linens |
Slipcovers, Decorative Pillows, or Cushions |
|
|
|
|
|
Room-size rugs or other Non-Permanent Floor Coverings |
Curtains or Drapes |
Blinds, Shades, or Other Window Coverings |
|
|
|
|
|
|
|
Appendix D: Screening Question
Must say “Yes” to the following question to participate.
Have you purchased any of the following items in the last three months? YES NO
Furniture
Computer Hardware, Software, Accessories, or Services
Household Linens
Appendix E: Consent Form
The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.
The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.
During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.
We estimate it will take you no longer than an hour to participate in this research (ranging from 45 minutes to 60 minutes).
Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.
Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires 2-28-2015.
------------------------------------------------------------------------------------------------------------
I have read and understand the statements above. I consent to participate in this study.
___________________________________ ___________________________
Participant's signature Date
___________________________________
Participant's printed name
___________________________________
Researcher's signature
OMB Control Number: 1220-0141
Expiration Date: 02-28-2015
In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.
| File Type | application/msword |
| File Title | October 14, 2003 |
| Author | Yu, Erica - BLS |
| Last Modified By | kincaid_n |
| File Modified | 2013-07-09 |
| File Created | 2013-07-09 |