May 3, 2012
NOTE TO THE REVIEWER OF: |
OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
|
FROM: |
Jennifer Edgar Research Psychologist Office of Survey Methods Research
|
SUBJECT: |
Submission of Materials for the CE Recall Cognitive Study |
Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.
The total estimated respondent burden hours for the study is a maximum of 40.
If there are any questions regarding this project, please contact Jennifer Edgar at
202-691-7528.
Introduction and Purpose
Proxy Reporting
CE has long identified proxy reporting, having a single respondent report expenditures for the whole household, as a source of underreporting. There are two likely reasons for these omissions: knowledge and recall. Lack of knowledge, the fact that respondents simply do not know what other household members have purchased or what the cost was, cannot be addressed with survey questions. The second reason, that respondents may forget to consider other household members, can be addressed through survey questions. This study seeks to explore whether identifying expenditures, and expenditure categories, relevant to other household members at the beginning of the CEQ can improve proxy reporting.
Reference Period
The CEQ currently uses a three month reference period for all expenditure categories. One way to reduce the length of the survey without compromising the expenditure estimates is to lengthen the reference period for some expenditure categories where respondents would be able to recall the expenses for a longer period of time. For example, the CEQ currently asks about new vehicle purchases every quarter, but a respondent probably can recall this information for a longer period. This study seeks to explore the use of longer reference periods for select expenditure categories. Participant ability to recall the occurrence of an expense and the details will be identified for multiple reference periods longer than three months.
2. Research Design
To address the research questions, two methods will be used: (1) an interviewer-administered cognitive interview, and (2) a self-administered web survey. All participants will complete a background form to collect demographic information (Attachment A).
Interviewer administered
Participants will complete a standard cognitive interview in the OSMR lab, using the protocol included as Attachment B.
Web Survey
A separate group of participants will complete a web survey (Attachment C) through the TryMyUI service. Three groups will be randomly assigned, each receiving one reference period condition. The current protocol has not been pretested, and the length may be reduced if it is found to exceed 20 minutes.
TryMyUI
TryMyUI (www.TryMyUI.com) is an online usability testing company which allows participants to participate in usability studies from their homes. Once they have joined the TryMyUI participant group, they are screened based on various demographic characteristics and sent links to usability tests they qualify for. Each usability test is limited to twenty minutes and one URL. During the testing session, participants follow online instructions and use a think-aloud approach to complete the task described. A video showing the participant completing the task is then provided.
For this study, this usability testing site is being used as a way to collect a large amount of data in an efficient (both in terms of financial and staff resources) way. TryMYUI is responsible for recruitment, screening, administration of the survey, and data collection. The company has been found to be able to provide data within hours of receiving the study from the sponsor. The same data would take weeks for OSMR to collect in the usability lab.
Participants will receive a link to an online survey (hosted by SurveyMonkey) as noted in Attachment C. For most of the tasks, they will be instructed to simply answer the questions, typing or clicking their response as on a standard web survey. To take advantage of the video and talk aloud capabilities of TryMYUI, a few questions will have instructions for participants to describe their thought processes as they answered (retrospective) or answer (concurrent) the questions.
3. Participants
Interviewer-administered Interview
Participants will be recruited from the OSMR participant database. Efforts will be made to select participants with varying levels of age, gender, and education based on self-reported information provided during the initial recruitment process. All participants will be screened to ensure that they have a multi-person household and have had some expenditures in a majority of the categories of interest. Representations of different household members (e.g., wife, husband, parents of teenage kids, head of a large household, member of an extended family household) will be attempted.
Web Survey
Participants are recruited by TryMyUI. TryMyUI qualifies candidate users based on their ability to stay on task and to clearly voice everything they, see, read, think, or do as they are interacting with a website. They are volunteers who respond to a recruitment ad on their website. Participants will be selected with varying levels of age, gender, and education. Only participants from multi-person households (with 2, 3 or more than 4 people each) who have a major appliance, furniture, or a vehicle purchase in the last year and had an expense for a subscription for a newspaper or magazine or book in the last three months will be eligible for the study (See Attachment C).
4. Payment
Interviewer-administered Interview
Participants will receive $40.
Self-administered Survey
In the TryMyUI framework, participants are paid $10 per completed survey. This is a fixed amount set by the company.
5. Burden Hours
The self-administered survey is limited to twenty minutes, with up to 60 participants completing it. The interviewer-administered interview is expected to take up to 60 minutes, with up to 20 participants completing it, for a maximum of 40 burden hours.
5. Data Confidentiality
Interviewer-administered Interview
Participants will be informed as to the voluntary nature of the study, and that the study will be used for internal purposes to improve the design of a national consumer expenditure interview survey. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice. OMB Control Number: 1220-0141. Prior to the interview, participants will be given a consent form to read and sign (Attachment D).
Self-administered Survey
Participants obtained through Trymyui sign separate agreements with Trymyui, and although BLS follows steps to ensure that their confidentiality is maintained, no pledge of confidentiality is provided to these respondents.
The following notice will be placed on the first page of the survey: “Note: This survey is being administered by surveymonkey.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.”
Attachment A: Background Questions
How many people live in your household? ___________
What is your relationship to each of them? (example, sister, father, aunt, friend)
_______________
_______________
_______________
_______________
_______________
_______________
_______________
What is the highest level of education you’ve completed?
Some high school
High school diploma
Some college
College degree
Advanced degree
What is your household income?
Less than or equal to $40,000
$40,001 to $60,000
$60,001 to $75,000
More than $75,000
PID___________
Attachment B: Interviewer administered Cognitive Interview Protocol
Introduction
Hi! Thank you for coming in today! I am …...
We are going to be working with some questions from one of the BLS surveys which asks questions about how households spend their money.
What we are going to do today is go through some questions from the survey to get your feedback to them.
The purpose of today’s session is to help us find out more about how people, like you, respond to these questions.
We are not here to evaluate you, we are looking to improve the questions, so there are no right or wrong answers.
We are actually more interested in how easy the questions are to understand more than your actual answers.
All the information you give us will be kept completely confidential, and will be used to improve the survey questions.
Any questions?
What I’m going to do is go through some questions and get your answers. Then we will talk about what you thought about when you answered the question, how easy or difficult it was to answer, and your reactions to the questions.
Before we get started, I have three things for you to complete.
The first is a consent form. Take a minute to read it over, and let me know if you have any questions
The second is this form, which is basically a receipt saying you received your check.
Have participant sign consent form and check voucher
Finally, this is a form which asks some background questions. This information, like everything you say, will be kept confidential, just used to help us better understand your feedback.
Have participant complete background form
Let me know at any point if you have a question or comment, or if you need to take a break.
Finally, before we get started I’d like your permission to audio tape our interview. The recording will be used to supplement the notes I’m going to take, and will be kept confidential.
Proxy Reporting Questions (to be asked after household roster and before first expenditure section)
You said that [Read roster names] all live in your household. I’d like to spend a few minutes talking about ways they spend their money. This will help us move faster later in the interview, and help you remember things that you might otherwise forget since it’s not an expense you had yourself.
To start, I want to ask about expenses that the other people in your household have that you may not. For example, one person might buy all the clothing for the family, while another handles all the vehicle expenses.
Are there some types of things that [name1] spends money on that you don’t?
Are there some types of things that [name2] spends money on that you don’t?
Are there some types of things that [name3…] spends money on that you don’t?
Now I’d like to ask some other questions about people in your household. Again, this information will help us move through the survey faster and I’ll use your answers later in the interview to remind you to include these types of things as you answer.
Do other people in your household have any hobbies or activities they like to do?
Have other people in your household taken any vacations or trips without you last year?
Has anyone in your household bought anything unusual or out of the ordinary in the past three months?
For each question, the expenditure category will be probed enough to identify the relevant CEQ subsection for the expense, and the name of the household member, but detailed expenditure information will not be collected
For each CEQ subsection identified, a flag will be placed in that section. The participant will be prompted to include other household members purchases in that section
The CEQ will be asked in the current production order.
Proxy Debriefing 1
What did you think of those questions?
Do you think you were able to answer them accurately?
How well do you know how other people in your household spend their money?
Do you know the types of things they spend their money on generally?
Do you know the amounts of money they spend, generally?
Do you know exactly?
Did you find those questions too personal?
Do you think other people would find them too personal?
Proxy Flag for CEQ Questions
Proxy Probe [to be used in each expenditure subsection identified during the initial proxy questions]. It will follow the section transition (e.g. “Now I am going to ask about….”).
When we started, you said that [name1] might have had some of these expenses. Please keep them in mind as you answer the questions.
Each identified CEQ subsection will be asked about following the probe
If one of the reference period sections (6a, 8a,, or 17) was identified, the probe will be added to the beginning of that section, and the experimental reference period will be used
If some or none of the reference period sections were identified, the identified sections will be asked in conjunction with the reference period sections, following the order of the standard CEQ,
CEQ interviewing will continue for 40 minutes. At that time, the CEQ portion of the study will end and the debriefing will be conducted.
Recall Period CEQ Questions
The following sections of CEQ will be asked using the specified reference period.
The full CEQ subsection will be asked with the changed reference period. All follow up questions for each expenditure report will be asked.
CEQ interviewing will continue for 40 minutes. At that time, the CEQ portion of the study will end and the debriefing will be conducted.
Section |
Expenditure |
Reference Period |
6A: Major Appliances |
Microwave, stove, range hood, fridge, dishwashers, garbage disposal, clothes washer, clothes dryer |
12 months |
8A: Home furnishings |
Furniture (not decorative items, closet storage items, travel items or dishes) |
6 months |
S11 Owned vehicles |
Purchased or acquired vehicles only |
12 months |
S17A. Subscriptions or memberships |
All |
6 months |
S17B. Books and entertainment |
All |
3 months (same as current, focus on switch within section) |
Second Proxy Reporting Debriefing
Ok, now I’d like to go back and talk about the questions you just answered. I’m interested in hearing your reactions to them, as well as hearing what you thought about as you gave your answers.
When we started, I asked a series of questions about other people in your household. The questions asked about their hobbies, trips, unusual expenses or ways they spend money that you typically don’t.
Based on your answers to those questions, when we got to the sections about [insert expenditure section with proxy probes], I reminded you about purchases made by other members in your household.
If I had not instructed you to remember their purchases, do you think you would have?
Did you include expenses from all other household members?
Do you think you included all the expenses they had?
I did not give that instruction for [expenditure category asked without proxy probe, if possible clothing or phones]. I just said [section transition] Let’s go back and review those questions.
I asked about this list of items show info book page. Did anyone else in your household have any of these expenses?
Did you include those expenses in your answers?
If yes: did you include expenses from all household members?
If yes: do you think you included all expenses they had?
If no: why did you not include other household member’s expenses?
We’re interested in helping people tell us as much as they can about expenses for all the members of their household. Do you have any suggestions for how we could have helped you report everyone’s expenses more accurately?
Reference Period Debriefing
I want to change focus now, and talk about the other questions I asked about. I’ll go through the categories one at a time and get your feedback.
Section |
Reference Period |
6A: Major Appliances |
12 months |
8A: Home furnishings |
6 months |
S11 Owned vehicles |
12 months |
S17A. Subscriptions or memberships |
6 months |
S17B. Books and entertainment |
3 months |
6A. Major Appliances
When is the last time you purchased one of the major appliances listed here [show info book page]?
And before that, when did you last buy a major appliance?
How often have you bought an appliance in the last six months? In the last year?
How easy or difficult was it to remember when you last purchased an appliance?
How easy or difficult was it for you to remember the details of that purchase, such as the cost and date?
How far back are you able to easily remember that you bought an appliance?
How far back are you able to easily remember the details of those purchases?
8A. Home furnishings
When is the last time you purchased furniture listed here [show info book page]?
And before that, when did you last buy furniture?
How often have you bought furniture in the last six months? Last year?
How easy or difficult was it to remember when you last purchased furniture?
How easy or difficult was it for you to remember the details of that purchase, such as the cost and date?
How far back are you able to easily remember that you bought furniture?
How far back are you able to easily remember the details of those purchases?
S11. Owned Vehicles
When is the last time you purchased a new or used vehicle?
And before that, when did you last buy a vehicle?
How often have you bought vehicle in the last three years? Last six years?
How easy or difficult was it to remember when you last purchased a vehicle?
How easy or difficult was it for you to remember the details of that purchase, such as the cost and date?
How far back are you able to easily remember that you bought a vehicle?
How far back are you able to easily remember the details of that vehicle purchases?
S17A. Subscriptions or memberships
When is the last time you purchased or renewed a subscription or membership listed here [show info book page]?
And before that, when did you have expenses for something listed here?
How often have you had an expense for one of these things in the last six months? Last year?
How easy or difficult was it to remember when you last purchased a subscription or membership?
How easy or difficult was it for you to remember the details of that purchase, such as the cost and date?
How far back are you able to easily remember that you bought a subscription or membership?
How far back are you able to easily remember the details of that subscription or membership purchase?
S17B. Books and entertainment
When is the last time you had expenses for any of the books or entertainment items listed here [show info book page]?
And before that, when did you have expenses for something listed here?
How often have you had an expense for one of these things in the last six months? Last year?
How easy or difficult to was it to remember when you last purchased something from this list?
How easy or difficult was it for you to remember the details of that purchase, such as the cost and date?
How far back are you able to easily remember that you bought something from this list?
How far back are you able to easily remember the details of that purchase?
Overall debriefing
Here’s a table showing the types of expenditures we just talked about. Please enter the time frame (e.g. one day, one year, five years) that you think you would be able to accurately remember purchasing that type of item, as well as the cost and month of the purchase.
Expense Type |
How long you think you can accurately remember |
6A: Major Appliances |
|
8A: Home furnishings |
|
S11 Owned vehicles |
|
S17A. Subscriptions or memberships |
|
S17B. Books and entertainment |
|
Do you have any final comments or suggestions as we work to improve our survey?
Attachment C: Self-administered Web Survey
Pilot test will be used to test timing, sections may be reduced if current plan takes more than 20 minutes
Order of sections will not be randomized, following current CEQ order
Screening Questions (to appear on TryMyUI screening screen to determine participant eligibility)
How many people current live in your household? ___
In the past year, have you purchased a major appliance (such as a refrigerator, stove or washing machine)? Yes No
In the past year have you purchased any furniture (such as a couch or table?) Yes No
In the past year have you purchased a vehicle? Yes No
In the past three months have you purchased or renewed any newspaper or magazine subscriptions? Yes No
In the past three months have you purchased any books? Yes No
Welcome!
Unlike some other TryMyUI studies, you will not be using a website to complete a task for this study. Instead you’ll be answering survey questions.
You’ll only be asked to talk aloud as you complete some of the sections, but feel free to do so at any point. The instructions will tell you when you need to talk aloud.
This voluntary study is being collected under OMB No. 1220-0141 and it is estimated it will take you no longer than 20 minutes to complete.
The Bureau of Labor Statistics is exploring ways to better measure how Americans spend their money. We will be asking you about recent purchases. Please read instructions for each question carefully, explaining your answer when instructed to.
Note: This survey is being administered by surveymonkey.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.”
Recall Period CEQ Questions
Current CEQ questions will be asked for all noted expenditures/sections
The reference period group (A, B or C) will be randomly assigned to each participant
Section |
Expenditure |
Reference Period A |
Reference Period B |
Reference Period C |
6A: Major Appliances |
Microwave, stove, range hood, fridge, dishwashers, garbage disposal, clothes washer, clothes dryer |
12 months |
6 months |
3 months |
8A: Home furnishings |
Furniture (not decorative items, closet storage items, travel items or dishes) |
12 months |
6 months |
3 months |
S11 Owned vehicles |
Purchased or acquired vehicles only |
12 months |
6 months |
3 months |
S17A. Subscriptions or memberships |
All |
6 months |
1 month |
3 months |
S17B. Books and entertainment |
All |
2 months |
1 month |
3 months |
Follow Up Questions
Now please provide some feedback about the questions you just answered. You’ll go through the categories one at a time and provide your feedback.
6A. Major Appliances
How many appliances from this list have you bought in the last [assigned reference period]? List from info book page will be shown on page
Please list all the appliances you bought in the last [assigned reference period] and how much each cost
Please rate how confident you are that the price is accurate (very, somewhat, not very, not at all)
Appliance Name |
Price |
Confidence that price is accurate |
|
|
(try to have likert scale) |
|
|
|
|
|
|
|
|
|
How confident are you that you listed every appliance you bought in the last [assigned reference period]?
Very confident
Somewhat confident
Not very confident
Not at all confident
We are trying to determine how far in the past people can remember details about a major appliance purchase such as the brand name, price, and the date of purchase. Think about the appliance you purchased the longest ago that you think you can remember accurately. List the:
Appliance name:
Appliance cost:
Month of purchase:
How confident are the cost and month of that appliance purchase is accurate?
Very confident
Somewhat confident
Not very confident
Not at all confident
8A. Home furnishings
How many pieces of furniture from this list have you bought in the last [assigned reference period]? List from info book
Please list all the furniture you bought in the last [assigned reference period]and how much each cost
Please rate how confident you are that the price is accurate (very, somewhat, not very, not at all)
Furniture Description |
Price |
Confidence that price is accurate |
|
|
(try to have likert scale) |
|
|
|
|
|
|
|
|
|
How confident are you that you listed all the furniture you bought in the last[assigned reference period]?
Very confident
Somewhat confident
Not very confident
Not at all confident
We are trying to determine how far in the past people can remember details about a furniture purchase such as the brand name, price, and the date of purchase. Think about the furniture you purchased the longest ago that you think you can remember accurately. List the:
Furniture type:
Furniture cost:
Month of purchase:
How confident are the month of that furniture purchase is accurate?
Very confident
Somewhat confident
Not very confident
Not at all confident
How confident are the cost of that furniture purchase is accurate?
Very confident
Somewhat confident
Not very confident
Not at all confident
S11. Owned Vehicles
How many vehicles from this list have you bought in the last[assigned reference period]? List from info book
Please list all the vehicles you bought in the last [assigned reference period]and how much each cost
Please rate how confident you are that the price is accurate (very, somewhat, not very, not at all)
Vehicle Description |
Price |
Confidence that price is accurate |
|
|
(try to have likert scale) |
|
|
|
|
|
|
|
|
|
How confident are you that you listed all the vehicles you bought in the last [assigned reference period]?
Very confident
Somewhat confident
Not very confident
Not at all confident
We are trying to determine how far in the past people can remember details about a vehicle purchase such as the brand name, price, and the date of purchase. Think about the vehicle you purchased the longest ago that you think you can remember accurately. List the:
Vehicle description:
Vehicle cost:
Month of purchase:
How confident are the month of that vehicle purchase is accurate?
Very confident
Somewhat confident
Not very confident
Not at all confident
How confident are the cost of that vehicle purchase is accurate?
Very confident
Somewhat confident
Not very confident
Not at all confident
S17A. Subscriptions or memberships
How many subscription or memberships from this list have you purchased or renewed in the last [assigned reference period]? List from info book
Please list all the subscriptions or memberships you purchased or renewed in the last [assigned reference period]and how much each cost
Please rate how confident you are that the price is accurate (very, somewhat, not very, not at all)
Subscription or Membership Description |
Price |
Confidence that price is accurate |
|
|
(try to have likert scale) |
|
|
|
|
|
|
|
|
|
How confident are you that you listed all the subscription or memberships you purchased or renewed in the last [assigned reference period]?
Very confident
Somewhat confident
Not very confident
Not at all confident
We are trying to determine how far in the past people can remember details about a subscription and membership purchases such as the brand name, price, and the date of purchase. Think about the oldest subscription or membership expense that you think you can remember accurately. List the:
Subscription or membership description:
Subscription or memberships cost:
Month of purchase:
How confident are the month of that subscription or membership purchase is accurate?
Very confident
Somewhat confident
Not very confident
Not at all confident
How confident are the month of that subscription or membership purchase is accurate?
Very confident
Somewhat confident
Not very confident
Not at all confident
S17B. Books and entertainment
How many books or entertainment items from this list have you bought in the last[assigned reference period]? List from info book
Please list all the books or entertainment items you bought in the last [assigned reference period]and how much each cost
Please rate how confident you are that the price is accurate (very, somewhat, not very, not at all)
Book or Entertainment Item Description |
Price |
Confidence that price is accurate |
|
|
(try to have likert scale) |
|
|
|
|
|
|
|
|
|
How confident are you that you listed all the book or entertainment item you bought in the last [assigned reference period]?
Very confident
Somewhat confident
Not very confident
Not at all confident
We are trying to determine how far in the past people can remember details about a book and entertainment purchases such as the brand name, price, and the date of purchase. Think about the oldest book or entertainment expense that you think you can remember accurately. List the:
Vehicle description:
Vehicle cost:
Month of purchase:
How confident are the month of that book or entertainment purchase is accurate?
Very confident
Somewhat confident
Not very confident
Not at all confident
How confident are the cost of that book or entertainment purchase is accurate?
Very confident
Somewhat confident
Not very confident
Not at all confident
Overall debriefing
Here’s a table showing the types of expenditures we just talked about. Please enter the time frame (for example: one week, one month, one year, five years, etc. ) hat you think you would be able to accurately remember the details (type of item, cost, and month) about purchasing that item.
Expense Type |
How long you think you can accurately remember |
6A: Major Appliances |
|
8A: Home furnishings |
|
S11 Owned vehicles |
|
S17A. Subscriptions or memberships |
|
S17B. Books and entertainment |
|
Here’s a table showing the time frame we used when we asked you about each type of purchase. What do you think about those time periods?
Do you think that we could make any of them longer and you’d still be able to answer the questions?
Do you think any of them should be made shorter to help you remember the purchases?
Expense Type |
How long you think you can accurately remember |
6A: Major Appliances |
Assigned reference period |
8A: Home furnishings |
Assigned reference period |
S11 Owned vehicles |
Assigned reference period |
S17A. Subscriptions or memberships |
Assigned reference period |
S17B. Books and entertainment |
Assigned reference period |
Attachment D: Consent Form
The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.
The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.
During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.
We estimate it will take you an average of 60 minutes to participate in this research (ranging from 45 minutes to 70 minutes).
Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.
Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2015.
------------------------------------------------------------------------------------------------------------
I have read and understand the statements above. I consent to participate in this study.
___________________________________ ___________________________
Participant's signature Date
___________________________________
Participant's printed name
___________________________________
Researcher's signature
OMB Control Number: 1220-0141
Expiration Date: 02-28-2015
In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | October 14, 2003 |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |