July 1, 2013
NOTE TO THE REVIEWER OF: |
OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
|
FROM: |
Erica Yu Research Psychologist Office of Survey Methods Research
|
SUBJECT: |
Submission of materials for the CE Proxy Reporting Follow-up Study |
Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.
The total estimated respondent burden for the study is 30 hours.
If there are any questions regarding this project, please contact Erica Yu at 202-691-7924.
Introduction and Purpose
Surveys often must rely on proxy reporting, having a single respondent provide responses on behalf of others, when it is not possible to interview all targeted respondents. The Consumer Expenditure Survey interview (CEQ) asks individuals to report on behalf of household members. However, there is strong evidence from the methodology literature that proxy-reports are consistently different from self-reports due, in the case of expenditure reporting, to the differential knowledge of own and others’ expenditures and also possibly due to the differential recall and judgment for others’ expenditures.
A previous OSMR study (CE Recall Cognitive Study, OMB submission date: May 3, 2012) sought to address the challenge of differential recall for others’ expenditures by investigating the impact of prompts to encourage respondents’ recall of other household members’ expenditures. The author reported that the probes and prompts tested led to an increase in proxy-related expenditures (median = $63), with 10 out of 18 participants adding at least one expenditure report. Such results are particularly notable given respondents’ initial CEQ answers may have already benefited from the cues provided during the initial general probing at the start of the interview (e.g., ‘are there some types of things that [name1] spends money on that you don’t?’). Given the low time and burden cost of the additional questions (median time spent = 2m 15s; 13 out of 17 participants reporting the questions were not sensitive at all), a follow-up study to confirm the initial results is warranted. A result of additional expenditure items across participants and across CEQ sections due to proxy probes would provide strong evidence in support of testing the protocol in the field.
Open-ended questions of the nature of these proxy-reporting probes are subject to possible reductions in data quality due to interviewer behavior. Research suggests that the way interviewers use probes can have a significant impact on the number of responses a respondent gives to an open-ended question (Groves & Magilavy, 1986) and that the amount of time allowed to access events and the amount of time used to formulate responses can impact what type of strategy is used and ultimately the type of information that is retrieved. (Reiser, Black, & Ableson, 1985; Blair & Burton, 1987; Burton & Blair, 1991). Longer response times are often associated with more accurate recalls while shorter response times are associated with inferior response strategies such as estimation. This procedural difference may affect the quality of responses given to the proposed proxy-reporting probes. Although some progress on this general question could be made by analyzing recordings of field interviews to find the relationship between number of items reported and amount of time spent, an experimental manipulation is necessary to understand the causes of any such relationship. Question pacing will be manipulated experimentally to assess the impact of minimum response interval length, with half of the participants experiencing a short minimum interval (2s) and half receiving a longer interval (10s).1 The short minimum interval length is intended to approximate the speed of normal conversational pace. A result of increased reporting of items in the longer minimum interval length condition would suggest that quick interview pace lowers the effectiveness of the proxy probes. However, an analysis of burden as assessed by participant ratings of the experience of listening to questions and responding to answers and perceived length will be critical for understanding whether implementing a minimum retrieval time is feasible while managing respondent burden.
The goals of the present study are twofold: (1) to refine proxy-reporting question wording to ensure clarity and maximize the number of expenditures reported and (2) to test interviewer question pacing to minimize the impact on burden due to additional questions for both the interviewer and the respondent.
Research Design
Proxy Study
The protocol and procedures for the present study are based on the 2012 CE Recall Cognitive Study. The study will follow the format of a standard cognitive interview in the OSMR lab, beginning with the introduction to the study purpose and task (see Appendix A). Participants will provide informed consent and complete background questions including a household roster (see Appendix B). The study will begin with the initial proxy questions (see Appendix C; Proxy Reporting Questions). Immediately after completing the Proxy Reporting Questions, the interviewer will ask the Proxy Debriefing questions (see Appendix C). After completion of the debriefing, the interviewer will administer three sections of the CEQ as per standard production directions. After completion of the CEQ, the interviewer will administer the Overall Debriefing questions (see Appendix C).
Study protocol is based on the previous OSMR proxy-reporting study with one major change: the CEQ sections used in the study will be tailored for each individual to align with the expenditure categories reported in response to the proxy questions. Where possible, two relevant CEQ sections will be chosen for use with proxy probes and a third CEQ section will be chosen for use without proxy probes for comparison purposes. Given that the entire CEQ cannot be administered during the lab study due to time constraints, this personalization will ensure that the expenditure questions that are asked are relevant to each participant and lab time and respondent burden are used effectively. Additionally, there have been minor modifications to the content and order of the proxy probe questions asked, based on participant and interviewer feedback from the 2012 study.
Pacing Study
A minimum response time interval will be imposed on the proxy-reporting questions asked at the start of the study (see Appendix C, Proxy Reporting Questions); there will be no minimum response time interval length imposed on the CEQ questions. Participants will be randomly assigned to receive a minimum response time interval that is short (2s) or long (10s), to be measured from the end of each question until the start of the next question. Interviewers will not provide any additional probing during the interval; the minimum length is only implemented if a respondent stops talking, in which case it is used to extend the amount of time available for the respondent to continue formulating his or her response.
Participants
Participants will be recruited from the OSMR participant database. Efforts will be made to select participants with varying levels of age, gender, and education based on self-reported information collected during an initial recruitment process and stored in the database. Individuals will be screened to ensure that he or she lives in a multi-person household with shared expenses; individuals who share living quarters with roommates but do not share expenses will not be selected because they would not be proxy-reporting expenses on behalf of others. Additionally, individuals will be screened to ensure that the household has had expenditures in the major expenditure categories and reference period of interest (see Appendix D). Representations of the different household member categories (e.g., wife, husband, parents of teenage kids, head of large household, member of an extended family household) will be attempted.
Burden Hours and Payment
Up to 30 participants will be recruited for interviews estimated to take up to 60 minutes each. This will result in a maximum of 30 burden hours. Each participant will receive compensation of $40.
Data Confidentiality
Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of a national consumer expenditure survey only. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice. Prior to the interview, participants will be given a consent form to read and sign (see Appendix E).
Appendix A: Introductory Protocol
Hi, thank you for coming in today. I am … and this is … who will be sitting in with us.
Have
you ever participated in one of our studies or any study like this
before?
… If you have any questions you can always feel
free to ask me at any time.
We’re going to ask you to do two tasks today.
The
first is to answer a few questions from one of our national surveys
on household expenditures.
But we’re not really
interested in your answers to those survey questions today.
We’re
interested in what you think about the questions, what you think the
questions were asking you, and how you go about answering them.
So
in the second part of our time today, we’re going to ask you to
tell us what it was like to answer the survey questions.
There
are no right or wrong answers. We’re here to get your feedback
on the questions so we’re actually more interested in how easy
the questions are to understand, than the information about your
spending.
Any information you give us will remain confidential
and used only to improve the survey questions.
Any questions so far?
Before we get started, I have three things for you to complete:
The
first is a consent form. Take a minute to read it over and let me
know if you have questions.
Have participant sign consent
form (Appendix E).
The
second is this receipt that confirms that you received your
check.
Have participant sign check voucher.
The
third is some background we’ll need before we can get started.
The survey questions we’ll be talking about today are
about how US households spend money, so we’ll need a list of
the people who live in your household.
As with everything we
talk about, this will remain confidential – we’ll just be
using it for the purposes of the questions today.
Have
participant orally complete background questions (Appendix B).
May I have your permission to audio tape our interview? The recording will be used only to supplement the notes I’m going to take and will be kept confidential.
Appendix B: Background questions
Name |
Relationship |
Gender |
Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
What is the highest level of education you’ve completed?
Some high school
High school diploma
Some college
College degree
Advanced degree
What is your household income?
Less than or equal to $40,000
$40,001 to $60,000
$60,001 to $75,000
More than $75,000
Appendix C: Proxy Reporting Protocol
Proxy Reporting Questions
To be asked immediately following background questions; the length of time required to complete these questions will be recorded.
You said that [read all roster names] live in your household. I’d like to spend a few minutes talking about ways they spend their money. This will help us move faster later in the interview, and help you remember things that you might otherwise forget since they’re not your expenses.
To start, I want to ask about expenses that the other people in your household have that you may not. For example, one person might buy all the clothing, while another handles all the vehicle expenses.
Are there some types of things that [name1] spends money on that you don’t?
Are there some types of things that [name2] spends money on that you don’t?
…
Thinking
about the last three months since [reference month], has [name1],
[name2], or [name-k] taken any vacations or trips without you?
Thinking about the last three months since [reference month], has [name1], [name2], or [name-k] made any changes to their normal day-to-day routine that might have changed what they did or did not spend money on?
Thinking about the last three months since [reference month], has [name1], [name2], or [name-k] bought anything unusual or out of the ordinary?
Does [name1], [name2], or [name-k] have any hobbies or activities that they spend time on?
For each question, the interviewer will discretely set a timer at the end of the reading of each question to monitor the minimum amount of time that the interviewer must allow for the respondent to formulate his or her response (short: 2s or longer: 10s).
If the household size is greater than four people, questions with lists of individual names will replace the list of names with the phrase “anyone in your household”.
For each question, the reported expenditure will be probed to identify the relevant household member and CEQ subsection for the expense (as necessary); detailed expenditure information will not be collected.
For each CEQ subsection identified, the interviewer will note that section. Those sections will then be asked during administration of the CEQ. The participant will be prompted to include the household members’ purchases in that section.
Proxy Reporting Debriefing
To be asked immediately after completing the Proxy Reporting Questions
Before we continue, I want to take a quick break and ask you for your thoughts on the questions you just answered. We are trying to improve these questions to make sure that people understand what the questions are asking and are able to respond to them accurately.
What did you think of the questions we just asked you about what the other people in your household spend money on, their trips, routines, unusual purchases, and hobbies?
Why do you think
we asked those questions?
Do you think you
were able to answer them accurately?
How much do you know
about how other people in your household spend their
money?
(OPEN-ENDED);
(A GREAT DEAL/QUITE A
BIT/SOMEWHAT/VERY LITTLE/NOT AT ALL)
Do you know the
types of things they spend their money on generally?
(YES/NO)
Do you know the
amounts of money they spend, generally?
(YES/NO)
Do you know
exactly?
(YES/NO)
How long would you say that first set of questions was? Long or short?
(LONG/SHORT)
You said the question set was [LONG/SHORT]. On a scale of 1 to 5, how [LONG/SHORT] would you say it felt?
(1,2,3,4,5)
How sensitive, or personal, would you say those questions were? On a scale from 1 to 5, where 1 is ‘not at all sensitive’ and 5 is ‘very sensitive’.
(1, 2, 3, 4, 5)
What do you think other people would say, on a scale from 1 to 5?
(1, 2, 3, 4, 5)
How did you feel about the experience of listening to the questions and giving your answers? Did it seem normal or awkward?
(NORMAL/AWKWARD)
You said the experience was [NORMAL/AWKWARD]. On a scale of 1 to 5, how [NORMAL/AWKWARD] would you rate the experience?
(1,2,3,4,5)
Proxy Flag for CEQ Questions
To be asked in each expenditure subsection identified during the initial proxy questions, following the section transition (e.g. “Now I am going to ask about….”).
When we started, you said that [name1] might have had some of these expenses. Please keep them in mind as you answer the questions.
After the respondent provides initial answer to a proxy-relevant subsection, ask the following questions, conditional on the answers to the initial proxy probes.
If regarding “Are there some types of things that [name1] spends money on that you don’t?”:
You mentioned that [name1] spends money on [reported item]. Are there any other expenses that you want to add related to that?
If regarding “Thinking about the last three months since [reference month], has [name1] taken any vacations or trips without you?”:
You mentioned that [name1] took a trip recently. Are there any other expenses that you want to add related to that trip?
If regarding “Thinking about the last three months since [reference month], has [name1] made any changes to their normal day-to-day routine that might have changed what they did or did not spend money on?”:
You mentioned that [name1] made a change to their day-to-day routine. Are there any other expenses that you want to add related to that?
If regarding “Thinking about the last three months since [reference month], has [name1] bought anything unusual or out of the ordinary?”:
You mentioned that [name1] bought something unusual or out of the ordinary. Are there any other expenses that you want to add related to the unusual item?
If regarding “Does [name1] have any hobbies or activities that they spend time on?”:
You mentioned that [name1] has a hobby or activity they like to do -- [reported hobby or activity]. Are there any other expenses that you want to add related to that?
Overall Debriefing
To be asked after finishing the CEQ.
Now I’d like to go back and talk about the questions you just answered. I’m interested in hearing your reactions to them, as well as hearing what you thought about as you gave your answers.
When we started, I asked a series of questions about other people in your household. The questions asked about their trips, unusual expenses, hobbies, or ways they spend money that you typically don’t.
Based on your answers to those questions, when we got to the sections about [insert expenditure section with proxy probes], I reminded you about the purchases made by other members in your household.
If I had not instructed you to remember their purchases, do you think you would have? (YES/NO)
How helpful were the questions at the start about what others spend their money in reminding you about others’ expenses, on a scale from 1 to 5, where 1 is ‘not at all helpful’ and 5 is ‘very helpful’?
(1,2,3,4,5)
Do you think you included all the expenses that people in your household had?
(YES/NO)
I did not give extra instructions for [insert
expenditure category asked without proxy probe OR clothing]. I just
said [insert expenditure category transition]. Let’s go back
and review those questions now. I asked you about this list of
items.
Show information booklet page
Do you think you included all the expenses that people in your household had?
(YES/NO)
How helpful would it have been to have additional questions to jog your memory during this section, on a scale from 1 to 5, where 1 is ‘not at all helpful’ and 5 is ‘very helpful’?
(1,2,3,4,5)
We’re interested in helping people tell us as much as they can about expenses for all the members of their household. Do you have any suggestions for how we could have helped you report everyone’s expenses more accurately?
(OPEN-ENDED)
Appendix D: Screening Questions
Begin by asking the candidate the following:
|
|
If answer ONE (SELF
or ALONE), then do not ask any further questions; candidate does not
qualify for participation.
If answer GREATER THAN ONE, then ask
the following:
|
|
If answer ONE (SELF
ONLY), then do not ask any further questions; candidate does not
qualify for participation.
If answer GREATER THAN ONE, then ask
the following:
|
YES / NO |
|
YES / NO |
If answer YES to both Q3 and Q4, then candidate may participate in the study.
If answer NO to either Q3 or Q4, then candidate does not qualify for participation.
Appendix E: Consent Form
The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.
The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.
During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.
We estimate it will take you an average of 60 minutes to participate in this research (ranging from 45 minutes to 70 minutes).
Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.
Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2015.
------------------------------------------------------------------------------------------------------------
I have read and understand the statements above. I consent to participate in this study.
___________________________________ ___________________________
Participant's signature Date
___________________________________
Participant's printed name
___________________________________
Researcher's signature
OMB Control Number: 1220-0141
Expiration Date: 02-28-2015
In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.
1 Precise interval lengths will be piloted internally with OSMR staff in advance of data collection.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Erica Yu |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |