Note to Reviewer - Testing Respondent Burden

July 13, 2015

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Scott Fricker Senior Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for Cognitive Testing of Respondent Burden Questions

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 33.

If there are any questions regarding this project, please contact Scott Fricker at

202-691-7390.

1. Introduction and Purpose

Survey researchers have long speculated that the burden respondents experience in a survey may negatively impact response rates and data quality. In recent years, falling response rates and the establishment of regulations to reduce the time and effort required of government survey respondents have contributed to increased attention to this topic by survey organizations and in the empirical literature.

In late 2011, the U.S. Consumer Expenditure Quarterly Interview Survey (CEQ) began collecting data on respondents’ perceptions of burden through a set of items administered at the end of the final interview. These items asked respondents for their reactions regarding interview length, item difficulty and sensitivity, the effort expended, the frequency of interviews, number of contact attempts, and a single item that asked “how burdensome was this survey to you.” Although previous studies have demonstrated the potential value of these items (e.g., in their associations with key outcome measures), and examined some of their psychometric properties (e.g., through Item Response Theory analyses), they have never been qualitatively tested. Moreover, the results of these previous studies have suggested ways in which burden measurement could be improved (e.g., by modifying response options of existing questions and adding items to better differentiate dimensions of burden).

The purpose of this study is to (1) evaluate respondents’ interpretations of the existing CEQ burden items, (2) assess the extent to which these items capture respondents’ perceptions of burden (and identify other topic areas that may need to be addressed), and (3) test a new set of questions designed to improve burden assessment. The results of this study will be used to refine the set of burden questions used in CEQ production, and serve as inputs for a larger study of a multiple-methods approach to item development being conducted by the Office of Survey Methods Research (OSMR) staff.

2. Study Design

This test will employ cognitive interviews. The interviews will be conducted in the Office of Survey Methods Research (OSMR) cognitive laboratory, and the audio of each session will be digitally recorded. Study participants first will be administered five sections from the CEQ (Utilities and Fuels, Clothing and Jewelry, Medical and Health Expenses; Subscriptions, Memberships, and Entertainment; and Miscellaneous) and then asked a set of burden items. Half of the respondents will be asked the existing nine burden items (see Appendix A), and the remainder will be administered the 20 revised burden items (see Appendix B). After respondents have answered the burden questions, the researchers will conduct a semi-structured debriefing to gain insight into participants’ response processes and reactions to the burden items (see Appendix C for the test protocol). All study interviews will be conducted by OSMR staff experienced in cognitive interviewing.

3. Participants

This study will recruit approximately 30 participants from the OSMR participant database. Efforts will be made to select participants who can report expenditures for their consumer unit, with varying levels of age, education, income, and household composition, based on self-reported information provided during the initial recruitment process. The final number of participants used in this study will be dependent upon the results and the point at which diminishing returns is reached in terms of the information is being gained.

4. Burden Hours

We anticipate that each session will last no longer than one hour, for a total maximum of 30 burden hours (20 minimum if few participants are required). Participants will be compensated $40 for their time. Recruiting prospective participants is estimated to take an additional 5 minutes per person, with an estimated response rate of 80%. Burden hours for screening and participation are expected to total 33 burden hours.

5. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants also will be informed that the study will be used for internal purposes to improve the design of questions used in national Consumer Expenditure interview survey. Participants will be given a consent form to read and sign (See Appendix D). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

Appendix A: Existing CEQ Burden Items

Variable Name	Question
BN_INTRO	I would like to change topics and ask you a few questions about your experience completing this interview. Your responses will help us improve the survey. Continue
BURDEN01	Do you feel that the length of today’s interview was too long, too short, or about right? 1. Too long 2. Too short 3. About right
BURDEN02	How interesting was this survey to you? 1. Very interesting 2. Somewhat interesting 3. A little interesting 4. Not at all interesting
BURDEN03	How difficult or easy was it for you to answer the questions in this survey? 1. Very easy 2. Somewhat easy 3. Somewhat difficult 4. Very difficult
BURDEN04	How burdensome was this survey to you? 1. Very burdensome 2. Somewhat burdensome 3. A little burdensome 4. Not at all burdensome
BURDEN05	How sensitive did you feel the questions I asked today were? 1. Very sensitive 2. Somewhat sensitive 3. A little sensitive 4. Not at all sensitive
BURDEN06	How agreeable would you be to take another survey like this in the future? 1. Very agreeable 2. Somewhat agreeable 3. Not at all agreeable
BURDEN07	If we had had to extend this survey for another 15 minutes, how willing would you have been to continue the interview? 1. Very willing 2. Somewhat willing 3. Somewhat unwilling 4. Very unwilling
BURDEN08	Thinking about the amount of effort that you put forth into answering today’s survey, would you say that you put forth: 1. A lot of effort 2. A moderate amount of effort 3. A little effort
BURDEN09	Please tell me if you agree or disagree with the following statement. I trust the U.S. Census Bureau to safeguard the information that I have provided them: 1. Strongly agree 2. Agree 3. Neither agree nor disagree 4. Disagree 5. Strongly disagree

Appendix B: Revised CEQ Burden Items

Variable Name	Question
rBN_INTRO	I would like to change topics and ask you a few questions about your experience completing this interview. Your responses will help us improve the survey. Continue
rBURDEN01 BURDEN	How burdensome was today’s interview for you? Extremely burdensome Very burdensome 3. Somewhat burdensome 4. A little burdensome 5. Not at all burdensome
rBURDEN02a	In the past six months, how many surveys, polls, or other questionnaires have you been asked to complete? ______________________ [Open-ended response – if not ‘none’ or 0’, go to BURDEN02b, else go to BURDEN03] Follow-up probes (if necessary): Would you say that it was: Just this one survey 2 – 5 surveys 2. More than 5
BURDEN
rBURDEN02b	[If BURDEN02a is not ‘none’ or ‘0’, then ask:] How burdened would you say you feel by those requests? Extremely burdened Very burdened 3. Somewhat burdened 4. A little burdened 5. Not at all burdened
BURDEN
rBURDEN03 LENGTH	Compared to how long you expected this interview would take, would you say that the length of today’s survey was: 1. Much longer than you expected 2. Somewhat longer than you expected 3. About as long as expected 4. A little shorter 5. Much shorter

Variable Name	Question
rBURDEN04	Do you feel that the pace of today’s interview was: 1. Much too fast 2. A little too fast 3. Neither too fast nor too slow 4. A little too slow 5. Much too slow
LENGTH
rBURDEN05 INTEREST	How enjoyable was this survey to you? Extremely enjoyable Very enjoyable 3. Somewhat enjoyable 4. A little enjoyable 5. Not at all enjoyable
rBURDEN06 INTEREST	How interesting was this survey for you? Extremely interesting Very interesting 3. Somewhat interesting 4. A little interesting 5. Not at all interesting
rBURDEN07 DIFFICULTY	How easy or difficult was it for you to answer the questions in this survey? 1. Very easy 2. Somewhat easy 3. Neither easy nor difficult 4. Somewhat difficult 5. Very difficult
rBURDEN08 DIFFICULTY	How capable do you feel you were to answer the questions in the survey? Extremely capable Very capable 3. Somewhat capable 4. Not very capable 5. Not at all capable
rBURDEN09	Thinking about your level of effort, how hard would you say that you tried on this survey: 1. Extremely hard 2. Very hard 3. Somewhat hard 4. Not very hard 5. Not at all hard
EFFORT
rBURDEN10 EFFORT	How important was it to you to do a good job answering today’s survey? Extremely important Very important 3. Somewhat important 4. A little important 5. Not at all important
rBURDEN11	How motivated were you to participate in this survey? Would you say you were: Extremely motivated Very motivated 3. Somewhat motivated 4. A little motivated 5. Not at all motivated
EFFORT

Variable Name	Question
rBURDEN12	How sensitive did you feel that the questions were on today’s survey? Would you say that they were: Extremely sensitive Very sensitive Somewhat sensitive A little sensitive Not at all sensitive
SENSITIVITY
rBURDEN13	How tense or nervous did you feel when answering today’s survey questions? 1. Not at all nervous 2. A little nervous 3. Somewhat nervous 4. Very nervous 5. Extremely nervous
SENSITIVITY
rBURDEN14	Generally speaking, how accurate do you think that government statistics are? 1. Extremely accurate 2. Very accurate 3. Somewhat accurate 4. A little accurate 5. Not at all accurate
TRUST
rBURDEN15	How important do you feel that surveys like this one are to research and government policy decisions: 1. Extremely important 2. Very important 3. Somewhat important 4. A little important 5. Not at all important
VALUE
rBURDEN16 VALUE	How beneficial do you feel this survey might be to you or your family personally? Extremely beneficial Very beneficial 3. Somewhat beneficial 4. A little beneficial 5. Not at all beneficial
rBURDEN17 WILLINGNESS TO PARTICIPATE	If we had to extend the length of this survey, how many more minutes would you be willing to continue the interview? 1. 0 minutes (not willing to do anything longer) 2. 1 – 5 more minutes 3. 6 – 10 more minutes 4. 11 – 20 more minutes 5. More than 20 minutes
rBURDEN18 WILLINGNESS TO PARTICIPATE	How willing would you be to take another survey like this in the future? 1. Very willing 2. Somewhat willing 3. Neither willing nor unwilling 4. Somewhat unwilling 5. Very unwilling

Variable Name	Question
rBURDEN19	If friends of yours were asked to participate in this survey, how likely would you be to encourage them to do so? 1. Extremely likely 2. Very likely 3. Somewhat likely 4. A little likely 5. Not at all likely
WILLINGNESS TO PARTICIPATE
rBURDEN20 TRUST	To what extent do you trust the U.S. Census Bureau to keep your information confidential? 1. Not at all 2. A little 3. Somewhat 4. A lot 5. Very much

Appendix C: Test Protocol

Introduction

Hello, my name is __________, and I work for the Bureau of Labor Statistics, which is part of the Department of Labor. The purpose of this interview is to find out what you think about the Consumer Expenditure Quarterly Interview Survey and its follow-up questions. We have found that the best way to develop new surveys/questions is to try them out with a variety of people and see how well they work for them. Your feedback will help us refine the survey and questions.

I’d also like to make a recording of this interview with your permission. The recording is for my own note taking and will help me to write an accurate report later. Your answers are confidential. Your name will not be associated with any of your answers, and will not be used in any of our reports. Your participation is voluntary and you can decline to answer any questions you don’t want to answer.

CHECKLIST:

□ OBTAIN RESPONDENT CONSENT

□ OBTAIN PERMISSION TO RECORD SESSION, START RECORDER

□ ASK IF RESPONDENT HAS QUESTIONS PRIOR TO STARTING

CEQ Administration

First, we will go through a portion of the Consumer Expenditure Survey.

[Collect modified household roster:]

Let’s start by getting the names of all of the people who currently live in your household.

_____________ ______________ _______________ _____________ ___________

What is [Name’s] relation to you?

_____________ ______________ _______________ _____________ ___________

[Administer CEQ Sections 4A, 4C, 9A, 15A, 17, and 19A]

SECTION 4 – UTILITIES AND FUELS FOR OWNED AND RENTED PROPERTEIS

4A. Telephone Expenses

Now I am going to ask about telephone bills. Please remember to include any bills you receive or pay online or have automatically deducted. Report any telephone bill you have received, even if the bill has not been paid.

Since the first of the reference month, have you or any members of your household received any bills for telephone services, including cellular and Voice Over IP? Do not include bells for telephones used entirely for business purposes.

• 1. YES

• What is the name of the company which provides the services? _______________

• How much were you billed for in the [reference period]? ____________________

• How much were you billed for in [two months ago]? _______________________

• How much were you billed for in [last month]? ___________________________

• What is your usual billing period for this service?

  • 1. MONTH

  • 2. 2 MONTHS

  • 3, QUARTER

  • 4. ANNUAL

  • 5. OTHER

• 2. NO [go to 4C]

4C. Internet Service Expenses

Now I am going to ask about cable and satellite TV service, satellite radio service, and internet service expenditures.

Since the first of the reference month, have you or any members of your household had any expenses for cable or satellite TV (not already reported), satellite radio services (NAR), internet connection or an internet service provider (NAR), internet services away from home such as web cafes or internet kiosks, or none?

Service	What was the expense for?	In what month was the expense?	How much was this expense?	What internet service was this combined with, if any? [enter 1-4]	Did you have any other internet service expenses? [if yes, go to next item] [if no, go to Section 9]
Cable or satellite TV
Satellite radio services
Internet connection or an ISP
Internet services away from home

SECTION 9 – CLOTHING AND JEWELRY

9A. Clothing, Watches, and Jewelry

Item	What did you buy?	Was this purchased for someone inside or outside your household, or both?	When did you purchase it? [month]	How much did it cost?	Did this include sales tax?	Did you purchase any other of these types of clothing? [if yes, go to next item] [if no, go to next table]
Coats, jackets, or furs
Sport coats or tailored jackets
Suits
Vests
Shirts, sweaters, blouses, or tops
Pants, jeans, or shorts
Dresses
Skirts
Undergarments
Hosiery
Nightwear or loungewear

Now I am going to ask about clothing and footwear, for infants, children, and adults. Please include any shipping and handling charges with the cost of any item that was shipped.

Since the first of [the reference month], have you or has any member of your household purchased any of the following items, either for members of your household or for someone outside your household? [read items in first table]

[read items in second table]

Item	What did you buy?	Was this purchased for someone inside or outside your household, or both?	When did you purchase it? [month]	How much did it cost?	Did this include sales tax?	Did you purchase any other clothing? [if yes, go to next item] [if no, go to next table]
Accessories
Swimsuits or warm-up or ski suits
Uniforms, for which the cost is not reimbursed
Costumes
Footwear, including athletic footwear
Diapers
Layettes
Watches
Jewelry

SECTION 15 – MEDICAL AND HEALTH EXPENDITURES

15A. Screening Questions for Medical Payments

Now I am going to ask some questions about medical payments. By payments I mean any co-pays and out-of-pocket expenses. Include all payments, even those for persons who are outside of your household.

Since the first of [the reference month], have you or any members of your household made any payments for the following? [read items in first table]

Care/service/item	Describe the care/service/item	Was this for a member of your household or someone outside of your household?	In what month was/were the payment(s) made?	What was the total amount paid in that month?	Did you or any members of your household make any other of these medical payments? [if yes, go to next item] [if no, go to next table]
Eye examinations, treatment, or surgery
Purchase of eyeglasses or contact lenses
Dental care
Hospital room or hospital services
Services by med. pros. other than physicians
Physician services

[read items in second table]

Care/service/item	Describe the care/service/item	Was this for a member of your household or someone outside of your household?	In what month was/were the payment(s) made?	What was the total amount paid in that month?	Did you or any members of your household make any other medical payments? [if yes, go to next item] [if no, go to Section 17]
Lab tests or x-rays
Care in convalescent or nursing homes
Care for invalids, convalescents, handicapped, or elderly persons in the home
Adult day care centers
Other medical care and services
Hearing aids
Prescription drugs
Purchase/rental of supportive/ rehabilitative equipment
Purchase/rental of medical/surgical equipment for general use

SECTION 17 – SUBSCRIPTIONS, MEMBERSHIPS, BOOKS, AND ENTERTAINMENT EXPENSES

Now I am going to ask about expenses for subscriptions, memberships, books, and entertainment. Please remember to include any payments you made online or had automatically deducted. Also, include any shipping and handling charges with the cost of any item that was shipped.

Since the first of the reference month have you or any member of your household had any membership costs or other expenses related to any of the following? [read items]

Item	Describe the item [ask for name of club, organization, card, or service if applicable]	Was this purchase for your household or someone outside your household?	In what month did you have this expense?	What was the total amount of the expense?	Did you purchase any other of these items? [if yes, go to next item] [if no, go to next table]
Golf courses or country clubs
Health clubs, fitness centers, swimming pools, weight loss centers, other sports & rec organizations
Fees for participating in sports
Vacation clubs
Civic, service, fraternal, or other social organizations
Credit card membership fees
Shopping club memberships (Sam’s Club, Amazon, etc.)
Direct or online dating services

Since the first of [reference month], have you or any member of your household purchased any of the following items for your household or for someone outside your household? [read items]

Item	Describe the item [ask for name of club, organization, card, or service if applicable]	Was this purchase for your household or someone outside your household?	In what month did you have this expense?	What was the total amount of the expense?	Did you purchase any other of these items? [if yes, go to next item] [if no, go to next table]
Single or season tickets to spectator sports events
Single or season tickets to plays, operas, or concerts
Tickets to movies, parks, or museums
Single copies of newspapers, magazines, or periodicals
Subscriptions to newspapers, magazines, or periodicals
Encyclopedias or other reference books
Books purchased through a book club
Books or digital books not purchased through a book club (excluding school books)

Have you purchased any of the following items? [read items]

Item	Describe the item [ask for name of club, organization, card, or service if applicable]	Was this purchase for your household or someone outside your household?	In what month did you have this expense?	What was the total amount of the expense?	Did you purchase any other items? [if yes, go to next item] [if no, go to Section 19]
Photographic film or disposable cameras
Film processing or printing digital photos
Purchased music files, CDs, or records
Subscription music services
Purchased video files, Blue-Ray discs, or DVDs
Rented video files or DVDs
Streaming video subscriptions
Purchased video games, not including computer games
Rented video games
Apps, games, or ringtones for mobiles
Online games or other internet entertainment sites

SECTION 19 – MISCELLANEOUS EXPENSES

19A. Miscellaneous Expenses

Now I am going to ask about miscellaneous expenses which have not been collected anywhere else in this survey. Please remember to include any payments you made online or had automatically deducted. Also, please include any shipping and handling charges with the cost of any item that was shipped.

Since the first of [reference month], have you or any members of your household had expenses for any of the following, either for you or any members of your household or for someone outside your household? [read items]

Item	What was the expense for?	In what month did you have this expense?	Was this expense for someone inside or outside your household?	What was the total amount of this expense?	Did you purchase any other of these items? [if yes, go to next item] [if no, go to next table]
Fresh flowers or potted plants
Professional photography
Services of lawyers or other legal pros.
Accounting fees
Occupational expenses (union dues, licenses)
Gardening or lawn care services
Housekeeping services
Home security system service fees
Other home services or small repair jobs
Moving, storage, or freight
Stamp or coin collecting
Lotteries or games of chance

Have you had expenses for any of the following? [read items]

Item	What was the expense for?	In what month did you have this expense?	Was this expense for someone inside or outside your household?	What was the total amount of this expense?	Did you purchase any other of these items? [if yes, go to next item] [if no, go to next table]
Babysitting, nanny services, or other child care
Toys or games
Arts or crafts kits
Sewing, knitting, or quilting materials
Purchase of pets, pet supplies, or medicine for pets
Pet services
Veterinarian expenses for pets
Have you had expenses for any of the following? [read items]
Food and beverages for catered affairs					Did you purchase any other items? [if yes, go to next item] [if no, go to Section 19B]
Live entertainment
Rental of party supplies
Purchase or upkeep of cemetery lots of vaults
Funerals, burials, or cremation

[Administer Burden Items]

Thank you for answering those questions about your household expenses. Now, I’d like to take a few minutes to ask you some questions about your reactions to the survey you just completed.

[Half of the respondents will be given Version 1 of the burden items (Appendix A) and half will be given Version 2 (Appendix B)]

Debriefing

Great – that completes the first portion of today’s session. For the remainder of our time, I’d like to have a more informal conversation with you about your experiences. I’ve got a number of questions I’d like to go through, so I’ll be using this script as a guide, but we are free to deviate from this script whenever we want to.

So, let’s start with getting your overall thoughts about the expenditure survey itself.

General CEQ Probes

What was your overall impression of the expenditure questions we went through?

Were there any topics or questions in the expenditure survey that were unclear to you? Why?

Were there any parts of survey that you felt were particularly difficult to answer? Why?

Burden Items

Now I’d like to move on to those follow-up questions that we asked at the end of the survey – e.g., those questions that asked about your interest in the survey, your perceptions of its length, sensitivity, etc. The intent of these questions is to collect information from respondents that we hope can be used to improve the survey, but in order to do that, we need to have a better understanding of how those questions were interpreted, if they made sense, if there are other types of questions that we should be asking, and so on.

So, again, I’d like to start by asking you for your general reactions to those questions, and then we’ll take a few minutes to walk through some of the individual questions together, okay?

Overall, what were your reactions to this group of questions that came at the end of the survey?

• Did they seem useful?

• Did they make sense to you? (If not, which ones/why not?)

• Any other general reactions to these questions?

VERSION 1 Targeted item questions

The first question I asked you was

“Do you feel that the length of today’s interview was too long, too short, or about right?”

• What reactions did you have when I asked you this question?

• How did you go about answering it (i.e., what led you to answer XXX? What factors were important, what did you think about?)?

• Did the response categories – too long, too short, or about right – seem appropriate to you? If not, why is that and what changes would you think would improve this question?

The next question I asked was: How interesting was this survey to you?

• Can you tell me in your own words what this question is asking?

• What did you think about when you answered this question

• Were there features of the survey, of your own expenditure situation, or the interview itself that made the survey more or less interesting?

• Did you have any problems selecting between the response options? If so, why?

• In general, in thinking about the surveys that you’ve been asked to do in your day-to-day life – at home/school/work, on the phone, by mail, or email – do you find them to be interesting, boring, tedious, fun, or what?

How easy or difficult was it for you to answer the questions in the survey?

We talked a moment ago in general terms about how easy or difficult the survey was for you. When we asked you this question in our initial follow-up you said that it was XX [fill with Respondent’s answer].

• How did you arrive at this answer?

• Were there features of the survey, of your own expenditure situation, or the interview itself that made the survey more or less difficult? What could we have done to make it easier to answer the CEQ questions?

• Any problems with or suggested improvements for this question or the response options?

How burdensome was this survey to you?

We next asked you how burdensome the survey was.

• What does that term “burdensome” mean to you?

• What were your reactions to this question?

• What factors contributed to the “burden” you felt?

  • [Probe on influence of questions themselves, the instructions, the interview process, mood, recall issues, etc. as they come up – do not lead Respondent]

• What would have caused you to answer “very burdensome”

• What would have caused you to answer “not at all burdensome”

• How might we improve this question?

How sensitive did you feel the questions I asked today were?

We also asked you how sensitive you felt the questions were – you said XX.

• What came to mind as you were answering this question?

• What caused you to say XX? What if any specific questions did you think were sensitive?

• What sorts of items would you view as particularly sensitive on a federal government survey?

How agreeable would you be to take another survey like this in the future?

When I asked you how agreeable you would be to taking another survey like the Consumer Expenditure Survey, you said XX.

• How did you arrive at this answer?

• What factors would affect your decision about whether or not to participate in another survey like the CEQ?

• Do you think that your answer to this question is indicative of how likely you would actually be to say “yes” to another survey request like this? If not, why not?

If we had had to extend this survey for another 15 minutes, how willing would you have been to continue the interview?

I also asked you how willing you would be to spend another 15 minutes on this survey, and you said XX.

• What caused you to say that you would be XX?

• We are interested in the things that can affect a person’s willingness to continue with a longer survey. What sorts of things would impact your decision to stick with a longer interview – either positively or negatively?

Thinking about the amount of effort that you put forth into answering today’s survey, would you say that you put forth:

I next asked you about the amount of effort you put into answering the survey.

• What was your reaction to this question?

• What kinds of things did you think about when arriving at your answer? That is, did you recall specific instances from the interview, or did you answer based more an overall feeling about how you approached the survey, or how you approach activities like this in general?

• Did you feel that people will feel comfortable answering this question? If not, why not? How could we improve this question?

The last question I asked you was how much you trust the Census Bureau to protect your data:

I trust the U.S. Census Bureau to safeguard the information that I have provided them:

• What was your reaction to this question?

• What does that phrase mean to you: “safeguard the information you provided?”

• How did you come up with your answer? That is, what sorts of things impact the trust you have in Census Bureau’s (or BLS’s) ability to keep your data confidential?

Do you have any other comments, concerns, or questions about this set of questions that were designed to capture respondents’ reactions to the survey process?

• Are there other questions that you think we should be asking that might help us to form a more accurate understanding of respondents’ experiences?

• Any questions that we should drop or revise?

VERSION 2 Targeted item questions

The first question I asked you was

How burdensome was today’s interview for you?

• What does that term “burdensome” mean to you?

• What were your reactions to this question?

• What factors contributed to the “burden” you felt?

  • [Probe on influence of questions themselves, the instructions, the interview process, mood, recall issues, etc. as they come up – do not lead Respondent]

• What would have caused you to answer “very burdensome”

• What would have caused you to answer “not at all burdensome”

• Any issues selecting between the different response options? If so, why?

• How might we improve this question?

The next question I asked was:

In the past six months [SINCE December 2014], how many surveys, polls, or other questionnaires have you been asked to complete?

• Again, what was your reaction when you heard this question?

• How did you arrive at an answer?

• Are there any words or terms in this question that you found confusing or difficult?

I followed up this question with one that asked:

How burdened would you say you feel by those requests?

• Can you tell me in your own words what you think this question is asking?

• Any problems answering this question – either in coming up with a response, or selecting one of the response categories?

• To what extent would you say that the amount of burden you felt in this survey was impacted by recent experiences with other surveys/survey requests?

I asked you a couple of questions about the length of the survey.

Compared to how long you expected this interview would take, would you say that the length of today’s survey was:

• Can you restate this question in your own words please? What did it mean to you?

• You said that today’s interview was XX – why is that?

• [Unless R replies “about as long as expected,” ask:]

  • How long did you anticipate the survey would last?

• To what extent does the fact that the interview was [longer/shorter] than you expected impact your overall reactions to the survey, how much effort you gave, etc.

• [If not obvious, ] Did the interview seem too long, too short, about right?

Do you feel that the pace of today’s interview was:

When I asked you about the pace of today’s interview, you said that it was XX.

• What made you say that? What made it seem XX?

• To what extent do you feel that the pace of the interview affects your overall reactions to the survey?

• Thinking both of the overall length and the pace of the expenditure survey, how much would you say these factors impact your reaction to the survey? How important is interview duration and pacing?

How enjoyable was this survey to you? / How interesting was this survey for you?

I asked you a couple of questions about how enjoyable and interesting the survey was to you.

• What were your reactions to these questions?

• What does “enjoyable” mean to you in this question? How about “interesting?”

• Any problems selecting one of the response options?

• To your mind, is there a clear distinction between “somewhat enjoyable” and “a little enjoyable” or “extremely enjoyable” and “very enjoyable”?

  • If not, why not?

  • How could we improve the response categories? (is it any clearer to have “not very enjoyable” [or interesting] vs. “somewhat enjoyable” [or interesting]?

  • Could you give me an example of a survey that was “extremely enjoyable” [or extremely interesting]?

How easy or difficult was it for you to answer the questions in the survey?

We talked a moment ago in general terms about how easy or difficult the survey was for you. When we asked you this question in our initial follow-up you said that it was XX [fill with Respondent’s answer].

• How did you arrive at this answer?

• Were there features of the survey, of your own expenditure situation, or the interview itself that made the survey more or less difficult? What could we have done to make it easier to answer the CEQ questions?

• Any problems with or suggested improvements for this question or the response options?

How capable do you feel you were to answer the questions in the survey?

• Could you tell me in your own words what this question is asking?

• How did you arrive at your answer – that you felt XX? What went into that determination (e.g., confidence in comprehension, knowledge of HH spending, general ability, ease of questions)?

• Any reactions to this question or its response options?

• Would it be better if we asked how confident you were in your ability to answer the questions in the survey?

• Other suggested changes?

Thinking about your level of effort, how hard would you say that you tried on this survey?

I next asked you about the amount of effort you put into answering the survey.

• What was your reaction to this question?

• What kinds of things did you think about when arriving at your answer? That is, did you recall specific instances from the interview, or did you answer based more an overall feeling about how you approached the survey, or how you approach activities like this in general?

• Did you feel that people will feel comfortable answering this question? If not, why not?

• How could we improve this question?

How important was it to you to do a good job answering today’s survey?

When I asked you this question, you said XX.

• How did you arrive at that answer?

• What does it mean to you to “do a good job answering” the survey?

• Did you feel comfortable answering this question? If not, how might we revise?

How motivated were you to participate in this survey?

• When you heard this question, what were your reactions?

  • Positive/negative emotions?

• Definitional issues – what does it mean to you to be “motivated to participate in this survey?”

• Was this question easy or difficult for you to answer? Why is that?

How sensitive did you feel the questions I asked today were?

We also asked you how sensitive you felt the questions were – you said XX.

• What came to mind as you were answering this question?

• What caused you to say XX? What if any specific questions did you think were sensitive?

• What sorts of items would you view as particularly sensitive on a federal government survey?

How tense or nervous did you feel when answering today’s survey questions?

• When you heard this question, what were your reactions?

  • Positive/negative emotions?

• Definitional issues – what does it mean to you to be “tense or nervous?”

• Was this question easy or difficult for you to answer? Why is that?

Generally speaking, how accurate do you think that government statistics are?

We asked several questions that were designed to get at your views of government statistics. We started by asking you for your impressions of their accuracy. You said that you think that government statistics generally are XX.

• Why do you say that? What leads you to believe this?

• To what extent do you think your impressions about the accuracy of government statistics are likely to influence your decisions to participate in government survey requests? How impact?

How important do you feel that surveys like this one are to research and government policy decisions:

You said that you felt that surveys like the CEQ are XX to research and policy decisions.

• Why do you say that? What leads you to believe this?

• What are your reactions to this question?

• Can you give me an example of a survey that does impact research OR government policy decisions? [Probe to determine how those terms are understood.]

• Suggested changes/improvements?

How beneficial do you feel this survey might be to you or your family personally?

• What does “beneficial” mean to you in this question?

• What would make a survey beneficial to you?

• Suggested changes/improvements?

If we had had to extend the length of this survey, how many more minutes would you be willing to continue the interview?

I also asked you how many more minutes you’d be willing to extend this survey, and you said XX.

• What caused you to say XX?

• We are interested in the things that can affect a person’s willingness to continue with a longer survey. What sorts of things would impact your decision to stick with a longer interview – either positively or negatively?

• Do the response options seem reasonable?

How willing would you be to take another survey like this in the future?

When I asked you how willing you would be to taking another survey like the Consumer Expenditure Survey, you said XX.

• How did you arrive at this answer?

• What factors would affect your decision about whether or not to participate in another survey like the CEQ?

• Do you think that your answer to this question is indicative of how likely you would actually be to say “yes” to another survey request like this? If not, why not?

If friends of yours were asked to participate in this survey, how likely would you be to encourage them to do so?

• What was your reaction to this question?

• Does the wording make sense? Any difficulties answering this question (if so, why)?

The last question I asked you was how much you trust the Census Bureau to protect your data:

To what extent do you trust the U.S. Census Bureau to keep your information confidential:

• What was your reaction to this question?

• What does that phrase mean to you: “keep your information confidential?”

• How did you come up with your answer? That is, what sorts of things impact the trust you have in Census Bureau’s (or BLS’s) ability to keep your data confidential?

Do you have any other comments, concerns, or questions about this set of questions that were designed to capture respondents’ reactions to the survey process?

• Are there other questions that you think we should be asking that might help us to form a more accurate understanding of respondents’ experiences?

• Any questions that we should drop or revise?

• Any issues with/suggested improvements for the order or flow of questions?

Appendix D: Informed Consent

CONSENT FORM

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take 60 minutes to participate in this research.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires April 30, 2018.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: April 30, 2018

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

36