Form Appendix C Appendix C Test Protocol

Appendix C: Test Protocol

Introduction

Hello, my name is __________, and I work for the Bureau of Labor Statistics, which is part of the Department of Labor. The purpose of this interview is to find out what you think about the Consumer Expenditure Quarterly Interview Survey and its follow-up questions. We have found that the best way to develop new surveys/questions is to try them out with a variety of people and see how well they work for them. Your feedback will help us refine the survey and questions.

I’d also like to make a recording of this interview with your permission. The recording is for my own note taking and will help me to write an accurate report later. Your answers are confidential. Your name will not be associated with any of your answers, and will not be used in any of our reports. Your participation is voluntary and you can decline to answer any questions you don’t want to answer.

CHECKLIST:

□ OBTAIN RESPONDENT CONSENT

□ OBTAIN PERMISSION TO RECORD SESSION, START RECORDER

□ ASK IF RESPONDENT HAS QUESTIONS PRIOR TO STARTING

CEQ Administration

First, we will go through a portion of the Consumer Expenditure Survey.

[Collect modified household roster:]

Let’s start by getting the names of all of the people who currently live in your household.

_____________ ______________ _______________ _____________ ___________

What is [Name’s] relation to you?

_____________ ______________ _______________ _____________ ___________

[Administer CEQ Sections 4A, 4C, 9A, 15A, 17, and 19A]

SECTION 4 – UTILITIES AND FUELS FOR OWNED AND RENTED PROPERTEIS

4A. Telephone Expenses

Now I am going to ask about telephone bills. Please remember to include any bills you receive or pay online or have automatically deducted. Report any telephone bill you have received, even if the bill has not been paid.

Since the first of the reference month, have you or any members of your household received any bills for telephone services, including cellular and Voice Over IP? Do not include bells for telephones used entirely for business purposes.

• 1. YES

• What is the name of the company which provides the services? _______________

• How much were you billed for in the [reference period]? ____________________

• How much were you billed for in [two months ago]? _______________________

• How much were you billed for in [last month]? ___________________________

• What is your usual billing period for this service?

  • 1. MONTH

  • 2. 2 MONTHS

  • 3, QUARTER

  • 4. ANNUAL

  • 5. OTHER

• 2. NO [go to 4C]

4C. Internet Service Expenses

Now I am going to ask about cable and satellite TV service, satellite radio service, and internet service expenditures.

Since the first of the reference month, have you or any members of your household had any expenses for cable or satellite TV (not already reported), satellite radio services (NAR), internet connection or an internet service provider (NAR), internet services away from home such as web cafes or internet kiosks, or none?

Service	What was the expense for?	In what month was the expense?	How much was this expense?	What internet service was this combined with, if any? [enter 1-4]	Did you have any other internet service expenses? [if yes, go to next item] [if no, go to Section 9]
Cable or satellite TV
Satellite radio services
Internet connection or an ISP
Internet services away from home

SECTION 9 – CLOTHING AND JEWELRY

9A. Clothing, Watches, and Jewelry

Item	What did you buy?	Was this purchased for someone inside or outside your household, or both?	When did you purchase it? [month]	How much did it cost?	Did this include sales tax?	Did you purchase any other of these types of clothing? [if yes, go to next item] [if no, go to next table]
Coats, jackets, or furs
Sport coats or tailored jackets
Suits
Vests
Shirts, sweaters, blouses, or tops
Pants, jeans, or shorts
Dresses
Skirts
Undergarments
Hosiery
Nightwear or loungewear

Now I am going to ask about clothing and footwear, for infants, children, and adults. Please include any shipping and handling charges with the cost of any item that was shipped.

Since the first of [the reference month], have you or has any member of your household purchased any of the following items, either for members of your household or for someone outside your household? [read items in first table]

[read items in second table]

Item	What did you buy?	Was this purchased for someone inside or outside your household, or both?	When did you purchase it? [month]	How much did it cost?	Did this include sales tax?	Did you purchase any other clothing? [if yes, go to next item] [if no, go to next table]
Accessories
Swimsuits or warm-up or ski suits
Uniforms, for which the cost is not reimbursed
Costumes
Footwear, including athletic footwear
Diapers
Layettes
Watches
Jewelry

SECTION 15 – MEDICAL AND HEALTH EXPENDITURES

15A. Screening Questions for Medical Payments

Now I am going to ask some questions about medical payments. By payments I mean any co-pays and out-of-pocket expenses. Include all payments, even those for persons who are outside of your household.

Since the first of [the reference month], have you or any members of your household made any payments for the following? [read items in first table]

Care/service/item	Describe the care/service/item	Was this for a member of your household or someone outside of your household?	In what month was/were the payment(s) made?	What was the total amount paid in that month?	Did you or any members of your household make any other of these medical payments? [if yes, go to next item] [if no, go to next table]
Eye examinations, treatment, or surgery
Purchase of eyeglasses or contact lenses
Dental care
Hospital room or hospital services
Services by med. pros. other than physicians
Physician services

[read items in second table]

Care/service/item	Describe the care/service/item	Was this for a member of your household or someone outside of your household?	In what month was/were the payment(s) made?	What was the total amount paid in that month?	Did you or any members of your household make any other medical payments? [if yes, go to next item] [if no, go to Section 17]
Lab tests or x-rays
Care in convalescent or nursing homes
Care for invalids, convalescents, handicapped, or elderly persons in the home
Adult day care centers
Other medical care and services
Hearing aids
Prescription drugs
Purchase/rental of supportive/ rehabilitative equipment
Purchase/rental of medical/surgical equipment for general use

SECTION 17 – SUBSCRIPTIONS, MEMBERSHIPS, BOOKS, AND ENTERTAINMENT EXPENSES

Now I am going to ask about expenses for subscriptions, memberships, books, and entertainment. Please remember to include any payments you made online or had automatically deducted. Also, include any shipping and handling charges with the cost of any item that was shipped.

Since the first of the reference month have you or any member of your household had any membership costs or other expenses related to any of the following? [read items]

Item	Describe the item [ask for name of club, organization, card, or service if applicable]	Was this purchase for your household or someone outside your household?	In what month did you have this expense?	What was the total amount of the expense?	Did you purchase any other of these items? [if yes, go to next item] [if no, go to next table]
Golf courses or country clubs
Health clubs, fitness centers, swimming pools, weight loss centers, other sports & rec organizations
Fees for participating in sports
Vacation clubs
Civic, service, fraternal, or other social organizations
Credit card membership fees
Shopping club memberships (Sam’s Club, Amazon, etc.)
Direct or online dating services

Since the first of [reference month], have you or any member of your household purchased any of the following items for your household or for someone outside your household? [read items]

Item	Describe the item [ask for name of club, organization, card, or service if applicable]	Was this purchase for your household or someone outside your household?	In what month did you have this expense?	What was the total amount of the expense?	Did you purchase any other of these items? [if yes, go to next item] [if no, go to next table]
Single or season tickets to spectator sports events
Single or season tickets to plays, operas, or concerts
Tickets to movies, parks, or museums
Single copies of newspapers, magazines, or periodicals
Subscriptions to newspapers, magazines, or periodicals
Encyclopedias or other reference books
Books purchased through a book club
Books or digital books not purchased through a book club (excluding school books)

Have you purchased any of the following items? [read items]

Item	Describe the item [ask for name of club, organization, card, or service if applicable]	Was this purchase for your household or someone outside your household?	In what month did you have this expense?	What was the total amount of the expense?	Did you purchase any other items? [if yes, go to next item] [if no, go to Section 19]
Photographic film or disposable cameras
Film processing or printing digital photos
Purchased music files, CDs, or records
Subscription music services
Purchased video files, Blue-Ray discs, or DVDs
Rented video files or DVDs
Streaming video subscriptions
Purchased video games, not including computer games
Rented video games
Apps, games, or ringtones for mobiles
Online games or other internet entertainment sites

SECTION 19 – MISCELLANEOUS EXPENSES

19A. Miscellaneous Expenses

Now I am going to ask about miscellaneous expenses which have not been collected anywhere else in this survey. Please remember to include any payments you made online or had automatically deducted. Also, please include any shipping and handling charges with the cost of any item that was shipped.

Since the first of [reference month], have you or any members of your household had expenses for any of the following, either for you or any members of your household or for someone outside your household? [read items]

Item	What was the expense for?	In what month did you have this expense?	Was this expense for someone inside or outside your household?	What was the total amount of this expense?	Did you purchase any other of these items? [if yes, go to next item] [if no, go to next table]
Fresh flowers or potted plants
Professional photography
Services of lawyers or other legal pros.
Accounting fees
Occupational expenses (union dues, licenses)
Gardening or lawn care services
Housekeeping services
Home security system service fees
Other home services or small repair jobs
Moving, storage, or freight
Stamp or coin collecting
Lotteries or games of chance

Have you had expenses for any of the following? [read items]

Item	What was the expense for?	In what month did you have this expense?	Was this expense for someone inside or outside your household?	What was the total amount of this expense?	Did you purchase any other of these items? [if yes, go to next item] [if no, go to next table]
Babysitting, nanny services, or other child care
Toys or games
Arts or crafts kits
Sewing, knitting, or quilting materials
Purchase of pets, pet supplies, or medicine for pets
Pet services
Veterinarian expenses for pets
Have you had expenses for any of the following? [read items]
Food and beverages for catered affairs					Did you purchase any other items? [if yes, go to next item] [if no, go to Section 19B]
Live entertainment
Rental of party supplies
Purchase or upkeep of cemetery lots of vaults
Funerals, burials, or cremation

[Administer Burden Items]

Thank you for answering those questions about your household expenses. Now, I’d like to take a few minutes to ask you some questions about your reactions to the survey you just completed.

[Half of the respondents will be given Version 1 of the burden items (Appendix A) and half will be given Version 2 (Appendix B)]

Debriefing

Great – that completes the first portion of today’s session. For the remainder of our time, I’d like to have a more informal conversation with you about your experiences. I’ve got a number of questions I’d like to go through, so I’ll be using this script as a guide, but we are free to deviate from this script whenever we want to.

So, let’s start with getting your overall thoughts about the expenditure survey itself.

General CEQ Probes

What was your overall impression of the expenditure questions we went through?

Were there any topics or questions in the expenditure survey that were unclear to you? Why?

Were there any parts of survey that you felt were particularly difficult to answer? Why?

Burden Items

Now I’d like to move on to those follow-up questions that we asked at the end of the survey – e.g., those questions that asked about your interest in the survey, your perceptions of its length, sensitivity, etc. The intent of these questions is to collect information from respondents that we hope can be used to improve the survey, but in order to do that, we need to have a better understanding of how those questions were interpreted, if they made sense, if there are other types of questions that we should be asking, and so on.

So, again, I’d like to start by asking you for your general reactions to those questions, and then we’ll take a few minutes to walk through some of the individual questions together, okay?

Overall, what were your reactions to this group of questions that came at the end of the survey?

• Did they seem useful?

• Did they make sense to you? (If not, which ones/why not?)

• Any other general reactions to these questions?

VERSION 1 Targeted item questions

The first question I asked you was

“Do you feel that the length of today’s interview was too long, too short, or about right?”

• What reactions did you have when I asked you this question?

• How did you go about answering it (i.e., what led you to answer XXX? What factors were important, what did you think about?)?

• Did the response categories – too long, too short, or about right – seem appropriate to you? If not, why is that and what changes would you think would improve this question?

The next question I asked was: How interesting was this survey to you?

• Can you tell me in your own words what this question is asking?

• What did you think about when you answered this question

• Were there features of the survey, of your own expenditure situation, or the interview itself that made the survey more or less interesting?

• Did you have any problems selecting between the response options? If so, why?

• In general, in thinking about the surveys that you’ve been asked to do in your day-to-day life – at home/school/work, on the phone, by mail, or email – do you find them to be interesting, boring, tedious, fun, or what?

How easy or difficult was it for you to answer the questions in the survey?

We talked a moment ago in general terms about how easy or difficult the survey was for you. When we asked you this question in our initial follow-up you said that it was XX [fill with Respondent’s answer].

• How did you arrive at this answer?

• Were there features of the survey, of your own expenditure situation, or the interview itself that made the survey more or less difficult? What could we have done to make it easier to answer the CEQ questions?

• Any problems with or suggested improvements for this question or the response options?

How burdensome was this survey to you?

We next asked you how burdensome the survey was.

• What does that term “burdensome” mean to you?

• What were your reactions to this question?

• What factors contributed to the “burden” you felt?

  • [Probe on influence of questions themselves, the instructions, the interview process, mood, recall issues, etc. as they come up – do not lead Respondent]

• What would have caused you to answer “very burdensome”

• What would have caused you to answer “not at all burdensome”

• How might we improve this question?

How sensitive did you feel the questions I asked today were?

We also asked you how sensitive you felt the questions were – you said XX.

• What came to mind as you were answering this question?

• What caused you to say XX? What if any specific questions did you think were sensitive?

• What sorts of items would you view as particularly sensitive on a federal government survey?

How agreeable would you be to take another survey like this in the future?

When I asked you how agreeable you would be to taking another survey like the Consumer Expenditure Survey, you said XX.

• How did you arrive at this answer?

• What factors would affect your decision about whether or not to participate in another survey like the CEQ?

• Do you think that your answer to this question is indicative of how likely you would actually be to say “yes” to another survey request like this? If not, why not?

If we had had to extend this survey for another 15 minutes, how willing would you have been to continue the interview?

I also asked you how willing you would be to spend another 15 minutes on this survey, and you said XX.

• What caused you to say that you would be XX?

• We are interested in the things that can affect a person’s willingness to continue with a longer survey. What sorts of things would impact your decision to stick with a longer interview – either positively or negatively?

Thinking about the amount of effort that you put forth into answering today’s survey, would you say that you put forth:

I next asked you about the amount of effort you put into answering the survey.

• What was your reaction to this question?

• What kinds of things did you think about when arriving at your answer? That is, did you recall specific instances from the interview, or did you answer based more an overall feeling about how you approached the survey, or how you approach activities like this in general?

• Did you feel that people will feel comfortable answering this question? If not, why not? How could we improve this question?

The last question I asked you was how much you trust the Census Bureau to protect your data:

I trust the U.S. Census Bureau to safeguard the information that I have provided them:

• What was your reaction to this question?

• What does that phrase mean to you: “safeguard the information you provided?”

• How did you come up with your answer? That is, what sorts of things impact the trust you have in Census Bureau’s (or BLS’s) ability to keep your data confidential?

Do you have any other comments, concerns, or questions about this set of questions that were designed to capture respondents’ reactions to the survey process?

• Are there other questions that you think we should be asking that might help us to form a more accurate understanding of respondents’ experiences?

• Any questions that we should drop or revise?

VERSION 2 Targeted item questions

The first question I asked you was

How burdensome was today’s interview for you?

• What does that term “burdensome” mean to you?

• What were your reactions to this question?

• What factors contributed to the “burden” you felt?

  • [Probe on influence of questions themselves, the instructions, the interview process, mood, recall issues, etc. as they come up – do not lead Respondent]

• What would have caused you to answer “very burdensome”

• What would have caused you to answer “not at all burdensome”

• Any issues selecting between the different response options? If so, why?

• How might we improve this question?

The next question I asked was:

In the past six months [SINCE December 2014], how many surveys, polls, or other questionnaires have you been asked to complete?

• Again, what was your reaction when you heard this question?

• How did you arrive at an answer?

• Are there any words or terms in this question that you found confusing or difficult?

I followed up this question with one that asked:

How burdened would you say you feel by those requests?

• Can you tell me in your own words what you think this question is asking?

• Any problems answering this question – either in coming up with a response, or selecting one of the response categories?

• To what extent would you say that the amount of burden you felt in this survey was impacted by recent experiences with other surveys/survey requests?

I asked you a couple of questions about the length of the survey.

Compared to how long you expected this interview would take, would you say that the length of today’s survey was:

• Can you restate this question in your own words please? What did it mean to you?

• You said that today’s interview was XX – why is that?

• [Unless R replies “about as long as expected,” ask:]

  • How long did you anticipate the survey would last?

• To what extent does the fact that the interview was [longer/shorter] than you expected impact your overall reactions to the survey, how much effort you gave, etc.

• [If not obvious, ] Did the interview seem too long, too short, about right?

Do you feel that the pace of today’s interview was:

When I asked you about the pace of today’s interview, you said that it was XX.

• What made you say that? What made it seem XX?

• To what extent do you feel that the pace of the interview affects your overall reactions to the survey?

• Thinking both of the overall length and the pace of the expenditure survey, how much would you say these factors impact your reaction to the survey? How important is interview duration and pacing?

How enjoyable was this survey to you? / How interesting was this survey for you?

I asked you a couple of questions about how enjoyable and interesting the survey was to you.

• What were your reactions to these questions?

• What does “enjoyable” mean to you in this question? How about “interesting?”

• Any problems selecting one of the response options?

• To your mind, is there a clear distinction between “somewhat enjoyable” and “a little enjoyable” or “extremely enjoyable” and “very enjoyable”?

  • If not, why not?

  • How could we improve the response categories? (is it any clearer to have “not very enjoyable” [or interesting] vs. “somewhat enjoyable” [or interesting]?

  • Could you give me an example of a survey that was “extremely enjoyable” [or extremely interesting]?

How easy or difficult was it for you to answer the questions in the survey?

We talked a moment ago in general terms about how easy or difficult the survey was for you. When we asked you this question in our initial follow-up you said that it was XX [fill with Respondent’s answer].

• How did you arrive at this answer?

• Were there features of the survey, of your own expenditure situation, or the interview itself that made the survey more or less difficult? What could we have done to make it easier to answer the CEQ questions?

• Any problems with or suggested improvements for this question or the response options?

How capable do you feel you were to answer the questions in the survey?

• Could you tell me in your own words what this question is asking?

• How did you arrive at your answer – that you felt XX? What went into that determination (e.g., confidence in comprehension, knowledge of HH spending, general ability, ease of questions)?

• Any reactions to this question or its response options?

• Would it be better if we asked how confident you were in your ability to answer the questions in the survey?

• Other suggested changes?

Thinking about your level of effort, how hard would you say that you tried on this survey?

I next asked you about the amount of effort you put into answering the survey.

• What was your reaction to this question?

• What kinds of things did you think about when arriving at your answer? That is, did you recall specific instances from the interview, or did you answer based more an overall feeling about how you approached the survey, or how you approach activities like this in general?

• Did you feel that people will feel comfortable answering this question? If not, why not?

• How could we improve this question?

How important was it to you to do a good job answering today’s survey?

When I asked you this question, you said XX.

• How did you arrive at that answer?

• What does it mean to you to “do a good job answering” the survey?

• Did you feel comfortable answering this question? If not, how might we revise?

How motivated were you to participate in this survey?

• When you heard this question, what were your reactions?

  • Positive/negative emotions?

• Definitional issues – what does it mean to you to be “motivated to participate in this survey?”

• Was this question easy or difficult for you to answer? Why is that?

How sensitive did you feel the questions I asked today were?

We also asked you how sensitive you felt the questions were – you said XX.

• What came to mind as you were answering this question?

• What caused you to say XX? What if any specific questions did you think were sensitive?

• What sorts of items would you view as particularly sensitive on a federal government survey?

How tense or nervous did you feel when answering today’s survey questions?

• When you heard this question, what were your reactions?

  • Positive/negative emotions?

• Definitional issues – what does it mean to you to be “tense or nervous?”

• Was this question easy or difficult for you to answer? Why is that?

Generally speaking, how accurate do you think that government statistics are?

We asked several questions that were designed to get at your views of government statistics. We started by asking you for your impressions of their accuracy. You said that you think that government statistics generally are XX.

• Why do you say that? What leads you to believe this?

• To what extent do you think your impressions about the accuracy of government statistics are likely to influence your decisions to participate in government survey requests? How impact?

How important do you feel that surveys like this one are to research and government policy decisions:

You said that you felt that surveys like the CEQ are XX to research and policy decisions.

• Why do you say that? What leads you to believe this?

• What are your reactions to this question?

• Can you give me an example of a survey that does impact research OR government policy decisions? [Probe to determine how those terms are understood.]

• Suggested changes/improvements?

How beneficial do you feel this survey might be to you or your family personally?

• What does “beneficial” mean to you in this question?

• What would make a survey beneficial to you?

• Suggested changes/improvements?

If we had had to extend the length of this survey, how many more minutes would you be willing to continue the interview?

I also asked you how many more minutes you’d be willing to extend this survey, and you said XX.

• What caused you to say XX?

• We are interested in the things that can affect a person’s willingness to continue with a longer survey. What sorts of things would impact your decision to stick with a longer interview – either positively or negatively?

• Do the response options seem reasonable?

How willing would you be to take another survey like this in the future?

When I asked you how willing you would be to taking another survey like the Consumer Expenditure Survey, you said XX.

• How did you arrive at this answer?

• What factors would affect your decision about whether or not to participate in another survey like the CEQ?

• Do you think that your answer to this question is indicative of how likely you would actually be to say “yes” to another survey request like this? If not, why not?

If friends of yours were asked to participate in this survey, how likely would you be to encourage them to do so?

• What was your reaction to this question?

• Does the wording make sense? Any difficulties answering this question (if so, why)?

The last question I asked you was how much you trust the Census Bureau to protect your data:

To what extent do you trust the U.S. Census Bureau to keep your information confidential:

• What was your reaction to this question?

• What does that phrase mean to you: “keep your information confidential?”

• How did you come up with your answer? That is, what sorts of things impact the trust you have in Census Bureau’s (or BLS’s) ability to keep your data confidential?

Do you have any other comments, concerns, or questions about this set of questions that were designed to capture respondents’ reactions to the survey process?

• Are there other questions that you think we should be asking that might help us to form a more accurate understanding of respondents’ experiences?

• Any questions that we should drop or revise?

• Any issues with/suggested improvements for the order or flow of questions?