Form Appendix A-C Appendix A-C Survey Protocol

Appendices and Attachments

Appendix A: In-lab cognitive interviewing protocols

Appendix B: Online cognitive testing protocols

Appendix C: Informed Consent form and Privacy Act statement

Appendix A: In-Lab Cognitive Interviewing Protocol

Because these interviews are scheduled to occur in March, reference months have been filled as “December”.

The protocol includes the questions that will be tested, additional questions that provide context for the tested question, and debriefing questions. The goal is to identify the most effective wording for questions, response choices, and Information Book examples by understanding the associations that participants have with terms used in the question wording and how they interpret question meanings. The interviewer may ask unscripted follow-up questions (e.g., “Can you tell me more about that?”) if a participant’s answer suggests additional, pertinent information can be gained. Note that the survey questions and the debriefing questions are listed together in this document, but all survey questions will be asked first and all debriefing questions will be asked at the end of the interview.

Introductory Material

• Hi! Thank you for coming in today.

• I am …... [This is my colleague ( ) who will be taking notes for us today]

• Have you participated in any of our studies before? (if yes, Which ones?)

• [Sign Consent Form] (see Appendix C)

• Explanation:

  • We are going to be working with the Consumer Expenditure Survey, an ongoing survey that provides information about the buying habits of American consumers.

  • Our purpose in bringing you in today is to have you provide feedback on the survey - we’re trying to find out whether our survey questions are working as we intended and how hard they are to answer. We are not here to evaluate you, we are looking to improve the survey, so there are no right or wrong answers. When you tell us that you find something about the survey to be difficult to understand or do, then that tells us that we need to change the survey. All the information you give us will be kept completely confidential, and will be used to improve the survey.

• Any questions before we begin?

• The Consumer Expenditure Survey is a survey of all members of a household who live together so I am going to ask you to report expenses that you or anyone else in your household have made during the last three months. To get started I’d like to get some information about you and the people in your household.

• How many people are in your household, including yourself?

• [if more than 1 in HH] OK, and who is the next person? I don’t need their name, just how they are related to you.

Name	Relationship	Age	Gender
	Participant		M F
			M F
			M F
			M F
			M F

Next we’re going to go through a short version of the interview. Typically, when the Bureau of Labor Statistics interviews people for the Consumer Expenditure Survey, we go through lots of questions and it takes about an hour for just the interview itself. What we’re going to do today is much shorter – we’re just trying out some new questions, and I will leave out a lot of the survey – then we’ll go over what it was like for you to answer those questions.

As we start, please understand that we ask the same topics of everybody we talk to. I realize some of these questions may not apply to your household. Most questions that I will be asking refer to a specific time period. During this interview, the time period, unless I state otherwise is for the past three months, that is, from the 1st day of December to today. Most of my questions are about expenses your household had or bills you've received. It may be helpful to have your checkbook register, credit card statements and other records as you answer the questions but I understand if you do not have those with you for today’s interview.

Since the first of December, have you or any members of your household purchased any of the following items?

Read aloud each item on the list one at a time. If the participant has the expense, go to follow-ups; otherwise continue to next item on the list.

Small electrical kitchen appliances

Video game hardware or accessories

Musical instruments, supplies, or accessories

General sports equipment (exclude athletic shoes for sports related use, such as football, baseball, soccer or bowling)

Bicycles or bicycle equipment

Tricycles or battery powered riders

Playground equipment

Fresh flowers or potted plants?

Toys or games?

Arts or crafts kits?

Purchase of pets, pet supplies, or medicine for pets? Do not include pet food.

If yes to any item, immediately ask follow-up; repeat as needed.

How much did it cost?

Any other [fill]?

Now we are going to ask about payments and contributions to people outside of your household.

Since the first of December, have you or any members of your household given any money by cash, check, or gift card to

Read aloud each item on the list one at a time. If the participant has the expense, go to follow-ups; otherwise continue to next item on the list.

College students living away from home?

Any other people not in your household, such as friends, co-workers, or homeless persons?

Child support?

Alimony?
Educational institutions?

Political organizations?

Religious organizations, including churches, temples and mosques?

Charities or other organizations?

Stocks, bonds, or mutual funds to persons or organizations outside of your household?

If yes to any item, immediately ask follow-up; repeat as needed.

What was the total amount of the payment?

Any other [fill]?

Now please think about all other expenses for people outside of your household not already reported, including payments toward housing, education, and care, as well as expenses for gifts for souvenirs, holidays, and life events.

Since the first of December, have you or anyone in your household purchased any

Read aloud each item on the list one at a time. If the participant has the expense, go to follow-ups; otherwise continue to next item on the list.

Gifts of clothing, footwear, jewelry, or watches for people outside of your household?

Gifts of food for people outside of your household?

Other expenses or gifts for people outside of your household?

If yes to any item, immediately ask follow-up.

What was the total amount spent?

Now I am going to ask about expenses for men’s suits, as well as watches, jewelry, and clothing services.

Have you or any members of your household had expenses for any of the following?

Read aloud each item on the list one at a time. If the participant has the expense, go to follow-ups; otherwise continue to next item on the list.

Men’s suits

Watches

Jewelry

Repair, alteration or tailoring for clothing and accessories

Shoe repair or other shoe services

Watch or jewelry repair

Clothing or accessory rental

Clothing storage outside the home

If yes to any item, immediately ask follow-up; repeat as needed.

What was the total amount spent?

Any other [fill]?

Now I am going to ask about clothing and footwear expenses for infants, children, and adults. Please do not include any expenses already reported.

Since the first of December, how much have you and your household spent on

Read aloud each item on the list one at a time.

Men’s clothing?

Women’s clothing?

Boy’s clothing?

Girl’s clothing?

Infant’s clothing?

Footwear?

Debriefing

Earlier I asked you about expenses for clothing, food, and everything else that you had purchased to give to people outside of your household. You had said you had these expenses: For clothing you said $[amount], for food you said $[amount], and for all other categories, you said $[amount]. What expenses did you include in your answers to get to those totals? I don’t need details but just the basics, like “Toys as Christmas presents”, and we’ll total it up together.

Description of gift	CE Category	Cost
	Clothing/Food/Other gifts

Earlier I asked you about expenses for (ask each one at a time) “gifts of clothing, footwear, jewelry or watches for people outside your household”/“gifts of food for people outside your household”/“other expenses or gifts for people outside of your household”. What does that mean to you? Even if you didn’t have the expense yourself, what kinds of things do you think we are asking about?

Take a look at this list. Look at each item and mark whether the item would definitely be included in any of the categories or none of them. If you don’t know what an item is or are not sure whether it should be in the category, mark “not sure”

Give printed list and pencil to participant to mark responses

	Definitely IN a category:			Definitely OUT of all categories	Not Sure
	Clothing, footwear, jewelry, or watches	Food	Other expenses/gifts
Cash loan to a family member who doesn’t live with you
Paid for car maintenance for someone else
Paid rent for your kid in college
Bought housekeeping supplies for elderly parent who lives in another state
Bought house decorations for a friend who moved to a new home
Bought outfits for your sibling’s baby
Helped elderly parent with healthcare payments
Paid for grandchild’s daycare
Bought wine and beer to serve at a dinner party at your home
Bought a bag of chips and dip to take to a friend’s party
Took someone out for a meal at a restaurant
Helped with expenses for medical supplies for friend that got in an accident
Treated extended family to tickets to the movies or a sports game
Bought toys for nieces and nephews
Bought flowers for a funeral
Bought gift card for a thank you present

Does this list remind you of any gift or other expense for people outside of your household that your household had since the first of December but you forgot to include earlier?

If yes, record the following:

Description of gift	CE Category	Cost
	Clothing/Food/Other gifts

Can you tell me a bit about why these expenses came to mind now but not earlier?

If applicable:

So looking at all the gifts and expenses for others that you have reported, it looks like there is some overlap in the types of expenses, like you reported [toys; fill as appropriate] as a gift and I also asked you about [toys] during the earlier part of the interview when we were going through the long lists of items? Did your earlier reports include the expenses for gifts? Tell me a bit more about that.

Now I’m going to repeat the instructions that I read to you earlier in the interview when I asked you about gift expenses. Here goes:

Now please think about all other expenses for people outside of your household not already reported, including payments toward housing, education, and care, as well as expenses for gifts for souvenirs, holidays, and life events.

Is there any way to change these instructions to make them more useful, now that you’ve had practice reporting your expenses for others outside of the household?

Now I want to return to clothing, footwear, and accessories, for infants, children, and adults. Take a look at this list. Look at each item and mark whether the item would definitely be included in “clothing” for men, women, boys, girls and infants, or not. If you don’t know what an item is or are not sure whether it should be in the category, mark “not sure”.

Category: Clothing for Men, Women, Boys, Girls and Infants

	Definitely IN	Definitely OUT	Not sure
Coats, jackets, or snowsuits
Sport coats, tailored jackets, or blazers
Suits
Vests
Shirts, sweaters, blouses, or tops
Pants, jeans or shorts
Dresses
Skirts
Undergarments
Hosiery
Nightwear or loungewear
Accessories
Swimsuits or swim-wear
Uniforms, for which the cost is not reimbursed
Costumes
Footwear, including athletic footwear
Diapers
Layettes

Earlier, I asked you some very general questions to find out how much your household had spent on clothing for men, women, boys, girls, and infants, but now let’s go through some really detailed questions.

Since the first of December, have you or any members of your household purchased any of the following items?

Read aloud each item on the list one at a time. If the participant has the expense, go to follow-ups; otherwise continue to next item on the list.

Coats, jackets, or snowsuits

Sport coats, tailored jackets, or blazers

Suits

Vests

Shirts, sweaters, blouses, or tops

Pants, jeans or shorts

Dresses

Skirts

Undergarments

Hosiery

Nightwear or loungewear

Accessories

Swimsuits or swim-wear

Uniforms, for which the cost is not reimbursed

Costumes

Footwear, including athletic footwear

Diapers

Layettes

(if yes to any, ask follow-up; repeat as needed)

How much did it cost?

Any other [fill]?

If applicable, ask:

Great. So now let’s compare your responses to these questions and the earlier set of general questions I asked you. It looks like the totals don’t quite match up – we’ve got [more/fewer] expenses just now from the detailed questions than we got from the earlier set of general questions. Can you talk a bit about the ways you answered each of these sets of questions?

Appendix B: Online Testing Protocol

Because testing is scheduled to occur in March, reference months have been filled as “December”.

Potential participants will see the following information on mTurk:

Description:	We'll ask you to recall expenses you have made recently and answer a few follow-up questions.
Keywords:	memory, survey, research
Full Description:	We are conducting a study to improve the way we ask questions in a government survey. We need to understand whether the questions are clear and easy to understand. Select the link below to complete the survey. At the end of the survey, you will receive a code to paste into the box below to receive credit for taking our survey. Make sure to leave this window open as you complete the survey. When you are finished, you will return to this page to paste the code into the box.

If the participant clicks the link to the study, they will be forwarded to a screen on SurveyMonkey with the introductory text on the next page. If they choose to proceed, they will be asked several demographic questions, the expense questions, and the debriefing questions. Any text shown in italics will not be shown to the participant.

---page break---

Welcome! Thanks for your interest in our survey. You’re here because we have asked you to participate in our research. We are asking you and hundreds of other people to tell us about what they think.

Unlike some surveys or online tasks you may be familiar with, we ask that you complete this survey all at one time and that you only start once you are in a quiet place where you won't be disturbed. The survey takes about 10 minutes, on average. Only share information you're comfortable with - nothing too personal - but please be honest and follow the instructions.

 Please do not use your browser's back button. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: March 31, 2021). Without this approved OMB number, BLS would not be able to conduct this survey. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Survey Monkey and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

---page break---

As we start, please understand that we ask the same topics of everybody we talk to. We realize some of these questions may not apply to your household and that your responses will not be a complete picture of your household’s spending.

We’ll start by asking a few questions about your household’s expenses and then we will ask you some additional questions to better understand your answers.

Let’s get started!

---page break---

How many people live in your household and are related to each other or share major expenses, such as housing, food and other living expenses? Include children.

[number]

Please think about all of these people when reporting expenses for the rest of the survey.

---page break---

Since the first of December have you or any members of your household had any membership costs or other expenses related to any of the following?

Each item will be presented on its own page with the question stem, with follow-up questions immediately afterwards. Resume this list after the follow-up items.

(1) Health clubs, fitness centers, swimming pools, weight loss centers, or other sports or recreational organizations

(2) Fees for participating in sports such as golf, bowling, biking, hockey, football, or swimming

(3) Bike, electric scooter, or moped sharing?  

Yes

No

---page break---

Follow-up questions for any item that the participant says they had an expense for.

(If Yes to 1, 3) What is the name of the (health, fitness or recreation club; bike, electric scooter, or moped sharing company)?

[open text entry box]

(If Yes to 2) Describe the expense.

[open text entry box]

What was the total amount of this expense since the first of December?

[open text entry box]

Did you or anyone in your household pay any other fees for (health, fitness or recreation clubs; participating in sports; bike, electric scooter, or moped sharing)?

Yes

No

Collect up to 2 expenses per item

---page break---

Now I am going to ask about clothing and footwear expenses for infants, children, and adults.

Since the first of December, how much have you and your household spent on

Clothing for men? [numeric response]

Clothing for women? [numeric response]

Clothing for boys? [numeric response]

Clothing for girls? [numeric response]

Clothing for infants? [numeric response]

Footwear for everyone? [numeric response]

---page break---

During the past 12 months, did you or anyone in your household contribute any money to retirement plans such as 401(k)s or Individual Retirement Accounts, also known as IRAs? Do not include rollovers.

Yes

No

Not sure

(If yes)

How much?

[numeric response]

---page break---

Thanks for answering those questions. Now we’d like to ask for your feedback about the experience of answering those questions. Your answers to these questions will help us to improve the way we ask about retirement plans.

Take a look at this list. Look at each item and mark whether the item would definitely be included in the category or definitely not included in the category. If you don’t know what an item is or are not sure whether it should be in the category, mark “not sure”.

Category: Retirement Plans

	Definitely IN	Definitely OUT	Not Sure
401(k)
403(b)
IRA
Roth IRA
SIMPLE IRA
SEP IRA
SAR-SEP IRA
Payroll Deduction IRA
Employee Stock Ownership Plan (ESOP)
Keogh plan
529 plan
Profit sharing
Treasury bonds
Other – please specify

(random order)

---page break---

Do you or anyone in your household personally have any of the following?

Please select all that apply. Do not include any personal information.

401(k)

403(b)

IRA

Roth IRA

SIMPLE IRA

SEP IRA

SAR-SEP IRA

Payroll Deduction IRA

Employee Stock Ownership Plan (ESOP)

Keogh plan

Other – please specify

Does looking at this list remind you of any retirement plans that you forgot to consider earlier in the survey when we asked about how much your household paid into retirement accounts?

Yes

No

If you excluded any, please describe which ones and why.

[open ended text box]

---page break---

Did you find it easy or difficult to answer the original question earlier in the survey about retirement plans –

During the past 12 months, did you contribute any money to retirement plans such as Individual Retirement Accounts, also known as IRAs, or 401(k)s? Do not include rollovers.

Very easy

Somewhat easy

Neither easy nor difficult

Somewhat difficult

Very difficult

---page break---

(If somewhat or very difficult)

Please describe what made the question difficult to answer. We’ll use your feedback to try to make the question better.

[open text box]

---page break---

Now we’ll ask about your knowledge of and experience with bike, electric scooter, and moped sharing.

Take a look at this list. Look at each item and mark whether the item would definitely be included in the category or definitely not included in the category. If you don’t know what an item is or are not sure whether it should be in the category, mark “not sure”

Category: Bike, electric scooter, or moped sharing

	Definitely IN	Definitely OUT	Not Sure
Bike-share day pass
Bike-sharing annual membership fee
Single trip on an electric scooter
Purchase of a bike
Citi Bike
Hubway
Lime
Bird
Revel moped
E-scooter
Car2Go
ZipCar
Uber
Divvy
Capital Bikeshare
Wheel Fun Rentals

(random order)

---page break---

Did you or anyone in your household pay for any expenses for any of the following in the last three months?

Please select all that apply. Do not include any personal information.

Bike-share day pass

Bike-sharing annual membership fee

Single trip on an electric scooter

Citi Bike

Hubway

Lime

Bird

Revel moped

E-scooter

Divvy

Capital Bikeshare

Wheel Fun Rentals

Yes

No

Does looking at this list remind you of any expense you forgot to include earlier in the survey when we asked about how much your household had spent on bike, electric scooter, and moped sharing?

Yes

No

If you excluded any, please describe which ones.

[open ended text box]

Did you find it easy or difficult to answer the original question earlier in the survey –

Since the first of December have you or any members of your household had any membership costs or other expenses related to any of the following? Do not include contributions to or membership in religious, professional, business, or other tax deductible organizations.

Bike, electric scooter, or moped sharing or rental?

Very easy

Somewhat easy

Neither easy nor difficult

Somewhat difficult

Very difficult

---page break---

(If somewhat or very difficult)

Please describe what made the question difficult to answer. We’ll use your feedback to try to make the question better.

[open text box]

---page break---

Now let’s return to the category of clothing. Take a look at this list. Look at each item and mark whether the item would definitely be included in “clothing” for men, women, boys, girls or infants, or not. If you don’t know what an item is or are not sure whether it should be in the category, mark “not sure”.

Category: Clothing for Men, Women, Boys, Girls, and Infants

	Definitely IN	Definitely OUT	Not sure
Coats, jackets, or snowsuits
Sport coats, tailored jackets, or blazers
Suits
Vests
Shirts, sweaters, blouses, or tops
Pants, jeans or shorts
Dresses
Skirts
Undergarments
Hosiery
Nightwear or loungewear
Accessories
Swimsuits or swim-wear
Uniforms, for which the cost is not reimbursed
Costumes
Diapers
Layettes

---page break---

And now please use the table below to detail your household’s clothing expenses since the first of December.

Clothing item	Amount Spent
Coats, jackets, or snowsuits
Sport coats, tailored jackets, or blazers
Suits
Vests
Shirts, sweaters, blouses, or tops
Pants, jeans or shorts
Dresses
Skirts
Undergarments
Hosiery
Nightwear or loungewear
Accessories
Swimsuits or swim-wear
Uniforms, for which the cost is not reimbursed
Costumes
Footwear, including athletic footwear
Diapers
Layettes
Other clothing

---page break---

Does anyone in your household receive a clothing “subscription box”, or recurring delivery of clothing items, such as StitchFix or Trunk Club?

Yes

No

Not sure

---page break---

(If yes, ask follow-ups)

Did you include the cost of the subscription in your reports of clothing expenses?

Yes

No

Does the subscription company provide you with a cost for each item?

Yes

No

Not sure

---page break---

And now a few final questions about you.

What is your age in years?

[numeric text box]

What is your sex?

Male

Female

What is the highest level of school you have completed or the highest degree you have received?

No schooling completed

Elementary school diploma

High school diploma or the equivalent (GED)

Associate degree

Bachelor’s degree

Master’s degree

Professional or doctorate degree

Are you Hispanic or Latino?

Yes

No

What is your race?

White

Black or African American

American Indian or Alaska native

Asian

Native Hawaiian or Pacific Islander

---page break---

Thank you for participating in this survey! If you have any comments or feedback, please use the box below. Otherwise, please continue on to the next page for your mTurk validation code.

[open-ended entry box]

Appendix C: Consent Form

Consent Form

OMB Control Number: 1220-0141

Expiration Date: March 31, 2021

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act (44 U.S.C. 3572) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 60 minutes to participate in this research.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

The OMB control number for this information collection is 1220-0141 and expires March 31, 2021 Without this approved OMB number, BLS would not be able to conduct this survey.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974 as amended (5 U.S.C. 552a), this study is being conducted by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under the authority of 29 U.S.C. 2.  Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections.  The information will only be used by and disclosed to BLS personnel and contractors who need the information for activities related to improving BLS information collection.  Information on routine uses can be found in the system of records notice,  DOL/BLS – 14, BLS Behavioral Science Research Laboratory Project Files (81 FR 47418). 

Page 31 of 31