Gen IC Justification

1220-0141 Note to Reviewer - OMB CEQ 2017 Changes FINAL (3Feb16).docx

Cognitive and Psychological Research

Gen IC Justification

OMB: 1220-0141

Document [docx]
Download: docx | pdf


February 3, 2016




NOTE TO THE

REVIEWER OF:

OMB CLEARANCE 1220-0141

“Cognitive and Psychological Research”


FROM:

Brandon Kopp

Research Psychologist

Office of Survey Methods Research


SUBJECT:

Submission of Materials for the Pretesting of proposed changes for the CE Quarterly 2017




Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.


The total estimated respondent burden hours for this study is 123 hours.


If there are any questions regarding this project, please contact Brandon Kopp at

202-691-7514.


  1. Introduction and Purpose

The Bureau of Labor Statistics has completed several projects with the goal of improving the Consumer Expenditure Quarterly Interview Survey (CEQ). Every two years, the questions are evaluated by the program office and questions in need of revision are identified. Before the revisions are made, they are reviewed and tested by the Office of Survey Methods Research.

One of the main goals of these improvements is to keep the market basket of goods and services, which provides item weights for the Consumer Price Index (CPI), current. Given the rapid changes in the marketplace, changes need to be made to the CEQ to ensure that the data accurately reflect the state of consumer buying habits. Based on input from CPI commodity analysts and economists, a few areas of the survey were identified as needing attention to maintain the most up-to-date market basket possible.

In a parallel effort to improve CEQ data quality, economists from the Branch of Production and Control (P&C) identified changes that need to be addressed for reasons of respondent burden, outdated questions, and general instrument issues.

In addition to these recommendations, independent teams within the Bureau of Labor Statistics (BLS) evaluated products and services which are becoming more common and made recommendations for changes to the questionnaire to improve reporting for these areas.


2. Research Design

The purpose of this study is to ensure that proposed changes to the 2017 CE Quarterly Interview instrument and information booklet are clear and understandable to respondents, and are interpreted by respondents as intended. A complete list of the question revisions to be tested is included in Appendix A.

To accomplish this goal, question variations will be tested using a mixture of in-lab cognitive interviewing and online testing. Because the number of proposed changes exceeds what can be tested in a single session, the full list of items is broken down into modules of related CEQ items and associated debriefing questions that will be administered together. The set of questions (modules) presented to each participant and their order will vary. Participants will be assigned modules based on their experience with and knowledge of the subject matter, as determined by self-report. Although the content for each participant will vary, the length of the protocol will be fixed for all participants. The full protocol is shown in Appendix B.

As noted, cognitive interviewing is one technique that will be used to gain insight into the topics addressed in the changes. Cognitive interviews provide an in-depth understanding of the respondent’s thought processes and reactions to the questions. These interviews will be done one-on-one in the Office of Survey Methods Research (OSMR) laboratory, and will be audiotaped. Interviews will be conducted by staff from the OSMR who are experienced in conducting these types of interviews. The testing protocol is included in Appendix B, Part 1.

Several of the CEQ changes will be tested using a sample of participants recruited through Amazon’s Mechanical Turk (mTurk) service. These participants will complete an online form which asks them the survey questions of interest and then several debriefing questions, including questions to probe the accuracy of their response to the CEQ questions and about the experience of answering those questions. This online testing is an unmoderated cognitive interview. The benefit of online testing is that feedback can be gained from a larger number of participants that are more diverse geographically and in terms of their expenditures than would be possible in the OSMR research lab in Washington, DC. This is especially helpful for testing changes to expenditure questions in which there is a low incidence rate (e.g., finding participants who drive a “flex-fuel” vehicle). The testing protocol for online testing is included in Appendix B, Part 2. The findings from both modes of testing will be evaluated qualitatively. For the online testing, quantitative analyses will be considered but statistical significance is not a goal. Rather, the debriefing feedback from both modes is the primary outcome to be used to revise question wordings.

As this is pretesting, we expect that modifications may be made during the course of the study based on initial results. Although the goals of the testing, and overall design, will remain the same, findings from preliminary results may be used to improve the interview questions. Modifications are likely to range from slight changes to question wording (e.g., naming examples of the expense category) or additional instructions (e.g., inclusion or exclusion instructions).


3. Participants

Cognitive Interviews. Thirty-five participants will be recruited from a participant database maintained by OSMR. Efforts will be made to select participants with varying levels of education, income, and occupation, and familiarity with the expenditures being tested based on self-reported information provided during the initial recruitment process.


Online Testing. Seven hundred participants will be recruited using a convenience sample from Amazon Mechanical Turk of adult U.S. citizens (18 years and older).


4. Burden Hours

The total burden hours for this study will be 123.

Cognitive Interviews. The number of participants needed for testing a given CEQ question is dependent upon the results and the point at which saturation is reached and no further information is being gained from additional interviews. Given that, our goal is to interview no more than 35 participants in the OSMR lab. We anticipate that each session will average 45 minutes and last no longer than one hour (or 35 burden hours). Participants will be asked several screening questions about their recent expenditures as part of recruitment (see Appendix B, Part 1). Screening questions and scheduling an appointment is expected to take no longer than 10 minutes and participants will not be compensated for time spent on scheduling. We anticipate a high qualification rate for the screening questions (approximately 90%), so estimate we will need to contact approximately 40 people in order to get 35 interviews. This will add an additional 6.7 burden hours (40 participants X .1667 hours) to the study, for a total of 41.7 burden hours for cognitive interviews.


Online Testing. Each of the seven revisions that will be tested online will be treated as an independent module (see Appendix B, Part 2 for details). Our goal is to obtain feedback from 100 participants for each module. We anticipate that the task will take no longer than 7 minutes, including screening, demographics, target questions, and debriefing questions for a total of 81.7 burden hours (7 modules X 100 participants X 0.1167 hours).


5. Payment to Participants

A total of $2100 will be spent on payments to participants.

Cognitive Testing. The 35 participants will each be paid $40 for their participation in testing for a total of $1400.


Online Testing. Each of the 700 participants will be paid $1.00 for participating in the study, a typical rate provided by Mechanical Turk for similar tasks; a total of $700 will be paid directly to Amazon Mechanical Turk for participant fees.


6. Data Confidentiality

Cognitive Interviews. Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of a national Consumer Expenditure interview survey. Participants will be given a consent form to read and sign (See Appendix C). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.


Online Testing. Recruiting of participants will be handled by Amazon Mechanical Turk. Once participants are recruited into the study, they will be provided a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics’ servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.


Appendix A: List of Changes to be Tested

Each of the following proposed changes to the CEQ will be tested using cognitive interviews and/or online testing, as specified below. Below is a list of changes to be tested in each mode. Notice that several of the revisions listed under cognitive interviews are also being tested under online testing.

Cognitive Interviews

  1. Add gateway questions for whether respondent has retirement accounts or checking/savings accounts.
    Add follow-up questions for when participants report having $0 in their accounts a year ago to ask whether they had an account a year ago.

  2. Remove “major” from credit card questions. Same additional question about presence of accounts one year ago.

  3. Add question for internet service away from home.
    Reword telephone bill questions to include internet, cable TV, and satellite TV.

  4. Ask for expenditure amount as soon as possible after “did you have an expense for ___?” gateway question.
    Reword questions related to book purchases.
    Combine questions related to rental of DVDs and streaming video subscriptions.
    Test for context effects when satellite radio service question is moved to subscription service section.
    Change order of photographic film and photo printing questions.

  5. Major changes to Health Insurance section.

  6. Changes to the mass transit questions in Expense Patterns section.

  7. Add questions related to the assessment of respondent burden.



Online Testing

  1. Reword alternative fuel vehicle and alternative fuel consumption questions.

  2. Reword mortgage origination date questions.

  3. Reword questions related to book purchases.

  4. Combine questions related to rental of DVDs and streaming video subscriptions.

  5. Change order of photographic film and photo printing questions.

  6. Test for context effects when satellite radio service question is moved to subscription service section.
    Change satellite radio service to Sirius/XM satellite radio service
    Update subscription music service examples

  7. Determine best examples for including e-cigarettes in the Information Booklet.



Appendix B, Part 1: Cognitive Testing Protocol

Cognitive Interview Screening Questions


Participants for this study will be recruited from a database maintained by OSMR. Several additional questions will be asked to ensure that at least some of the participants who come in to the lab will have experience with topics asked about in the study. With the questions below, we are interested in finding participants who have some sort of government administered healthcare or a special healthcare plan like dental or eyecare insurance. We are also interested in those that have had expenses for public transportation (which should be a large proportion of our DC area participant pool). Our recruiter will ensure that we have at least 10 participants that meet each of these requirements. These groups can overlap. For example, a participant who has a dental insurance plan and takes public transportation would meet two of these requirements. Once these recruiting needs are met, these questions will no longer be asked of interested participants.


OMB Control Number: 1220-0141

Expiration Date: 04-30-2018


Thank you for calling in. I am recruiting for a study by the Bureau of Labor Statistics that asks people to recall purchases that they have recently made and then answer some follow-up questions about those purchases. We need to talk to people who have had certain types of expenses during the last three months. I have a few questions that will help us see if you are eligible to participate in this study. OK?


Shape1
  1. Do you or anyone in your household have any of the following types of health insurance?

Shape2

10 who say yes to at least one in this group

    • Medicare

    • Medicaid

    • SCHIP (State Child Health Insurance Program)

      Shape3

      10 who say yes to one in this group

    • VA (Veterans Administration)

    • Tricare

      Shape4
    • Dental Insurance

    • Eyecare Insurance?


Shape5

10 who say yes to one in this group

  1. Do you pay out-of-pocket for travel or commuting on public transportation?

    • Yes

    • No



If participant meets eligibility requirements:

You are eligible to participate in this study. Let’s get you scheduled.

If participant does not meet eligibility requirements:

I’m sorry but you are not eligible to participate in this study. We will keep you in mind for future studies. Thank you for your time.

If the person requests to know why they were not eligible. The recruiter will explain the eligibility requirements.


According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 10 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to the U.S. Department of Labor, OASAM/OCIO 200 C Street NW Washington DC 20210 and reference the OMB Control Number 1220-0141.

The protocol includes the questions that will be tested, additional questions that provide context for the tested question, and debriefing questions. The goal is to identify the most effective wording for questions and response choices by understanding the associations participants have with terms used in the question wording and how they interpret question meanings. The interviewer may ask unscripted follow-up questions (e.g., “Can you tell me more about that?”) if a participant’s answer suggests additional, pertinent information can be gained. Note that the survey questions and the debriefing questions are listed together in this document, but all survey questions will be asked first and all debriefing questions will be asked at the end of the interview.


Introductory Material

OMB Control Number: 1220-0141

Expiration Date: 04-30-2018


  • Hi! Thank you for coming in today.

  • I am …... [This is my colleague ( ) who will be taking notes for us today]

  • Have you participated in any of our studies before? (if yes, Which ones?)

  • [Sign Consent Form/Permission to audiotape] (see Appendix C)

  • Explanation:

    • We are going to be working with some questions from the Consumer Expenditure Survey, an ongoing survey that provides information about the buying habits of American consumers. This information is used for a lot of things but its primary purpose is to help calculate inflation. Are you familiar with the inflation rate?

    • What we are going to do today is go through a few questions from the survey and get your reactions to them. The purpose of this session is to help us find out more about how people respond to these questions. Basically, we’re trying to find out what you think a question is asking and how hard it is to answer. We are not here to evaluate you, we are looking to improve the questions, so there are no wrong answers. All the information you give us will be kept completely confidential, and will be used to improve the survey questions.

  • Any questions before we begin?

  • The Consumer Expenditure Survey is a household survey so I am going to ask you to report expenses that you or anyone else in your household have made during the last three months. To get started I’d like to get some information about you and the people in your household.

  • How many people are in your household, including yourself?

Name

Relationship

Age

Gender


Participant


M F




M F




M F




M F




M F

Module A

Revisions to be tested:

  • Add gateway questions for whether respondent has retirement accounts or checking/savings accounts.

  • Add follow-up questions for when participants report having $0 in their accounts a year ago to ask whether they had an account a year ago.


Protocol:

The next few questions are about financial assets, credit, and loans. We know people aren't used to discussing their debt and financial assets, but we use this information to get a picture of how spending relates to changes in debt and savings. Be assured that, like all other information you have provided, these answers will be kept strictly confidential.


NEW QUESTION

Do ^YOU_ANYMEM have any retirement accounts such as 401(k)s, IRAs, Thrift Savings Plans?

  • Yes

  • No


As of TODAY --

What is the total value of all retirement accounts such as 401(k)s, IRAs, Thrift Savings Plans that ^YOU_YRCU ^own_owns?


What was the total value of all retirement accounts ONE YEAR AGO TODAY?

  • If $0, ask followup


NEW QUESTION

Did ^YOU_ANYMEM have any retirement accounts such as 401(k)s, IRAs, Thrift Savings Plans ONE YEAR AGO TODAY?

  • Yes

  • No

________________________________________

Stocks & Bonds


Do ^YOU_ANYMEM have any directly-held stocks, bonds, or mutual funds ^Retirement?

  • Include U.S. savings bonds

  • Yes

  • No


As of TODAY --

What is the total value of all directly-held stocks, bonds, and mutual funds ^Retirement that ^YOU_YRCU ^own_owns?

  • Include US savings bonds


What was the total value of all directly-held stocks, bonds, and mutual funds ^Retirement ONE YEAR AGO TODAY?

  • Include US savings bonds

  • If $0, ask followup



NEW QUESTION

Did ^YOU_ANYMEM have any directly-held stocks, bonds, or mutual funds ^Retirement ONE YEAR AGO TODAY?

  • Include U.S. savings bonds

  • Yes

  • No

________________________________________

Checking/Savings Accounts


NEW QUESTION

Do ^YOU_ANYMEM have any checking, savings, money market accounts, or certificates of deposit or CDs?

  • Yes

  • No


As of TODAY --

What is the total value of all checking, savings, money market accounts, and certificates of deposit or CDs ^YOU_YRCU ^have_has2?


What was the total value of all checking, savings, money market accounts, and certificates of deposit or CDs ONE YEAR AGO TODAY?

  • If $0, ask followup


NEW QUESTION

Did ^YOU_ANYMEM have any checking, savings money market accounts, or certificates of deposit or CDs ONE YEAR AGO TODAY?

  • Yes

  • No

________________________________________

Annuities/Trusts


^DoDoes_C ^YOU_YRCU have any other financial assets, such as annuities, trusts, and royalties?

  • Yes

  • No


As of TODAY --

What is the total value of these other financial assets?


What was the total value of these other financial assets ONE YEAR AGO TODAY?

  • If $0, ask followup


NEW QUESTION

Did ^YOU_YRCU have any other financial assets, such as annuities, trusts, and royalties ONE YEAR AGO TODAY?

  • Yes

  • No

[Monitor participants’ answers and probe if there is any apparent difficulty (e.g., “You had trouble with ____. Can you tell me more about that?”)]


Debriefing Questions:

  1. Do you have any accounts that we missed? I don’t need to know how much is in those other accounts, I’m just interested in what type of account they are.

  2. [if no to having an account one year ago today] When answering the earlier questions, you reported that you did not have [type of account] one year ago today. When did you open that account?



Module B

Revisions to be tested:

  • Remove “major” from credit card questions. Same additional question about presence of accounts one year ago.

MODIFIED QUESTION

^DoDoes_C ^YOU_YRCU have any credit cards including store cards and gas cards?

  1. Yes

  2. No


What is the total amount owed on all cards as of TODAY?


What was the total amount owed on all cards ONE YEAR AGO TODAY?

  • If $0, ask followup


NEW QUESTION

Did ^YOU_YRCU have any credit cards including store cards and gas cards ONE YEAR AGO TODAY?

  • Yes

  • No

________________________________________

Student Loans


As of TODAY --

^DoDoes_C ^YOU_YRCU have any student loans?

  1. Yes

  2. No


What is the total amount owed on all student loans?


What was the total amount owed on all student loans ONE YEAR AGO TODAY?

  • If $0, ask followup


NEW QUESTION

Did ^YOU_YRCU have any student loans ONE YEAR AGO TODAY?

  • Yes

  • No


Debriefing Questions:

  1. One of the questions I asked you referred to “any credit cards including store cards and gas cards.” What are some examples of that category? Any others? [Try to get at least 5 examples]

  2. And what about the term “major credit cards”. What are some examples of that category? Name whatever comes to mind, even if it overlaps with what you told me before.

  3. Do you have any of those cards?


Module C

Revisions to be tested:

  • Add question for internet service away from home.

  • Reword telephone bill questions to include internet, cable TV, and satellite TV.

Now I am going to ask about utilities including telephone bills.


Report any bill you have received, even if the bill has not been paid. First, I'll ask you about telephone, internet, cable TV, and satellite TV bills.

1. Enter 1 to Continue


MODIFIED QUESTION

Since the first of (reference month), have (you/you or any members of your household) received any bills for telephone, internet, cable TV, or satellite TV services? Do not include bills entirely for business purposes.

  • Yes

  • No


What is the name of the company which provides the service?


How much were you billed for in (reference month)? Do not include any unpaid charges from a previous billing period.


MODIFIED QUESTION

Which of the following service items were included in the bill(s) -

  • Read each item on list.

  1. Telephone service (mobile, residential, or Voice over IP)?

  2. Internet service (including broadband, DSL, and dial-up)?

  3. Cable or satellite television service?


MODIFIED QUESTION

Did (you/you or any members of your household) receive any other bills for telephones, internet, or cable or satellite television not used entirely for business purposes?

  • Yes

  • No


NEW QUESTION

Since the first of (reference month) not including this month, have (you/you or any members of your household) paid for internet or wireless internet access away from home, such as at cafés, hotels, or airplanes?

  • Yes

  • No


What has been the total cost of internet service away from home since the first of (reference month)?


Debriefing Questions:

  1. I asked about telephone, internet, and television bills. You said you had received bills for [categories for which they received bills]. Is that correct?

  2. Is that all of the expenses you had for telephone? For Internet? For cable or satellite TV?

    1. [if No] Please explain?

  3. [if not already mentioned during questioning] Are any of these expenses combined?

  1. [if no expense for internet away from home] Did you travel anywhere since the first of (reference month)? Did you use any public internet sources during that time? Did you pay for any of those sources?

  2. [if no expense for internet away from home] You said that you did not have any expenses for internet away from home. Have you accessed the internet away from your home since the first of (reference month)?

    1. [if yes] And there was no charge for that?


Module D

Revisions to be tested:

  • Ask for expenditure amount as soon as possible after “did you have an expense for ___?” gateway question.

  • Reword questions related to book purchases.

  • Combine questions related to rental of DVDs and streaming video subscriptions.

  • Test for context effects when satellite radio service question is moved to subscription service section.

  • Change order of photographic film and photo printing questions.

Since the first of the reference month, have you or any members of your household purchased or rented any of the following items?

* Read each item on list

  1. Small electrical kitchen appliances

  2. Electrical personal care appliances

  3. Electric floor cleaning equipment


What did you purchase or rent?


Was this item

  1. Purchased for someone inside the household?

  2. Rented?

  3. Purchased for someone outside the household?


What did it cost?


Did this include sales tax?

  • Yes

  • No


When did you purchase it?


Any other expenses for [ ]?

[if yes] Loop through follow-up questions


________________________________________

Furniture


Since the first of the reference month, have you or any members of your household purchased for you or your household or for someone outside of your household any -

* Read each item on list

  1. Sofas?

  2. Living room chairs?

  3. Mattresses or box springs?


What did you purchase?


What was the purchase price?


Did this include sales tax?

  • Yes

  • No


In what month did you purchase it?


Was this purchased for your household or for someone outside of your household?

  • For use by household

  • For someone outside the household



________________________________________

Clothing


Since the first of the reference month, have you or has any member of your household purchased any of the following items, either for members of your household or for someone outside your household?

* Read each item on list.

  1. Coats, jackets or furs

  2. Pants, jeans, or shorts

  3. Shirts, blouses, or tops


What did you buy?


How much did it/they cost?


Did this include sales tax?

  • Yes

  • No


Was this (were these) purchased for someone inside or outside of your household?

  • Inside your household

  • Outside your household

  • Both inside and outside household


For whom was this purchased?

* Enter all age/sex categories that apply to the purchase, separate with commas.

  • Male 16 and over

  • Female 16 and over

  • Male 2-15

  • Female 2-15

  • Children under 2 years old


When did you purchase it/them?


________________________________________

Entertainment


Since the first of reference month have you purchased any of the following items for your household or for someone outside your household?

* Read each item on list.

  1. Books, digital books, or book subscriptions. Do not include school books or reference books.

  2. Rented video files or DVDs or streaming video subscriptions

  3. Satellite Radio Services

  4. Photo printing or processing

  5. Photographic film or disposable cameras


* Ask if not apparent. Describe the item.


What was the total amount of this expense?


Was this purchased for your household or someone outside your household?

  • For household

  • For someone outside your household


In what month did you have this expense?

[Monitor participants’ answers and probe if there is any apparent difficulty (e.g., “You had trouble with ____. Can you tell me more about that?”)]


Debriefing Questions:

  1. [if book expense] You said that you purchased books, digital, books, or book subscriptions. Can you tell me more about those purchases?

  2. One of the questions I asked referred to “book subscriptions.” What does “book subscription” mean to you?

  3. Do you have, or have you ever had, any book subscriptions?

    1. [if yes] Tell me more about that?

  4. Have you purchased any books through a book club?

  5. [if rented video expense] You said you had an expense for “rented video files or DVDs or streaming video subscriptions.” What businesses or services did you use for those?

  6. Are there any video files or streaming video subscriptions that we didn’t talk about? How about any DVD rentals?

  7. Do you think of satellite radio services as a vehicle expense, a utility expense, an entertainment expense, or a subscription expense?

  8. Did you have any expenses related to photography since the first of (reference month)?

  9. Did you print any photos in the last 3 months?

    1. [if no] When was the last time you had photos printed?



































Module E

Revisions to be tested:

  • Major changes to Health Insurance section.

The next questions are about health insurance. Include health insurance obtained through employment or purchased directly as well as government programs like Medicare and Medicaid that provide Medical care or help pay medical bills.


NEW QUESTION

[fill: Are you/Are any members of your household] covered by any kind of health insurance or some other kind of health care plan? Include policies that you pay for on behalf of someone outside your household.

  • Yes

  • No


NEW QUESTION

What kind of health insurance or health care coverage [do you/does your household] have or pay for? INCLUDE those that pay for only one type of service; for example, nursing home care, vision care, or dental care. EXCLUDE private plans that only provide extra cash while hospitalized.

  • Read each item on list

  • Enter all that apply, separate with commas

  1. Private health insurance, including employer and healthcare exchange plans?

  2. Medicare?

  3. Medi-Gap?

  4. Medicaid^MDCDSTfill?

  5. SCHIP, also known as Children’s Health Insurance Program?

  6. Military health care, also known as Tricare, VA, or CHAMP-VA?

  7. Indian Health Service?

  8. Single service plan, such as dental, vision, long-term care?

  9. No coverage of any type


NEW QUESTION

[if 9 selected & AGE > 65] [Are you/Is anyone in this household] covered by Medicare?

  • Yes

  • No


NEW QUESTION

[if 9] There is a program called Medicaid that pays for health care for persons in need. In this state it is also called [state name]. [Are you/Is anyone in this household] covered by Medicaid?

  • Yes

  • No


NEW QUESTION

[if 8 not selected] [Do you/Does anyone in your household] have or pay for a separate insurance plan that pays for only one type of service such as dental, vision, or long-term care?

  • Yes

  • No


What type of service or care does your single service plan pay for?

  1. Dental care

  2. Long-term care

  3. Prescriptions

  4. Vision care

  5. Other (specify)

[Monitor participants’ answers and probe if there is any apparent difficulty (e.g., “You had trouble with ____. Can you tell me more about that?”)]


Debriefing Questions:

  1. [if health insurance] I asked you what kind of health insurance or health care coverage you and the people in your household have and you said you have [PARTICIPANT’S ANSWERS]. How did you go about answering this question?

  2. Did you answer only for yourself or for your whole household? Do the other people in your household have the same type of insurance or something else?

    1. [if something else] What type of coverage do they have?

  3. Do you pay for the health care of anyone outside your household, perhaps a member of your family?

  4. Did you include coverage for your household, even if you don’t pay for it?



Module F

Revisions to be tested:

  • Changes to the mass transit questions in Expense Patterns section.

MODIFIED QUESTION

Do ^YOU_ANYMEM pay out-of-pocket for public transportation services such as bus, subway, train, or passenger ferry?

  • Include all forms of public transportation

  • Yes

  • No


NEW QUESTION

What are the names of all household members who use public transportation?

  • Enter line numbers for all that apply, separate with commas

  1. Person 1

  2. Person 2


NEW QUESTION

What is ^NAME’s usual out-of-pocket cost?

  • Select time period (per day, week, or month) on next screen


  • Select time period

  • Day

  • Week

  • Month


NEW QUESTION

[if expense is Daily] How many days per week usually?


NEW QUESTION

Is any of the $[fill: amount] per [fill: time period] for Amtrak?

  • Yes

  • No


NEW QUESTION

[if yes] How much?

  • Enter amount per [fill: time period]


MODIFIED QUESTION

Did ^NAME receive a transit subsidy?

  • Yes

  • No



[if yes] What is the usual monthly amount?


  1. What types of transit did you consider?

  2. What time period were you thinking of when answering the question?

  3. Did you include transit taken for work and for leisure?

  4. Did you include any expense for transit while you were out of town on a trip?

  5. How accurate do you think you were in reporting expenses for yourself?

  6. [if HHSize > 1] How accurate do you think you were in reporting expenses for the rest of your household?

  7. [if expense] How often do you (and your family members) use each type of public transportation?

  8. [if subsidy] Do you receive you subsidy monthly or on some other time period?


Module G

Revisions to be tested:

  • Add questions related to the assessment of respondent burden.

Next I will ask you a few questions about your experience taking this survey. Be completely honest. The important thing is to improve the survey for others and your honest feedback will help me to do that.


NEW QUESTION

How burdensome was this survey to you? Would you say:

  • Not at all burdensome

  • A little burdensome

  • Somewhat burdensome

  • Very burdensome

  • Extremely burdensome


NEW QUESTION

How difficult was it for you to answer the questions in this survey?

  • Not at all difficult

  • A little difficult

  • Somewhat difficult

  • Very difficult

  • Extremely difficult


NEW QUESTION

How sensitive did you feel the questions I asked today were?

  • Not at all sensitive

  • A little sensitive

  • Somewhat sensitive

  • Very sensitive

  • Extremely sensitive


NEW QUESTION

In thinking about the length of today’s survey, would you say that it was…

  • Very short

  • Somewhat short

  • Neither short nor long

  • Somewhat long

  • Very long

Debriefing Questions:

  1. If a friend told you they were going to participate in this study next week, how would you describe it to them?

  2. Earlier, I asked how burdensome this survey was to you and you said [ANSWER]. What does “burdensome” mean to you?

  3. How did you arrive at your answer to this question about burden?

  4. [if not at all burdensome] Do you think others would find this survey burdensome? Tell me more about that.

  5. What are activities that you would rate as not at all burdensome? As extremely burdensome?

  6. [if > not at all difficult] You also said that the questions were [ANSWER] difficult to answer. In what ways were the questions [ANSWER] difficult? Which questions were you thinking of?

  7. [if > not at all difficult] Is there another task that you’ve done that you would rate as [ANSWER] difficult?

  8. How did you arrive at your answer to this question about difficulty?

  9. [if > not at all sensitive] Which questions did you find to be sensitive?

  10. One of the questions I asked was about the length of today’s survey you said it was [ANSWER]. How did you arrive at your answer to this question?



Appendix B, Part 2: Online Testing Protocol

For online testing, debriefing questions will focus on assessing the accuracy of responses to the target questions. The goal is to monitor the responses from a large number of online participants to detect irregular responses. Each of the revisions will be listed on mTurk as a separate study (these studies are referred to as modules below) with an initial screening question. The screening criteria for the study will be listed in the study invitation. Potential participants will see the following information on the mTurk site.

OMB Control Number: 1220-0141

Expiration Date: 04-30-2018

Description:

We'll ask you to recall expenses you have made recently and answer a few follow-up questions about those expenses.

Keywords:

memory, survey, research

Qualifications Required:

[THIS SECTION WILL CONTAIN INFORMATION INCLUDED AS SCREENING QUESTIONS BELOW. FOR EXAMPLE, MODULE B WILL SAY…]

Participants in this study must have refinanced the mortgage on their current home.

If they click the link to the study, they will be asked the initial screening question.

If the participant’s answer fits the screening criteria for that module, they will be forwarded to a screen with the introductory text on the next page. If they choose to proceed, they will be asked several demographic questions, the focal questions for that module, and then the debriefing questions. If the participant’s answer does not fit the screening criteria, they will see the following text.


Thank you for your time, but you do not meet the qualifications for this study.


One hundred participants will complete each module. In two cases, Modules A and E, two alternatives are being tested. In these cases, 50 participants will be randomly assigned to one version of the question and 50 will be assigned to the alternative version.





Welcome! Thanks for your interest in our survey. You’re here because we have asked you to participate in our research. We are asking you and hundreds of other people to tell us about what they think.


Unlike some surveys or online tasks you may be familiar with, we ask that you complete this survey all at one time and that you only start once you are in a quiet place where you won't be disturbed. The survey takes about 7 minutes, on average. Only share information you're comfortable with - nothing too personal - but please be honest and follow the instructions.

 Please do not use your browser's back button. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this study.

---page break---


On the following pages, you’ll be asked to answer questions about expenditures you may or may not have made over the last three months. We will then ask you some additional questions to better understand your answers.


Let’s get started!


---page break---



We’ll start by asking a few questions about you and your household.



What is your age in years?


What is your gender?

  • Male

  • Female


What is the highest level of school you have completed or the highest degree you have received?

  • Grades 1-12/No Diploma

  • High School Diploma or Equivalent

  • Some College

  • Associate’s Degree

  • Bachelor’s Degree

  • Master’s Degree

  • Doctorate or Professional Degree


Are you Hispanic or Latino?

  • Yes

  • No


What is your race?

  • American Indian or Alaska Native

  • Asian

  • Black or African-American

  • Native Hawaiian or Other Pacific Islander

  • White


How many people live in your household, including yourself?

  • 1

  • 2

  • 3

  • 4

  • 5 or more


[if HHSize > 1] How many of those people are under the age of 18?

  • 1

  • 2

  • 3

  • 4

  • 5 or more


---page break---



Module A

Revisions to be tested:

  • Reword alternative fuel vehicle and alternative fuel consumption questions.

Screening Question:


Do you or anyone in your household own one of the following vehicle makes/models?

  • Chevy

    • Impala

    • Lacrosse

    • Silverado

    • Spark

    • Suburban

    • Tahoe

    • Volt

  • Dodge

    • Dart

    • Durango

    • Ram

  • Ford

    • Explorer

    • F-150/250/350

    • Focus

    • C-MAX Hybrid

  • Honda

    • Accord Hybrid

    • Civic

    • Clarity

    • CR-Z

    • Fit

  • Jeep

    • Cherokee

    • Grand Cherokee

  • Nissan

    • Leaf

  • Toyota

    • Avalon

    • Highlander

    • Prius

  • Volkswagen

    • Jetta Hybrid


  • Yes

  • No

Do you or anyone in your household own any automobiles, trucks, minivans, vans or SUVs that are not used entirely for business?

  • Yes

  • No


What is the model year of the vehicle?


What is the make of this vehicle?


What is the model of this vehicle?


Is this vehicle an automobile or a truck, minivan, van, or SUV?

  • Automobile

  • Truck, van, minivan, or SUV


MODIFIED QUESTION

Is it fueled by –

  • Gasoline only?

  • Diesel fuel?

  • Hybrid electric/gasoline?

  • Electric only?

  • FlexFuel/E85?

  • Other – specify

ALTERNATIVE MODIFIED QUESTION

What types of fuel can this vehicle use –[Mark all that apply]

  • Gasoline

  • Diesel

  • Electricity

  • E85 Ethanol (also known as Flexfuel)

  • Something Else – specify


Are there any other vehicles in the household?

[if yes] loop through all household vehicles


Since the first of (reference month) not including this month –

What has been your household’s AVERAGE MONTHLY expense for gasoline and all other fuels for all vehicles?


Was any of this expense for diesel fuel?

  • Yes

  • No


How much?


NEW QUESTION

Was any of this expense for E85 fuel?

  • Yes

  • No


NEW QUESTION

How much?


Was any of the AVERAGE MONTHLY cost counted as a business expense?

        • Yes

        • No


What percentage of the AVERAGE MONTHLY COST was counted as a business expense?

  • Less than 10%

  • 10% - 20%

  • 21% - 30%

  • 31% - 40%

  • 41% - 50%

  • 51% - 60%

  • 61% - 70%

  • 71% - 80%

  • 81% - 90%

  • 91% - 100%


NEW QUESTION

Since the first of (reference month) not including this month –

Have you or any member of your household paid for electric vehicle charging away from home?

  • Yes

  • No


NEW QUESTION

How much?


NEW QUESTION

Was any of this amount counted as a business expense?

  • Yes

  • No


NEW QUESTION

What percentage of [fill: Amount of electric expense] was counted as a business expense?

  • Less than 10%

  • 10% - 20%

  • 21% - 30%

  • 31% - 40%

  • 41% - 50%

  • 51% - 60%

  • 61% - 70%

  • 71% - 80%

  • 81% - 90%

  • 91% - 100%


Debriefing Questions:

Questions will only be asked of those who report owning a vehicle

  1. One of the questions we asked you was:

Is it fueled by –

    • Gasoline only?

    • Diesel fuel?

    • Hybrid electric/gasoline?

    • Electric only?

    • FlexFuel/E85?

    • Other – specify


What types of fuel can this vehicle use –[Mark all that apply]

  • Gasoline

  • Diesel

  • Electricity

  • E85 Ethanol (also known as Flexfuel)

  • Something Else – specify


What do you think this question is asking? [order of response options randomized]

  • The type of fuel I usually put in the vehicle

  • The types of fuel my vehicle can use

  • The types of fuel I have ever put in the vehicle, even once

  • Something else – Specify

  1. [if # of Vehicles > 1] Were you able to answer this question accurately for all vehicles in your household?

    • Yes

    • No

  1. [if no] Please explain why you were not able to answer the question about fuel.

  2. How confident are you that your answer to this question was correct?

    • Not at all confident

    • A little confident

    • Somewhat confident

    • Very confident

    • Extremely confident

  1. Have you ever heard of E85?

    • Yes

    • No

  1. [if yes] Can you explain what E85 is?

    • Yes

    • No

  1. Have you ever heard of FlexFuel?

    • Yes

    • No

  1. [if yes] Can you explain what FlexFuel is?

    • Yes

    • No


Module B

Revisions to be tested:

  • Reword mortgage origination date questions.

Screening Question:


Have you or any member of your household ever refinanced a mortgage for your current home?

        • Yes

        • No

Do you own the property in which you live?

  • Yes

  • No


[if yes] Do you have a mortgage or lump sum home equity loan for this property?

  • Yes

  • No


NEW QUESTION

[if yes] Have you or anyone in your household ever refinanced this mortgage?

  • Yes

  • No


MODIFIED QUESTION

[if yes to previous] In what year did you make the first payment on this mortgage or lump sum home equity loan after refinancing?


MODIFIED QUESTION

In what month did you make the first payment on this mortgage or lump sum home equity loan after refinancing?

  • January

  • February

  • March

  • April

  • May

  • June

  • July

  • August

  • September

  • October

  • November

  • December


MODIFIED QUESTION

What was the amount of this mortgage or lump sum home equity loan when you obtained it, not including any interest?


Debriefing Questions:

  1. [if refinance] Earlier we asked if you had refinanced a mortgage or home equity loan. You reported you made the first payment in [month], [year]. Is that when you made your first payment on the original loan or the refinanced loan?

    • Original loan

    • Refinanced loan

    • Something else - specify

  1. [if homeowner with mortgage] Have you heard the term “refinance” regarding mortgages before?

    • Yes

    • No

  1. [if yes] Could you explain what refinance means?

    • Yes

    • No


Module C

Revisions to be tested:

  • Reword questions related to book purchases.

Screening Question:


Have you or any members of your household purchased any books or enrolled in any book subscription services in the last three months?

  • Yes

  • No

Since the first of (reference month) have you purchased any of the following items for your household or for someone outside your household?

  1. Books, digital books, or book subscriptions. Do not include school books or reference books.

    • Yes

    • No


Describe the item.


What was the total amount of this expense?


Was this purchase for your household or someone outside your household?

  • For household

  • For someone outside your household


In what month did you have this expense?

  • October

  • November

  • December

  • January

  • February

Debriefing Questions:

  1. [if book expense] You said that you purchased books, digital books, or book subscriptions. What exactly did you purchase?

    • Books (paperback, hardcover)

    • Digital Books (e.g., books for Kindle, iPad, etc.)

    • Audio Books

    • Book Subscriptions

  1. One of the questions I asked referred to “book subscriptions.” What are some examples of book subscription services?

  2. Do you have, or have you ever had, any book subscriptions?

    • Yes

    • No

    1. [if yes] What businesses are or were these subscriptions through?

  1. Have you purchased any books through a book club?

    • Yes

    • No


Module D

Revisions to be tested:

  • Combine questions related to rental of DVDs and streaming video subscriptions.

Screening Question:


Have you had any expenses for the rental of DVDs or for a streaming video subscription service in the past 3 months?

  • Yes

  • No

Since the first of (reference month) have you purchased any of the following items for your household or for someone outside your household?

  • Subscription music services such as Rhapsody or Pandora

  • Rented video files or DVDs or streaming video subscriptions

    1. Yes

    2. No


Describe the item.


What was the total amount of this expense?


Was this purchase for your household or someone outside your household?

  • For household

  • For someone outside your household


In what month did you have this expense?

  • October

  • November

  • December

  • January

  • February

Debriefing Questions:

  1. [if rented video expense] You said you had an expense for “rented video files or DVDs or streaming video subscriptions.” What businesses or services did you use for those? [mark all that apply]

    • Netflix

    • Hulu Plus

    • Amazon Prime

    • HBO Now

    • iTunes

    • Google Play

    • Redbox

    • Video rental store

    • Other - specify


Module E

Revisions to be tested:

  • Change order of photographic film and photo printing questions.

Screening Question:


Have you had any photography related expenses in the past 3 months?

  • Yes

  • No


PROPOSED ORDER

Since the first of (reference month) have you purchased any of the following items for your household or for someone outside your household?

  1. Photo printing or processing

  2. Photographic film or disposable cameras


CURRENT ORDER

Since the first of (reference month) have you purchased any of the following items for your household or for someone outside your household?

  1. Photographic film or disposable cameras

  2. Photo printing or processing


    • Yes

    • No


Describe the item.


What was the total amount of this expense?


Was this purchase for your household or someone outside your household?

  • For household

  • For someone outside your household


In what month did you have this expense?

  • January

  • February

  • March

  • April

  • May

  • June

  • July

  • August

  • September

  • October

  • November

  • December

Debriefing Questions:

  1. Did you have any expenses related to photography since the first of (reference month)?

    1. Yes

    2. No

  2. Did you print any photos in the last 3 months?

      • Yes

      • No

    1. [if no] Have you ever had photos printed?

      • Yes

      • No

      1. When was the last time you had photos printed?

    1. [if yes] Were they printed commercially or at home (mark all that apply)?

      • Printed commercially

      • Printed at home

    1. [if “home”] In the expense you reported to us earlier, did you include the cost of ink and paper?

      • Yes

      • No


Module F

Revisions to be tested:

  • Test for context effects when satellite radio service question is moved to subscription service section.

  • Change satellite radio service to Sirius/XM satellite radio service

  • Update subscription music service examples

Screening Question:


Have you had any expenses for a subscription to satellite radio services (e.g., Sirius/XM or any other) during the past 3 months?

  • Yes

  • No

Since the first of (reference month) have you purchased any of the following items for your household or for someone outside your household?

  1. Subscription music services such as Spotify or Pandora

  2. Sirius/XM satellite radio service

  • Yes

    • No


Describe the item.


What was the total amount of this expense?


Was this purchase for your household or someone outside your household?

  • For household

  • For someone outside your household


In what month did you have this expense?

  • January

  • February

  • March

  • April

  • May

  • June

  • July

  • August

  • September

  • October

  • November

  • December

Debriefing Questions:

  1. Have you used or paid for any of the following streaming music services in the past three months??

Service

Have Not Used

Used for Free

Used and Paid For

Pandora

iHeartRadio

iTunes Radio

Spotify

Amazon Music

Rhapsody

Google Play All Access

TuneIn Radio

Radio.com

Slacker

Beats Music

Tidal


  1. [if satellite radio] What service(s) do you use for satellite radio?

  2. [if satellite radio] Have you ever used any other services for satellite radio?

    • Yes

    • No

  1. [if yes] Which ones?

  1. Which of these best matches the way you think of satellite radio services:

    • Vehicle expense

    • Utility expense

    • Entertainment expense

    • Subscription expense


Module G

Revisions to be tested:

  • Determine best examples for including e-cigarettes in the Information Booklet.

Screening Question:


Have you purchased any e-cigarettes or vaping products in the last 3 months?

  • Yes

  • No

Since the first of (reference month), have you or any members of your household purchased cigarettes?

  • Yes

  • No


MODIFIED QUESTION

Have you or any members of your household purchased other tobacco products such as cigars, pipe tobacco, chewing tobacco, or vaping products?

  • Yes

  • No


Debriefing Questions:

  1. [if cigarettes] You reported that you purchased cigarettes. Did this include electronic cigarettes (i.e., e-cigarettes)?

  2. [if expense for other tobacco products] What did you purchase?

    • Cigars

    • Pipe tobacco

    • Chewing tobacco

    • Vaping products

  1. [if no to vaping products] Have you ever purchased vaping products?

    • Yes

    • No

  1. What does “vaping products” mean to you?

  2. What terms would you use to describe vaping products?

Appendix C: Consent Form


CONSENT FORM


The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.


The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.


During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.


We estimate it will take you an average of 45 minutes to participate in this research (ranging from 40 minutes to 60 minutes).


Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.


Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires 04-30-2018.


------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.



___________________________________ ___________________________

Participant's signature Date



___________________________________

Participant's printed name



___________________________________

Researcher's signature





PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.



File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleOctober 14, 2003
AuthorYu, Erica - BLS
File Modified0000-00-00
File Created2021-01-22

© 2024 OMB.report | Privacy Policy