Note Reviewer - Extended Recall Period

September 4, 2018

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Madeleine Saxton and Erica Yu
SUBJECT:	Submission of Materials for “Research on extended recall periods to collect outlet information in the CE”

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 198 hours.

If there are any questions regarding this project, please contact Madeleine Saxton at [email protected] or 202-691-6874 or Erica Yu at [email protected] or 202-691-7924.

1. Introduction

The Telephone Point of Purchase Survey (TPOPS) uses variable recall periods to solicit the names and addresses of stores at which people shop. Items and services vary in purchase prevalence (how many households in the US have purchased the item or service at least once), and survey recall periods that reflect purchase behavior can reduce burden on respondents and increase reporting rates for the survey. The majority of the recall periods have not been changed since the survey’s inception in 1997, despite changes in consumption. As the Consumer Price Index moves its collection of stores from TPOPS to the Consumer Expenditure Surveys (CE), it is a good time to evaluate the recall periods and optimize them for ideal reporting.

Optimizing Recall Periods

In TPOPS, the length of a recall period (along with other factors) has been used to balance the need for a sufficient sampling frame with the need to minimize respondent burden and collect high quality data. In general, the recall periods are short (one or two weeks) for frequently purchased items like gasoline and longer (up to five years) for infrequently purchased items like new cars.

In the past, TPOPS item response rates have been reviewed to assess specific recall periods and adjust them. For example, the recall period for the category Cars, Trucks and Van Rentals was shortened from five years to two years when analysis showed the sample frame was greater than needed. Also, there was a concern regarding memory quality for vehicle rentals up to five years in the past. While recall periods for a few items have been reviewed and changed, there has not been an analysis of all of the recall periods in over two decades.

Preliminary research based on historical TPOPS reporting rates and assumptions about purchase frequencies was conducted to model expected extended recall outlet yields. However, we are aware that these historical data were collected using an instrument that has not significantly changed since TPOPS first began in 1998 and may therefore misrepresent current purchase behavior.

The ongoing movement of outlet collection from TPOPS to the CE adds an additional motivation to analyze and revise the recall periods. Outlets will be collected for a subset of items reported in the CE Interview’s regular three month recall period; however, given that the three month recall period is significantly shorter than the recall period currently used in TPOPS for many items (six months to five years), we anticipate that the CE Interview will yield significantly fewer outlets.

To address this insufficient yield in production, if a respondent does not report any expenditures in a POPS category during the three month recall period, then the interviewer will ask the respondent an additional question about POPS category expenditures within a recall period that is longer than three months. We believe that new research is needed to collect data on expenditure recall to provide valuable information for identifying optimal recall periods for these new questions.

Because this study is based on an online convenience sample, we do not expect the results to represent purchase behavior among the CE survey population. We will use these results to understand how much detail people remember about selected purchases, and then combine these data with our preliminary research based on historical TPOPS data to converge on recommended recall periods. We plan to continually monitor production data for possible future revisions.

2. Methodology

The purpose of this study is to collect information about how long ago people purchased items and if they can remember details about the expense that are necessary for the CPI’s outlet frame. Unlike other studies where we are primarily interested in comprehension or question wording, in this study we are interested in recall of expenditure details and so participants must have had the target expense in order to provide usable data. This consideration drives the design of the item modules, sample size, protocol, and mode.

Item Modules

The items to be tested were identified from the current TPOPS survey as having a longer recall period than the CE Interview survey and anticipated low purchase prevalence as reported in TPOPS. Because the full list of items to be tested is longer than can be asked of any single participant, the list of items will be split such that each participant sees only a module, or subset of items.

Each module consists of 13-20 items. For these calculations, we assume that participants will have had expenses for 4 or fewer of the items. Items that the participant did not purchase take approximately 10 seconds total to respond to (the time taken to consider the item category and respond that they did not have the expense) whereas items that the participant did purchase take approximately 75 seconds total (the time taken to respond to all follow-up questions). We anticipate participants will need no more than 10 minutes to complete the full task.

A list of the items included in the testing is included in Appendix A, broken out by modules.

Sample Size

Due to the nature of our research questions being focused on recall of expenditure details, the number of participants from whom we can collect usable data will vary by item based on item purchase prevalence. We propose a sample design based on anticipated purchase prevalence in the general population, based on TPOPS data.

• For 60 items that we anticipate have been purchased by only a small portion of the population, we plan for a relatively large sample to be asked about those expenses - 300 participants for each item.

• For 39 items that we anticipate have been purchased by a large portion of the population, we plan for a smaller sample - 96 participants for each item.

Furthermore, given that the actual purchase prevalence for each item among the study sample is currently unknown, we propose a responsive design such that items for which sufficient data have been collected (at least 30 participants reporting having had the expense) can be removed from protocols and replaced with other items that still need additional data. This design is expected to reduce overall respondent burden compared to using a large sample size for all items while still ensuring a sufficient number of expenditure reports are captured. Data will be collected in batches of 100 participants and reviewed before the next batch is begun.

Protocol

Participants will be asked whether they have ever had an expense for a target item and, if so, they will be asked to:

• Recall details about the most recent expense, including when and from where the purchase was made and how much it cost

• Answer debriefing questions to probe the accuracy of their responses and their confidence in those recollections

These follow-up questions are the same for each item, varying only by what item category is the target of the questions. If a participant reports not having had the expense, then they are not asked any follow-up questions. The template for the testing protocol is included in Appendix B and shows the questions that will be asked for each item category.

Mode

Online testing will be used to conduct unmoderated cognitive interviews. The benefit of online testing for this study is that feedback can be gained from a larger number of participants quickly and at lower cost than would be possible in the OSMR research lab in Washington, DC.

3. Participants

Participants will be recruited as a convenience sample from Amazon Mechanical Turk of adults located in the United States that are 18 years or older¹. This research design requires a sample of no more than 1188 participants in order to sufficiently explore the range of variables of interest. These participants will be randomly assigned to a grouping of items as described above.

4. Burden Hours

Individuals may participate only one time and cannot respond to any other calls for participation. Our goal is to collect data from no more than 1188 participants. We anticipate that each survey will take no more than 10 minutes, including expenditure questions, debriefing questions, and demographics, for a total of 198 burden hours (1188 participants x 0.1667 burden hours)

5. Payment to Participants

Participants will be compensated $1.50 for participating in the study, a typical rate provided by Mechanical Turk for similar tasks; a total of $1,782 will be paid directly to Amazon Mechanical Turk for participant fees

6. Data Confidentiality

Recruiting of participants will be handled by Amazon Mechanical Turk. Once participants are recruited into the study, they will be given a link to the survey, which is hosted by SurveyMonkey. The data collected as part of this study will be stored on SurveyMonkey servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. This survey will take approximately 10 minutes to complete. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

7. Attachments

Appendix A: Protocol template

Appendix B: Items for testing

Appendix A: Items for testing

Module 1

Good or Service	Item category name
S	Adult day care
S	Catered events or board such as weddings or college meal plans
G	Elementary or high school textbooks
G	Unpowered Boats or trailers
G	Outboard motors or powered sports vehicles
G	Leased cars, trucks or vans
S	Day care providers including nursery schools
G	Infants’ furniture
S	Tuition or fixed fees for elementary or high schools
S	Tuition or fixed fees for technical or vocational schools
G	Calculators, typewriters, or other information processing equipment
G	Women’s suits or suit components
S	Reupholstery of household furniture
G	Fuel oil for household heating
G	Audio Equipment for automobiles or home
S	Funeral services
G	Washers or dryers
G	Refrigerators or home freezers
G	Entertainment centers, bookcases or desks
S	Taxi or cab fares

Module 2

Good or Service	Item category name
S	Photographer’s fees
G	DVD players, camcorders or other video equipment
S	Shoe repair or other shoe services
G	Hunting, fishing, or camping equipment
G	Outdoor furniture
G	Stoves, ovens, or microwave ovens
S	Moving, storage or freight services
G	Heating or cooling equipment or home safety devices
G	Musical instruments or musical accessories
S	Legal services
S	Recreational lessons, classes or instruction
G	Men’s suits or blazers
G	Infants’ equipment such as strollers, car seats, bottles or dishes
S	Ship travel or passenger cruises
S	Services by medical professionals other than MDs
S	Tuition or fixed fees for a college or university
S	Housekeeping services
S	Appliance repair
G	College or university level textbooks
G	General sports equipment

Module 3

Good or Service	Item category name
G	Video game hardware, games or accessories
S	Motor vehicle body work
G	Propane, firewood, coal or charcoal used for household heating or cooking
G	Bedroom furniture including mattresses or springs
G	Jewelry
G	Cameras or other photographic equipment excluding film
S	Clothing rental, alterations or repairs
S	Watch or jewelry repair
S	Pet services such as grooming, boarding, or training
G	Vacuums or other electric floor cleaning equipment
G	Watches
S	Admissions to sporting events
G	Men’s coats, jackets, or snow wear
G	Sewing machines, fabric or sewing supplies
G	Living room, dining room, or kitchen furniture
G	Computer software and blank media
G	Televisions
G	Bicycles or bicycling accessories
G	Power tools
G	Luggage, briefcases, or other carrying cases

Module 4

Good or Service	Item category name
G	Men’s accessories including hats, ties, belts or wallets
G	Barbeque grills, powered lawn and garden equipment or other outdoor items excluding outdoor furniture
G	Window coverings such as curtains, drapes or blinds
G	Hard liquor for home use
G	Household decorative items including clocks or lamps
G	Vehicle parts or accessories
G	Boys’ or girls’ clothing or accessories
G	Women’s dresses
G	Floor coverings such as hard surface tiling, carpets or scatter rugs
G	Paint, wallpaper tools, or related supplies
G	Nonpowered tools or miscellaneous hardware
S	Motor vehicle repair
S	Car, truck or van rentals

Module 5

Good or Service	Item category name
G	Small kitchen electric appliances or clothing irons
S	Inside home maintenance and repair
G	Electric personal care appliances such as shavers or hair dryers
S	Veterinarian services
G	Dishes, glassware or flatware
G	Motor oil, coolants, or fluids
S	Club Membership Dues for Shopping Clubs, Fraternal, or Other Organizations or Fees for Participant Sports
G	Toys, Games, Hobby Supplies, or Playground Equipment
G	Boys' or Girls' Footwear
G	Gardening or Lawncare Services
S	Automobile Service Clubs
G	Women’s Coats, Jackets, or Snow Wear
G	Tires

Module 6

Good or Service	Item category name
G	Lawn and Garden Supplies or Insecticides
G	Personal Computers or Peripheral equipment
G	Wine for Home Use
G	Men's Footwear
G	Men's Pants or Shorts
G	Eyeglasses or Eye Care
G	Men's Shirts, Sweaters, or Vests
G	Household Linens, such as Kitchen or Bathroom Towels, Bedding, or Tablecloths
S	Lodging Away From Home such as Hotels, Motels, or Vacation Rentals
S	Garbage or Trash Collection Service
S	Tax Return Preparation or Other Accounting Services
G	Women's Footwear
S	Dental Services

Appendix B: Protocol Template

[This is the first screen they see after they sign up for the study]

Thank you for your interest in our research study.

The Consumer Price Index (CPI) is an important economic indicator that measures how prices change. To calculate the CPI, we collect data on what US households purchase, how much they spend, and what stores and businesses they purchase them from. Your participation in this study will help us to design our surveys to ensure we get the best data possible.

The goal of this study is to understand what people can remember about their past expenses. This survey will ask you whether you had a particular expense and what you can remember about it.

Please be honest about what you remember. We know that some expenses are easier to remember than others and you may or may not remember the details about all of them. There are no penalties for saying that you can’t remember something.

Please do not use your browser's back button.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. This survey will take approximately 10 minutes to complete. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

[Next screen]

You will be asked questions about household expenses. When answering the survey, please include expenses made by:

• You

• Anyone you currently live with

• Anyone you lived with at the time the item was purchased

And please remember:

• It is OK to give estimates

• It is OK to say that you don’t remember something

Thank you for your help! Let’s get started.

[Next screen]

[Fill for a service: e.g., Adult day care]

1. Have you or anyone in your household had expenses for [fill: adult day care]?

   1. Yes

   2. No (skip to next target category)

   3. Yes, but I don’t remember many details

[Next screen]

(If Yes or Yes, but I don’t remember many details)

[Fill for a service: e.g., Adult day care]

The following questions are about the most recent time you or someone in your household had an expense for [fill: service title].

2. Briefly describe the expense. (open text entry)

3. When did you have this expense?

   1. (dropdown for month)

   2. (dropdown for year)

   3. I don’t remember

4. How much did you pay? $ _______

5. How accurate is that amount?

   1. Exactly accurate

   2. Very close

   3. Somewhat close

   4. I don’t remember how much was paid

6. What was the name of the [fill: adult day care] provider?

   1. Enter the name of provider ______________________

   2. I don’t remember

7. In what city and state was this provider located?

   1. Name of city ______________________

   2. Two character state __

   3. Online

   4. I don’t remember

[Next screen]

[Fill for a commodity: e.g., Film, film development supplies or disposable cameras]

1. Have you or anyone in your household EVER had expenses for [fill: film, film development or disposable cameras]?

   1. Yes

   2. Yes, but I don’t remember many details

   3. No (skip to next target category)

[Next screen]

(If Yes or Yes, but I don’t remember many details)

[Fill for a commodity: e.g., Film, film development supplies or disposable cameras]

The following questions are about the most recent time you or someone in your household had an expense for [fill commodity name].

2. Briefly describe the expense. (open text entry)

3. When did you have this expense?

   1. Enter the two digit month __

   2. Enter the four digit year ____

   3. I don’t remember

4. How much did you pay? $ _______

8. How accurate is that amount?

   1. Exactly accurate

   2. Very close

   3. Somewhat close

   4. I don’t remember how much was paid

5. What was the name of the business?

   1. Enter the name of business ______________________

   2. I don’t remember

6. In what city and state was this business located?

   1. Name of city ______________________

   2. Two character state __

   3. Online

   4. I don’t remember

[Next screen]

And now a few final questions about you, to help us better understand your answers.

1. What is your age in years?

(integer)

2. What is your gender?

• Male

• Female

• Other/Prefer not to say

3. What is the highest level of school you have completed or the highest degree you have received?

• Grades 1-12/No Diploma

• High School Diploma or Equivalent

• Some College

• Associate’s Degree

• Bachelor’s Degree

• Master’s Degree

• Doctorate or Professional Degree

4. Are you Hispanic or Latino?

• Yes

• No

5. What is your race?

• American Indian or Alaska Native

• Asian

• Black or African-American

• Native Hawaiian or Other Pacific Islander

• White

[Next screen]

Thank you for participating in our research study!

If you have any feedback about the survey, please leave your comments here; otherwise, leave this space blank.

Please proceed to next page for your completion code.

[Next screen]

Your completion code is: [fill code]

Please copy and paste this code into the HIT page on mTurk.

1 Mechanical Turk is an online platform that offers a crowdsourcing “marketplace”. Individuals can volunteer to participate in short tasks and be paid small amounts of money ($4-6/hour pro-rated, on average). “Workers” may not be demographically representative of the United States but social science researchers are increasingly using Mechanical Turk for large-scale studies. Eligibility criteria can be set to restrict participation to only individuals over 18 years old and located in the United States.

20