Note to Reviewer - CE Testing

October 15, 2020

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Erica Yu and David Biagas Office of Survey Methods Research
SUBJECT:	Submission of Materials for Cognitive Testing of the Consumer Expenditure Survey streamlined questionnaire revisions

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 173 hours.

If there are any questions regarding this project, please contact Erica Yu at 202-691-7924.

1. Introduction

The Consumer Expenditure surveys (CE) are undergoing a phased implementation of their original Gemini redesign plan¹. As part of that phased implementation, the interview survey is being streamlined, with the revisions to the questionnaire being made over time in phases rather than all at one time. In the first phase, three parts of the questionnaire are being revised: transportation, family care and education, and expense patterns. The proposed revisions, which range from the re-locating of questions within the interview to major revisions to existing questions, are intended to improve data quality through a verifiable reduction in measurement error.

Before the revisions can be made, they are reviewed by the Office of Survey Methods Research (OSMR) for question wording and possible measurement error concerns. For revisions that are more than minor tweaks, OSMR recommends conducting testing in addition to the expert review. The goal of the pre-testing is to evaluate whether the revised questions are clear and understandable to respondents, whether the revised questions are interpreted by respondents as intended, to identify appropriate examples for the Information Book that accompanies the interview, and to make any refinements as needed.

2. Methodology

Due to the Coronavirus pandemic, traditional in-person research methods are not available. To collect feedback about the questionnaire, this study uses remote cognitive interviews and online surveys. Both have been used in the past and are known to be a valid approach to collect useful information. In both modes, participants will answer survey questions about their expenses and answer debriefing questions to enable us to better understand their responses and reactions to the questions, including comprehension tasks.

As this is pretesting, we expect that modifications may be made during the course of the study based on initial results. The data will be collected in small iterative rounds (interviews: up to 3 participants per round; online: 30 participants per round), with findings evaluated qualitatively after each round. Although the goals of the testing, and overall design, will remain the same, findings from preliminary results may be used to improve the questions. In some cases, we may test two different versions of questions and compare results to evaluate which version appears to result in better respondent comprehension or data quality. In other cases, we may iteratively modify a question based on preliminary results. Versions and modifications are likely to range from slight changes to question wording (e.g., naming examples of the expense category) to additional instructions (e.g., inclusion or exclusion instructions).

Remote Cognitive Interviews

Cognitive interviews provide an in-depth understanding of the participant’s thought processes and reactions to the questions. Interviews will be conducted using a protocol similar to what would be used in-person, but done via video calls by staff who are experienced in conducting these types of interviews. The interviews will be conducted with participants from anywhere in the United States. Observers from CE may attend interviews.

The interview protocol and showcards are included in Appendix A. The topics covered in interviews include:

• Gift expenses

• School meal expenses

• Grocery expenses

• Whether the more conversational wording of “did you pay for” rather than “did you have an expense for” is inclusive of credit card payments.

Online Surveys

Online surveys collect information from a larger group that is more diverse geographically and in terms of expenditures than would otherwise be possible. This enables us to find a range of participants with the expenses that we are targeting. The online surveys will be available to participants from anywhere in the United States.

The online protocol is included in Appendix B. The topics covered in the surveys include:

• Construction expenses

• Vehicle maintenance, repair, parts, and accessories expenses

• School bus expenses

• Tobacco and related expenses

• Grocery expenses

• Whether the more conversational wording of “did you pay for” rather than “did you have an expense for” is inclusive of credit card payments.

For the online surveys, quantitative analyses will be performed but statistical significance is not a goal. Rather, the debriefing feedback from all modes is the primary outcome to be used to evaluate the proposed revisions and make any revisions for the next round.

3. Participants

Remote Cognitive Interviews

Up to 20 participants will be recruited for cognitive interviews using two different strategies:

• Advertisements in online bulletin boards in Florida, Pennsylvania, and Texas where schools are open for in-person instruction (Appendix C)

• Calls to potential participants in OSMR’s existing participant database

Participants for the cognitive interviews will be screened for eligibility using a screening script (Appendix D). We aim to select participants for the below characteristics:

• Paying for school meals

• Elementary or middle (n=12), high school (n=8)

• Private (n=8), public (n=12)

Online Survey

Up to 2,000 participants will be recruited as a convenience sample from Amazon Mechanical Turk of adults (18 years and older) residing in the U.S for a screening survey (Appendix E). From this sample, we aim to recruit up to 500 participants in the following subgroups, which we expect will enable us to find a range of participants with the targeted expenses:

• Own or lease a car (n=125)

• Pay for school bus (n=125)

• Smoker (n=125)

• Home owner (n=125)

Given the range of expenses being targeted in the online surveys, we anticipate it is unlikely to find households that have all of these expenses. The topics have been separated into modules and participants will be offered the opportunity to respond to only one based on their responses to the initial screener survey. For example, the module about vehicle maintenance and repair expenses will be asked only of participants who own or lease cars; the module about construction expenses will asked only of participants who own property. An individual may participate in only one module.

4. Burden Hours

Total expected burden for this study is 173 hours, as shown in the table below:

	Number Contacted	Time spent Recruiting	Screening Burden	Number of Participants	Session Length	Session Burden	Total Burden
Remote Cognitive Interviews	35	5 minutes	2.9 hours	20	60 minutes	20 hours	22.9 hours
Online Survey	2000	2 minutes	66.7 hours	500	10 minutes	83.3 hours	150 hours
Overall Total Burden Hours							173 hours

5. Payment to Participants

Remote Cognitive Interviews

A $40 incentive will be used to recruit participants ($600.00 total), to cover the costs associated with participation including internet access and child care.

Online Survey

Participants from Amazon Mechanical Turk will be paid $0.10 for completing the screener and $1.50 for participating in the study ($950.00 total). This aligns with typical rates offered on the Amazon Mechanical Turk platform for similar tasks.

6. Data Confidentiality

Remote Cognitive Interviews

Interview participants will be asked for their verbal consent to participate in the study, outlined in Appendix F. Participants will be given a pledge of confidentiality with the CIPSEA pledge and Privacy Act statement shown on the screen at the start of the interview.

Online Survey

Once online survey participants are recruited into the study, they will be given a link to the survey, which is hosted by SurveyMonkey. The data collected as part of this study will be stored on SurveyMonkey servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: March 31, 2021). Without this currently approved number, we could not conduct this survey. We estimate that it will take on average 10 minutes to complete this survey. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

7. Attachments

Appendix A: Cognitive Interviewing protocol

Appendix B: Online Survey protocol

Appendix C: Online bulletin board electronic advertisement

Appendix D: Recruitment and eligibility script

Appendix E: Screener Survey

Appendix F: Remote testing verbal consent

1 Gemini Project to Redesign the Consumer Expenditure Surveys. https://www.bls.gov/cex/geminiproject.htm

6