Attachment A - Pilot Overview

Individual Diaries Feasibility Test – Pilot Test

1. Study Overview

The current Consumer Expenditure Diary Survey (CED) at the Bureau of Labor Statistics (BLS) uses a pencil-and-paper instrument (PAPI) to collect expenditure information from respondents. PAPI diary data collection has a number of inherent drawbacks, such as limiting entry to a single individual in a single location, and requiring that the main diary keeper carry the diary with them throughout the day. Additionally, having one person in the consumer unit (CU) maintain the diary for all members of the CU has drawbacks (Edgar et al., 2006).

In addition, the Consumer Expenditure Survey (CE) program would like to leverage data collection technologies and present CUs with more timely data entry options, such as a mobile-optimized web option for Smartphones or Tablets and a web diary. These modes of data collection have been recommended by several researchers (Westat, 2011; Mathiowetz et al., 2011). The technologies have the potential to improve unit and item response rates as well as measurement error. Additionally, a web component and a mobile-optimized web option component have potential cost savings over PAPI due to reduced or eliminated materials, scanning, and data entry expenses.

This study will test the feasibility and impact of using both individual diaries and multiple modes to collect CE diary expenditures. CU members in the research sample would be offered sequentially a mobile-optimized web option and then a web survey. These modes would be available for each member of the CU to complete individually.

This study will attempt to answer the following research questions:

1. What are the operational issues related to implementing a mobile-optimized web option for CED data collection versus web mode with an individual log-in?

   1. Debrief with field representatives (FRs) to determine:

1. What issues if any did respondents have with the materials provided?

2. What technical issues if any were raised by the respondents?

3. How should respondent materials be revised?

1. Debrief with respondents to determine:

   1. What did they like about the mode they were offered?

   2. What did they dislike about the mode they were offered?

   3. What extra information would have been helpful to them?

2. Debrief with the Regional Office (R.O.) and Demographic Survey Divisions (DSD) staff that handle respondent help requests to determine:

1. What kinds of questions did respondents ask?

2. How should respondent materials be revised?

4. Debrief with the National Processing Center (NPC) staff to determine:

   1. How would they change the data keying and coding processes to improve efficiency if the modes were put into Production?

2. How do high-level expenditure reporting rates and data quality differ by research group and control?

   1. Expenditures

   2. Item-missingness

3. How do acceptance and completion rates differ by research group and control?

   1. Week 1 acceptance rate

   2. Week 1 completion rates

   3. Week 2 completion rates

4. What are web respondents’ and mobile-optimized web option respondents’ specific data entry patterns?

   1. Number of log-ins per case

   2. Start/stop time stamps by diary day (to determine multiple times per day versus multiple days per week versus only data entry at the end of the week)

2. Study Design

1. Field period

The field period is scheduled for March 2013.

2. Respondents

The Research sample will serve as the test group. The control group will come from the Production sample for whatever quarter is used for fielding.

3. Modes and diary changes

The control group will use the current CED methodology. The test group will use multiple modes: an in-person first visit with some CAPI data collection, a telephone reminder instead of visit 2, completion of individual diaries via web or mobile-optimized web, and an in-person third visit with some CAPI data collection.

Changes to the web diary and mobile-optimized web diary formats and content were based on one round of usability tests conducted by the Office of Survey Methods Research (OSMR) at the Bureau of Labor Statistics¹.

4. CAPI changes

The CAPI survey instrument will have to be modified to add three questions for both the Research and Production samples: internet access, mobile access, and how accessed. The first question will allow for matching between Research and Production samples since households/consumer units with internet access are different from those without internet access.²

There will also be questions added into CAPI to de-brief both the respondents and FRs. These will gauge respondents’ general comments, use of records, and burden as well as FRs’ assessments of the respondents and their general comments.

5. Methodology

The control group will consist of production sample from the study’s field period, and will be administered the current diary using standard procedures. The control group will be matched on socio-demographic characteristics to the test group after the field period concludes, as mentioned above.

Test Group

The test group will consist of research sample cases, and will be administered the web diary and/or mobile-optimized web diary using modified procedures. The contact schedule for the Test Group will be as follows:

1. Visit 1 - All respondents randomly assigned to the treatment group will be asked the internet access screening question and a follow-up questions on how the internet is accessed. The web diaries and/or mobile-optimized web diaries would be placed in person. If eligible respondents are not available at placement, the head-of-household will determine which instrument the unavailable respondent will receive. Respondents would be given materials describing the web diary and/or mobile-optimized web diary and instructions on how to log in and complete the diary. If the CU does not have any internet connection, then the CU is screened out and CAPI questions are not asked. Any CAPI questions would be asked as currently scheduled:

   1. Front section: http://www.bls.gov/cex/ced/2011/csxfront.htm

   2. Household Characteristics: http://www.bls.gov/cex/ced/2011/csxsection1.htm

   3. CU characteristics: http://www.bls.gov/cex/ced/2011/csxsection2.htm

   4. Back section (not asked, but FR-completed): http://www.bls.gov/cex/ced/2011/csxback.htm

Non-English households in the Research group will be screened out as Type C-language barrier (“WD Language Barrier”). (Type Cs are ineligibles for the FRs response rates.) There will not be a Spanish web survey or mobile-optimized web survey, but this will be revisited if the web survey and/or mobile-optimized web survey are put into Production.

Spawned cases (multiple CUs found in one address) will also be coded out as Type Cs.

2. Visit 2 - The second-week visit will not occur; instead, CUs will be reminded about the diary via telephone. The FRs will not have access to the respondents’ web diaries or mobile-optimized web diaries. Therefore, the question for the FRs will be modified such that it will be directly asked of respondents: “Did you or anyone in your household record any expenses in the Web Diaries or Mobile-Optimized Web Dairies during the first week?”

FRs would not visit week 1 non-contacts. The following sections of CAPI questions would be asked as currently scheduled:

1. Coverage (not asked, but FR-completed): http://www.bls.gov/cex/ced/2011/csxcoverage.htm

2. Back section (not asked, but FR-completed): http://www.bls.gov/cex/ced/2011/csxback.htm#WK2_ST2

3. Visit 3 - The CUs would be visited in-person at the end of the diary period to thank them for their participation. Any CAPI questions would be asked as currently scheduled:

   1. Work experience and income: http://www.bls.gov/cex/ced/2011/csxsection4a.htm

   2. Work experience and income for the CU: http://www.bls.gov/cex/ced/2011/csxsection4b.htm

   3. Coverage (not asked, but FR-completed): http://www.bls.gov/cex/ced/2011/csxcoverage.htm

   4. Back section (not asked, but FR-completed): http://www.bls.gov/cex/ced/2011/csxback.htm#WK2_ST2

FRs will also collect receipts if provided by respondents. FRs would enter any receipts into a modified CAPI instrument. (The CAPI instrument will be modified to include a new tab that mimics the web survey instrument. FRs won’t have access to respondents’ web or mobile-optimized web surveys. FRs will be trained on the new web survey procedures and follow the documented procedures.)

At the end of the last visit, respondents will be debriefed, as well as FRs (both using the CAPI instrument).

There will be no reinterviews for Research cases.

FRs will be debriefed regarding operational issues after the completion of the study via paper instrument. The questions may include items such as:

1. Were there issues with giving respondents the website and login information?

2. Were there technical issues raised by the respondents?

3. How should respondent materials be revised?

4. General comments about the diary

6. Sampling and sample size

The Production sample will serve as the control group (meaning not offered the web diary or mobile-optimized web diary options). The research sample will be randomly selected from a starting sample of 60 addresses.

Table 1: Estimated sample sizes.

	Research sample size for 48 total completed diaries
Starting sample addresses	60
Occupied housing units (occupancy rate = 80%)	48
Eligible CUs, will be asked CAPI questions (screening rate = 68%)	33
Completed diaries for week 1 (CED response rate = 70%)	23
Completed diaries for week 2 (CED attrition rate = 0%)	23
Total completed diaries (weeks 1 and 2)	46

7. Available Materials

The following materials are available to OMB for consideration of this project:

1. OSMR’s Round 1 Usability Report

2. Screenshots of the web diary survey

3. Screenshots of the mobile-optimized wed diary survey

4. Analysis Plan

5. Draft survey materials, such as the Respondent User Guides and the FR Talking Points

III. Burden Hour Estimate

BLS estimates that this pilot study will require 25.19 burden hours. We base this estimate upon the following assumptions:

	Mins.	Sample	Total Hours
Estimated time to be screened for Research sample	2	48	1.60
Estimated time to complete diary (first week)	203	23	7.67
Estimated time to complete diary (second week)	20	23	7.67
Additional time for CAPI questions, total of all visits, asked for all CU’s where a diary was placed	15	33	8.25
TOTAL			25.19

4. References

1. Edgar, J., Davis, J., Spell, S., Verlander, L., and Wetzel, G. (2006). Individual Diary Feasibility Test. BLS, Division of Consumer Expenditure Survey internal report.

2. Westat. (2011). Data Capture Technologies and Financial Software for Collecting Consumer Expenditure Data. BLS, Division of Consumer Expenditure Survey internal report.

3. Mathiowetz, N., Olson, K., and Kennedy, C. (2011). Redesign Options for the Consumer Expenditure Survey. Prepared for National Academies of Sciences. BLS, Division of Consumer Expenditure Survey internal report.

1 Usability testing by OSMR was covered by their generic clearance and they submitted their own clearance packages.

2 Consumer units will also be matched by other variables, such as geography and income.

3 BLS estimates that the web diary and/or mobile-optimized web diary will take 20 minutes to complete per week, based on results of the Web Diary Feasibility Test.

6