Attachment F - Nonresponse Bias Analysis 2013

Attachment F - Nonresponse Bias Analysis 2013.docx

Telephone Point of Purchase Survey

Attachment F - Nonresponse Bias Analysis 2013

OMB: 1220-0044

Document [docx]
Download: docx | pdf












An Updated Nonresponse Bias Analysis of the Telephone Point of Purchase Survey:

Gabriela Arcos and Erik Bergmann

Based Upon Work by Patrick Falwell and Madeleine Saxton1

July 22th, 2013





  1. Introduction


The Telephone Point of Purchase Survey (TPOPS) is a random digit dialed (RDD) computer-assisted-telephone-interview (CATI) survey that collects expenditure information from households, including the list of grocery stores, retailers, service providers, and shops where purchases are made. The survey is used exclusively by the U.S. Bureau of Labor Statistics (BLS) as the primary source of the retail establishment sampling frame for the Commodities and Services (C&S) Pricing Survey. A sample of the retail establishments (hereafter, outlets) reported in TPOPS is selected for participation in C&S, and prices of select consumer goods and services at these outlets are monitored for use in calculating the U.S. Consumer Price Index (CPI). The TPOPS is a rotating panel survey with each household participating over four consecutive quarters. The TPOPS is conducted by the Census Bureau for the Bureau of Labor Statistics (BLS).


This paper is an update to selected portions of the 2009 work, “Nonresponse Bias in the Telephone Point of Purchase Survey: A summary of three studies by Patrick Falwell and Madeleine Saxton. It features amended tables and analysis, as well as preliminary results from a cellular phone frame, which began in the second quarter of 2012. Up until that point, respondents in the TPOPS were selected exclusively from a landline-based frame. Because of the declining number of landlines in the U.S, coupled with an increased prevalence of cellular-only households, a cell phone frame was added to the TPOPS. 2 The addition of the cell phone frame was expected to mitigate possible bias in the survey resultsattributed to the underrepresentation of certain demographic groups within the landline-only frame.



TPOPS response rates are examined in Section II below. Section III compares demographics reported in TPOPS against the Census Bureau’s American Community Service (ACS). In Section IV, outlet frame characteristics are compared and contrasted for various cohorts of the total TPOPS sample.

  1. Cellular and Landline Frame Results


Table 1 and Figure 1 show the response rate from the 1st quarter in 1997 to the 1st quarter in 2013. Response rates are calculated according to OMB’s response rate definition


Unweighted Response Rate3 = C a

C + R + NC + O + e(U)

Where,


C = number of completed interviews or sufficient partially completed interviews

R = number of refusals

NC = number of non-contacted sample units known to be eligible

O = number of eligible sample units not responding for reasons other than refusal

U = number of sample units of unknown eligibility, not completed

e = estimated proportion of sample units of unknown eligibility that are eligible,

(set to 0.27)


TPOPS response rates show a decreasing trend since BLS switched to an RDD/CATI survey in 1997. In the late 1990s, a response rate of close to 70% was achieved. By 2010, the response rate had fallen below 50%. From 2009 through 2011, response rates held somewhat steady, ending at 48.5% in the last quarter of 2011. With the introduction of the cell phone frame in the second quarter of 2012, overall response rates appear to have gone down, explained mainly by a lower response rate in the cell phone frame compared to the landline frame.

Table 1: Historic Response Rates by Quarter

Quarter

Frame

Interviews

Non Contacts

Refusals

Estimate of Unknown Population

Total

Response Rate

Q971

Landline

3,453

429

1,008

175

5,065

68.20%

Q972

Landline

6,330

940

1,736

357

9,363

67.60%

Q973

Landline

9,217

978

2,994

522

13,711

67.20%

Q974

Landline

12,086

1,266

3,800

741

17,893

67.60%

Q981

Landline

11,946

2,026

3,354

769

18,095

66.00%

Q982

Landline

12,000

2,419

3,363

748

18,530

64.80%

Q983

Landline

11,969

2,690

3,340

740

18,739

63.90%

Q984

Landline

11,996

2,312

3,601

911

18,820

63.70%

Q991

Landline

11,752

1,496

3,909

984

18,141

64.80%

Q992

Landline

11,514

1,284

4,034

793

17,625

65.30%

Q993

Landline

10,884

1,907

3,668

736

17,195

63.30%

Q994

Landline

11,502

1,667

4,019

738

17,926

64.20%

Q001

Landline

17,205

3,081

6,736

1,368

28,390

60.60%

Q002

Landline

16,666

4,004

6,455

1,303

28,428

58.60%

Q003

Landline

18,041

3,481

6,378

1,293

29,193

61.80%

Q004

Landline

17,904

2,320

6,094

1,300

27,618

64.80%

Q011

Landline

17,821

2,819

5,671

1,345

27,656

64.40%

Q012

Landline

13,680

2,357

4,731

963

21,731

63.00%

Q013

Landline

13,641

2,411

4,503

945

21,500

63.50%

Q014

Landline

13,824

2,290

4,424

936

21,474

64.40%

Q021

Landline

13,240

1,934

4,753

892

20,819

63.60%

Q022

Landline

13,166

2,559

5,144

997

21,866

60.20%

Q023

Landline

13,602

2,001

5,087

984

21,674

62.80%

Q024

Landline

13,684

2,178

5,136

955

21,953

62.30%

Q031

Landline

14,210

2,233

5,431

1,033

22,907

62.00%

Q032

Landline

14,084

2,920

5,019

1,016

23,039

61.10%

Q033

Landline

13,705

2,352

5,402

1,021

22,480

61.00%

Q034

Landline

13,759

2,132

5,618

1,076

22,585

60.90%

Q041

Landline

13,303

2,611

5,714

1,056

22,684

58.70%

Q042[1]

Landline

11,297

5,042

5,363

1,558

23,260

48.60%

Q043

Landline

12,643

4,364

4,664

1,535

23,206

54.50%

Q044

Landline

13,999

4,160

5,097

1,200

24,456

57.20%

Q051

Landline

14,245

4,562

5,223

1,271

25,301

56.30%

Q052

Landline

15,021

4,822

5,934

1,275

27,052

55.50%

Q053

Landline

14,961

4,796

5,808

1,226

26,791

55.80%

Q054

Landline

14,851

5,978

6,732

1,254

28,815

51.50%

Q061

Landline

15,009

6,983

7,825

1,338

31,155

48.20%

Q062

Landline

15,121

6,649

7,414

1,498

30,682

49.30%

Q063

Landline

14,939

7,532

5,857

1,489

29,817

50.10%

Q064

Landline

15,197

5,391

7,391

1,238

29,217

52.00%

Q071

Landline

13,276

7,130

6,553

1,365

28,324

46.90%

Q072

Landline

13,869

8,006

7,102

1,576

30,553

45.40%

Q073

Landline

13,538

7,332

7,032

1,386

29,288

46.20%

Q073

Landline

13,264

6,881

6,457

1,216

27,818

47.70%

Q081

Landline

13,207

6,506

5,898

1,101

26,712

49.40%

Q082

Landline

9,996

7,315

4,742

1,252

23,305

42.90%

Q083

Landline

11,507

5,992

5,458

1,125

24,082

47.80%

Q084

Landline

12,687

6,125

6,877

1,277

26,966

47.10%

Q091

Landline

13,083

7,487

6,193

1,370

43,144

46.50%

Q092

Landline

13,642

7,609

5,973

1,309

43,393

47.80%

Q093

Landline

14,107

6,842

6,293

1,329

43,988

49.40%

Q094

Landline

13,466

6,998

6,274

1,351

44,170

47.90%

Q101

Landline

13,761

6,586

6,139

1,400

43,996

49.40%

Q102

Landline

13,867

6,185

6,492

1,404

43,757

49.60%

Q103

Landline

13,705

6,024

6,773

1,317

43,526

49.30%

Q104

Landline

12,985

6,236

6,513

1,331

43,424

48.00%

Q111

Landline

13,434

5,860

6,066

1,324

42,940

50.40%

Q112

Landline

12,714

6,379

6,439

1,393

43,512

47.20%

Q113

Landline

13,057

6,461

6,209

1,199

43,984

48.50%

Q114

Landline

12,847

6,453

5,998

1,215

43,884

48.50%

Q121

Landline

12,765

6,308

5,970

1,249

43,303

48.60%

Q122

Combined

11,911

8,580

8,262

1,143

47,085

39.80%

Q122

Cell Phone

942

2,710

2,650

89

11,228

14.70%

Q122

Landline

10,969

5,870

5,612

1,054

35,857

46.70%

Q123

Combined

12,658

12,029

10,567

1,312

61,162

34.60%

Q123

Cell Phone

1,964

5,814

4,956

170

19,889

15.20%

Q123

Landline

10,694

6,215

5,611

1,142

41,273

45.20%

Q124

Combined

12,087

11,066

8,891

905

51,090

36.70%

Q124

Cell Phone

2,694

6,190

4,527

135

19,017

19.90%

Q124

Landline

8480

3881

4031

640.98

17032.98

49.8%

Q131

Combined

11748

10789

8942

812.43

32291.43

36.4%

Q131

Cell Phone

3,268

6,908

4,911

171.45

15,258

21.42%

Q131

Landline

9,393

4,876

4,364

769

32,073

48.40%


Figure 1


Table 2: Collection Results

Quarter

Frame

Interviews

Non Contacts

Refusals

Unknown

Q091

Landline

41.1%

23.5%

19.5%

15.9%

Q092

Landline

42.5%

23.7%

18.6%

15.1%

Q093

Landline

43.9%

21.3%

19.6%

15.3%

Q094

Landline

42.4%

22.0%

19.8%

15.8%

Q101

Landline

43.4%

20.8%

19.4%

16.4%

Q102

Landline

43.7%

19.5%

20.5%

16.4%

Q103

Landline

43.7%

19.2%

21.6%

15.5%

Q104

Landline

42.3%

20.3%

21.2%

16.1%

Q111

Landline

44.4%

19.4%

20.0%

16.2%

Q112

Landline

41.4%

20.8%

21.0%

16.8%

Q113

Landline

43.3%

21.4%

20.6%

14.7%

Q114

Landline

43.1%

21.7%

20.1%

15.1%

Q121

Landline

43.0%

21.3%

20.1%

15.6%

Q122

Combined

36.1%

26.0%

25.0%

12.8%

Q122

Cell Phone

14.2%

40.9%

40.0%

5.0%

Q122

Landline

41.6%

22.3%

21.3%

14.8%

Q123

Combined

31.6%

30.0%

26.3%

12.1%

Q123

Cell Phone

14.7%

43.5%

37.1%

4.7%

Q123

Landline

40.0%

23.2%

21.0%

15.8%

Q124

Combined

34.1%

31.3%

25.1%

9.5%

Q124

Cell Phone

19.4%

44.5%

32.5%

3.6%

Q124

Landline

43.7%

22.7%

20.3%

13.3%

Q124

Combined

34.1%

31.3%

25.1%

9.5%

Q124

Cell Phone

19.4%

44.5%

32.5%

3.6%

Q124

Landline

43.7%

22.7%

20.3%

13.3%

Q131

Combined

34.1%

31.3%

25.9%

8.7%

Q131

Cell Phone

20.8%

43.9%

31.2%

4.0%

Q131

Landline

45.2%

20.7%

21.5%

12.7%


  • From 2009-2011, response rates appear to be stationary around the 50% mark. This trend continues in the landline frame from the second quarter in 2012 through the first quarter of 2013. Underlying this trend is a large decrease in non contacts which is slightly offset by a small increase in refusals, as shown in Table 2.

  • At the time of this report, there are not enough data to analyze any long term trends in the cell phone frame, or the overall combined results. However, as evidenced in Table 2, the cell phone frame appears to have much higher refusal rates and non contact rates than the landline frame.


Table 3: Response Rate by Region 2005-2012

Area

Y2005

Y2006

Y2007

Y2008

Y2009

Y2010

Y2011

Y2012

East

Boston

52.7%

47.3%

45.3%

45.5%

46.5%

48.0%

49.5%

42.3%

Philadelphia

50.2%

44.0%

47.3%

48.5%

44.2%

48.3%

46.0%

42.0%

Pittsburgh

60.6%

56.8%

52.4%

52.5%

53.2%

59.4%

56.2%

45.8%

New York

39.3%

34.1%

29.5%

30.0%

33.2%

35.0%

33.9%

30.1%

NY-CT Suburbs

46.9%

44.1%

43.4%

42.8%

43.7%

47.4%

44.3%

40.0%

NJ Suburbs

49.9%

44.3%

42.0%

43.7%

42.6%

45.7%

45.7%

39.7%

X-Sized East

59.2%

44.9%

58.7%

55.8%

55.3%

56.6%

56.5%

47.7%

Average

51.3%

45.1%

45.5%

45.5%

45.5%

48.6%

47.4%

41.1%

Midwest

Chicago

51.5%

48.2%

45.4%

45.5%

44.8%

47.1%

45.2%

37.4%

Detroit

57.9%

55.3%

50.3%

50.4%

51.4%

51.4%

51.2%

40.6%

St. Louis

65.1%

56.5%

56.0%

56.2%

58.6%

54.8%

54.8%

38.0%

Cleveland

61.4%

56.7%

51.3%

54.3%

54.7%

52.6%

55.4%

45.3%

Minneapolis

61.7%

59.9%

58.1%

58.9%

58.6%

58.6%

60.1%

48.0%

Milwaukee

60.1%

59.0%

54.5%

58.0%

57.0%

60.4%

62.1%

51.7%

Cincinnati

60.8%

55.5%

58.1%

57.0%

54.9%

55.4%

55.6%

45.6%

Kansas City

61.6%

60.2%

58.2%

61.3%

58.2%

55.8%

61.8%

52.3%

X-Sized Midwest

63.8%

49.8%

58.5%

58.4%

58.9%

57.8%

59.3%

48.2%

Y-Sized Midwest

68.6%

61.7%

64.1%

61.7%

62.6%

61.3%

61.2%

45.8%

Average

61.3%

56.3%

55.5%

56.2%

56.0%

55.5%

56.7%

45.3%

South

Washington, DC

56.9%

51.3%

47.5%

49.0%

48.3%

49.3%

52.5%

44.7%

Baltimore

57.2%

55.3%

48.5%

48.1%

47.0%

47.9%

47.6%

39.2%

Dallas

55.3%

49.6%

44.1%

45.9%

44.7%

45.0%

46.0%

35.1%

Houston

50.2%

45.4%

42.9%

41.6%

38.8%

39.5%

37.9%

33.9%

Atlanta

44.7%

40.8%

40.2%

41.1%

41.4%

41.9%

43.7%

38.2%

Miami

37.7%

33.7%

29.9%

30.4%

33.4%

35.8%

32.7%

27.5%

Tampa

49.8%

45.9%

45.0%

46.7%

42.9%

46.4%

48.2%

41.8%

X-Sized South

55.5%

50.7%

47.5%

47.0%

47.6%

50.1%

49.8%

37.4%

Y-Sized South

59.2%

55.7%

50.2%

52.1%

50.5%

55.3%

51.6%

32.4%

Average

51.8%

47.6%

44.0%

44.7%

43.8%

45.7%

45.6%

36.7%

West

Los Angeles

44.6%

38.0%

35.8%

37.5%

38.4%

41.3%

36.8%

31.4%

LA Suburbs

53.3%

44.4%

40.0%

41.4%

43.7%

42.4%

45.1%

36.7%

San Francisco

50.6%

45.6%

42.9%

41.3%

42.2%

44.5%

42.7%

39.8%

Seattle

57.4%

53.2%

51.2%

50.3%

51.5%

53.5%

53.1%

44.9%

San Diego

52.8%

45.0%

41.7%

43.1%

43.2%

45.7%

45.2%

39.9%

Portland

62.6%

57.4%

56.1%

55.9%

57.4%

54.1%

54.4%

40.7%

Honolulu

46.3%

41.7%

41.7%

40.4%

41.4%

43.0%

43.9%

33.5%

Anchorage

61.4%

52.8%

51.4%

46.2%

46.5%

47.8%

45.7%

43.0%

Phoenix

55.1%

54.3%

48.4%

46.0%

47.2%

46.3%

45.6%

41.2%

Denver

59.0%

51.7%

52.2%

50.9%

51.5%

53.2%

53.2%

44.4%

X-Sized West

61.0%

48.7%

48.1%

51.4%

52.4%

51.6%

50.0%

39.4%

Y-Sized West

62.7%

62.2%

60.6%

58.2%

61.4%

61.6%

57.6%

41.2%

Average

55.6%

49.6%

47.5%

46.9%

48.1%

48.8%

47.8%

39.7%



  • Historically, the response rate of the Midwest has been much higher than the response rates of the other three regions. In 2011, for example, Midwestern areas had an average response rate of 56.7%, while non Midwestern areas averaged 47.0%.

  • The Midwest’s A-sized (self-representing) PSUs also had higher response rates than PSUs in other regions. In 2011, the Midwest’s A-sized PSUs had an average response rate of 55.8%, while the East, the South, and the West had average response rates of 45.9%, 44.1%, and 46.6%, respectively.

  • Additionally, the average decline in response rate from 2005 to 2011 for the Midwest has been low, at 7.5%, compared to the West and South’s respective declines of 14.0% and 12.0%. Though at a lower absolute level than the Midwest, the East’s decline has also been slow, at 7.7%.

  • New York City (A109) has traditionally had the lowest response rate, but was surpassed in recent years by Miami. The best performing PSU from 2009-2012 was Pullman, Washington.4

  • The X-sized PSUs (metropolitan non-self-representing) and the Y-sized PSUs (nonmetropolitan non-self-representing) (53.9% and 56.8% respectively in 2011) had higher average response rates than A-sized PSUs.

  • The introduction of the cell phone frame in 2012 has lowered response rates drastically on average.


Cell Phone Frame

Starting with the second quarter of 2012, the TPOPS dual frame design was introduced with the first panel of respondents. Over four quarters, additional panels were added to the cell phone frame until reaching its steady state sample size the first quarter of 2013.


Figure 2


Four key characteristics of the two frames (landline and cellular) and the respondents from each frame stand out:

  • The cell phone frame contains better telephone numbers than the landline frame; the percent of eligible panel 1 cases is 57% for cell phone and 36% for landline.

  • The cell phone frame includes a higher percentage of geographic screen outs, refusals and non interviews (grouped as eligible non interview in the graph).

  • There are almost four times as many geographic screen outs in the cell phone frame than the landline frame.

  • The sample inflation ratio (SIR) of cases that go out for interviews has held steady for the landline frame between 6 and 7 cases per interview. The cell phone frame fluctuated as the survey reached its steady state, and in the first quarter of 2013 the SIR was about 10.3 cases per interview.5


The cell phone frame is providing a quality frame of eligible respondents, but it is not as productive as the landline frame. It takes a larger cell phone sample size to reach the target number of interviews.


Summary

The TPOPS suffers from high nonresponse in both frames, however low response rates are not uncommon for RDD surveys. Furthermore, a survey with a low response rate but with respondents who are still representative of the target population suffers very little nonresponse bias. The low response rates in TPOPS, then, are only a problem if the nonrespondents differ from respondents. In the following section, we examine the demographics of TPOPS respondents as a judge for how closely they represent our target population.6


  1. Demographic Comparisons to ACS


A comparison of TPOPS demographic data to the Census Bureau’s American Community Survey’s (ACS) demographic data is necessary to analyze potential non-response bias in TPOPS. The American Community Survey, an extension of the Decennial Census, is widely considered the most accurate estimate of the actual composition of the population, due to its breadth and extremely high response rate (97%).7

TPOPS collects limited demographic data from the respondent, allowing only some demographic comparisons to ACS estimates. The TPOPS demographic data are limited to the respondent, and not the other members of the household. The TPOPS respondents’ demographics are compared against the demographic data of ACS householders, from 3-year estimates limited to the same geographic frame as the TPOPS.8 Because TPOPS weights on non-demographic factors, the data below are weighted TPOPS percentages.9 The observed weighted number of households in each demographic category were then compared with the expected number of households from ACS percentages. Using a chi square goodness of fit test, the population estimates in TPOPS for all demographic groups studied were statistically significantly different than the ACS estimates, below an alpha of <.001. That is to say there is less than a 0.1% chance that the distributions are actually equal.

  1. Race and Origin of Householder:


2008 2009


Table 4 Table 5

2008

Race of Householder

ACS Percentage

TPOPS Weighted Percentage

American Indian or Alaska Native

0.5%

0.5%

Asian

5.6%

2.9%

Black or African American

13.4%

8.0%

Native Hawaiian or Pacific Islander

0.1%

0.2%

Other

1.4%

4.2%

Two or More Races

5.5%

1.6%

White

73.5%

82.7%

 

 

 

Origin of Householder

ACS Percentage

TPOPS Weighted Percentage

Hispanic or Latino Origin

13.9%

8.8%

Not Hispanic or Latino Origin

86.1%

91.2%

2009

Race of Householder

ACS Percentage

TPOPS Weighted Percentage

American Indian or Alaska Native

0.4%

0.6%

Asian

5.6%

3.5%

Black or African American

13.5%

8.4%

Native Hawaiian or Pacific Islander

0.1%

0.3%

Other

1.5%

4.0%

Two or More Races

5.0%

1.7%

White

73.8%

81.5%

 

 

 

Origin of Householder

ACS Percentage

TPOPS Weighted Percentage

Hispanic or Latino Origin

13.9%

9.1%

Not Hispanic or Latino Origin

86.1%

90.9%



2010 2011


2010

Race of Householder

ACS Percentage

TPOPS Weighted Percentage

American Indian or Alaska Native

0.4%

0.5%

Asian

6.0%

4.4%

Black or African American

13.9%

9.5%

Native Hawaiian or Pacific Islander

0.1%

0.3%

Other

1.7%

4.2%

Two or More Races

4.6%

2.1%

White

73.2%

78.9%

 

 

 

Origin of Householder

ACS Percentage

TPOPS Weighted Percentage

Hispanic or Latino Origin

14.3%

10.1%

Not Hispanic or Latino Origin

85.7%

89.9%

Table 6 Table 7

2011

Race of Householder

ACS Percentage

TPOPS Weighted Percentage

American Indian or Alaska Native

0.4%

0.5%

Asian

6.1%

5.3%

Black or African American

13.9%

9.7%

Native Hawaiian or Pacific Islander

0.1%

0.4%

Other

1.8%

4.1%

Two or More Races

4.5%

1.8%

White

73.2%

78.2%

 

 

 

Origin of Householder

ACS Percentage

TPOPS Weighted Percentage

Hispanic or Latino Origin

14.5%

10.5%

Not Hispanic or Latino Origin

85.5%

89.5%


2012



Table 8

2012

Race of Householder

ACS Percentage
(2011)

TPOPS Weighted Percentage

American Indian or Alaska Native

0.4%

0.5%

Asian

6.1%

4.5%

Black or African American

13.9%

9.5%

Native Hawaiian or Pacific Islander

0.1%

0.3%

Other

1.8%

4.6%

Two or More Races

4.5%

1.7%

White

73.2%

79.0%

 

 

 

Origin of Householder

ACS Percentage
(2011)

TPOPS Weighted Percentage

Hispanic or Latino Origin

14.5%

9.6%

Not Hispanic or Latino Origin

85.5%

90.4%


Age Comparison


2008 2009


Table 9 Table 10


2008

Age of Householder

ACS Percentage

TPOPS Weighted Percentage

Under 35 years

20.2%

10.9%

35 to 44 years

21.5%

16.8%

45 to 54 years

22.3%

23.1%

55 to 64 years

16.8%

21.9%

65 to 74 years

9.9%

15.3%

75 to 84 years

6.9%

9.4%

85 years and over

2.5%

2.7%

2009

Age of Householder

ACS Percentage

TPOPS Weighted Percentage

Under 35 years

20.7%

8.9%

35 to 44 years

20.8%

16.4%

45 to 54 years

22.2%

23.4%

55 to 64 years

16.9%

22.2%

65 to 74 years

10.0%

15.8%

75 to 84 years

6.9%

10.1%

85 years and over

2.6%

3.2%


2010 2011


Table 11 Table 12


2010

Age of Householder

ACS Percentage

TPOPS Percentage

Under 35 years

20.4%

8.9%

35 to 44 years

20.1%

16.2%

45 to 54 years

22.1%

23.1%

55 to 64 years

17.5%

23.1%

65 to 74 years

10.4%

15.5%

75 to 84 years

6.8%

9.9%

85 years and over

3.5%

3.4%

2011

Age of Householder

ACS Percentage

TPOPS Percentage

Under 35 years

20.0%

7.7%

35 to 44 years

19.7%

14.9%

45 to 54 years

22.0%

22.5%

55 to 64 years

18.0%

24.8%

65 to 74 years

10.7%

16.4%

75 to 84 years

6.8%

10.3%

85 years and over

2.8%

3.4%


2012


Table 13

2012

Age of Householder

ACS Percentage
(2011)

TPOPS Percentage

Under 35 years

20.0%

14.6%

35 to 44 years

19.7%

14.2%

45 to 54 years

22.0%

19.6%

55 to 64 years

18.0%

22.8%

65 to 74 years

10.7%

17.4%

75 to 84 years

6.8%

8.6%

85 years and over

2.8%

2.8%






Figure 3

Group 43


Figure 4

Group 38



Comparison of Frames


Table 14

 

Landline

Cell

ACS (2011)

TPOPS Percentage

Under 35 years

6.1%

30.5%

20.0%

14.6%

35 to 44 years

12.5%

16.1%

19.7%

14.2%

45 to 54 years

20.5%

16.9%

22.0%

19.6%

55 to 64 years

24.7%

19.6%

18.0%

22.8%

65 to 74 years

21.3%

11.6%

10.7%

17.4%

75 to 84 years

11.5%

3.6%

6.8%

8.6%

85 years and over

3.4%

1.8%

2.8%

2.8%


Table 15

 

Landline

Cell

ACS (2011)

TPOPS Weighted Percentage

American Indian or Alaska Native

0.5%

0.5%

0.4%

0.5%

Asian

3.8%

4.7%

6.1%

4.5%

Black or African American

8.6%

10.4%

13.9%

9.5%

Native Hawaiian or Pacific Islander

0.3%

0.2%

0.1%

0.3%

Other

3.1%

6.8%

1.8%

4.6%

Two or More Races

1.4%

2.1%

4.5%

1.7%

White

82.3%

75.4%

73.2%

79.0%

 

 

 

 

 

 

Landline

Cell

ACS (2011)

TPOPS Weighted Percentage

Not Hispanic or Latino Origin

92.3%

88.4%

85.5%

90.4%

Hispanic or Latino Origin

7.7%

11.6%

14.5%

9.6%




Summary


As stated above, the differences between the TPOPS and ACS samples are statistically significant for race and age, for all years between 2008 and 2012, at an alpha <.001. The TPOPS race demographics are also significantly different from ACS, do not seem to be as systematically different as the age cohorts. The individual race cohorts appear to be systematically different than the ACS. The white, American Indian or Alaskan Native, and other cohorts were always overrepresented, while the black, Asian and multiracial cohorts were always underrepresented.10 The Hispanic cohort has also been traditionally underrepresented. With only 3 quarters of data, it remains to be seen what long-term effect the addition of the cell phone frame will have on these numbers. However, preliminary results show that it appears to mitigate these results.


The TPOPS has had a historic problem underrepresenting householders who are under 35 years old. Again, preliminary results from the cell phone frame show a marked increase in that percentage, as can be seen in Table 14. 2012 ACS data were unavailable at the time of publication, but when compared with the 2011 values, the cell phone frame appears to reach younger respondents, on average. Respondents 55 years of age and older are still overrepresented, but the effect seems to be mitigated. Since Falwell and Saxton also found the under 35 cohort to be the most systematic underrepresentation in TPOPS, close monitoring of this cohort is recommended as additional results from the cell phone frame and ACS become available.


As stated in the previous section, a survey can suffer from nonresponse bias if its respondents are not representative of the target population. The TPOPS sample is not perfectly representative of the target population. However, it should be noted that unless a demographic cohort differs markedly from others in its outlets and expenditures, under- or overrepresentation of a particular cohort in the TPOPS does not necessarily result in biased outlet sampling frames for the CPI. The next section examines these factors.

  1. Outlet Frame Comparisons Between Demographic Groups


This section evaluates whether the coverage issues first identified in 2009 produce an outlet bias in the survey. We compare reported expenditures and outlet names from the dual frame by age groups, race, and frame, to analyze whether survey results could be biased due to an underrepresentation of the race and age cohorts discussed in section three.

  • Hispanic Origin

  • Race Black

  • Age groups under 55 and over 55


The data analyzed include an average of 48,813 gross outlets reported per quarter. In both frames, each interview yields an average of 4 outlets.


Table 16

TPOPS Dual Frame Data by Quarter

 

2012 Qtr 2

2012 Qtr 3

2012 Qtr 4

2013 Qtr 1

Respondents from landline

10,982

10,705

9,401

8,483

Respondents from cell phone

943

1,971

2,699

3,274

Gross outlet reported landline

43,817

42,774

37,636

34,775

Gross outlet reported cell phone

4,084

7,990

11,078

13,098

Landline outlet yield per interview

4.0

4.0

4.0

4.1

Cell phone outlet yield per interview

4.3

4.1

4.1

4.0


Data collected during these four quarters are used to perform this preliminary outlet bias analysis. In the cell phone frame, these four quarters of data represent only one fourth of a complete administration of the survey due to its split questionnaire design where subsets of questions are asked in each geographic area. Results for all categories in all areas will be available in 4 years.


Expenditure Levels


Six separate Wilcoxon rank sum tests were performed on reported Point of Purchase (POPS) category expenditures by frame and four different demographic characteristics11. Figure 5 summarizes the number of categories within a major group found to have statistically different expenditures when tested by frame, age group, race, gender, or Hispanic origin 12

Figure 5


The high rates of statistically different expenditures were found in the major categories for Food and Beverages, Apparel, and Other Goods and Services where most of the data points in the shaded portion of the graph. More than half of the categories for Food and Beverages were found to be statistically different in five of the six groups. The race group for Black versus all other non White races is the only group where more Food POPs categories are similar than different.


The range of Food categories where expenditures were found to be statistically different included a variety of groups such as food away from home, spices, seafood, milk, and bread. The highest rates of statistically different food categories were between age groups under 55 and over 55, where 80% of categories were found to have differences in reported expenditures between groups. High rates of statistically different expenditures are also observed when evaluating groups by Hispanic origin, race, and frame. The high rate of statistically different expenditures for food categories may be driven by types of outlets that cater to specific demographic groups.


The next highest count of POPs categories having statistically different expenditures were categories for the major group Apparel. When testing expenditure differences between age groups under 55 and over 55 there are 48% of apparel categories found to have statistically different expenditures. When testing by Hispanic origin 52% of apparel categories had statistically different expenditures. This may be attributed to an income effect but this cannot be tested since TPOPS does not request respondents to provide information on household income.


Of the six tests run for age, race, gender, and Hispanic origin the tests of expenditures between age groups under 55 and over 55 had the most categories found to be statistically different. Not only were the majority of Apparel and Food categories found to be different but also Other Goods and Services, Housing, and Medical Care. Overall 54% of POPS categories were found to have expenditures statistically different between under 55 and over 55. This result is consistent with expenditure data collected from the Consumer Expenditure Survey which has found older cohorts have different spending patterns than younger cohorts. 13


Figure 6


Comparing the percent reporting by POPs Categories and demographic groups shows that Food categories are the most reported outlets across frames, age groups, and races. POPs Categories in the Medical care major group have the second highest percent reporting for cohorts that are over represented in the survey (older, white, and not Hispanic). Groups that are under-represented in TPOPS (younger, non-white, and Hispanic) on average have a higher rate of reporting for Apparel categories.


These differences in reporting likely contribute to the expenditure differences shown in Figure 5. Both results indicate there may be different shopping habits of the under represented groups.



  1. Outlet Rankings by Demographic Groups


The previous section examined differences in outlet expenditures and percent reporting between demographic groupings, while this section examines the rankings of reported outlets. Before analyzing the lists of outlets reported, the data were cleaned to minimize variation in outlet spelling. 14 Using four quarters of data from the dual frame, the frequency of outlet names were ranked within each demographic cohort at the Major Group level15

  • Top ranked outlets reported by Hispanic origin were compared to the list of all outlets reported by non Hispanic origin.

  • Top ranked outlets reported by all non white race groups were compared to the list of all outlets reported by the race group for white.

  • Top ranked outlets reported by the race group for black were compared to the list of all outlets reported by the race group for white.

  • Top ranked outlets reported by the age group for under 35 were compared to the list of all outlets reported by the age group over 35.

  • Top ranked outlets reported by in the cell phone frame were compared to the list of all outlets reported in the landline frame.


The results of the outlet overlap comparisons are summarized in Table 17 at the Major Group level for frame, race, Hispanic origin, and age group. In the list of top 10 through top 50 outlets for every group there is a high degree of overlap reporting across all major groups. Hispanic origin has the least amount of overlap in the larger list of top 100 and top 200 for most major groups. Similar results are observed when comparing the top 200 outlets reported by the race group Black to the race group White; six out of the eight major groups have less than 100% overlap.


Table 17


The final report on the TPOPS cell phone test included a similar analysis of outlet overlap on test data collected in the Consumer Expenditure Interview Survey (CE) in the second quarter of 201116. The data from CE reported a high degree of overlap in the top 20 outlets between landline and cell phone households. These results were found to be inconclusive due to the small sample. The results from four quarters of TPOPS data also show a higher rate of overlap between cell phone and landline respondents when evaluated at the major group level across all PSUs. This result is also preliminary given only four quarters of data were available at the time of the study.


Do any of these differences impact outlet selection for the CPI? To answer this question, analysis was also conducted at the PSU/POPs category level which is consistent with the administrative design of the TPOPS survey. The dual frame is compared to results of the landline portion of the survey to measure the impact of adding the cell phone frame.


Collected outlet data from the two most recent quarters, 2012 quarter 4 and 2013 quarter 1, along with expenditure and weighting results from the dual frame design, are used to calculate outlet probabilities at the PSU/POPs category level for the dual frame and for the landline only portion of the frame. The outlets for the two groups are ranked using the weighted probabilities within a PSU/POPs category cell and truncated at the top 5 outlets. The two ranked lists are matched on outlet names to perform a comparison of top 5 to top 5 outlets. A ratio is calculated for the number of outlet name matches to the number of ranked outlets (5 in most cases) within each PSU/POPs category. The table below summarizes the average rate of matching outlets in the top 5 from the dual frame and the top 5 from landline frame. The average percent of matching outlets for Education categories in PSUs from the Northeast is 89%, the highest among the regions and major groups. The lowest percent of matching outlet is observed in the major group Medical Care in PSUs from the West, where on average there is 52% matching outlets in these cells. Overall the dual frame does provide a high degree of matching outlets with the landline frame at the PSU/POPs category level when evaluating the top 5 ranked outlets.


Table 18


Although similar outlets appear in the top 5, the position of those outlets by their rankings in the dual frame and the landline only frame are different. The next part of the analysis compars the outlet rankings across the two dual frame and landline frame at the PSU/POPs category level. Over all PSU/POPs category combinations (2065 cells), 669 cells have outlet rankings that are the same in only 10% of cases. In 262 cells (PSU/POPs categories) the top 5 ranked outlets are identical in the dual frame data compared to the landline frame data. The cells with the largest amount of differences in outlet rankings are Food categories, followed by Housing.

Table 19


The data in Table 18 reveal that across PSUs and POPs categories almost 70% of the same outlets appear in the dual frame data and the landline only data of weighted and ranked outlets. Table 19 shows that there are many differences in the actual outlet rankings of the top 5 outlets from the dual frame to the landline only frame.


  1. Conclusion


The TPOPS still has trouble reaching respondents under 44 years of age, especially those under 35. However, it appears that in most cases, unique outlets and outlet expenditure do not appear to differ significantly between these and other age groups when comparing data across PSUs and at the major group level. The impact from the cell phone frame is only apparent when evaluating the ranking of outlets at the PSU/POPs category level. Furthermore, as outlined in Section II, the cell phone frame appears to somewhat mitigate the historic underrepresentation of younger respondents.


A potential concern is TPOPS’s underrepresentation of blacks and those of Hispanic origin. As shown in the previous sections, outlet reports and expenditures do differ in many cases between blacks and whites, and Hispanics and non-Hispanics. While no allowances were made for potential confounding variables (i.e. regional or income differences) in these analyses, the results still hint at a potential non-response bias in the TPOPS due to underrepresentation of these demographic cohorts. However, we cannot state conclusively that the underrepresentation translates to biased CPI outlet frames. Outlet sample sizes within each elementary item-area cell are relatively small, on average 2.3 outlets are selected per POPS Category-PSU cell. Even though the underrepresented groups do have different expenditure patterns, which result in different outlet rankings, it is not certain that their higher rates of nonresponse have an effect on realized sampling outcomes. This could be quantified in the future once more dual-frame data become available, via Monte Carlo simulations.




1 This paper is an updated version of “Nonresponse Bias in the Telephone Point of Purchase Survey:

A summary of three studies” August 4, 2009, by Patrick Falwell and Madeleine Saxton.


2 See, for example, NHIS estimates of cell-phone only households: http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201212.pdf

3 Response rate is calculated according to OMB’s response rate definition.

4 For a complete list of response rates by PSUs, see the Appendix, Table A.1.

5 The sample inflation ratio calculated here reflects phone numbers which have gone through screening to remove business phone numbers and non working numbers as much as possible.

8 ACS 3-year estimates were used due to the unavailability of more accurate 5-year estimates, and in favor of less accurate 1 year estimates. In the previous version of this paper, 1-year estimates were used, which were forgone this time in favor of accuracy. However, while the 3-year estimates are more accurate, they were unavailable in 2005-2007 (due to the ACS being a new survey), so analysis from that time period has been dropped from this paper.

9 TPOPS uses a variety of adjustment factors in weighting, including those accounting for probability of selection, household size, and non interview adjustment factors.

10 Falwell and Saxton found the black cohort to be overrepresented in 2006, however, this difference could be due to using the less accurate 1-year ACS estimates.

11 The Wilcoxon rank sum test is a non parametric alternative to the two sample t-test that does not make assumptions of normality and only requires a minimum of 5 observations in each sample.

12 The test results by non responses show a high degree of similarity in reported expenditures for all major groups, minimizing the concern that groups of non responders could be different than those who respond.

13 Cashell, Brian W. “A Seprate Consumer Price Index for the Elderly?” January 20, 2010. http://www.aging.senate.gov/crs/aging9.pdf.

14 The cleaning of outlet data was only performed on the names and not street address, so the analysis did not take into account attributes related to location.

15 The rankings are based off of the frequency of outlets reported, in the case where there’s a tie the same ranking is assigned. As result when listing the top 10 ranked outlets there might be more than 10 outlets if multiple outlets were reported at the same frequency.

16 Stockburger, Anya. Memo to OMB: Update on the Telephone Point of Purchase Survey cell phone frame planning. June 6, 2012.

40

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleAssessing nonresponse bias in the Telephone Point of Purchase Survey:
AuthorSaxton_M
File Modified0000-00-00
File Created2021-01-27

© 2024 OMB.report | Privacy Policy