CPQS: Summary of the Nonresponse Followup Analysis

Date:

January 8, 2009

To:

Martin Rater, USPTO

From:

Tom Krenzke and David Ferraro, Westat

Subject:

CPQS: Summary of the Nonresponse Followup Analysis

Memo: 37v1

Introduction
After the field period for Waves 6 and 7, a postcard with the overall examination quality item (Q7)
was sent to nonrespondents that were rotating out of the sample. The objective was to get a
picture of how the nonrespondents would have responded to the main survey had they actually
participated. Nonresponse can induce bias into survey estimates, and the amount of bias is
commonly thought of as a function of two components: the response rate (which we know), and
the difference between those that respond and those who do not respond (which we don’t know).
Sending a followup postcard to nonrespondents and comparing their responses to the same item
from the main survey is an attempt at measuring that difference in the second component.
Results Summary
There were about 200 customers in each wave that sent back their postcard out of about 600.
The wave samples were combined to allow for more statistical power when making comparisons.
A chi-square test shows a significant relationship between the Q7 responses and whether or not
the respondent came from the wave sample or the followup sample. As you can see, follow-up
responses were 9 percentage points higher for the Good/Excellent category.
Wave Respondents
Q7
Very poor/Poor
Fair
Good/Excellent

Follow-up Respondents

Estimated
Percent

Standard
Error

Estimated
Percent

24.78
46.70
28.52

1.452
1.762
1.580

21.86
40.17
37.97

Difference

Standard
Error Estimate
2.627
3.570
3.107

-2.92
-6.53
9.45

Standard
Error
3.024
4.237
3.841

ChiSquare
p-value
0.0397

-2-

Impact of Results
While there is likely potential for bias in the followup sample estimates, the results are an
indication that the overall examination quality is more favorable than is currently presented in the
wave analysis reports. In addition, based on this result, waves with lower response rates relative to
other waves could actually have a higher ‘hidden’ boost than waves with higher response rates.