Wave 19 Nonresponse Follow-Up Study

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

Authors
Shelley Brock Roth
Tom Krenzke

October 31, 2013

Prepared for:
Martin Rater
Office of Patent Quality Assurance (OPQA)
The United States Patent and Trademark Office

Prepared by:
Westat
An Employee-Owned Research Corporation®
1600 Research Boulevard
Rockville, Maryland 20850-3129
(301) 251-1500

Table of Contents
Chapter
1

Page
Introduction ........................................................................................................

1-1 

Methodology .......................................................................................................

2-1 

Analysis ................................................................................................................

3-1 

4

Results ..................................................................................................................

4-1 

5

Conclusions .........................................................................................................

5-1 

References ............................................................................................................

R-1 

The Benjamini-Hochberg Procedure...............................................................

A-1 

Percentage distribution of Q7 by response status: Overall ..........................

5-2

Percentage distribution of Q7 by response status: by Panel 19 ...................

5-2

Percentage distribution of Q7 by response status: by Panel 20 ...................

5-3

Average response of Q7 by selected categorical variables: Overall ............

5-4

Average response of Q7 by selected categorical variables: by
Panel 19 ................................................................................................................

5-5

Average response of Q7 by selected categorical variables: by
Panel 20 ................................................................................................................

5-6

2

3

References
R
Appendix
A
Tables
5-1. 

5-2. 

5-3. 

5-4. 

5-5. 

5-6. 

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

i

Introduction

1

The USPTO Customer Panel Quality Survey (CPQS) has been conducted for nineteen waves, with
each wave occurring every three or six months, using a rotating panel design. Each wave consists of
two panels, where one panel is surveyed for the first time and one panel is surveyed in its second
consecutive wave. Customers are randomly sampled within strata defined by the size of the firm to
which they belong, in terms of both number of customers and number of patent applications
submitted. The purpose of this report is to document the results of a nonresponse follow-up survey
that was conducted after wave 19. The follow-up sample consisted of nonrespondents1 that were
known to be eligible2 in the original sample, which resulted in a total of 1,033 follow-up cases that
were mailed follow-up postcards. Both panels of wave 19 within the rotating panel design were
eligible for the follow-up survey (panels 19 and 20). These panels were analyzed both together and
separately. The main wave 19 survey concluded on August 26, 2013. The nonresponse follow-up
survey mail out was on September 9, 2013, and data collection ended on October 9, 2013.
The nonrespondents were sent a postcard with one question (modification to Q7):
“Consider your experiences with USPTO Patent Examiners in the past three months. How would
you rate overall examination quality for this time period?”
The possible answers were:
1.

Very Poor

2.

Poor

3.

Fair

4.

Good

5.

Excellent

6.

Have not communicated with patent examiners in the past 3 months

1

These included nonrespondents, refusals, and cases that were ineligible in the prior 3 months but received a mailing in wave 19.

2

Cases with unknown eligibility status were dropped since they cannot be contacted (e.g., they are no longer at the firm).

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

1-1

Introduction

1

This question is the same item as Q7 in the original questionnaire, except that answer choice 6) was
only included in the follow-up survey. The follow-up survey was conducted mainly to help
determine how different wave 19 main survey respondents are from main survey nonrespondents.
This was done by comparing the responses to the Q7 question between main survey respondents
and follow-up postcard respondents. The assumption was that the respondents to the postcard
follow-up (who were originally main survey nonrespondents) would be more like main survey
nonrespondents than they are like main survey respondents in terms of their response to Q7. If this
assumption held, it would potentially indicate some bias due to nonresponse.

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

1-2

Methodology

2

Nonresponse bias is measured by two terms: the nonresponse rate, and differences between
respondents and nonrespondents. To explain further, we introduce the following expression for
nonresponse bias for a sample mean ( y R ):
Bias ( y R )  (1  WR )(YR  YN ),

where WR is the weighted unit response rate, YR is the population mean of the respondent stratum,
and YN is the population mean for the nonrespondent stratum. While the response rate (first
component) is universally recognized as a measure of survey quality, it is not by itself a good
indicator of nonresponse bias. The difference between respondents and nonrespondents (second
component) is just as important. Theoretically, even if the response rate is 55 percent, as it was in
wave 19, if there is no difference in the mean of the characteristic y between respondents and
nonrespondents, then bias does not exist. In practice, the second component is unknown; however,
typically proxies (auxiliary data) are used to estimate the difference. Weighting adjustments are used
to reduce nonresponse bias, although it is widely recognized that some nonresponse bias remains in
survey estimates.
For all analyses, survey base weights were used to account for the unequal within-household
probabilities of selection, and were adjusted to account for cases with unknown eligibility status.
Replicate weights were used to adequately reflect the impact of the sample design on variance
estimates. The weights for the main survey respondents were the wave 19 base weights, adjusted for
unknown eligibility status. The weights for the follow-up respondents were additionally adjusted to
account for individuals who did not participate in both the main survey and the follow-up. This
assumes that main survey nonrespondents were more similar to the follow-up respondents than they
were to the main survey respondents. Together with the main survey respondents, the weights
account for the entire eligible population.
For the nonresponse adjustment, the classification software package Search was used to create the
initial adjustment cells for nonresponse. Search employs a hierarchical tree algorithm described in
Sonquist, Baker, and Morgan (1974). Cell sizes were limited to 23 or more cases in each analysis.
(The Search software is a freeware product developed and maintained by the University of
Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

2-1

Methodology

2

Michigan.) The chi algorithm in Search produces a classification tree, which reveals the domains as
defined by combinations of variables with the most differential response rates, thereby leading to
domains with the highest potential for nonresponse bias. This is the same approach as is used in the
main survey weighting procedures.

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

2-2

Analysis

3

Analyses were performed using 1,329 main survey respondents compared with 202 follow-up
respondents; all with nonmissing responses to Q7. A bivariate analysis (response indicator versus
each auxiliary variable) was used to compare the distribution of the main survey participants to the
distribution of the follow-up sample participants for both Q7 and for the mean of Q7 across several
auxiliary variables.
Follow-up respondents who answered “No applications in the past 3 months” were excluded from
the analysis as ineligible. Respondents from both the main survey and the follow-up who were
missing answers to Q7 were also excluded from the analysis.
Two approaches were used to test for statistical differences between main survey respondents and
follow-up respondents. First, to test differences in the categorical responses to Q7, the hypothesis of
independence between the characteristic and participation status was tested using a Rao-Scott
modified Chi-square statistic at the 10 percent level (see Appendix B of the WesVar User’s Guide
for more information at http://www.westat.com/Westat/pdf/wesvar/WV_4-3_Manual.pdf).
Secondly, to see if there were differences within subgroups, we computed a mean score of the
categorical responses to Q7 for each subgroup, treating Q7 as a continuous variable, with a larger
mean score indicating a more favorable response. The difference between means was tested using a t
test, which was adjusted using the Benjamini-Hochberg procedure (Benjamini and Hochberg, 1995;
Thissen, 2002), to control the overall false discovery rate for a family of comparisons. The B-H
critical values are shown in the appendix.
The absolute and relative differences between the respective estimates for the main survey
respondents and the follow-up respondents are given in each table. The relative difference is
calculated as the absolute difference divided by the estimate from the main survey respondents. The
relative difference is a measure of the size of the difference compared to the main survey estimate.
As mentioned earlier, all analyses are presented for main survey respondents and follow-up
respondents both together and separately for panels 19 and 20.

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

3-1

Results

4

The results are shown in tables 5-1 through 5-6. In tables 5-1 through 5-3, we present the
distribution of Q7 with the five categorical responses, and also with three categories by collapsing
“very poor” with “poor” and “good” with “excellent”, both overall and separately for panels 19 and
20. Secondly, in tables 5-4 through 5-6, we show the overall mean response of Q7 and also the mean
of Q7 by selected characteristics, both overall and separately for panels 19 and 20.
For the overall set of main survey and follow-up respondents, the chi-square test result is significant
for the full distribution of Q7 (table 5-1), with the follow-up respondents 9 percentage points higher
in the response of “excellent”. Table 5-2 (second time panelists, i.e. panel 19) also shows a
significant chi-square result with the follow-up respondents more than 13 percentage points higher
in the “excellent” category. While a difference of about 4 percent for the “excellent” category was
observed for new panelists (panel 20 in table 5-3), it is not significant. The recode of Q7 does not
show a significant difference between the main survey and the follow-up, overall or for either of the
panels. However, the “good/excellent” category is 2 percent higher for follow-up respondents
overall.
The difference between main survey respondents and follow-up respondents for the mean of Q7
was not statistically significant overall or for either of the two panels. For the subgroups, after
adjusting for multiple comparisons, there were six significant differences, as follows:


Table 5-4: Census region=Northeast (both panels together)



Table 5-4: newest registered customers (both panels together)



Table 5-5: Census region=Northeast (panel 19)



Table 5-5: type of customer=attorney (panel 19)



Table 5-5: second oldest registered customers (panel 19)



Table 5-6: type of customer=agent (panel 20)

In all but one of these results (table 5-4, newest registered customers) the follow-up respondents had
a more favorable response.
Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

4-1

Results

4

Finally, table 5-4 also shows the overall estimated bias for the mean of Q7 and for each subgroup,
calculated using the corresponding weighted wave 19 response rate for each one. These results
show that the overall bias is small but the amount of bias varies across the subgroups, as expected
given the differences among the subgroup estimates.

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

4-2

Conclusions

5

Based on this analysis, the conclusions are:


Statistically significant differences that were detected between the main survey and
follow-up respondents in the categorical responses to Q7 seem to indicate a somewhat
brighter outlook regarding the quality of patent examinations, particularly in second
time panelists (panel 19).



For the mean responses to Q7, both overall and by panel, there were no significant
results.



There are only a few significant differences by characteristic while controlling the
overall false discovery rate using the B-H approach. The subgroups with significantly
brighter outlooks regarding the quality of patent examinations are:

–

Customers in the northeast region (both panels together and panel 19)

–

Customers who are attorneys (panel 19)

–

Customers who are in the second oldest registered group (panel 19)

–

Customers who are agents (panel 20)

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

5-1

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

Table 5-1.

Percentage distribution of Q7 by response status: Overall

Characteristic
Q7
1-very poor
2-poor
3-fair
4-good
5-excellent
Q7 collapsed
1/2 very poor/poor
3-fair
4/5 good/excellent

Table 5-2.

Main survey
(percent)

Standard error

Follow-up
(percent)

Standard error

Difference

Relative
difference

5-2

1.84
7.29
38.16
48.90
3.81

0.49
0.96
1.64
1.56
0.56

1.58
8.98
34.65
42.16
12.64

1.16
2.30
3.63
4.48
3.30

-0.263
1.683
-3.515
-6.736
8.830

-0.143
0.231
-0.092
-0.138
2.318

9.14
38.16
52.71

1.04
1.64
1.55

10.56
34.65
54.80

2.55
3.63
3.85

1.420
-3.515
2.095

0.155
-0.092
0.040

Difference

Relative
difference

-1.126
2.692
-9.001
-5.977
13.412

-0.611
0.386
-0.230
-0.126
3.113

Chi-square
p-value
0.0251

0.6617

Percentage distribution of Q7 by response status: by Panel 19

Characteristic
Q7
1-very poor
2-poor
3-fair
4-good
5-excellent
Q7 collapsed
1/2 very poor/poor
3-fair
4/5 good/excellent

Main survey
(percent)
2.06
6.97
39.07
47.60
4.31

Standard error
0.74
1.29
2.20
2.20
0.85

Follow-up
(percent)
0.94
9.66
30.06
41.62
17.72

Standard error
0.89
3.42
5.43
5.70
4.91

Chi-square
p-value
0.0113

0.2920
9.03
39.07
51.91

1.45
2.20
2.23

10.59
30.06
59.34

3.52
5.43
5.57

1.566
-9.001
7.435

0.173
-0.230
0.143
Conclusions

5

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

Table 5-3.

Percentage distribution of Q7 by response status: by Panel 20

Characteristic
Q7
1-very poor
2-poor
3-fair
4-good
5-excellent
Q7 collapsed
1/2 very poor/poor
3-fair
4/5 good/excellent

Main survey
(percent)

Standard error

Follow-up
(percent)

Standard error

Difference

Relative
difference

1.62
7.63
37.23
50.23
3.30

0.66
1.20
2.28
2.17
0.73

2.20
8.32
39.05
42.68
7.75

2.12
3.06
5.31
6.43
3.47

0.583
0.690
1.823
-7.552
4.457

0.317
0.090
0.049
-0.150
1.352

9.25
37.23
53.53

1.36
2.28
2.21

10.52
39.05
50.43

3.68
5.31
5.90

1.273
1.823
-3.095

0.138
0.049
-0.058

Chi-square
p-value
0.6349

0.8794

5-3
Conclusions

5

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

Table 5-4.

Average response of Q7 by selected categorical variables: Overall

Characteristic

5-4

Q7
Census region (CREG)
Northeast
Midwest
South
West
Agent/attorney (TYPE)
Agent
Attorney
Sample domain (DOMAIN)
Large firms, 50 customers or less
Large firms, more than 50 customers
Firms, number of applications between
150 and 275
Firms, less than 150 applications
Top-filer firms or independent inventors
Registration number (REG_NO_R)
Oldest registered customers
Second oldest registered customers
Second newest registered customers
Newest registered customers

Main
survey
(mean)
3.46

Standard
error
0.02

Follow-up
(mean)
3.55

Standard
error
0.07

Difference
0.098

Relative
difference
0.028

t test
p-value
0.2260

Estimate of
bias
0.044

3.49
3.47
3.46
3.40

0.05
0.05
0.04
0.06

3.89
3.53
3.53
3.27

0.16
0.12
0.13
0.18

0.399
0.059
0.068
-0.128

0.114
0.017
0.020
-0.038

0.0182*
0.6592
0.6013
0.5183

0.178
0.026
0.031
-0.055

3.40
3.47

0.06
0.03

3.55
3.56

0.18
0.08

0.156
0.096

0.046
0.028

0.3849
0.2953

0.066
0.043

3.44
3.50

0.03
0.04

3.50
3.52

0.09
0.14

0.061
0.019

0.018
0.005

0.5336
0.9025

0.029
0.010

3.44
3.45
3.50

0.06
0.04
0.19

3.30
3.69
3.40

0.14
0.15
0.22

-0.138
0.239
-0.100

-0.040
0.069
-0.029

0.3974
0.1182
0.7165

-0.059
0.097
-0.056

3.52
3.49
3.38
3.43

0.06
0.05
0.05
0.05

3.67
3.71
3.51
3.12

0.14
0.14
0.20
0.11

0.153
0.221
0.132
-0.309

0.044
0.063
0.039
-0.090

0.3158
0.1374
0.5115
0.0107*

0.063
0.100
0.061
-0.140

* Significant under B-H approach

Conclusions

5

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

Table 5-5.

Average response of Q7 by selected categorical variables: by Panel 19
Characteristic

5-5

Q7
Census region (CREG)
Northeast
Midwest
South
West
Agent/attorney (TYPE)
Agent
Attorney
Sample domain (DOMAIN)
Large firms, 50 customers or less
Large firms, more than 50 customers
Firms, number of applications between
150 and 275
Firms, less than 150 applications
Top-filer firms or independent inventors
Registration number (REG_NO_R)
Oldest registered customers
Second oldest registered customers
Second newest registered customers
Newest registered customers

Main survey
(mean)
3.45

Standard
error
0.04

Follow-up
(mean)
3.66

Standard
error
0.11

Difference
0.204

Relative
difference
0.059

t test
p-value
0.0700

3.45
3.55
3.39
3.44

0.07
0.07
0.07
0.08

4.07
3.57
3.54
3.57

0.20
0.15
0.21
0.23

0.629
0.022
0.151
0.130

0.183
0.006
0.045
0.038

0.0030*
0.8950
0.4810
0.6090

3.38
3.46

0.10
0.04

3.28
3.74

0.24
0.12

-0.099
0.278

-0.029
0.080

0.6900
0.0270*

3.46
3.50

0.04
0.05

3.38
3.70

0.13
0.22

-0.081
0.201

-0.023
0.057

0.5710
0.3690

3.41
3.43
3.87

0.08
0.07
0.22

3.57
3.84
3.28

0.26
0.21
0.39

0.153
0.418
-0.590

0.045
0.122
-0.152

0.5760
0.0570
0.1770

3.53
3.46
3.31
3.49

0.08
0.07
0.09
0.06

3.70
3.87
3.44
3.22

0.22
0.17
0.20
0.20

0.168
0.409
0.129
0.269

0.048
0.118
0.039
0.077

0.4770
0.0220*
0.5540
0.2020

* Significant under B-H approach

Conclusions

5

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

Table 5-6.

Average response of Q7 by selected categorical variables: by Panel 20
Characteristic

5-6

Q7
Census region (CREG)
Northeast
Midwest
South
West
Agent/attorney (TYPE)
Agent
Attorney
Sample domain (DOMAIN)
Large firms, 50 customers or less
Large firms, more than 50 customers
Firms, number of applications between
150 and 275
Firms, less than 150 applications
Top-filer firms or independent inventors
Registration number (REG_NO_R)
Oldest registered customers
Second oldest registered customers
Second newest registered customers
Newest registered customers

Main survey
(mean)
3.46

Standard
error
0.03

Follow-up
(mean)
3.46

Standard
error
0.10

Difference
-0.005

Relative
difference
-0.001

t test
p-value
0.9660

3.53
3.39
3.54
3.35

0.08
0.06
0.06
0.08

3.75
3.50
3.52
2.92

0.24
0.19
0.13
0.22

0.221
0.106
-0.014
-0.426

0.063
0.031
-0.004
-0.127

0.3870
0.6000
0.9210
0.0840

3.41
3.48

0.08
0.04

4.00
3.41

0.00
0.11

0.588
-0.070

0.172
-0.020

<0.0001*
0.5850

3.41
3.49

0.04
0.07

3.67
3.34

0.12
0.20

0.257
-0.155

0.075
-0.044

0.0630
0.5000

3.46
3.47
3.25

0.07
0.06
0.26

3.09
3.55
3.50

0.15
0.19
0.24

-0.365
0.079
0.250

-0.106
0.023
0.077

0.0480
0.7040
0.4840

3.50
3.52
3.45
3.35

0.08
0.06
0.08
0.06

3.64
3.52
3.58
3.05

0.18
0.19
0.32
0.13

0.140
0.001
0.131
-0.298

0.040
0.000
0.038
-0.089

0.4930
0.9940
0.6900
0.0420

* Significant under B-H approach

Conclusions

5

References

R

Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and
powerful approach to multiple testing. Journal of the Royal Statistical Society, B(57), 289-300.
Sonquist, J., Baker, E., and Morgan, J. (Eds.). (1974). Searching for structure. (Rev. ed.). Ann Arbor, MI:
Michigan Institute for Social Research.
Thissen, D., Steinberg, L., and Kuang, D. (2002). Quick and easy implementation of the BenjaminiHochberg Procedure for controlling the false positive rate in multiple comparisons. Journal of
Educational and Behavioral Statistics, Spring, 2002(27), 77-83.

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

R-1

Appendix A
The Benjamini-Hochberg Procedure

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

A-1

Appendix

A

The Benjamini-Hochberg Procedure

To improve the power of tests involving multiple comparisons, the Benjamini-Hochberg (1995)
procedure offers an approach to control the overall false discovery rate (FDR); that is, it controls the
proportion of significant results that are Type I errors. The B-H critical value (αi) for the each test i as
sorted in descending order by p-value, is computed as follows:
Pi 

(G  i  1)  0.10
G

Benjamini-Hochberg critical values for the USPTO wave 19 nonresponse follow-up survey:
Index
2 levels
1
2
4 levels
1
2
3
4
5 levels
1
2
3
4
5

Customer Panel Quality Survey (CPQS) Wave 19
Nonresponse Follow-up Study

B-H critical value
0.100
0.050
0.100
0.075
0.050
0.025
0.100
0.080
0.060
0.040
0.020

A-2