0651-0057 SuptStmnt Part B Dec 20 2012

SUPPORTING STATEMENT
United States Patent and Trademark Office
Patents External Quality Survey (formerly Customer Panel Quality Survey)
OMB Control Number 0651-0057
(December 20, 2012)

B.

COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1.

Universe and Respondent Selection

The respondent pool for this survey is made up of the businesses, organizations, and
individuals who frequently file patent applications. The USPTO plans to survey large,
medium, and small-sized domestic corporations, universities and other non-profit
research organizations, and independent inventors. Foreign entities will not be included
in the sample frame.
The target population consists of individuals associated with USPTO top filers (e.g.,
firms at a given address who have filed six or more patent applications in the past year).
The sample unit will be the USPTO-registered agents/attorneys associated with the top
filers and also independent inventors that filed six or more patents in the past 12
months. The target population typically accounts for over 85% of all patent applications
filed in a given fiscal year.
2.

Procedures for Collecting Information

The Patents External Quality Survey will use a longitudinal, rotating panel design. The
USPTO has developed a sampling plan which is included in this submission. The
sampling plan also contains information about the respondent pool and the response
rate.
The sample is drawn from a frame of USPTO customers, all of whom are either
associated with a particular firm or are considered independent. There are six sampling
domains for which different sampling rates are used. One of these six sampling
domains is identified for each customer on the frame, using counts of the number of
applications within each firm in conjunction with a count of agents associated with that
firm. Then a sampling rate is computed for each domain.
The USPTO uses a rotating panel design for the sample, such that customers are
assigned to waves (survey period) and then to one of two panels within each wave.
The second panel from each wave is fielded in the subsequent wave, in addition to a
new panel.
After being selected for two consecutive waves, customers must stay out of the sample
for at least 18 months. Because of this 18 month leave of absence from the sample, it
is necessary to control for when the old sample can rotate back into the sample. A
complication is that there is some potential for panel conditioning effects from being in
the old cycle. Therefore, to reduce the impact from the distributional differences

between frames, newly sampled cases from old panels are spread out evenly across
the new panels.
Following a pre-notification letter that is sent to all potential respondents informing them
of the purpose of the survey and including instructions for completing the survey online,
the USPTO’s survey contractor, Westat, will mail the survey to all of the sampled
respondents. A personalized label will be inserted on the survey packet envelope in
order to reach the specified respondent. The survey packet will contain the paper
version of the questionnaire and a cover letter explaining that the USPTO is sponsoring
the survey, that all responses will only be used for internal analysis, and that no
identifying information will be linked to the results. . The cover letter will also contain
the username, password, and the 5-digit survey ID number. The electronic and paper
surveys will mirror each other.
During the follow-up non-response prompting calls, Westat employees will use a script
developed in collaboration with the USPTO. For all non-respondents a reminder
postcard will be sent to encourage survey participation.
The survey packet will include the three-page questionnaire and a postage-paid preaddressed return envelope. The cover letter will be printed on USPTO letterhead and
signed by the Commissioner of Patents.
3.

Methods to Maximize Responses

In order to maximize the number of responses received from the survey, the USPTO
plans to follow several well-established survey procedures.
First, all sampled
respondents will receive a pre-notification letter signed by the Commissioner. The letter
will explain the importance of the study and encourage respondent cooperation. Next,
all sampled respondents will receive the paper survey in the mail. Follow-up contact will
be made after the initial survey is sent. One week after the initial survey mailing, all
non-respondents will be sent a thank you/reminder postcard in the mail. Two weeks
after the initial survey is mailed, we will telephone all of the non-respondents to prompt
them to answer either the paper or internet version of the survey. A script has been
developed for these phone calls so that everyone conducting these interviews asks the
same questions, in the same manner.
Historic response rates for the Patents External Quality Survey are shown below.

2

In order to determine how the non-response bias affected the results of the survey,
Westat conducted a non-response follow-up study during Waves 6 and 7 for the survey.
The USPTO was provided with the analysis of the findings. The objective of the study
was to try to get a picture of how the non-respondents would have responded to the
main survey if they had actually responded to the survey. The study was conducted
because the non-responses can cause bias in the survey estimates, which is itself
affected by the response rate to the survey and the differences between those who
responded to the survey and those who did not respond.
As part of the study, Westat sent a postcard to those who did not respond to the original
survey and who now were rotating out of the survey sample. There were 200
customers that sent back their postcard out of about 619 mailed for the follow-up study.
The postcard contained one question concerning overall examination quality that was
asked in the original survey. This question was also asked in the original survey; the
only difference between the two was that the follow-up question had an additional
answer that was not included in the original survey. Half of the sample in the study
received a white postcard and the other half received a colored postcard to see if the
colored card would help increase the response rates.
The follow-up study compared the responses to the overall examination quality question
between those who responded to the question in the original survey in the outgoing
panel with those that responded to the follow-up postcard (who were also in the
outgoing panel). In this study, it is assumed that the respondents to the follow-up
survey are like the non-respondents to the original survey and that there are indications
that the survey non-responses are causing a potential bias. The results of the follow-up
study were used to help answer the following questions:
1. How different are the Wave 6/7 respondents from the follow-up respondents?

3

2. How different are the follow-up respondents from the follow-up non-respondents?
3. Do the results impact what can be done in weighting to reduce the bias due to
non-response?
4. What is the impact of the colored postcard on the follow-up response rates?

Non-response bias is affected by two different items: the non-response rate and the
differences between respondents and non-respondents. While the response/nonresponse rate is known, the differences between those who respond and those who do
not is unknown. The follow-up study attempts to measure the difference between the
respondents and non-respondents. The non-response bias is calculated using the
following equation for a sample mean:
Bias( yR )  (1  WR )(YR  YN ),

where WR is the weighted unit response rate, YR is the population mean of the
respondent stratum, and YN is the population mean of the respondent stratum and
is the population mean for the non-respondent stratum. While the response rate is
universally recognized as a measure of survey quality, the difference between the
respondents and non-respondents is just as important in determining the non-response
bias. Weighting adjustments are used to reduce the non-response bias (although some
non-response bias will remain in the survey estimates).
However, in the case with the non-response follow-up sample, the bias can be written
as





Bias y R   1  WR  YR  YFU  YNR



where YNR is the population mean of the follow-up non-respondent stratum, and YFU is
the population mean for the follow-up respondent stratum.
A bivariate analysis (response indicator versus each auxiliary variable) compares the
distribution of the participating households to the distribution of the total eligible sample
of households for several auxiliary variables. Survey base weights were used to
account for the unequal within-household probabilities of selection, and replicate
weights were used to adequately reflect the impact of the sample design on variance
estimates. The weights for the follow-up respondents were adjusted to account for nonrespondents to both the main survey and the follow-up. This assumes that nonrespondents were more similar to the follow-up respondents than the original survey
respondents. Together with the main sample respondent, the weights account for the
entire eligible population. Adjustment cells were created using the Search software
(WesSearch) using the same approach used in the normal weighting procedure.

4

To test for statistical differences, the distribution of the patent examination quality
question for the wave respondents was compared with the distribution for follow-up
respondents and similarly within the follow-up study for the white and colored postcards.
To test the categorical responses, the hypothesis of independence between the
characteristic and participation status was tested using a Rao-Scott modified Chi-square
statistic at the 10 percent level. The average score of the categorical responses was
computed as a continuous variable, with the larger average score the more favorable
the response. The difference between means was tested using a t test. The
continuous variables were tested using the Benjamin-Hochberg procedure to control the
overall false discovery rate for a family of comparisons.
Westat analyzed the results of this study, prepared a report and a summary of the
report, and submitted both to the USPTO. Some of the conclusions made concerning
the survey were:


There are no statistically significant differences detected between the main
survey and follow-up respondents in their categorical responses to the patent
examination questions for either Wave 6 or 7.



There are fairly large relative differences in both waves. These differences are
not detectable due to the large standard errors of the estimates from the followup study. The responses were generally more positive for the follow-up.



For the average responses, the overall averages were not significant.



There are only a few significant differences by characteristic while controlling the
overall false discovery rate using the B-H approach. It is expected that 10% of
the difference would be significant by chance. In Wave 6, only one of the fifteen
differences tested (6.7%) was significant, the sample domain for firms with less
than 150 applications. In Wave 7, two of the fifteen differences tested (13.3%)
were significant, agents and other registration numbers (those recently
registered).



In regards to postcard type, there were significant differences between the
different colors for response rates and categorical responses only in Wave 7.
The response rate was higher for the white postcard as was the proportion of the
fair category. The direction of the differences was not consistent in Wave 6.



For the average responses by postcard type, the overall averages were not
significant. There was only one significant difference (6.7%) by characteristic,
firms with registration numbers between 44155 and 50724, in Wave 7.

The USPTO and Westat plan on conducting another non-response study in Spring
2014.

5

4.

Testing of Procedures

To ensure the survey questions are meaningful to respondents and easy to understand,
Westat conducted four cognitive interviews with customers identified by the USPTO.
These customers are similar to the sampled respondents for the Patents External
Quality Survey study. The wording of the survey questions was then revised based on
feedback from these customers.
Low response rates have typically been observed in previous customer surveys
administered by the USPTO. The USPTO believes that offering both a paper and a
web response option will enhance response rates for this effort. The Patents External
Quality Survey was designed to focus only on key aspects of examination quality to
keep the time burden to a minimum and to help response rates.
When sending the mail survey out to sampled customers, we will use the wellestablished procedures documented by Dillman (2002). After the online version of the
survey is programmed, Westat will test the web survey internally to ensure respondents’
answers are properly captured and the survey is easy to navigate online. Westat will
also ensure that all computer security requirements are met.
5.

Contact for Statistical Aspects and Data Collection

The Office of Patent Quality Assurance of the USPTO is responsible for conducting the
Patents External Quality Survey. Martin Rater is the point of contact for this survey and
can be reached by phone at 571-272-5966 or by e-mail at [email protected]. The
names and telephone numbers for the individuals from Westat who consulted on the
statistical aspects of the survey and who are conducting the survey under the direction
of the USPTO are:
Kerry Levin, Ph.D.
Senior Study Director
Westat
301-738-3563
[email protected]
David Marker, Ph.D.
Senior Statistician
Westat
301-251-4398
[email protected]

6