External Quality Sampling Plan

Westat’s Technical Capabilities
2.0

Requirements as Set Forth in the RFQ

2.1

Survey/Project Management

2

In order to meet the timeline for deliverables and the technical requirements of this work, Westat
proposes a strong management structure with ample technical knowledge of USPTO’s research
needs, flexibility, and staff coordination to collect, analyze, and report on the EQS. The assignment
of responsibilities to staff are based on matching appropriate corporate and individual skills to the
tasks and timeline. It emphasizes complementary skills, role relationships, and provision of sufficient
resources to maintain the work schedule at the high levels of quality that the EQS demands. Our
project plan, developed and refined over the course of 16 administrations of the EQS, details major
milestones in data collection, data processing, and data delivery. In keeping with our current
practice, discrepancies between proposed and actual dates will be documented along with remedies
for resuming project progress.

2.2

Design, Recruit, and Maintain Customer Panel

Under this task, we will design, select, and maintain a sample of rotating panels of continuous patent
customers. We understand that the current as well as intended field schedule for the EQS requires
two data collection waves per year (January and July). The sample design objectives include:

1

1.

Minimizing reporting burden;

2.

Accounting for, at a minimum, the frequency of customer contact, volume of patent
activity, and technology areas;

3.

Producing reliable estimates of wave-to-wave change;

4.

Producing estimates for each of the waves;

5.

Maintaining the customer panel to preserve representativeness;

6.

Adding new members to the panel to compensate for attrition; and

7.

Tracing lost members to maintain high response rates. 1

Since FY09-Q1, weighted response rates to the EQS have averaged about 50 percent.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

2

WPN 12-525

Westat has completed work on the next sample frame for the EQS. Having conducted this task
twice before, we employed proven processes and procedures to execute the necessary steps to
obtaining the cleaned frame.

Step 1: Creating the FIRMS File
Below is a description of the processes and procedures involved in creating the FIRMS file. An
initial listing of 4,938 top-filing firms was provided for use in sampling frame development and
matching to a separate file of customers. From this initial file, 192 firms were dropped due to a
location outside of the United States. The sample cleaning process began by de-duplicating the file
of top-filing firms. In addition to creating a single record for each of the top-filer firms, it was
necessary to sum the number of patent applications filed by the firm across all duplicate entries.
De-duplication involved a combination of an iterative automated process using SAS and manual
cleaning. We began by running a SAS program to identify all exact duplicate entries in the firms file,
based on state, city, and organization name. However, as the information for the firms was not
entered in a standardized fashion (both the text entry for a single firm differed across duplicate
records as well as the placement of the information across the columns), extensive manual review
and editing were required to de-duplicate the firms records.
Manual cleaning involved arranging the file so information was placed in the appropriate columns,
sorting the file by state, city, and organization name and then reviewing the file to identify and
standardize duplicate firms. During this process we used the following guidelines:


If multiple locations were listed for a top filing firm, these different locations were
retained as separate records in the firms file.



If there were separate records for different people at a top filing firm, these records
were de-duplicated to a single listing for the top filing firm.

A total of 535 records were identified as duplicate firms. The final number of firms available for
sampling and matching was 4,211. A total of 807 of these firms did not have matches to the
customer file; therefore, as discussed in the next section, 3,404 firm records had at least one
matching customer record, and comprise the dataset referred to as FIRMS.

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

3

WPN 12-525

Step 2: Creating the CUSTOMERS File
An initial file of 41,981 customers was provided to match to the listing of top-filer firms. From this
initial file, 1,170 customers were dropped due to a location outside of the United States; 1,330 were
dropped due to recent registration; and 27 customers were dropped due to both issues. The final
number of customers available for sampling and matching was 39,454.

Step 3: Matching the FIRMS file to the CUSTOMERS File
The process of matching top-filing firms with registered customers also consisted of several phases.
First, a shortened version of organization name, with all commas, spaces, and other characters
removed was created. Then the FIRMS file and the CUSTOMER files were sorted by STATE,
ORG NAME, and CITY, and automated matching was done. A flag was set for any records where a
firm matched to a customer record. A total of 10,980 records were matched to a firm during this
process.
Next, the matched customer file was split into three files for research assistants to do manual
matching. Again, the matching was based on STATE, ORG NAME, and CITY, and if the matcher
found a match between the files, the value for the matching firm, FIRMID, was entered in Excel on
the customer file. An additional 4,567 customer records were matched to firms during this process,
including reviewing all non-matching firms manually in an attempt to match the records.

Sample Selection Summary
Now that the cleaned frame is ready, we are prepared to select the next four waves of customers to
contact for the next four administrations of the EQS. The USPTO sample is drawn from a frame of
PTO customers, all of whom are either associated with a particular firm or are considered
independent. There are six sampling domains for which different sampling rates are used. These are
described generally as follows:
1.

Large firms (more than 275 applications) with 50 or fewer customers, where all
customers will be sampled;

2.

Large firms (more than 275 applications) with more than 50 customers, where a sample
of customers will be drawn;

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

4

WPN 12-525

3.

Firms with 150 to 275 applications where a sample of customers will be drawn;

4.

Firms with less than 150 applications where a sample of customers will be drawn;

5.

Non-matched firms and independent inventors with an associated last name, where a
sample of customers will be drawn; and

6.

Non-matched firms and independent inventors without an associated last name, where
no sample will be drawn.

One of these six sampling domains is identified for each customer on the frame, using counts of the
number of applications within each firm in conjunction with a count of agents associated with that
firm. A sampling rate is then computed for each domain.
The USPTO uses a rotating panel design for the sample, so that sampled cases are assigned to waves
and then to two panels within each wave. The second panel from each wave is fielded in the
subsequent wave, along with a new panel.
Once the sampling rates have been determined and the sample of customers drawn, sample cases are
assigned to the waves (which occur 6 months apart) and also to the two panels within each wave.
Customers must stay out of the sample for at least 18 months. Because of this 18-month leave of
absence from the sample, it is necessary to control for when the old sample can rotate back into the
sample. Complicating this is the potential for panel conditioning effects from being in the old cycle.
Therefore, to reduce the impact from the distributional differences between frames, newly sampled
cases from old panels are spread out evenly across the new panels.

2.3

Survey Instruments

With each approaching wave of data collection, Westat staff will prepare the mail and Web versions
of the surveys. Each will be updated to reflect the current wave date and checked for accuracy in
printing. The survey currently covers examination quality concerns, importance of targeted
improvement areas, and assessment of USPTO progress from the previous wave time period. In
addition to ongoing measurement of customers’ perceptions about these core survey topics, the
USPTO plans to use the panel study to provide timely information about emerging issues. That goal
will be accomplished by adding new modules to the core survey as needed. We will follow the same
guidelines and procedures (starting with discussions with the client) used for the core survey to
develop the new modules. We recommend including new modules as a single section, located at or

Use or disclosure of data contained on this page is subject
to the restriction on the title page of this proposal or quotation.

5

WPN 12-525