Supporting_Stmt_B_PRA

Part B. Collection of Information Employing Statistical Methods
1. Respondent Universe and Sampling Methods
Surveys of Eligible Professionals and Administrators
The respondent universe for the eligible professional survey consists of all professionals
eligible to participate in PQRS or eRx. The respondent universe for the administrator survey
is all administrators who work in the practices of professionals eligible to participate in
PQRS or eRx. A random sample of 8,400 eligible professionals and administrators will be
selected for the study. (Attachment A.)
Sample size and sampling techniques: Stratified random sampling will be used to select a
nationally representative sample of eligible providers. Strata are formed based on the
variables available in the frame and the specific requirement of the evaluation study. Strata
variables are specialty type, group practice reporting option (GPRO), participation status, and
reporting mechanism. Then a simple random sample will be selected with equal probability
within each stratum.
The eligible professional survey targets 2,100 completes, 1,050 from the participating group
and 1,050 from the nonparticipating group. With an estimated 50-percent response rate, we
expect to contact about 4,200 providers. With each provider practice, we will sample the
practice administrator—an additional 4,200 surveys yielding 2,100 completes. The
administrator sample size may be slightly smaller, as some providers share the same practice;
in this case, the administrator survey will only be completed once for the practice.
To allocate the 2,100 eligible professional completes with 1,050 in each participating and
nonparticipating group, we started with a proportional allocation that has the smallest design
effect. Then we oversampled the GPRO group and the providers whose reporting mechanism
is electronic health record (EHR), registry only, or claim and registry; these are very small
groups and proportional allocation would allocate very small sample sizes. Table 1 shows the
allocated target sample size in each stratum, as well as the marginal total for each strata
variable. We also included the sampling rate for each stratum, which is the ratio of the final
allocated completes and the proportionally allocated completes according to frame count by
stratum.

1

Table 1: Allocation of Completed Surveys and Sampling Rate
GPRO
Option

Reporting
Mechanism

Participation
Status

Target
Completes

Sampling
Rate

Nonspecialty group

Non-GPRO

Claim

Participants

65

0.43

Nonspecialty group

Non-GPRO

Claim and Registry

Participants

57

1.94

Nonspecialty group

Non-GPRO

EHR

Participants

70

129.56

Nonspecialty group

Non-GPRO

Registry

Participants

44

4.91

Nonspecialty group

Non-GPRO

Other

Nonparticipants

187

1.00

Specialty oversampling group

Non-GPRO

Claim

Participants

68

0.43

Specialty oversampling group

Non-GPRO

Claim and Registry

Participants

25

1.98

Specialty oversampling group

Non-GPRO

EHR

Participants

28

127.16

Specialty oversampling group

Non-GPRO

Registry

Participants

40

4.89

Specialty oversampling group

Non-GPRO

Other

Nonparticipants

425

1.00

Specialty Group

Specialty undersampling group

GPRO

Web

Participants

300

2.75

Specialty undersampling group

Non-GPRO

Claim

Participants

15

0.42

Specialty undersampling group

Non-GPRO

Claim and Registry

Participants

5

1.98

Specialty undersampling group

Non-GPRO

EHR

Participants

7

123.34

Specialty undersampling group

Non-GPRO

Registry

Participants

9

5.04

Specialty undersampling group

Non-GPRO

Other

Nonparticipants

30

0.45

Other specialties

Non-GPRO

Claim

Participants

152

0.43

Other specialties

Non-GPRO

Claim and Registry

Participants

63

1.97

Other specialties

Non-GPRO

EHR

Participants

45

127.89

Other specialties

Non-GPRO

Registry

Participants

57

4.96

Other specialties

Non-GPRO

Other

Nonparticipants

408

0.80

Total

2,100

Participation Status
Participants

1,050

Nonparticipants

1,050

Specialty Groups
Nonspecialty group

423

Specialty oversampling group

586

Specialty undersampling group

366

Other specialties

725

GPRO Option
GPRO

300

Non-GPRO

1,800

Reporting Mechanism
Web

300

Claim

300

Registry

150

Claim and Registry

150

EHR

150

Other

1,050

2

Based on the allocations in Table 1, we have determined the minimal detectable differences
(MDDs) for group comparisons at 80-percent power, as seen in Table 2.
Table 2: Minimal Detectable Differences
NON-PARTICIPANTS
Specialty
oversampling
group

Nonspecialty
group

Specialty
undersampling
group

Specialty oversampling group

0.124

Specialty undersampling group

0.281

0.270

Other specialties

0.124

0.098

0.270

Nonspecialty
group

Specialty
oversampling
group

Specialty
undersampling
group

PARTICIPANTS

Specialty oversampling group

0.144

Specialty undersampling group

0.120

0.135

Other specialties

0.121

0.136

0.110

GPRO option
GPRO vs. Non-GPRO

0.096
PQRS only

eRx only

eRx only

0.128

PQRS and eRx

0.097

0.121

Web

Claim only

Registry only

Claim only

0.115

Registry only

0.141

0.141

Claim and Registry

0.141

0.141

0.163

EHR

0.141

0.141

0.163

Claim and
registry

0.163

NOTE: The calculations are based on the following assumptions: 80-percent power; two-tailed test at a 5-percent
significance level; the design effect due to oversampling and undersampling of 1.2; the design effect due to
nonresponse of 1.2; and covariates explaining 30 percent of the variance in outcomes with the mean of the outcome
equal to 0.50.

3

The MDD is used to determine the likelihood of detecting a significant difference between
groups; however, confidence intervals (CIs) can be used to determine, with 95-percent
confidence, the range in which a true point estimate falls. Table 3 contains the half-width CIs
for participants and nonparticipants, as well as for subgroups, for binary or categorical
variables.
Table 3: Confidence Intervals for Participants, Nonparticipants, and Subgroups

Target completes

Half-width of 95%
confidence level
(proportion near 0.50)

PARTICIPANTS
Overall

1,050

0.030

Specialty
Nonspecialty group

236

0.064

Specialty oversampling group

161

0.078

Specialty undersampling group

336

0.054

Other specialties

317

0.055

GPRO Option
GPRO

300

0.057

Non-GPRO

750

0.036

Reporting Mechanism
Web

300

0.057

Claim

300

0.057

Registry

150

0.081

Claim and Registry

150

0.081

EHR

150

0.081

Participation Status
PQRS only

353

0.052

eRx only

185

0.072

PQRS and eRx

512

0.044

1,050

0.030

Nonspecialty group

187

0.072

Specialty oversampling group

425

0.048

Specialty undersampling group

30

0.183

Other specialties

408

0.049

NONPARTICIPANTS
Specialty
Overall

NOTE: The calculations are based on the following assumptions: two-tailed confidence interval at 95-percent
confidence level; the design effect due to oversampling and undersampling of 1.2; the design effect due to
nonresponse of 1.2; and covariates explaining 30 percent of the variance in binary outcomes with the mean of the
outcome equal to 0.50.

4

Quality checks. SurveyMonkeyTM will be set up to indicate required items and the
appropriate skip patterns. We will also perform a frequency check of the data once we have
received between 50 and 100 completed electronic surveys. This helps ensure that there are
no problems with the electronic data entry system and also ensures that skip logic within the
instrument is working as intended. (Attachments B1 and B3.)
When surveys are returned by mail or fax, they will first be logged in (“receipted”) by staff.
They will then undergo a manual quality review check to ensure that all items deemed
critical are answered. (Attachments B2 and B4.) The instruments returned by mail and fax
will be entered into SurveyMonkeyTM. A random sample of entered surveys will be reviewed
by another team member. If one or more critical items are left unanswered, we will attempt to
contact the respondent to obtain the necessary information.
Training materials for data collectors. Econometrica team staff will be trained to respond to
any incoming phone calls or emails from professionals and office staff. A brief training
manual will be developed that contains the following:


Overview of the study including the study sponsor, study purpose, study team,
characteristics of sample members, and survey methodology used.



A fact sheet about the evaluation, the importance of the evaluation, and an
explanation that the decision to participate is voluntary and will not affect in any way
their interaction with the PQRS or eRx programs.



A copy of the surveys.



Reinforcement of Econometrica’s strict confidentiality rules as well as CMS’
requirements.

Obtaining and training interviewers or collectors. Training materials for data collectors will
not be necessary, as this survey is self-administered.
Data entry staff will receive training on both quality control procedures after receipt of the
survey and on specifications about data entry itself. Critical items will be reviewed, and staff
will be trained to determine what constitutes a valid response and what kind of responses
would require follow-up with the respondent. In terms of specifications for data entry, we
will review the skip logic embedded in the survey instrument and train staff in how to code
responses if the skip logic is not followed.
Computer record formats. The data will be imported into a SAS format. Label and format
statements will be documented.
Time table and cost burden. The survey is expected to be fielded from January 2014 through
February 28, 2014. The estimated cost burden on respondents is $112,014.00.

5

Interviews with Eligible Professionals and Administrators
Eligible professionals who respond to the Survey of Eligible Professionals will serve as the
universe for the eligible interviews. A convenience sample of 25 participating and nonparticipating eligible professionals will be selected for interviews. Similarly, a convenience
sample of 25 administrators who responded to the administrator survey will be interviewed.
Sample size and sampling techniques. Three percent (approximately 25 eligible professionals
and 25 administrators) of the surveyed eligible participating (Attachment C1) and nonparticipating professionals (Attachment C2) and group practices will be interviewed. In the
eligible professional and administrator surveys (which precede the interviews), we will
include a question asking if they would be willing to participate in a 30-minute program
evaluation interview. We will work in consultation with CMS to contact participating eligible
professionals and practices directly. The convenience sample will include professionals and
administrators who represent different specialties, practice sizes, individual versus group
reporting option, and measure submission approaches.
Quality checks. The team will discuss its notes and refer to the tape to settle differences
among staff members. A combined set of notes for all interviews will be created that
provides information on reasons for eligible professionals’ and administrators’ participation
and their experience and assessment of the program.
Training materials for data collectors. A procedures manual will be developed for
conducting the interviews. The manual will be updated as new circumstances arise.
Obtaining and training interviewers or collectors. Interviewer training will involve a review
of the purpose of the topic guide and practice interviewing each other. A debrief will occur
after each interview to improve subsequent interviews. Additional training will occur as
needed depending on the team’s experience conducting interviews.
Computer record formats. The data will be stored in Word format and in MP3 audio format.
Time table and cost burden. The interviews are expected to be conducted from January 2014
through February 28, 2014. The estimated cost burden on respondents is $1,666.88.
Survey of Medicare Beneficiaries
The Medicare enrollment database will serve as the universe for the beneficiary sample. The
sample will include approximately 4,200 beneficiaries with an anticipated ineligibility rate of
10 percent, yielding an eligible sample of 3,750. (Attachment D1.)
Sample size and sampling techniques. The sample of beneficiaries will be limited to those
beneficiaries who see participating and non-participating eligible professionals with
prescribing privileges. We plan to use 2012 claims data to select the sample. The beneficiary
sample will be drawn from the matched treatment and control eligible samples. We will pull
the Health Insurance Claim (HIC) numbers from a sample of beneficiaries assigned to
6

providers from each of the study groups and use the Medicare Enrollment Database (EDB) to
identify contact information. The sample will include approximately 4,200 beneficiaries with
an anticipated ineligibility rate of 10 percent, yielding an eligible sample of 3,750. We will
create replicate samples in case additional releases are necessary. This may occur if a larger
than anticipated number of respondents indicates they have not received services from the
provider of record within the past 12 months, rendering them ineligible. The estimated
response rate is between 30 and 35 percent, yielding approximately 1,275 completed surveys.
Quality checks. When mail surveys are returned to the Econometrica team, they will first be
logged in and then undergo a manual quality review check to ensure that all items deemed
critical are answered. The instrument will then go to data entry where all forms will undergo
100-percent verification (double-keyed) as an additional quality control measure. Imputation
procedures will be developed for unanswered questions.
Training materials for data collectors. The Econometrica team will train help desk staff so
they can respond to incoming phone calls and emails from beneficiaries or other interested
parties. The Econometrica team will develop a brief training manual that contains the
following:


Brief overview of the study, including the study sponsor, study purpose, study team,
characteristics of sample members, survey methodology used, and survey instrument.



A fact sheet about the evaluation, the importance of the evaluation, and an
explanation that the decision to participate is voluntary and will not affect in any way
their interaction with the PQRS or eRx programs.
A copy of the survey instrument.




Reinforcement of the Econometrica team’s strict confidentiality rules, as well as
CMS’ requirements.

Computer record formats. The data will be in a SAS format. Label and format statements
will be documented.
Time table and cost burden. The survey is expected to be fielded from January 2014 through
March 31, 2014. The estimated cost burden on respondents is $14,931.00.
2. Information Collection Procedures
Surveys of Eligible Professionals and Administrators
Data source and collection techniques. The source of data will be in the form of surveys
sent to the physicians and administrators. The following procedures are the same for all
members of the Surveys of Eligible Professionals and Administrators:


Pre-Notification Letter: Approximately 14 days after receipt of Office of
Management and Budget (OMB) clearance, we will mail a pre-notification letter on
7

CMS letterhead alerting the eligible professional and administrator to the survey and
its importance to program evaluation. Studies have found that pre-notification letters
improve response rates.


Initial Survey Request Letter: Approximately 1 week later, we will mail the eligible
professionals the following materials:
o A cover letter (Attachment B5) that explains the purpose of the survey and its
importance in evaluating the PQRS and eRx programs. This letter would also
clearly indicate that the professional (or the professional’s proxy administrator)
will be given a URL to complete the survey. We will also include language that
professionals who complete the survey electronically via the online link and send
back the survey will receive a $100 incentive in the form of a gift card for their
participation. A toll-free number and an email address will be included in the
packet in case the eligible professionals or administrators have questions.
o A fact sheet about the evaluation, the importance of the evaluation, and an
explanation that the decision to participate is voluntary and will not affect in any
way their interaction with the PQRS or eRx programs.



One week following the initial survey request letter, the first reminder letter will be
sent to nonresponders, which will include the electronic link and the following:
o A cover letter that explains the purpose of the survey and its importance in
evaluating the PQRS and eRx programs. We will also include language that
professionals (or proxies) who complete the survey electronically via the online
link and send back the survey will receive a $100 incentive in the form of a gift
card for their participation. A toll-free number and an email address will be
included in the packet in case the respondent has any questions.



Two more reminder letters will be sent if necessary. The letters will include a
package for the mail-in survey:
o A cover letter that explains the purpose of the survey and its importance in
evaluating the PQRS and eRx programs. This letter would also clearly indicate
that the professional (or the professional’s proxy administrator) can opt to take the
survey online or fax the survey back to Econometrica. We will also include
language that professionals (or proxies) who complete the survey electronically
will receive a $100 incentive and those who complete the hardcopy survey will
receive a $50 incentive in the form of a cash card incentive for their participation.
o A printed copy of the eligible professional and administrator surveys, whichever
is appropriate for the respondent.
o A fact sheet about the evaluation, the importance of the evaluation, and an
explanation that the decision to participate is voluntary and will not affect in any
way their interaction with the PQRS or eRx programs.
8

o A prefilled fax cover sheet that contains Econometrica’s contact information,
including our fax number.
o A self-addressed business-reply envelope for mailing the survey back to
Econometrica. We will use business-reply postage to minimize the postage cost of
unreturned paper-based surveys.


A final letter consisting of all of the materials in the initial packet will be sent to all
nonresponders on February 21, 2014.

The CMS advance letter, survey packet materials, and reminder letters can be found in
Attachment B5.
Interviews with Eligible Professionals and Administrators
Information will be collected using a brief modular topic guide with sections applicable for
professionals and administrators. Three Econometrica team staff (a facilitator and two notetakers) will conduct 30-minute interviews over the phone with each eligible professional or
administrator. With the interviewees’ permission, the interviews will be recorded,
transcribed, and analyzed to extract their experience with PQRS and eRx, any discussion of
improvement in care coordination, impact of participation on patient care, and cost to
participate in the program.
Data elements and instruments. Topic guides will be created with modules to account for
differences in clinical area (primary care versus specialty), responders (clinicians versus
administrative staff), and reporting options.
Sample size and sampling techniques. Three percent (approximately 25 eligible professionals
and 25 administrators) of the surveyed eligible participating and non-participating
professionals and group practices will be interviewed. In the Survey of Eligible Professionals
and Survey of Administrators (which precedes the interviews), we will include a question
asking if they would be willing to participate in a 30-minute program evaluation interview.
We will work in consultation with CMS to contact participating eligible professionals and
administrators directly. The convenience sample will include professionals and
administrators who represent different specialties, regions, and reporting options.
The topic guides and correspondence can be found in Attachment C.
Survey of Medicare Beneficiaries
The survey of Medicare beneficiaries will use the Consumer Assessment of Healthcare
Providers and Systems (CAHPS) Clinician and Group Survey with the Patient Centered
Medical Home (PCMH) module. Minor revisions will be made to address the unique needs
of the evaluation of the PQRS and eRx programs. A copy of the questionnaire is in
Attachment D1.

9

Data source and collection techniques. Data will be collected from Medicare beneficiaries
using a mail survey, as is common with CAHPS administration. The procedures are as
follows:


Pre-Notification Letter: The survey will begin with the mailing of an advance letter,
approximately 14 days after receipt of OMB clearance. It will be on CMS letterhead
to help legitimize the survey and encourage participation.



Initial Survey Request Letter: Approximately 1 week later, a survey packet containing
a cover letter, the survey instrument, a fact sheet, and a business-reply envelope will
be mailed to the beneficiary. We will also include a $2 incentive. A toll-free number
will be included in the packet in case beneficiaries have questions. We will also
include language indicating that the beneficiary can complete the survey on the phone
by calling a toll-free number.



Beneficiaries will receive three reminders via letters and one re-mail of the survey
packet. (Attachment D2.) Given the mail mode with a token incentive, we estimate a
response rate of 45 percent to 50 percent. This is slightly higher than that outlined in
the CAHPS Clinician & Groups Surveys and Instructions document.
o The first reminder, sent approximately 1 week after the survey packet mailing,
will be a letter that explains the purpose of the survey and its importance in
evaluating the PQRS and eRx programs.
o A second reminder, sent approximately 1 week after the first reminder, will
ask the beneficiary to return the completed questionnaire.
o A final letter consisting of all of the materials in the initial packet will be sent
approximately 1 month after the initial survey packet mailing.

The CMS advance letter, survey packet materials, and reminder letters can be found in
Attachment D2.
3. Methods to Maximize Response Rates
Surveys of Eligible Professionals and Administrators
Physicians and support staff can be a challenging population to contact and survey due to
workload and scheduling demands; therefore, the estimated response rate is 50 percent. The
survey topics related to quality measures and improving the quality of patient care are highly
salient to the universe of sampled eligible professionals. In addition to survey topic
relevance, numerous methods will be used to encourage response and reduce challenges to
participation.


CMS endorsement. A letter on CMS letterhead will be sent in advance of the survey.
This will serve to legitimize the survey and will also stress the importance of eligible
professionals’ input on the PQRS and eRx programs as CMS considers ongoing
program refinements.

10



Use of incentives. Eligible professionals and administrators will be provided with up
to a $100 postpay incentive. This will encourage participation among those who may
typically disregard such a survey without an incentive payment.



Survey length. The survey has been developed so that it can be completed in 10–15
minutes. This is important given the limited time that eligible professionals and
administrators will have available to participate.



Survey support. The contractor will set up a toll-free helpline and an email address
for this survey. This phone and email contact information will be made available to
sample members in the survey cover letter and fact sheet. This toll-free line and email
box has restricted access, limited to project personnel only.



Targeted nonresponse follow-up. The contractor will send up to four reminder
letters to nonresponders. The language of each successive letter will convey the
urgency of survey participation.



Flexible participation. For respondents whose preferred mode of completion is not
mail, the contractor will offer the opportunity to complete the survey by Web or by
fax. This flexibility will allow the respondent to participate in the most convenient
manner based on their own scheduling needs.

The contractor will compare the individual and practice characteristics of responders and
nonresponders using sample frame data, to the extent that they are available. This will help
determine the potential for nonresponse bias and inform the need for a nonresponse bias
analysis.
Interviews with Eligible Professionals and Administrators
Only those eligible professionals and administrators who have agreed to be interviewed will
be contacted.
Survey of Medicare Beneficiaries
The estimated response rate among Medicare beneficiaries is between 45 percent and 50
percent, based on the use of a token incentive ($2 bill) and current response rates to similar
CAHPS surveys. The topics covered in the survey are highly salient to Medicare
beneficiaries—these include the extent to which their health care needs are being met and
their perceptions of their health and mental and emotional well-being. In addition to survey
topic relevance, numerous methods will be used to encourage response and reduce challenges
to participation.


CMS endorsement. A letter on CMS letterhead will be sent in advance of the survey.
This will serve to legitimize the survey and will also stress the importance of
understanding how the PQRS and eRx programs are impacting the quality of care
received by patients. The letter will mention a $2 incentive to encourage survey
participation.
11



Survey design. The survey used in this data collection is one that has been used with
similar populations in the past (the CAHPS Clinician and Group Practice Survey with
the Primary Care Medical Home module). The survey will take approximately 10–15
minutes to complete.



Allowing for assistance with survey completion. Some Medicare beneficiaries may
be unable to complete the survey without assistance. Although self-response is
encouraged, we will allow the respondent to receive assistance completing the survey.
Information is collected at the end of the survey about the extent of help received by
the sample member.



Incentive. A $2 bill will be included with the survey packet mailing. Although $2 is a
small amount, it is a useful way to show our appreciation for survey participation.
Also, for individuals on a fixed income, $2 can be useful in meeting basic needs.



Survey support. The contractor will set up a toll-free helpline and an email address
for this survey. This phone and email contact information will be made available to
sample members in the survey cover letter and fact sheet. This toll-free line and email
box has restricted access, limited to project personnel only.



Targeted nonresponse follow-up. The contractor will send up to three reminder
letters to nonresponders. The language of each successive letter will convey the
urgency of survey participation.



Flexible participation. The survey can be completed over the phone, if preferred by
the sample member. A toll-free line will be made available for this purpose.

The contractor will compare the individual characteristics of responders and nonresponders
using sample frame data, to the extent that they are available. This will help determine the
potential for nonresponse bias and inform the need for a nonresponse bias analysis.
4. Tests of Procedures
Prior to implementation, the procedures were tested among nine eligible professionals and
administrators. The survey instrument for Medicare beneficiaries will not be tested because it
has been used over a period of time by different researchers.
Surveys of Eligible Professionals and Administrators
The questionnaires for the Surveys of Eligible Professionals and Administrators were
pretested in February 2013. The survey pretest results determined that the information being
requested in the survey is reasonable, clearly stated in coherent and unambiguous language,
and collected in the least burdensome way possible. Pretest participants were selected
purposively. As a result of the pretest, Econometrica will not make any recommendations for
survey revisions.

12

Pretest responses to the survey questionnaires were collected electronically. The pretest was
used to verify the average interview length, currently estimated at 10–15 minutes per
respondent.
Survey of Medicare Beneficiaries
There are no plans to pretest the beneficiary survey, as the CAHPS family of surveys has
been well validated and the survey methodology is consistent with that used to field CAHPS
surveys.
5. Statistical Consultants
The following persons were consulted on the statistical aspects of this study:
Monique A. Sheppard, Ph.D.
Econometrica, Inc.
Project Director
(301) 657-2491
[email protected]
Rafael Semansky, Ph.D., M.P.P.
Econometrica, Inc.
Program Evaluator
(240) 333-4802
[email protected]
Kirsten Barrett, Ph.D.
Mathematica Policy Research
Senior Survey Researcher
(202) 554-7564
[email protected]
Sheng Wang, Ph.D.
Mathematica Policy Research
Statistician
(609) 936-2799
[email protected]
Betty Fout, Ph.D.
IMPAQ International, LLC
Technical Lead
(202) 696-1010
[email protected]

13

The companies that will actually collect and analyze the data for CMS are:


Mathematica Policy Research will collect the Survey of Medicare Beneficiaries based on
the CAHPS Clinician and Group Survey with the PCMH supplemental questions.



Econometrica, Inc., will collect and analyze the Surveys of Eligible Professionals and
Administrators and the Interviews of Eligible Professionals and Administrators. In
addition, Econometrica will analyze the Survey of Medicare Beneficiaries based on the
CAHPS Clinician and Group Survey data with the PCMH supplement questions.

14