Response to comments

FDMS ID

Comment ID

CMS 60-Day Comment Tracking: Includes FDMS Public Comments and Other Lessons Learned

Date
Received

1

FDMS

Booz Allen
Hamilton

0001

4/16/2010

N/A

This is a test only.

No response required.

N/A.

2

FDMS

Medical
Mutual of
Ohio

0003

6/11/2010

Supporting
Statement

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

3

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

Data
Validation
Standards

Section B, #18, of the 508 Supporting
Statement says that plans are exempt from
data validation if they terminate their
contracts prior to the start of the collection
year. We are terminating our MA-PD
contracts effective 12/31/10. Would the
"collection year" in this case be 2011, so that
would make us exempt from validating our
2010 data? My email address is
[email protected]. Thank you.
Sec 2- Please clarify what "applicable CMS
contract" means. Can data be separated into
part C versus part D? or does data need to
be separated by Medicare HMO versus POS
etc.?

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

4

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

Data
Validation
Standards

Sec 5c- CMS in the past did not request
inpatient files. This element refers to
"concurrent" org determinations. Is this
referring to inpatient acute or only post acute,
home care and physical therapy?

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

5

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

Data
Validation
Standards

Sec 5i- Need confirmation on what is being
requested. Based on language, I interpret it
as a home care or physical therapy request
for additional visits should be excluded from
sample. Please confirm.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

6

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

Data
Validation
Standards

Validation standards: Is the expectation the
contractor will review programming codes? If
so, how will you ensure there are uniform
criteria since plans utilize different systems?

No change is required; comment is
already addressed in document.

N/A.

Method of
Comment

Organization

DV
Document

Comment

CMS Response

Burden
Impact

Page 1 of 58

7

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

Data
Validation
Standards

Is CMS going to provide a standardized
questionnaire for interviewing plan
management? How will you ensure the
contractor is asking all plans the same
questions during the interview process?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

8

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

Data
Validation
Standards

Validation standards: What criteria are you
using to determine if source & output
documents are properly secured? Is this
from a physical or IT perspective?

No
change.

9

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

N/A

We would like CMS to provide a list of
“certified contractors”.

Proper security of source and output
documents is required from both a
physical and IT standpoint. Hardcopy and
electronic copies of documents that
contain PII or PHI should be secured per
any corporate policies and HIPAA
requirements.
Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

10

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

N/A

Are you requiring the contractor to make a
site visit to validate data or is a desk review
acceptable.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

11

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

N/A

Is there an appeal process? Plans should
have the ability to comment within a certain
time period after the contractor submits to
CMS.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

12

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

N/A

Will plans have the ability to change
contractors during the audit?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

N/A.

Page 2 of 58

13

FDMS

Horizon Blue
Cross Blue
Shield of New
Jersey

0004

6/15/2010

N/A

One of the standards is to identify if the data
fields have meaningful, consistent labels.
This appears to be very subjective. Can CMS
specify what exactly they are looking for.

Data field labels should be descriptive and
have supporting documentation that
allows the reviewer to ascertain the data
field meaning and intended usage.

No
change.

14

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

Can sponsors use the same vendor for the
pre-assessment and the audit?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

15

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

Will CMS please clarify the timing of the audit
and exactly what CY data will be audited?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

16

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

Will the Pass/Fail (i.e. No-Pass) scoring be at
the contract level?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

17

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

Can a sponsor use the same vendor for both
the pre-assessment, including the completion
of the OAI and data validation audit without
any conflict of interest?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

18

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

In the case of a PBM which has, for example,
20 plans, do they need to have multiple
vendors or can they use one vendor to certify
the entire process?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 3 of 58

19

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

What documentation will CMS expect to
review from sponsors (i.e. such as flow
charts, policies & procedures)?

CMS will review the final findings from the
DV contractor, but can request working
papers or any documents that were
reviewed during the review. The DV
Contractor will review all documentation
that is requested in the OAI and
documentation from the site visit and
follow-up data requests.

No
change.

20

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

Would CMS allow sponsors to first validate
the Part C reports and then at a later date,
validate the Part D reports?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

21

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

Would it be feasible for CMS to do the Part C
and the Part D Data Validation Requirements
on a three year schedule with one third of the
reports being done each year considering for
larger plans, this may be very expensive and
labor intensive?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

22

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

When will the training be for Part C and Part
D sponsors related to the DVR audits in
2010?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

23

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

Please clarify the cost burden for a Medicare
Advantage sponsor to complete the Part C
and the Part D DVR for one calendar year.

The burden estimates have been revised.
Please see the updated Supporting
Statement for the updated assumptions
and calculations.

N/A.

Page 4 of 58

24

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

Part C and Part D Medicare Advantage
sponsors wish to practice due diligence when
selecting an outside auditor for the upcoming
audit engagement. CMS provided some
hourly estimates in the Supporting Statement
for Paperwork Reduction Act Submissions;
Medicare Part C and Part D Data Validation
(42 C.F.R. §422.516(g) and
§423.514(g));however, will CMS estimate the
cost per hour it believes is reasonable for a
sponsor to hire an auditor to complete the
Final Part C and Part D DVR guidance?
Furthermore, will CMS estimate the cost
burden to a MA sponsor by contract (i.e.
many sponsors have several contracts with
multiple plan benefit packages associated
with each contract)? If this estimate is not
available by contract, please explain how
CMS derived the estimated cost.
Will the Part C Benefit Utilization report be
audited during the Spring 2011 data
validation effort given the data reported at
that time will be for 2009?

The burden estimates have been revised.
Please see the updated Supporting
Statement for the updated assumptions
and calculations.

N/A.

25

FDMS

Blue Cross
Blue Shield of
TN

0005

6/15/2010

N/A

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

26

FDMS

Coventry
Health Plan

0006

6/18/2010

N/A

What methodology will CMS use to derive an
overall "pass" or "fail" determination?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

27

FDMS

Coventry
Health Plan

0006

6/18/2010

N/A

Will it be up to the vendor to assign pass/fail
or CMS to make final determination based
upon comments/results from vendor?

No change required. The "Findings Data
Collection Form Introduction" provides this
guidance (the Pass/Not Pass
Determination will be performed by CMS).

No
change.

Page 5 of 58

28

FDMS

Coventry
Health Plan

0006

6/18/2010

N/A

What will result if an element does not pass
based upon the sample selected?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

29

FDMS

Coventry
Health Plan

0006

6/18/2010

N/A

What rationale was used to determine which
elements required to be audited? For
example, Employer Group demographic
data?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

30

FDMS

Coventry
Health Plan

0006

6/18/2010

Would CMS be willing to share lessons
learned or results from best practices from
validation audits that have been piloted?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

31

FDMS

Coventry
Health Plan

0007

6/18/2010

What methodology will CMS use to derive an
overall "pass" or "fail" determination?

Duplicate of FDMS 0006.

N/A.

32

FDMS

Coventry
Health Plan

0007

6/18/2010

Duplicate of FDMS 0006.

N/A.

33

FDMS

Coventry
Health Plan

0007

6/18/2010

Will it be up to the vendor to assign pass/fail
or CMS to make final determination based
upon comments/results from vendor?
What will result if an element does not pass
based upon the sample selected?

Duplicate of FDMS 0006.

N/A.

34

FDMS

Coventry
Health Plan

0007

6/18/2010

Duplicate of FDMS 0006.

N/A.

35

FDMS

Coventry
Health Plan

0007

6/18/2010

What rationale was used to determine which
elements required to be audited? For
example, Employer Group demographic
data?
Would CMS be willing to share lessons
learned or results from best practices from
validation audits that have been piloted?

Duplicate of FDMS 0006.

N/A.

Page 6 of 58

36

FDMS

Argus Health
Systems

0008

6/18/2010

OAI

37

FDMS

Argus Health
Systems

0008

6/18/2010

OAI

Section 2.2 states "All documentation and
responses to questions should reflect the
organization’s systems and processes that
were in place during the reporting period(s)
undergoing the data validation review."
As the processes used to create reports may
include tools where source code
management is rudimentary, this proposed
requirement may introduce additional
complexity to an organization's IT
infrastructure to:
1. Maintain multiple versions of IT
components (code, data dictionary, etc.)
2. Implement a methodology that readily
tracks point-in-time changes to IT
components
3. Develop a capability to link to those IT
changes and access the variations in code.
Additionally, as some reports may result from
combining data from service providers with
data from the Part D sponsor, systems and
processes from multiple parties need to be
consolidated as a single flow.
It would be preferable to make this proposed
requirement effective for CY 2012 to provide
organizations the time needed to develop and
implement the needed capabilities.
Section 2.2 states "The organization is
responsible for ensuring that it has established
mutually agreeable methods for sharing
proprietary and/or secure (PHI/PII) data with the
reviewer and that the reviewer complies with all
HIPAA privacy and security requirements." The
Business Associate Agreement is the
standard mechanism for establishing HIPAA
compliance obligations with parties receiving
PHI. For Part D sponsor data this is the
responsibility of the Part D organization. We
recommend establishment of standard
accepted methods for such transmissions
and similar to what occurs for sending
secured information to CMS, reviewers
should be required to prove to CMS that they
have such existing capabilities.

No change required (no change in policy).

No
change.

It is up to the organization and contractor
to work out mutually agreeable and
HIPAA compliant methods for sharing
proprietary and/or secure data.

No
change.

Page 7 of 58

It is up to the organization and contractor
to work out mutually agreeable and
HIPAA compliant methods for sharing
proprietary and/or secure data.

No
change.

OAI

Section 5.1 states "For the contract(s) included
in this version of the OAI, organizations should
provide programming code/source code and
example output for computer programs used to
calculate the data collected for each of the CMS
data measures that are currently undergoing
data validation review (as identified in Section
3.3)."
Please clarify if a
code sample or complete program code is
proposed as the requirement. In either
approach, since code and all corresponding
components (e.g., data dictionary) are
proprietary information, a standard provision
of providing code must be an obligation with
the reviewer that stipulates:
1. The reviewer is responsible for protecting
the intellectual property of the providing
organization
2. Code will only be used for purposes of the
Data Validation efforts
3. The reviewer will indemnify the providing
organization if information is otherwise
disclosed or used for other purposes
Same comment as 5.1

It is up to the organization and contractor
to work out mutually agreeable and
HIPAA compliant methods for sharing
proprietary and/or secure data.

No
change.

6/18/2010

OAI

Same comment as 5.1

It is up to the organization and contractor
to work out mutually agreeable and
HIPAA compliant methods for sharing
proprietary and/or secure data.

No
change.

0008

6/18/2010

OAI

Same comment as 5.1

It is up to the organization and contractor
to work out mutually agreeable and
HIPAA compliant methods for sharing
proprietary and/or secure data.

No
change.

Argus Health
Systems

0008

6/18/2010

See comments in 2.2

No change required (no change in policy).

No
change.

FDMS

Argus Health
Systems

0008

6/18/2010

See comments in 2.2

No change required (no change in policy).

No
change.

FDMS

Argus Health
Systems

0008

6/18/2010

Data
Validation
Standards
Data
Validation
Standards
Data
Validation
Standards

See comments in 2.2

No change required (no change in policy).

No
change.

38

FDMS

Argus Health
Systems

0008

6/18/2010

OAI

39

FDMS

Argus Health
Systems

0008

6/18/2010

40

FDMS

Argus Health
Systems

0008

41

FDMS

Argus Health
Systems

42

FDMS

43

44

Page 8 of 58

45

FDMS

Argus Health
Systems

0008

6/18/2010

Instructions
for Findings
Data
Collection
Form

Any organization providing data should
receive the reviewers comments with regard
to their findings

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

No
change.

46

FDMS

Argus Health
Systems

0008

6/18/2010

Findings
Data
Collection
Form

Any organization providing data should
receive the reviewers comments with regard
to their findings

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

No
change.

47

FDMS

Argus Health
Systems

0008

6/18/2010

Sampling
Instructions

No action required (case is not same as
claim).

No
change.

48

FDMS

Argus Health
Systems

0008

6/18/2010

Sampling
Instructions

The "Data Extraction and Sampling
Instructions" have been updated to allow
more flexibility (document now allows for
secure storage devices beyond CDs).

No
change.

49

FDMS

Health Spring

0009

6/18/2010

Sampling
Instructions

No change required.

No
change.

50

FDMS

Health Spring

0009

6/18/2010

Sampling
Instructions

The line item Coverage
Determinations/Exceptions has an incorrect
Sampling Unit listed (Case ID). The sampling
unit should be Claim ID.
While a CD can be encrypted, it is
burdensome to provide information in this
manner and expands the risk that
organization IP and PHI may be used or
disclosed in a manner inconsistent with a
BAA or other form of agreement designed to
protect such. It is preferable to designate a
standard method of exchanging data
information that has more robust industryaccepted security. Organizations that do not
have this capability should not be accepted
as reviewers.
What does CMS consider the records to be
for the sampling unit? For example, the
document entitled Medicare Part C and Part
D Measure Sampling Instructions for Data
Validation shows the Appeals sample size as
150
What "records" are going to be reviewed to
validate that the Appeals were processed and
classified correctly? How are the auditors
going to know if the samples (from the
universe of reports numbers) are real appeals
and classified counted correctly? The
Document Supporting Statement for
Paperwork Reduction Act does not address
what this documentation will be. All the
information just talks about sampling and
submitting documentation.

The "Data Extraction and Sampling
Instructions" have been updated to add
clarification.

No
change.

Page 9 of 58

51

FDMS

Health Spring

0009

6/18/2010

Sampling
Instructions

52

FDMS

Health Spring

0009

6/18/2010

Sampling
Instructions

53

FDMS

Medicare
Cost
Contractors
Alliance

0010

6/18/2010

N/A

54

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

N/A

55

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

Part D realizes there will be other documents
that will be supplied; like P&P's, processes
and data pull criteria, but we interpret the 150
sample validation to mean extra
documentation that would be reviewed and
do not see what that documentation is
Our current PBM, Argus, also provided
comments directly to CMS. Bulletin number #
D834
Summarization of letter received from
Organization: Medicare Cost Plans should
not be required to report Part C data or have
a Part C data validation requirement

No change required. Refer to the
documentation request provided in the
"Organizational Assessment Instrument."

No
change.

No response required.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

We encourage CMS to allow PBMs, that
provide similar services and reports to
multiple Part D plans, to transfer such audit
results across plans. To the extent a given
Part D plan has a unique aspect to it; we fully
support additional Part D plan specific testing
and auditing. However, where the
parameters and processes are identical, a
single sampling across the plans the PBM
serves that would satisfy the Data Validation
requirements for all plans served by the PBM
would greatly improve efficiencies for CMS,
plans and PBMs. Upon completion of such
an audit, the PBM or CMS could provide the
plans with a “audit certification number” for
entry into HPMS at a plan level if tracking
such information via that repository is
required.
The timeline provided indicated that 2010
validation reviews will begin between March
and May of 2011. We recommend that this
timeframe is moved to the May, through July
timeframe because plan sponsors will be
handling the First Quarter 2011
reporting during March, April and May, and
such a shift would streamline the activity
required through the plan year for plans.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 10 of 58

56

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

OAI

57

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

Data
Validation
Standards

The OAI tool needs to be specific to that
year’s reporting requirements. Please see the
following comments regarding the
discrepancies noted between the 2010 Data
Validation Standards and the 2010 Reporting
Requirements. The discrepancies that are
noted appear to be requirements that were
suspended, removed or changed from prior
years reporting requirements and technical
specifications and are no longer applicable
for 2010.
This Data Validation Standard measure is
different from the approved and distributed
Reporting Requirements and Technical
Specifications. The 2010 Reporting
Requirements, Section VII: Grievances, page
16 : “Multiple grievances by a single
complainant should be tracked, followed, and
reported as separate grievances.”
The
2010 Reporting Requirements and the
Technical Specifications do not say to
combine complaints on the same topic into
one grievance.
We recommend that the
Measure Specific Criteria be updated to
reflect the actual 2010 Reporting
Requirements and Technical Specifications
verbiage.

No change required. The organization
appears to be commenting on earlier
versions than the documents posted for
public comment in April 2010.

No
change.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

Page 11 of 58

58

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

Data
Validation
Standards

59

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

Data
Validation
Standards

60

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

Data
Validation
Standards

This Data Validation Standard measure is
different from the approved and distributed
Reporting Requirements and Technical
Specifications.
The Reporting Requirements document, page
19 states: “A. The total number of pharmacy
transactions in the time period above. “
Neither the 2010 Reporting Requirements nor
the Technical Specifications state that the
transactions were to be included based on a
transaction’s date of service for this report.
Only the Data Validation Standards say to
use date of service.
We recommend that the final Data Validation
specifications be updated to indicate that the
number of pharmacy transactions counted
and reported is to be based upon those
claims with a processing date that falls within
the reporting period.
We recommend that the Data Validation
Standards match the published 2010 CMS
Reporting Requirements and the 2010
Reporting Technical Specifications
documents.
This is not a stated requirement in the 2010
Reporting Requirements or in the Technical
Specifications documents. We believe this is
a carry over from the 2009 requirements. We
recommend that this requirement be removed
from the Data Validation requirements.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

This Data Validation Standard measure is
different from the approved and distributed
Reporting Requirements and Technical
Specifications. We recommend that the Data
Validation Standards match the published
2010 CMS Reporting Requirements and the
2010 Reporting Technical Specifications
documents.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

Page 12 of 58

61

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

Data
Validation
Standards

This section contains the 2009 Reporting
Requirements and needs to be updated to
reflect the 2010 Reporting Requirements and
Technical Specification information.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

62

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

Data
Validation
Standards

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

63

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

Sampling
Instructions

4b. and 5 b. specify to only include
pharmacies that were active on the last day
of the reporting period. The CMS guidelines
do not state on data elements A, B, C, D & E
to include only pharmacies that are
contracted as of the last day of the reporting
period in any of the elements like the data
validation reflects. The Technical
Specifications under element D (only) does
state to report: “ Any pharmacy that is active
in the network for 1 or more days in reporting
period should be included.” We recommend
that the Data Validation Standards match the
published 2010 CMS Reporting
Requirements and the 2010 Reporting
Technical Specifications documents.
Please confirm how long does the sponsor or
PBM need to retain intermediary data sets
(interim and final stage data sets) after the
report is generated?

No
change.

64

FDMS

SilverScript
Insurance
Company and
Accendo
Insurance
Company

0011

6/18/2010

Sampling
Instructions

Sponsoring organizations are required to
maintain all data to support the reporting
requirements for 10 years, per 42 CFR §
422.504(d) and 42 CFR § 423.505(d). All
related entities, contractors, or
subcontractors must also maintain
supporting data for 10 years, per 42 CFR
§ 422.504(i)(2) and 42 CFR §
423.505(i)(2).
The "Data Extraction and Sampling
Instructions" have been updated to allow
pooling as an option. If a reviewer would
like to generate random samples of each
contract, this is also an option.

In regards to the statement that “one sample
must be randomly drawn from pooled data
from all contracts”. We believe that pooling
multiple contracts from a single sponsor will
cause a significant burden on the sponsor
and on any PBM servicing multiple sponsors
that have multiple contracts. The reporting is
currently created by contract and the pooling
the contracts together for a single sponsor
will likely require significant database
development if the current design for the
reporting databases purposefully keeps the
contract data separated.

No
change.

Page 13 of 58

65

FDMS

WellPoint Inc.

0012

6/18/2010

N/A

66

FDMS

Blue Care
Network of
Michigan

0013

6/18/2010

N/A

67

FDMS

Blue Care
Network of
Michigan

0013

6/18/2010

N/A

68

FDMS

Blue Care
Network of
Michigan

0013

6/18/2010

N/A

According to the HPMS memo released on
November 23, 2009 (Medicare Part C and D
Reporting Requirements and Data
Validation), CMS indicates that the CY 2010
data validation audits will occur during the
period of approximately March 2011 through
May 2011. Based on the answer to
question number 26 contained in the
November 23, 2009 HPMS memo, it is our
understanding that the data validation review
of the Benefit Utilization report due August
31, 2011 will be performed prospectively.
Additionally, based on the footnote contained
on page 7 of this document, CMS indicates
that all data validation reviews, with the
exception of Part C Benefit Utilization, will be
retrospective; however, there are several Part
C reports and one Part D report subject to CY
2010 data validation that are due on or after
May 31, 2011 (Procedure Frequency, Serious
Reportable Adverse Events, Special Needs
Plans Care Management, and Long-Term
Care Utilization). Since these reports are
due at the end of the projected time that the
data validation audits will occur, will CMS
also be performing a prospective review of
these reports? Any clarification that you can
provide is appreciated.
Reduce the required amount of information if
the health plan is already HEDIS audited as
much of the information is duplicative.
Review of Part D tech specs could cause a
considerable burden on some delegated
entities (i.e., PBMs). Consider allowing the
PBMs to have an external auditor review
those requirements that are applicable and
issue a report to the PBM clients (aka plans)
which would be acceptable to CMS. More
efficient and definitely cost effective
Tech Specs: Procedure Frequency
Clarification should be given to when the
exclusions apply for each condition.
Additionally clarification is requested for items
like Procedure Frequency Sections 2.3 and
2.4 which both contain CPT 35472
suggesting that the two buckets are double
counting the same procedures

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Language has been added to the "Data
Validation Standards" in the Procedure
Frequency Measure (2.2) to add
clarification.
Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

N/A.

Page 14 of 58

69

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

N/A

70

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

General

71

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

Sampling
Instructions

We recommend that CMS allow PBMs that
use common systems with identical
parameters and processes and provide
similar services and reports to multiple Part D
plans, that the PBM be able to obtain one
data validation audit for each of their systems
and processes and report the audit results to
each of their Part D plans. We believe that
where the parameters and processes are
identical, a single audit across the systems
and/or the plans the PBM serves should
satisfy the Data Validation requirements for
all Part D plans served by the PBM. Allowing
for this provision would greatly improve
efficiencies for CMS, Part D plans and
PBMs. Upon completion of such an audit, the
PBM could provide the plans with a “audit
certification number” to provide to their data
validation auditor or for use in HPMS if
tracking such information is required We
agree, that to the extent that a Part D plan
has other data systems or processes that
support the CMS reports, that a Part D plan
would need obtain specific data validation
testing and auditing.
We recommend that the timeline be moved to
May through July 2011 as Part D plans will be
handling the First Quarter 2011 reporting and
PDE submission from March through May. In
addition, because the final criteria for hiring a
data validation auditor will not be released
until fall 2010, it is going to be very difficult to
prepare an RFP and get an auditor retained
and have them complete their audit in the
March to May timeframe. A shift to a May
through July would streamline the activity
required throughout the plan year for our
plan.
Please confirm in the final instructions how
long the sponsor or PBM needs to retain
interim and final stage data sets after the
report is generated.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Sponsoring organizations are required to
maintain all data to support the reporting
requirements for 10 years, per 42 CFR §
422.504(d) and 42 CFR § 423.505(d). All
related entities, contractors, or
subcontractors must also maintain
supporting data for 10 years, per 42 CFR
§ 422.504(i)(2) and 42 CFR §
423.505(i)(2).

No
change.

Page 15 of 58

72

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

Sampling
Instructions

73

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

Data
Validation
Standards

74

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

Data
Validation
Standards

In regards to the statement that “one sample
must be randomly drawn from pooled data
from all contracts”: Our reporting is
currently created by contract and pooling the
contracts together will likely require significant
database development because the current
design for the reporting databases
purposefully keeps the contract data
separated. We believe that pooling multiple
contracts will create a significant burden on
us as a plan sponsor and on our PBM.
There appear to be discrepancies between
the 2010 Data Validation Standards and the
2010 Reporting Requirements. These appear
to be requirements that were suspended,
removed or changed from prior reporting
requirements and technical specifications and
are no longer applicable for 2010. Below you
will find the areas that do not correspond with
each other:

The "Data Extraction and Sampling
Instructions" have been updated to allow
pooling as an option. If a reviewer would
like to generate random samples of each
contract, this is also an option.

No
change.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

This Data Validation Standard measure is
different from the approved and distributed
Reporting Requirements and Technical
Specifications. The 2010 Reporting
Requirements, Section VII: Grievances, page
16 states “Multiple grievances by a single
complainant should be tracked, followed, and
reported as separate grievances.” The 2010
Reporting Requirements and the Technical
Specifications do not say to combine
complaints on the same topic into one
grievance. We recommend that the Measure
Specific Criteria be updated to reflect the
actual 2010 Reporting Requirements and
Technical Specifications verbiage.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

Page 16 of 58

75

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

Data
Validation
Standards

76

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

Data
Validation
Standards

77

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

Data
Validation
Standards

This Data Validation Standard measure is
different from the approved and distributed
Reporting Requirements and Technical
Specifications. The Reporting Requirements
document, page 19 states: “A. The total
number of pharmacy transactions in the time
period above. “ Neither the 2010 Reporting
Requirements nor the Technical
Specifications state that the transactions
were to be included based on a transaction’s
date of service for this report. Only the Data
Validation Standards say to use date of
service. We recommend that the final Data
Validation specifications be updated to
indicate that the number of pharmacy
transactions counted and reported is to be
based upon those claims with a processing
date that falls within the reporting period and
that the Data Validation Standards match the
published 2010 CMS Reporting
Requirements and the 2010 Reporting
Technical Specifications documents.
This Data Validation Standard measure is
different from the approved and distributed
Reporting Requirements and Technical
Specifications. We recommend that the Data
Validation Standards match the published
2010 CMS Reporting Requirements and the
2010 Reporting Technical Specifications
documents.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

This is not a stated requirement in the 2010
Reporting Requirements or in the Technical
Specifications documents. We believe was
carried over from the 2009 Reporting
Requirements and recommend that this
requirement be removed from the Data
Validation requirements. We recommend that
this requirement be removed from the Data
Validation requirements.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

Page 17 of 58

78

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

Data
Validation
Standards

This section contains the 2009 Reporting
Requirements and should be updated to
reflect the 2010 Reporting Requirements and
Technical Specification information.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

79

FDMS

National
Rural Electric
Cooperative
Association

0014

6/18/2010

Data
Validation
Standards

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions). The
organization appears to be commenting
on earlier versions than the documents
posted for public comment in April 2010.

No
change.

80

FDMS

Independent
Health

0015

6/18/2010

Supporting
Statement

Measure 4b. and 5 b. specify to only include
pharmacies that were active on the last day
of the reporting period. The CMS guidelines
do not state on data elements A, B, C, D & E
to include only pharmacies that are
contracted as of the last day of the reporting
period in any of the elements like the data
validation reflects. The Technical
Specifications under element D (only) does
state to report: “Any pharmacy that is active
in the network for 1 or more days in reporting
period should be included.”. We recommend
that the Data Validation Standards match the
published 2010 CMS Reporting
Requirements and the 2010 Reporting
Technical Specifications documents
Regarding 'Supporting Statement for
Paperwork Reduction Act Submissions:
Medicare Part C and Part D Data Validation
(42 C.F.R. 422.516(g) and 423.514(g))',
Section B, 18. Certification Statement,
recommend defining or clarifying "data
collection year." Is data collection year the
year of the data validation collection or the
year that the reporting data was collected?
For example, the data collection conducted
by the independent external reviewer will take
place in Spring 2011 while the initial data
itself that was collected and reported is from
2010. This clarification would be helpful for a
contract that terminates for 2011 determine
whether or not it is required to validate 2010
data.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 18 of 58

81

FDMS

Express
Scripts

0016

6/18/2010

Express Scripts appreciates the opportunity
to provide comments on CMS’ April 6thth,
2010 Regulation on Medicare Part C and Part
D Data Validation (42 C.F.R. §422.516(g) and
§423.514(g)), as required under 1857(e) and
1860D-12 of the Social Security Act. We
recognize that developing data standards that
would determine reliability, validity,
completeness, and comparability of
measures reported by plan sponsors is an
important undertaking for CMS. We also
realize that preparation for detailed external
data audits will be a significant challenge for
Part D sponsors and their subcontracted
pharmacy benefit managers (PBMs).
Express Scripts is one of the largest
pharmacy benefit management companies in
North America. Headquartered in St. Louis,
Express Scripts provides integrated PBM
services including network-pharmacy claims
management, home delivery services,
specialty benefit management, benefit-design
consultation, drug-utilization review, formulary
management and medical and drug data
analysis services to more than 50 million
Americans.
First and foremost, we appreciate CMS’
desire to utilize the experience and intimate
knowledge that Part D sponsors and their
subcontracted pharmacy benefit managers
have in collecting and processing data for
CMS reported measures. We agree with
CMS’ conclusion that validating reported data
is a necessary precursor to fulfilling its
responsibilities in responding to questions
from Congress, oversight agencies, and the
public. Hence, we are supportive of CMS’
decision to standardize data validation across
reported measures and data elements and
across plans.

No response required.

N/A.

Page 19 of 58

82

FDMS

Express
Scripts

0016

6/18/2010

While we are overall supportive of CMS’
guidance, we do offer the following
recommendations:
Audit Timeframe:
In the Supporting Statement for Paperwork
Reduction Act Submissions – B. Justification
-16.Publication/Tabulation Dates, CMS states
that the collection of the Part C and Part D
validation data will commence around March
1, 2011 and that the audits are expected to
occur each year over a three-month period.
Express Scripts Recommendation: First, we
would request CMS to confirm if we are right
to assume that the audits are expected to be
completed before May 31, 2011. If so, we
believe that the three months given as the
audit timeframe will not be sufficient for
PBMs. Many plans outsource their reporting
to PBMs, and to schedule and accommodate
large numbers of simultaneous on-site audits
is a significant administrative burden. Based
on past experience, we estimate at least five
business days of on-site visits, with probable
follow-up later. Hence, we strongly
recommend CMS to extend the audit period
to at least nine months, and preferably to
twelve months. This will help not only in
scheduling of the audits, but also will increase
audit quality and acceptance of results. In
addition, it will also alleviate resource and
bottleneck issues on the external auditor side,
since we believe there are, at present, a
limited number of audit firms that have the
qualifications that CMS is looking for.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 20 of 58

83

FDMS

Express
Scripts

0016

6/18/2010

Pass/Fail Explanation
In the Supporting Statement for Paperwork
Reduction Act Submissions – B. Justification1. Need and Legal Basis, CMS states that
reviewer will share their findings with the
organization and then submit the completed
Findings Data Collection Form to CMS, who
will process the measure-level or data
element-level findings for each measure’s
standards to derive an overall “Pass” or “Not
Pass” determination. In addition, in the
answer to Question 15 of the Q&A section of
the November 23, 2009 memorandum titled
Medicare Part C and Part D Reporting
Requirements and Data Validation, CMS has
stated that “a scoring system will be
developed and a “pass” or “not pass” will be
assigned based on information reported to us
by the independent data validation contractor
hired by the MAO or Part D sponsor.”
Express Scripts Recommendation: Part C
and D Data Validation Audit is the first of its
kind and presents a learning opportunity for
both the plans and CMS. Hence, instead of
deciding upon an evaluation system in
advance, we would recommend that CMS
use the findings as a guide toward the
selection of an evaluation system that would
ultimately best focus and direct plans toward
CMS’ goals of data validity, reliability and
comparability. We would also suggest that,
before finalizing it, CMS open the evaluation
system it proposes to public comment, which
could provide helpful insight about its
advantages and disadvantages. However, if
CMS has made a firm decision toward a
Pass/Fail system, then we would request
CMS explain the evaluation criteria and
methodology on how the Pass/Fail grade will
be given. Transparency of the methodology
would help plans prevent incorrect or
unnecessary weightings, would reduce
objections to the outcome of the audit and
give more credibility to the final evaluation.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

Page 21 of 58

84

FDMS

Express
Scripts

0016

6/18/2010

Appeals Process
In Question 22 of the Q&A section of the
November 23, 2009 memorandum titled
Medicare Part C and Part D Reporting
Requirements and Data Validation, CMS has
stated that the plan will not have the right of
appeal. CMS also states that the plan may
disagree with the results of the audit, but that
process solely involves the plan and the
external auditor. CMS states in Question 20
of the same memorandum that sponsors “that
are found to be deficient will be requested to
develop corrective action plans or could be
subject to other enforcement actions”. It adds
that a Fail grade will be considered noncompliant and may be used to adjust plan
performance measurement.
Express Scripts Recommendation: We would
like to understand the implications of failure to
pass the audit more clearly, especially the
scope of “other enforcement actions”
mentioned in the CMS memo. If a failure on
any part or whole of the audit can lead into
contractual sanctions including prohibition
from bidding or contract termination, then we
believe that the plans should be provided with
an appeals process. Such a process, by itself
and by its sheer availability, would reduce
issues that may originate from auditor errors
or bias, facilitate acceptance of the results
and also help CMS maintain accuracy and
impartiality in evaluating plans and informing
beneficiaries. Otherwise, the plans have no
recourse for significant disagreements. As an
example and possible precedent, we would
like to point out to the three-stage appeals
process for RADV payment error calculations
mentioned in the MA Part D Final Rule (CMS4085-F). While the aforementioned
interaction is entirely between CMS and the
plan, we believe that CMS can institute in this
case an appeals process that would allow
reexamination of the audit results by a third
party, as well as allow plans to appeal to
CMS’ cumulative evaluation itself.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 22 of 58

85

FDMS

Express
Scripts

0016

6/18/2010

86

FDMS

Express
Scripts

0016

6/18/2010

Findings
Data
Collection
Form

Long-term Care Utilization Reports
CMS has stated in Medicare Part C and Part
D Measure Instructions for Findings Data
Collection Form for Data Validation
Contractors that Long-term Care Utilization
will be included in the 2011 audit.
Express Scripts Recommendation: Long-term
Care Utilization report for PY 2010 is due on
June 30, 2011. Data Validation Audits, which
start in March 1, 2011, may end before this
report is submitted. In addition, operational
build-out, technical development and QA
testing of the report process may continue
beyond March 1, 2011, before the report is
submitted in June. Hence, while we support
the evaluation of Long-term Care Utilization
reporting, we would recommend that this
portion of the audit be carried out in 2012 for
PY 2010 data, when the full year of data and
the reports will be available.
Data Sharing Risk
CMS has stated in the Supporting Statement
for Paperwork Reduction Act Submissions
that the Findings Data Form “allows the
reviewer to record notes, data sources
referenced and findings for different
standards and criteria specified for a given
measure” to be submitted to CMS. In
Question 1 of the Q&A section of the
November 23, 2009 memorandum, CMS has
also mentioned that “audited data will ensure
that health and drug plans are on equal
footing for public reporting” and, in the 2010
Call Letter and elsewhere, has confirmed that
it “may adjust performance measurements to
reflect the plan’s non-compliance with CMS
audit specifications.”
Express Scripts Recommendation:
We are concerned with possible release of
sensitive, detailed information that can be
traced to plans or PBMs when CMS makes
public the results of the audit. We understand
CMS’ desire to provide as transparent a Part
D program as possible, however, we also
believe that the data collected by the auditors
may reveal to the auditing team and CMS
sensitive information, proprietary business
processes

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 23 of 58

87

FDMS

UnitedHealth
Group

0017

6/18/2010

General

88

FDMS

UnitedHealth
Group

0017

6/18/2010

General

89

FDMS

UnitedHealth
Group

0017

6/18/2010

General

Issue: It is unclear how CMS will derive the
"Pass" or "Not Pass" determination. In
addition, there is no indication what the
process will be for correcting an initial “Not
Pass” determination.
Recommendation: We recommend clarifying
how CMS will derive an overall "Pass" or "Not
Pass" determination. For example, will each
standard be rated the same or will there be
over/under weighting of certain standards? In
addition, please clarify if the "Pass" or "Not
Pass" determination will be made at the
organization or contract level.
We further recommend CMS clarify the
process for correcting a "Not Pass"
determination.
Annual Validation Issue: It appears that the
cost and total time involved in this data
validation audit has been underestimated.
The estimates appear to be for a small plan
with only a handful of contracts and a single
set of systems for delivering CMS reports.
Given the number of systems that may be
used by larger plans to report on the broad
scope of the measures, this places a
significant burden on the handful of qualified
Audit Contractors to review both the plan and
its delegated entities systems and
appropriate documentation within the 12
week timeframe.
Recommendation: We recommend that a
limited set of reports be validated each year,
as CMS had initially planned, with a three
year overall schedule. For example, 1/3 of
reports would be validated year 1, the next
1/3 in year 2 etc. We further recommend that
the Appeals/grievances measure be included
in the first set of reports to be validated.
Measures with the first reporting due in 2011
should be included in the third validation set.
In the alternative, we recommend allowing a
minimum of six months for the validation,
which would allow sufficient time for a
thorough, quality validation of plan and
delegated entities systems, as well as all
appropriate documentation.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

The burden estimates have been revised.
Please see the updated Supporting
Statement for the updated assumptions
and calculations.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 24 of 58

90

FDMS

UnitedHealth
Group

0017

6/18/2010

General

91

FDMS

UnitedHealth
Group

0017

6/18/2010

Findings
Data
Collection
Form

92

FDMS

UnitedHealth
Group

0017

6/18/2010

Findings
Data
Collection
Form

93

FDMS

UnitedHealth
Group

0017

6/18/2010

OAI

CMS intends the Data Validation Audits to
begin March 1 and occur over a three month
time period; however, this is the period in
which plans are preparing bids.
Recommendation: We recommend the Data
Validation Audit be scheduled in the summer
so it does not conflict with Plans' bid
development. This will also allow for
retrospective data review of the 2010 reports
that are not due to CMS until May and August
of 2011.
Issue: The Review findings instructions
indicate that any inaccuracies result in a "no"
finding. However, the sampling instructions
indicate that the process is designed to
"detect error rates of 15% or more."
Recommendation: We recommend clarifying
the standard that applies and that auditor(s)
have the discretion to determine that
negligible errors would not exclude a data
element from being met.
Issue: On both the Benefit Utilization and
Employer/Union Sponsored Group Health
Plan Sponsors (Part D) forms, Standard 3.a
"Data elements are accurately entered into
the HPMS tool…" does not apply. These
reports are file uploads.
Recommendation: We recommend marking
Standard 3.a as "N/A" on both forms, similar
to the manner that the file upload is marked
"N/A" for those that are entered via HPMS.
Issue: Provided in section 3.1 is a list of
contract types that includes CCP, SNP, PFFS
and Employer/Union "800 Series," etc.
However, these types are not mutually
exclusive as a contract may be a CCP or
PFFS and have some employer group and
non- group Plan Benefit Packages (PBPs)
within the contract.
Recommendation: Since there is specific
reporting for SNPs and Employer/Union
plans, we recommend identifying the overall
contract type, then indicating under that
contract type, if there are SNP or 800 series
type plans. To accomplish this, we
recommend adding an additional 2 columns
to identify underlying plan types (PBPs).
"Include SNP?" and "Include Employer/Union
"800 Series?" These should then be
answered as either Yes or No.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Part B of the supporting statement has
been updated to clarify the 15% error rate.

N/A.

Standard 3.a has been changed to "N/A"
on both the Benefit Utilization and the
Employer/Union Sponsored Group Health
Plan Sponsors findings data collection
forms.

No
change.

Section 3.1 of "Organizational
Assessment Instrument" has been revised
to accommodate for required plan detail.

No
change.

Page 25 of 58

94

FDMS

MCS
Advantage,
Inc.

0018

6/18/2010

OAI

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Supporting
Statement

According to this section the information
gathered by the data validation reviewers will
provide a better understanding of the scope
for the organization’s data validation review,
including which contract(s) will be reviewed.
1.Which criteria will be considering to select
the contract that will be reviewed? It is
possible that CMS provide to the MAO’s this
information?
Who will be responsible for the payment of
these services?

95

FDMS

MCS
Advantage,
Inc.

0018

6/18/2010

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

No
change.

96

FDMS

MCS
Advantage,
Inc.

0018

6/18/2010

Supporting
Statement

Will CMS recommend any contractors to
perform this data validation?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

97

FDMS

MCS
Advantage,
Inc.

0018

6/18/2010

Supporting
Statement

If the MAOs complete the OAI internally, do
results have to be submitted to CMS?

No action required. Already in the
"Organizational Assessment Instrument"
(see instructions).

No
change.

98

FDMS

MCS
Advantage,
Inc.

0018

6/18/2010

Supporting
Statement

The result of the data validation will be
processed by CMS to determine if those
findings “Pass” or “Not Pass”, will this
document be a formal finding to the MAO?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

99

FDMS

MCS
Advantage,
Inc.

0018

6/18/2010

Supporting
Statement

The independent yearly audit proposed, will
be performed by CMS or by auditors hired by
the MAO?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 26 of 58

10
0

FDMS

Kaiser
Permanente

0019

6/18/2010

10
1

FDMS

Kaiser
Permanente

0019

6/18/2010

In Section A. ("Background"), CMS states
that it will provide "a set of standards for
selecting a data validation organization", and
these standards will "describe the minimum
qualifications, credentials, and resources that
the selected data validation contractor must
possess." But CMS does not say when it will
do this. If CMS intends to make any
significant changes to the draft standards it
issued in September, 2009, it should do so
very quickly. MAOs and Part D sponsors
(including Medicare Cost contractors) are
even now trying to assess the qualifications
of, and select, data validation auditors. If that
assessment and selection process must
accommodate different qualifications for the
auditors, these organizations/sponsors need
to know that as soon as possible
In Section B, Subsection 16
("Publication/Tabulation Dates"), CMS states
that "Collection of the Part C and Part D
validation data will commence on March 1,
2011. The data validation audits are expected
to occur each year over a three year period."
From this statement, we infer that CMS
intends to require that all MAOs and Part D
sponsors undergo a data validation audit
every year, during the same 3 month period
each year. If this is a correct inference, we
urge CMS to reconsider. These audits are
intensely labor- and resource-intensive, and
the number of qualified data validation
auditors too small, to accommodate these
data validation audits for every
organization/sponsor in the country at the
same time every year. Kaiser strongly
believes that CMS should conduct these data
validation audits on a 3 year cycle, as it
currently does with the OFM financial
solvency audits, so that one-third of
MAOs/sponsors undergo data validation
audits each year

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 27 of 58

10
2

FDMS

Kaiser
Permanente

0019

6/18/2010

Supporting
Statement

10
3

FDMS

Kaiser
Permanente

0019

6/18/2010

OAI

There is another concern about the proposed
March 1, 2011 "start date". CMS does not
explain (and should) how a data validation
audit that begins on or around March 1, 2011
could validate measures for which 2010 data
is not reported until the end of May, 2011 (
Procedure Frequency and Serious
Reportable Adverse Events) or until the end
of August, 2011 ( Benefit Utilization). Kaiser
recommends that CMS withdraw these
measures from data validation audits
conducted during 2011, and reinstate them
for audits conducted in 2012.
CMS does not state whether this completed
tool, which it encourages MAOs/sponsors to
complete and give to the data validation
auditor before the actual audit, must be given
to CMS by either the auditor or the
MAO/sponsor. CMS should revise Section
2.1 to clarify that neither the MAO/sponsor
nor the auditor will be required to give this
completed tool to CMS. Such a clarification
will also make moot the issue of whether the
completed tool, replete with confidential and
proprietary information about an
MAO/sponsor's data systems and IT
networks, could be the subject of a FOIA
request, were it to come into CMS'
possession.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 28 of 58

10
4

FDMS

Kaiser
Permanente

0019

6/18/2010

Findings
Data
Collection
Form

10
5

FDMS

Kaiser
Permanente

0019

6/18/2010

Findings
Data
Collection
Form

In Section 1.1, CMS states that the data
validation auditor will use the Findings Data
Collection Form to record its audit findings,
and "will share these findings with the
organization, and then submit the
completed... Form to CMS, who will process
the measure- or data element-level findings
for each measure's standards to derive an
overall "Pass" or "Not Pass" determination." It
is not clear from this statement whether the
audited organization/sponsor has the right to
respond formally to its auditor's findings, and
the right to have that formal response
submitted with the completed Form to CMS.
As with other CMS audits, Kaiser strongly
believes that audited organizations/sponsors
should have the opportunity to hear
preliminary findings in an exit conference and
respond to those findings with the auditor, so
that any confusion or misunderstandings can
be resolved before the auditor's final report is
issued.
Organizations/sponsors should also be able
to have their response to the auditor's report
(the completed Form) included when the
auditor sends that Form to CMS.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 29 of 58

10
6

FDMS

Kaiser
Permanente

0019

6/18/2010

Findings
Data
Collection
Form

10
7

FDMS

Kaiser
Permanente

0019

6/18/2010

Findings
Data
Collection
Form

CMS states that it will "process" the auditor's
findings to "derive an overall "Pass" or "Not
Pass" determination, but CMS does not at all
explain or describe how it will do this
"processing" or how it will make this crucial
"overall determination" of "Pass" or "Not
Pass". We don't know how CMS will weight
the auditor's findings to determine "Pass" or
"Not Pass". For example, if the auditor's
findings indicate that the
organization/sponsor's data was satisfactorily
validated with respect to 15 measures, but
not for 2 other measures, would CMS'
processing determine that this resulted in a
"Pass" or "Not Pass" outcome? Based on
what CMS has issued to date, this very
important "processing", which will result in the
crucial "Pass" or "Not Pass" determination, is
a "black box". As such, it is capable of
producing arbitrary and capricious results that
can do significant damage to an
organization/sponsor.
Organizations/sponsored are entitled to have
a much more robust explanation of CMS'
intended "processing" and how it will make
"Pass" and "Not Pass" determinations.
There are several elements under Section
2.1.1 ("Benefit Utilization") that ask for "total
cost sharing paid by members directly to
providers" for various services. (See
elements 1.56, 1.64, 1.72, 1.80, 1.88, 1.96,
1.102, 1.108, 1.114, 1.120, 1.126, and
1.130). It is understandable that CMS would
want to measure the cost sharing that
members pay for various services, but the
phrase "paid...directly to providers" is not
appropriate for many MAOs. The phrase is
actually misleading for Kaiser members,
because the cost sharing they pay for these
services is not paid "directly to providers".
Instead, their cost-sharing is paid to Kaiser,
the MAO or Medicare Cost contractor.
Kaiser's contracted Permanente physicians
do not collect or retain cost-sharing that
members pay for covered MA plan services.
Kaiser recommends that CMS carefully reexamine what data it is trying to collect in
these elements and modify the language of
these elements accordingly.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

Page 30 of 58

10
8

FDMS

Kaiser
Permanente

0019

6/18/2010

Findings
Data
Collection
Form

In Section 2.2.2 ("Medication Therapy
Management Programs"), the word "was"
should be deleted in Element A. The last
sentence, referring to a "currency field", in
Element F should be deleted. The inclusion of
beneficiary-specific data fields at the end of
this Section is puzzling, because the
Supporting Statement (at page 10) states that
"CMS will not be requesting any beneficiary
identification information."

The Findings Data Collection Form
instructions have been updated to correct
this error. The beneficiary-specific data
fields are listed in the instructions for the
Findings Data Collection Form because
this document lists all data elements that
are requested as part of the Part D
Reporting Requirements for the MTMP
Data Measure, and therefore not covered
by the sentence on p 10 of the Supporting
Statement. The data collected on the
FDCF does not include beneficiary
identification information.

No
change.

10
9

FDMS

Kaiser
Permanente

0019

6/18/2010

Findings
Data
Collection
Form

No response required.

N/A.

11
0

FDMS

Kaiser
Permanente

0019

6/18/2010

Sampling
Instructions

The "Data Extraction and Sampling
Instructions" have been updated to correct
this error.

No
change.

11
1

FDMS

Kaiser
Permanente

0019

6/18/2010

Sampling
Instructions

In Section 2.2.6 ("Long Term Care
Utilization"), Element E requires certain data
"In aggregate, for all retail pharmacies in the
service area". CMS can not reasonably
request data about all retail pharmacies in the
service area, because there is no way a Part
D sponsor would have that information.
Surely CMS means "all network retail
pharmacies in the service area" or "all owned
and operated retail pharmacies in the service
area." CMS should correct this reference.
Table 1 in Section 1.0 ("Overview") indicates
that no sampling is required for the two
Employer Group Sponsors measures and the
Retail, Home Infusion and LTC Pharmacy
Access measure. This is repeated in Table 2
on page 4. However the last sentence of the
first paragraph in Section 3.2 ("Evaluating the
Sample Data") states that "The validation of
all criteria except for meeting deadlines will
be conducted using sample data." It is not
clear whether this last sentence contradicts
the "no sampling" notations in Tables 1 and
2. CMS should clarify.
In Section 2.0 ("Conceptual Framework for
Sampling"), #4 states "Data from interim
steps are combined into a detailed data set."
We believe the word "steps" should be "sets".

The "Data Extraction and Sampling
Instructions" have been updated to correct
this error.

No
change.

Page 31 of 58

11
2

FDMS

[none]

0020

6/18/2010

Sampling
Instructions

11
3

FDMS

[none]

0020

6/18/2010

Sampling
Instructions

11
4

FDMS

[none]

0020

6/18/2010

Sampling
Instructions

The sampling method appears to be a bit
ambiguous. For example, if you sample from
the denominator for procedure frequency or
SRAE, and you only select 205 members, it is
unlikely that your sample will include any of
the members for which there was a positive
match. However, if you sample from the
numerator, then it’s both not a random
sample and you would have to have an very
large plan and/or very poorly performing
hospitals in your network to even approach a
number from which 205 could be sampled.
Although the documentation asks for
“programming code”, it seems more
appropriate to provide pseudo code (i.e.
programming code translated into
understandable English).
File exchange using encrypted CD seems
both archaic and less secure. We suggest
using other means (for example, we use a
secure web server for most PHI
transmissions) as the primary method of file
exchange.

No change required. The "Data
Extraction and Sampling Instructions"
explain that the sampling originates from
the Final Stage Data Set.

No
change.

No changes required. Programming code
is required.

No
change.

The "Data Extraction and Sampling
Instructions" have been updated to allow
more flexibility (document now allows for
secure storage devices beyond CDs).

N/A.

Page 32 of 58

11
5

FDMS

PerformRX

0021

6/18/2010

OAI

11
6

FDMS

PerformRX

0021

6/18/2010

OAI

Section 2.2 states "All documentation and
responses to questions should reflect the
organization’s systems and processes that
were in place during the reporting period(s)
undergoing the data validation review."
As the processes used to create reports may
include tools where source code
management is
rudimentary, this proposed requirement may
introduce additional complexity to an
organization's IT infrastructure to:
1. Maintain multiple versions of IT
components (code, data dictionary, etc.)
2. Implement a methodology that readily
tracks point‐in‐time changes to IT
components
3. Develop a capability to link to those IT
changes and access the variations in code.
Additionally, as some reports may result from
combining data from service providers with
data
from the Part D sponsor, systems and
processes from multiple parties need to be
consolidated
as a single flow.
It would be preferable to make this proposed
requirement effective for CY 2012 to provide
organizations the time needed to develop and
implement the needed capabilities.
Section 2.2 states "The organization is
responsible for ensuring that it has
established mutually
agreeable methods for sharing proprietary
and/or secure (PHI/PII) data with the reviewer
and
that the reviewer complies with all HIPAA
privacy and security requirements."
The Business Associate Agreement is the
standard mechanism for establishing HIPAA
compliance obligations with parties receiving
PHI. For Part D sponsor data this is the
responsibility of the Part D organization.
We recommend establishment of standard
accepted methods for such transmissions
and
similar to what occurs for sending secured
information to CMS, reviewers should be
required
to prove to CMS that they have such existing
capabilities.

Duplicate of FDMS 0008.

N/A.

Duplicate of FDMS 0008.

N/A.

Page 33 of 58

11
7

FDMS

PerformRX

0021

6/18/2010

OAI

Duplicate of FDMS 0008.

N/A.

OAI

Section 5.1 states "For the contract(s)
included in this version of the OAI,
organizations should
provide programming code/source code and
example output for computer programs used
to
calculate the data collected for each of the
CMS data measures that are currently
undergoing
data validation review (as identified in Section
3.3)." Please clarify if a code
sample or complete program code is
proposed as the requirement. In either
approach, since
code and all corresponding components (e.g.,
data dictionary) are proprietary information, a
standard provision of providing code must be
an obligation with the reviewer that stipulates:
1. The reviewer is responsible for protecting
the intellectual property of the providing
organization 2. Code will only
be used for purposes of the Data Validation
efforts 3. The reviewer will
indemnify the providing organization if
information is otherwise disclosed or used for
other
purposes
Same as comment 5.1

11
8
11
9
12
0
12
1

FDMS

PerformRX

0021

6/18/2010

Duplicate of FDMS 0008.

N/A.

FDMS

PerformRX

0021

6/18/2010

OAI

Same as comment 5.1

Duplicate of FDMS 0008.

N/A.

FDMS

PerformRX

0021

6/18/2010

OAI

Same as comment 5.1

Duplicate of FDMS 0008.

N/A.

FDMS

PerformRX

0021

6/18/2010

OAI

Duplicate of FDMS 0008.

N/A.

FDMS

PerformRX

0021

6/18/2010

OAI

Duplicate of FDMS 0008.

N/A.

FDMS

PerformRX

0021

6/18/2010

Duplicate of FDMS 0008.

N/A.

FDMS

PerformRX

0021

6/18/2010

Data
Validation
Standards
Data
Validation
Standards

Would it be possible for the auditors to view
source codes on site (ie. PBM, plan, vendor)
and have SME explain. The auditor would not
be allowed to have laptop etc. to write source
codes down.
Please confirm what is meant by captured in
your data systems?
See comments 2.2

12
2
12
3
12
4

See comments 2.2

Duplicate of FDMS 0008.

N/A.

Page 34 of 58

12
5

FDMS

PerformRX

0021

6/18/2010

Data
Validation
Standards

12
6

FDMS

PerformRX

0021

6/18/2010

12
7

FDMS

PerformRX

0021

6/18/2010

12
8

FDMS

PerformRX

0021

6/18/2010

12
9

FDMS

PerformRX

0021

6/18/2010

13
0

FDMS

PerformRX

0021

6/18/2010

Data
Validation
Standards
Data
Validation
Standards
Instructions
for Findings
Data
Collection
Form
Findings
Data
Collection
Form
Sampling
Instructions

13
1

FDMS

PerformRX

0021

6/18/2010

Sampling
Instructions

13
2

FDMS

PerformRX

0021

6/18/2010

OAI

CMS provides a breakdown for the
calculation of covered Part D medications per
member per month. I t says we will need to
provide the coding/logic used to determine
the answer. What do you do when you are
the vendor receiving files that include only the
summary amount per member per quarter?
We do not receive the coding logic from the
client on how they create the files
When it reference to -on-site review. Where
does on-site mean (i.e. plan level, PBM,
vendor, etc.)
See comments 2.2

Duplicate of FDMS 0008.

N/A.

Duplicate of FDMS 0008.

N/A.

Duplicate of FDMS 0008.

N/A.

Any organization providing data should
receive the reviewers comments with regard
to their
findings

Duplicate of FDMS 0008.

N/A.

Any organization providing data should
receive the reviewers comments with regard
to their
findings
The line item Coverage
Determinations/Exceptions has an incorrect
Sampling Unit listed (Case
ID). The sampling unit should be Claim ID
While a CD can be encrypted, it is
burdensome to provide information n this
manner and
expands the risk that organization IP and PHI
may be used or disclosed in a manner
inconsistent with a BAA or other form of
agreement designed to protect such. It is
preferable
to designate a standard method of
exchanging data information that has more
robust industry accepted
security. Organizations that do not have this
capability should not be accepted as
reviewers.
Concerns: handing over vendor source
documents, prep work and the impact to
production staff, short CMS implementation
time frame, inconsistency among data
validation auditors, lack of field knowledge of
data validation auditors.

Duplicate of FDMS 0008.

N/A.

Duplicate of FDMS 0008.

N/A.

Duplicate of FDMS 0008.

N/A.

Duplicate of FDMS 0008.

N/A.

Page 35 of 58

13
3

FDMS

PerformRX

0021

6/18/2010

13
4

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

13
5

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

13
6

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

N/A

Concerns: handing over vendor source
documents, prep work and the impact to
production staff, short CMS implementation
time frame, inconsistency among data
validation auditors, lack of field knowledge of
data validation auditors.
ACG has examined the proposed DVA
requirements closely and believes that CMS’
has significantly under estimated the cost
burden to MAOs and PDPs of
implementing the Part C and Part D Data
Validation requirements, especially as
released within this package of DVA
Standards and Findings Documentation.

Duplicate of FDMS 0008.

N/A.

The burden estimates have been revised.
Please see the updated Supporting
Statement for the updated assumptions
and calculations.

N/A.

ACG is presenting the following comments,
many of which relate to clarifying CMS’s
definitions of specific data categories where
ACG believes that without further
definition by CMS of the data to be reported,
that it will be impossible to properly
validate the reported data because the
current definition is subject to a wide range
of interpretation at the MAO and PDP
Sponsor. ACG’s comments were developed
by ACG’s subject matter experts who have
many
years of hands-on experience supporting
MAOs and PDPs to maintain compliance
with the regulations that underlie the reporting
requirements. We respectively
submit that making the clarifications
suggested below will enhance MAO and PDP
reporting accuracy and allow for expedited
data validation.
MAOs and PDPs provide benefits or procure
administrative services through a
variety of downstream vendors including
Individual Practice Associations
(IPAs) and behavioral health organizations
under Part C, and Pharmacy
Benefit Managers (PBMs) for Part D benefits.
In many instances, the
downstream vendors are producing only a
portion of the data that is part of
an MAO or PDP plan’s reporting.

No response required.

N/A.

No response required.

N/A.

Page 36 of 58

13
7

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

N/A

None of the documents in the PRA Package
10305 address and CMS has not yet opined
on, whether a Data Validation Contractor
(DVC) can rely on MAO or PDP validation of
the data provided by downstream vendors, or
whether the DVC must validate the data from
downstream contractors.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

13
8

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

N/A

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

13
9

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

N/A

If the DVC must conduct primary verification
of data provided by downstream vendors,
especially transaction based data (medical
claims and pharmacy claims data,) where
that data is only a constituent or subset part
of the reported data, the process will be much
more costly for the MAO or PDP; at the same
time, if downstream contractors are subject to
ten’s DVC audits (e.g. for PBMs) this process
will become untenable for those vendors.
ACG recommends the following standard:
For transactional (claim data), where the
downstream contractor provides a portion of
the data (e.g. IPAs processing claims,) that a
DVC rely on, and be required to evaluate an
MAO’s validation of the data provided by the
downstream contractor.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

14
0

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

N/A

ACG recommends the following standard:
For transactional data and non-transactional
data where the downstream contractor is the
sole outsourcer, the DVC validate data as
they would with the MAP

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

14
1

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

N/A

Following are comments on measure specific
criteria contained in the Data Validation
Requirements. In some cases comments
suggest that CMS to develop more concrete
and detailed definitions in order to help
enhance reliability, validity, completeness and
comparability of data that is received in the
initial cycles of data validation.

No response required.

N/A.

Page 37 of 58

14
2

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

14
3

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

14
4

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

14
5

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

Validating data related to all these elements
may be difficult using the current loose
definition of services provided. MAOs may
opt to pay for a particular service that is not
covered under original Medicare
requirements as a value added benefit, as a
benefit exception, or for a variety of other
reasons. MAOs claims payments systems
generally do not note whether a service is
traditionally covered by Medicare or not, but
is being covered by the Plan on an exception
basis.
ACG recommends that CMS provide
clarification in the form of a more specific
definition of the services to be included in the
values in this reporting, encompassing all
benefits provided to MAO enrollees.
Almost all SRAEs occur in the hospital setting
and, because of the sensitive nature of the
data; many hospitals do not willingly provide it
to MAOs. DVCs could test a MAO’s policy
and procedures for identifying SRAEs as part
of its Quality of Care review process, but
would still not be able to assess
completeness of reported data. It is unlikely
that MAOs will have complete SRAE data in
the absence of contractual requirements that
hospitals provide it.
ACG recommends that consideration be
given to delaying the implementation of
measure specific requirement 5 while CMS
develops a regulation requiring that MAOs’s
contracts with hospitals contain a clause
requiring the report of SRAEs.
As currently construed the term “in the
network” is loosely defined. Since the data is
reported at a contract level and not a county
level as with MAO Health Service Delivery
Tables, further definitions regarding counting
would also be useful.
MAO’s differently construe “in the network” to
include 1) contract signed, pending
credentialing review process, 2) provider
contracted and completely
through credentialing process and eligible to
treat members, 3) in process with
credentialing committee.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

No response required.

N/A.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

Page 38 of 58

14
6

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

Further, for staff model or group practice
MAO delivery systems, there is no definition
around whether the numbers reported are to
count full time
equivalents (FTEs) or individual practitioners.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

14
7

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

If a provider has multiple offices (e.g. in
multiple counties) CMS should define
whether this provider should be counted only
once (which is assumed since
reporting is at the contract level.)

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

14
8

FDMS

0022

6/18/2010

0022

6/18/2010

ACG recommends further definition be
developed for the purpose of testing
“continuously” part of the network.
That providers be contracted on Day 1 and
Day 365 without a lapse in availability of
service provided

N/A.

FDMS

Data
Validation
Standards
Data
Validation
Standards

No response required.

14
9

ATTAC
Consulting
Group, Inc
ATTAC
Consulting
Group, Inc

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

15
0

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

That if a provider’s contract has expired, but
been extended during negotiations, that
provider also be counted.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

15
1

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

That is a provider has been sanctioned, or
served with termination notice but the internal
review is still ongoing and the provider has
not been finally
terminated that the provider be counted.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

15
2

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

ACG recommends that CMS clarify the
definition of “accepting new patients” for this
reporting category. Physicians may be
accepting new patients generally or only for
referred patients etc. ACG recommends that
this count include both.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

Page 39 of 58

15
3

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

ACG recommends that CMS clarify the
definition of a “Hospital”. Does the definition
of Hospital include: acute care only, specialty
hospitals, CORFs, Partial Stay
institutions, outpatient only facilities, etc?

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

15
4

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

CMS considered this approach in
developing the DV standards and
determined it would add an unacceptable
resource burden to the process

No
change.

15
5

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

15
6

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

Grievances are a measure where CMS
appears to be concerned regarding the wide
variance in reported numbers. The current
Validation Standard and related
sampling instructions focus on sampling
grievance data only. ACG recommends that
CMS add to the Grievance sampling criteria
that DVCs audit a sample of Part C
Customer Service logs to validate grievance
counts under the expected numbers rubric.
Without testing the customer service logs is a
chance that MAOs will receive validation for a
total number of Part C grievance data
remains incorrect because it only samples
data within the MAO’s or PDP’s actual
Grievance database, not actual complaints,
especially oral, filed with the MAO or PDP.
The use of the term “final decision” needs
clarification for proper recording and accurate
testing. Although re-openings are defined in
Managed Care Manual Ch.13, 130, as a
remedial action to taken to change a final
determination, MAO plans use this process to
manage cases where an adverse
determination was issued because requested
information was not received, and MAO plans
subsequently receive the information. Without
clarification that these cases are excluded,
MAOs might include them. Even with
clarification, the data validation procedure
should include assessing if the MAO has
classified these cases correctly
ACG recommends clarification regarding Sub
Element 5c. The statement that ODs “does
not combine fully favorable claims
determinations for the same approved
services” is vague. With this statement, CMS
may be trying to avoid double counting
approved final authorizations and paid claims.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

Page 40 of 58

15
7

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

15
8

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

15
9

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

16
0

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

As written, Sub Elements 5b and 5k
potentially conflict. 5b asks MAOs to include
all ODs covered by Medicare and Medicaid,
yet 5k allows MAOs to exclude ODs where
there is no member liability. In some States,
the Medicaid coverage for duals effectively
shields the member from all liability, as
Medicaid picks up what Medicare does not.
ACG recommends that CMS clarify whether
ODs should be counted in 5K in instances
where a State picks up the cost sharing
balance for dual eligible Medicaid members.
Currently, CMS currently requires that such
complaints be reported under each contract
an MAO or PDP has if the specific contract
can not be identified. One example would
complaints received during a canceled
enrollment or “beneficiary did not complete”
enrollment be reported since they are not tied
to a specific contract. Current guidance may
significantly inflate the number of actual
complaints, especially for large national plans
if the actual contract is not identified, which
can often be the case. As such there is
potentially built in bias in reported numbers.
ACG recommends that CMS change this
requirement to allow the MAO or PDP to
record the complaint once, for one contract
within the state where the complaint
occurred, if
the actual contract can not be identified,
thereby enhancing the “comparability” of the
supplied data and simplify testing.
Some states to not require appointment of
agents. ACG recommends that 4b read "all
licensed agents who are under contract
agreement to sell on behalf of the contract
and appointed during the reporting year,
except in states that do not require
appointment"
ACG believes that the term “Complaint” is
insufficiently defined to effectively support
comparable data. Complaints may range from
"Activities which mislead, confuse, or
misrepresent the MAO”, to the fact that an
agent was late for an appointment, or falsified
information on an application. ACG
recommends that CMS define what types of
“Complaints” are to be included in the report
values and tested by the DVC.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

Page 41 of 58

16
1

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

ACG recommends that CMS clarify in the
Technical Specifications and DVA Standards
whether revocation of selling privileges is
reported only when it is permanent or when
permanent or temporary (as when going
through retraining.)

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

16
2

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

16
3

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

CMS considered this approach in
developing the DV standards and
determined it would add an unacceptable
resource burden to the process

No
change.

16
4

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

Dates of LTC enrollment, MTM enrollment
and MTM opt-outs could occur multiple times
for the same enrollee within the time period.
ACG recommends that CMS clarify and
define whether beneficiaries with multiple optouts (over time) should be counted once or
per opt-out. (Assuming re-enrollment)
Grievances are a measure where CMS
appears to be concerned regarding the wide
variance in reported numbers. The current
Validation Standard and related
sampling instructions focus on sampling
grievance data only. ACG recommends that
CMS add to the Grievance sampling criteria
that DVCs audit a sample of Part C
Customer Service logs to validate grievance
counts under the expected numbers rubric.
Without testing the customer service logs is a
chance that MAOs will receive validation for a
total number of Part C grievance data
remains incorrect because it only samples
data within the MAO’s or PDP’s actual
Grievance database, not actual complaints,
especially oral, filed with the MAO or PDP.
Low Income Subsidy (LIS) status changes
frequently and will impact whether MAPD and
PDP quarterly reporting is accurate even
though it reflected the best
available information when the report was
filed. For example, a non-LIS member files a
grievance on May 28 with the grievance
closing on June 3, but in August
there is a retroactive change to the member’s
LIS status. ACG requests that CMS provide
guidance as to how this situation should
reported by the MA-PD or PDP, i.e. as an LIS
member or not.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

Page 42 of 58

16
5

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Data
Validation
Standards

“Pharmacy Transactions” should be better
defined. Does it include or not include EA
drugs transactions?

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

16
6

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Sampling
Instructions

Sample Sizes to not appear to be
appropriately stratified for smaller plans.

The "Data Extraction and Sampling
Instructions" have been updated to
indicate that stratified data is an option.
This is left to the reviewer's discretion.

No
change.

16
7

FDMS

ATTAC
Consulting
Group, Inc

0022

6/18/2010

Sampling
Instructions

Sampling instructions and testing overall do
not reflect the challenges of a delegated
services or processing model operated by
many MAOs and PDPs (e.g.
IPAs, PBMs) where source data may not be
easily obtainable.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

16
8

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

N/A

Rather than validating all these measures
every year, organizations would like to see a
more pointed audit that utilizes a phased
approach. Perhaps data validation could be
completed on a portion of the measures one
year and a different portion the following year.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

16
9

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

N/A

Part D data validation will commence around
March 1, 2011 and the data validation audits
are expected to occur each year over a threemonth period. However, measures like 2010
Benefit Utilization aren’t due until August
2011. Please clarify how the data will be
validated if the audits take place in March?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

17
0

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

N/A

Regarding the timing of the validation audits,
we suggest conducting these after the bid
submission due date. Since some of the work
on the bid creation feeds into our reporting
data (particularly benefit utilization), that work
would then be completed by the time of the
audit and not place undue burden on plans
while they are focusing on bid submission.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 43 of 58

17
1

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

Supporting
Statement

Please clarify whether contracts that are nonrenewing for CY2011 will be required to
undertake a data validation audit for data
reported in 2010

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

17
2

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

Supporting
Statement

The burden estimates have been revised.
Please see the updated Supporting
Statement for the updated assumptions
and calculations.

N/A.

17
3

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

OAI

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

17
4

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

Findings
Data
Collection
Form

No response required.

N/A.

17
5

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

Findings
Data
Collection
Form

Based on our analysis, CMS has grossly
underestimated the staff time and expense of
these audits. Our estimates indicate it will
take 5 to 10 times the staff time as CMS’ 120
hour estimate. The vendors we have spoken
to estimate their staff time will be 2 to 3 times
CMS’ estimate of 206 hours per contract.
Finally, based on quotes we’ve received from
audit vendors, the cost to procure and
support the auditor is anywhere from 5 to 10
times higher than CMS’ estimate of $5,177
per contract
Indicates that a separate OAI must be
completed for each contract if the information
provided varies by contract. We suggest
formatting the OAI in such a way as to
accommodate the inclusion of more than one
contract. This would be a relatively simple
change to the setup of the document and
would decrease the burden on organizations
who will otherwise have to fill out a separate
version for every contract they have.
States the reviewer will share findings with
the organization and then submit the
completed Findings Data Collection Form to
CMS who will process the measure- or data
element-level findings for each measure’s
standards to derive an overall “Pass” or “Not
Pass” determination.
Similar to other regulatory processes
(CMS/OIG audits), organizations should have
an opportunity to respond to or dispute the
reviewer’s findings.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 44 of 58

17
6

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

Findings
Data
Collection
Form

There is no information as to what constitutes
“Pass” and “Not Pass”, the ramifications of
receiving a “Not Pass” and what recourse
organizations have in that event. The
document suggests this will be an overall
determination. What is meant by “overall”
since some measures are at the PBP level
and some at the contract level?

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

17
7

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

Findings
Data
Collection
Form

We suggest that CMS prioritize the measures
for purposes of “pass” and “not pass”. For
example, there are 138 benefit utilization
measures –we suggest ranking the measures
and requiring a smaller number of measures
that CMS considers higher priority be met to
constitute a pass rather than all measures.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

17
8

FDMS

Blue Cross
Blue Shield of
Minnesota

0023

6/18/2010

Sampling
Instructions

In comparison to other types of audits (for
example CMS audits, NCQA) the sample
sizes are very large. This is burdensome for
plans as well as auditors and we feel these
sample sizes could be reduced substantially
while maintaining the integrity of the
validation.

No change required.

No
change.

Page 45 of 58

17
9

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

Standards
for Selecting
a Data
Validation
Contractor

Contractor Selection. In the Supporting
Statement, CMS explains that each plan
sponsor will be required to enter into
agreement with a data validation contractor
that meets CMS’ standards, which will
describe the minimum qualifications,
credentials, and resources that the contractor
must possess. During the CMS 2010
Medicare Advantage and Prescription Drug
Plan Spring Conference (2010 Spring
Conference), CMS indicated that CMS will
provide training this fall for plan sponsors and
interested contractors and that each plan
sponsor will be responsible for documenting
its contractor selection process, including
how the sponsor determines that the
contractor chosen meets the CMS standards.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

To serve CMS’ principal goal for the data
validation program, which is to ensure that
data for monitoring and performance
measurement are “reliable, valid, complete,
and comparable among sponsoring
organizations,” we strongly urge the agency
to certify contractors that meet CMS
requirements and to make a list of these
contractors available to plan sponsors. By
conducting a centralized review of
contractors, CMS or a contractor on the
agency’s behalf will be best positioned to
accomplish uniform and accurate application
of CMS standards, promote consistent data
validation reviews by contractors, and ensure
availability of a sufficient number of qualified
contractors to conduct data validation for all
plan sponsors. In addition, the agency’s
knowledge of the contractors as a result of
the certification process will position CMS to
evaluate the potential impact of contractor
performance on data validation results, which
will be particularly important in the first year of
implementation when CMS, plan sponsors,
and contractors will be gaining experience
with the new process.
This approach would be consistent with the
conduct of the Health Outcomes Survey
(HOS) and CMS’ plans for the Medicare
Consumer Assessment of Health Plans
Survey (CAHPS). As the agency explained in
the preamble to its October 22, 2009 (74 FR
54633) proposed regulation for the MA and
Part D programs and confirmed in the
preamble to the April 15, 2010 final regulation
(75 FR 19677), MA organizations will be

Page 46 of 58

18
0

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

Standards
for Selecting
a Data
Validation
Contractor

18
1

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

Standards
for Selecting
a Data
Validation
Contractor

18
2

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

Standards
for Selecting
a Data
Validation
Contractor

If CMS does not adopt AHIP’s
recommendation for CMS contractor
certification and plan sponsors remain
responsible for the contractor selection
process, we recommend that CMS provide
clear, detailed guidance about the agency’s
standards for qualified contractors and the
documentation plan sponsors must maintain
to meet CMS’ expectations for demonstrating
how they have applied these standards. We
urge CMS to issue these requirements as
soon as possible, because plan sponsors are
already working to identify potential
contractors to be prepared to contract for
timely pre-assessment and data validation
reviews.
We also recommend that CMS conduct a
training session for plan sponsor staff
involved in contractor selection to promote
consistent understanding of the requirements.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Further, we recommend that CMS develop a
strategy for addressing the possibility that
qualified contractors may not have sufficient
capacity to accommodate contracting with all
plan sponsors for simultaneous reviews.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 47 of 58

18
3

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

Standards
for Selecting
a Data
Validation
Contractor

To prepare for CMS desk and onsite program
audits based upon CMS’ Monitoring Guides,
it is common for plan sponsors not only to
conduct internal activities to review
compliance with CMS requirements but also
to engage contractors to conduct
independent reviews. As plan sponsors
prepare for implementation of the data
validation reviews, they are interested in
following similar practices. As CMS
establishes standards that contractors must
meet to conduct Part C and Part D data
validation, we recommend that the agency
explicitly provide that a plan sponsor may
utilize the same contractor to conduct both
pre-assessment and data validation reviews.
CMS indicated at the agency’s 2010 Spring
Conference that such arrangements with
qualified contractors would be permissible,
and we believe that this approach could not
only facilitate plan sponsor efforts to enter
into agreements with contractors but also has
the potential to facilitate data validation by
familiarizing the contractor with plan sponsor
systems in advance of the data validation
review.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 48 of 58

18
4

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

N/A

It is not uncommon for plan sponsors to
contract with PBMs and other delegated
entities to carry out functions that may include
collecting and maintaining data that are used
to satisfy Part C and/or Part D reporting
requirements and that would be included in
data validation reviews. Since a single PBM
or other delegated entity may contract with
multiple Part C and Part D plan sponsors, it
will be administratively complex and
significantly more burdensome for these
delegated entities to work with differing data
validation contractors selected by each plan
sponsor. To address this situation, we
recommend that CMS provide in the
guidance that, for data maintained by
delegated entities, it is permissible for plan
sponsors, at their election, to rely on reviews
performed by qualified data validation
contractors under contract to the delegated
entities. We also recommend that CMS
specify, if such an election is made, how
reporting of results from multiple data
validation contractors for a single Part C or
Part D contract number would be
accomplished, and we urge CMS to develop
these requirements in consultation with Part
C and Part D plan sponsors. We would be
interested in engaging in discussions with
CMS to assist the agency in identifying and
resolving practical issues that could merit
consideration.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 49 of 58

18
5

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

While data for a number of measures under
the Part C reporting requirements are due by
February 28 of the following year (e.g.,
provider network adequacy, plan oversight of
agents), several Part C measures have a
reporting deadline of May 31 of the following
year (e.g., procedure frequency, serious
reportable adverse events, SNPs care
management) and Part C benefit utilization
data are due by August 31 of the following
year. In addition, Part D data on long-term
care pharmacy utilization are due by June 30
of the following year. The Supporting
Statement and accompanying Appendices do
not address how the data validation process
for these measures would be coordinated
with the data collection deadlines. However,
CMS noted at the agency’s 2010 Spring
Conference, that for “measures with
submission dates of 5/31, 6/30, and 8/31, an
extension for data validation reviews and
submission of findings to CMS is being
considered.” (See CMS presentation on “Part
C & D Reporting Requirements and Data
Validation,” slide 10.)

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 50 of 58

18
6

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

Since a later data validation data collection
period than the proposed time frame of
approximately March – May would be better
aligned with the deadlines under the
Reporting Requirements, we recommend that
CMS reconsider the March – May time frame
for annual data validation data collection and
submission and establish a later period to
provide for consistent coordination with the
deadlines under the Reporting Requirements
across all measures. If CMS does not adopt
this recommendation, we recommend that the
agency establish specific timeframes for
validation of data for each measure that has a
deadline of May 31 or later under the Part C
and Part D Reporting Requirements.
Since a later data validation data collection
period than the proposed time frame of
approximately March – May would be better
aligned with the deadlines under the
Reporting Requirements, we recommend that
CMS reconsider the March – May time frame
for annual data validation data collection and
submission and establish a later period to
provide for consistent coordination with the
deadlines under the Reporting Requirements
across all measures. If CMS does not adopt
this recommendation, we recommend that the
agency establish specific timeframes for
validation of data for each measure that has a
deadline of May 31 or later under the Part C
and Part D Reporting Requirements.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 51 of 58

18
7

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

18
8

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

In response to a question at CMS’ 2010
Spring Conference, the agency indicated that
CMS would evaluate the first year of
experience with the data validation initiative
before considering whether the validation
program might move to requiring submission
of data on a subset of measures rather than
all measures. While AHIP appreciates that
CMS has conducted pilot testing of the data
validation program, the agency has indicated
that only one large MA-PD organization and
one large PDP participated in this testing. As
with any new program, it is likely that as all
MA and Part D plan sponsors begin
implementation, a variety of questions and
operational issues will arise as plan sponsors,
data validation contractors, and CMS gain
experience with the data validation
requirements.
Consequently, AHIP recommends that for the
first year, CMS revise its approach to focus
reporting on a subset of measures to permit
all parties to gain an understanding of the
process and allow an opportunity for any
needed adjustments to be put in place. We
also recommend that as CMS evaluates the
results, the agency take into consideration
that the results may be affected by start-up
issues. In addition, for the future, we
recommend that CMS rotate validation of
measures over a three year timeframe, rather
than requiring validation of the entire set of
measures in order to make the best use of
CMS and plan sponsor resources. We
believe that such an approach would permit
CMS to complete timely analysis of the data
validation results and utilize this information
effectively for oversight purposes.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 52 of 58

18
9

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

19
0

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

Standards
for Selecting
a Data
Validation
Contractor

In the Supporting Statement on page 5 and in
Appendix 1 (see below) CMS indicates that
the agency will “process the measure-level or
data element-level findings for each
measure’s standards to derive an overall
‘Pass’ or ‘Not Pass’ determination.” At CMS’
2010 Spring Conference, the agency
explained that scoring and thresholds for
these determinations are under development
and would be the subject of future CMS
guidance. We support the agency’s intent to
provide information to plan sponsors about
the methodology CMS will utilize to arrive at
“Pass” and “Not Pass” determinations. We
recommend that the forthcoming guidance
provide a detailed explanation of the
methodology and that CMS provide an
opportunity for plan sponsor review and
comment prior to finalizing the methodology
to allow the agency to consider relevant
operational and implementation issues.
The third paragraph under Section 1.1 states
that “The reviewer will share these findings
with the organization and then submit the
completed Findings Data Collection Form to
CMS, who will process the measure- or data
element-level findings for each measure’s
standards to derive an overall “Pass” or “Not
Pass” determination.” To promote consistent
understanding, we recommend that CMS
revise this language in the instructions to
clarify that the contractor will be required to
conduct an exit interview, as well as share
findings with the plan sponsor, and that
contractor findings submitted to CMS should
note any findings with which the plan sponsor
disagrees and the reason for the
disagreement. It appears the language in the
Appendix is intended to reference these wellestablished steps that are commonly followed
in CMS audits.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the data validation
training (expected Fall 2010), and the data
validation Manual. Regarding the
Pass/Not Pass Determination logic, CMS
plans to share this logic with industry once
it is completed (this logic does not require
OMB clearance and will be shared
separately).

N/A.

Policy questions will be addressed
separate from the documents undergoing
OMB clearance. These questions will be
addressed in future industry
communications, the Data Validation
Training (expected Fall 2010), and the
Data Validation Manual.

N/A.

Page 53 of 58

19
1

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

Data
Validation
Standards

19
2

FDMS

America's
Health
Insurance
Plans

0024

6/18/2010

Sampling
Instructions

In the Measure-Specific Criteria, items 5. and
8. list the criteria for determining whether the
MA organization accurately calculates the
total number of organization determinations
and reconsiderations, respectively. Criteria
5k. and 8f. indicate that the calculation
excludes organization determinations and
reconsiderations, respectively, where there is
no member liability. However, in the
Medicare Part C Reporting Requirements
Technical Specifications, Version Date: June
3, 2010, the specifications for measure “6.
Organization
Determinations/Reconsiderations,” no longer
include this statement, although it appeared
in the previous version (Version Date:
February 24, 2010). For clarity, we
recommend that CMS revise the Part C
Reporting Requirements Technical
Specifications to restore the statement. We
also recommend that CMS conduct a review
to ensure that the data validation standards
are consistent with the June 3 reporting
requirements for all measures.
In section “2.0 Conceptual Framework for
Sampling,” CMS notes in item 2. that “[m]any
organizations have analytic warehouses
where data is cleansed and put into database
structures to support analysis.” Subsequently
in section 2.0 and in item “4. Create ‘Source
Samples(s)’” of section “3.0 Sampling
Process Detail,” CMS indicates that source
samples will be drawn from either the plan
sponsor’s data warehouse or operational
systems. Consistent with the related
schematics, it appears that this language is
intended to indicate that if the plan sponsor
has established a data warehouse, the
source sample will be drawn from the
warehouse. If not, the source sample will be
drawn from the operational source systems.
We recommend that CMS revise the
language in the referenced sections to clarify
that this is the case.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

The "Data Extraction and Sampling
Instructions" have been updated to add
clarification.

No
change.

Page 54 of 58

19
3

FDMS

Health Net,
Inc.

0025

6/18/2010

Data
Validation
Standards

19
4

FDMS

Health Net,
Inc.

0025

6/18/2010

Data
Validation
Standards

19
5

FDMS

Health Net,
Inc.

0025

6/18/2010

Sampling
Instructions

This methodology would result in an
incomplete count of HRA completion. HRA
completion for members that were enrolled at
the end of the reporting period may have
completed the HRA subsequent to the
reporting period given the 90 day window
allowed for this requirement. Therefore it is
recommended that either 1) Data Element
13.3 extends the measurement period to 90
days after the reporting period so that
completion of an HRA within the 90 day
required time frame is included in this
measure for all newly enrolled SNP members
whose effective date of enrollment occurred
during the reporting period OR 2 ) The
requirement that Element 13.3 be a subset of
Element 13.1 is removed so that all HRAs
completed during the reporting period be
included in this data element regardless of
the effective date of enrollment.
Appendix 3 (p.9) Criteria 4 states: 4.
Organization accurately calculates the total
number of surgeries, including the following
criteria: a)Includes all surgeries with dates of
service that occur during the reporting period;
b)Includes only surgeries that occur in an
acute hospital setting(Data Element 3:1) But
5.b. states: c) Includes any supplemental
information provided by the hospital regarding
SRAEs that are confirmed during the
reporting period (even if the event actually
occurred during a previous reporting period).
Our Comment: There will be no
standardization or ability to trend over time or
across plans if events from prior periods are
included. The same direction is provided for
HACs.
Will CMS provide alternatives to submission
format (other than by CD)?

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

The measure-specific criteria in both the
"Data Validation Standards" and the
"Findings Data Collection Form" have
been updated to reflect the most recent
updates to the Technical Specifications
documents (June 2010 versions).

No
change.

The "Data Extraction and Sampling
Instructions" have been updated to allow
more flexibility (document now allows for
secure storage devices beyond CDs).

No
change.

Page 55 of 58

19
6

Pilot
Finding

CMS

N/A

6/16/2010

Sampling
Instructions

CMS Pilot Finding: After extracting sample
data sets for some measures and extracting
the entire data set or “census” for other
measures, it was determined that extraction
of the entire data set did not add an undue
burden to the organization undergoing
review.

The "Data Extraction and Sampling
Instructions" have been updated to
include the following CMS
recommendation: When possible, the data
validation reviewer should obtain the
census of data records used to report a
measure. This will ensure that the source
through final stage data sets support the
data reported via HPMS. An exact
determination of compliance with
validation standards can be determined
using the census, instead of relying on an
estimate generated by sampling. The use
of random sampling should be left to the
discretion of the data validation reviewer
and should be limited to situations where
pulling all records for a measure will
create too heavy a burden on the
organization. In addition, the "Data
Validation Standards" have updated
references of "sample data" to "census or
sample data."

Net
decrease
.

19
7

Pilot
Finding

CMS

N/A

6/16/2010

Sampling
Instructions

CMS Pilot Finding: Using sample data to
check manual processes or to check for
errors that occur relatively infrequently may
require larger sample sizes than those
currently outlined in the Sampling
Instructions.

The "Data Extraction and Sampling
Instructions" have been updated to
include the following CMS
recommendation: Allow data validation
reviewers the flexibility to request sample
data sets larger than the sizes prescribed
in the April 2010 Sampling Instructions
(i.e., more than 150 or 205 records) if
additional data are required to complete
the review.

Net
decrease
.

19
8

Pilot
Finding

CMS

N/A

6/16/2010

Sampling
Instructions

CMS Pilot Finding: It was no more difficult to
pull data for the entire year vs. pulling sample
data for only one reporting period (e.g., one
quarter).

The "Data Extraction and Sampling
Instructions" have been updated to
include the following CMS
recommendation: The data validation
reviewer should select and review the
entire year’s data for a measure, despite
the measure’s reporting frequency
requirements (e.g., quarterly, bi-annual).
This will simplify the process for the data
validation reviewer and allow thorough
examination of all reported data,
eliminating issues related to data
seasonality.

Net
decrease
.

Page 56 of 58

19
9

Pilot
Finding

CMS

N/A

6/16/2010

Sampling
Instructions

CMS Pilot Finding: While two to four gigabyte
flash drives were sufficient for collecting data
for the pilot tests, larger external drives may
be needed for data covering multiple
contracts and data measures.

The "Data Extraction and Sampling
Instructions" have been updated to
include the following CMS
recommendation: The data validation
reviewer should work with the sponsoring
organization prior to the site visit to
determine file sizes and ensure that data
storage requirements are sufficient for
data transport.
The "Data Extraction and Sampling
Instructions" have been updated to
include the following CMS
recommendation: Ensure that copies of
source, intermediate, and final stage files
are saved so that reporting requirements
can be re-generated at any given time for
validation purposes (e.g., so that counts in
the files match HPMS reported counts).

No
change.

20
0

Pilot
Finding

CMS

N/A

6/16/2010

Sampling
Instructions

CMS Pilot Finding: It was more likely that if
the source data relied on a transactional
database where records are often updated,
the source files were not archived. Similarly,
many of the intermediate files created using
query programs were not archived.

20
1

Pilot
Finding

CMS

N/A

6/16/2010

Sampling
Instructions

CMS Pilot Finding: Additional data fields
should have been included in the
organizations’ sample data sets, in order to
assess the accuracy of their reported data.

The "Data Extraction and Sampling
Instructions" have been updated to
include the following CMS
recommendation: An organization’s
measure report owners/data providers
should familiarize themselves with the
standards and criteria included in the
"Data Validation Standards" document.
This will ensure that the report
owners/data providers are prepared to pull
the appropriate data fields necessary
during the sampling process. The data
validation reviewer should also reference
this document as needed when
conducting the on-site review to confirm
that the required data fields are provided.

No
change.

20
2

Pilot
Finding

CMS

N/A

6/16/2010

Sampling
Instructions

CMS Pilot Finding: Without intermediate data
sets, it may be difficult for the review team to
determine whether data sets were extracted
properly (e.g., tables may have been joined
incorrectly, or records were
included/excluded improperly).

The "Data Extraction and Sampling
Instructions" have been updated to
include the following CMS
recommendation: For more complex
measures that draw data from multiple
databases or intermediate data source
files, a sample or census from each of the
intermediate data sets will aid the data
validation reviewer in determining if tables
are being joined properly.

No
change.

No
change.

Page 57 of 58

20
3

Pilot
Finding

CMS

N/A

6/16/2010

Sampling
Instructions

CMS Pilot Finding: An organization’s security
software may interfere with transferring data
to an encrypted flash or hard drive.

The "Data Extraction and Sampling
Instructions" have been updated to
include the following CMS
recommendation: The reviewer and
organization should confirm that the type
of device used to transfer data will be
compliant with the organization’s systems.

No
change.

20
4

Pilot
Finding

CMS

N/A

6/16/2010

OAI

CMS Pilot Finding: The level of detail in the
documentation provided by the pilot
organizations varied.

The "Organizational Assessment
Instrument" have been updated to include
the following CMS recommendation:
Include a data dictionary template as a
reference to help organizations more
effectively prepare their documentation for
reviewers. For example, the dictionary
template could illustrate pertinent content
such as data field name, data field
description, and code definitions.

No
change.

20
5

Pilot
Finding

CMS

N/A

6/16/2010

OAI

CMS Pilot Finding: There were gaps in the
documentation provided, and it was not
always possible to replicate a data report that
was submitted to CMS via HPMS.

The "Organizational Assessment
Instrument" have been updated to include
the following CMS recommendation:
Include instructions for the organizations
to reference the CMS Reporting
Requirements Technical Specifications to
ensure documentation is provided for all
data elements. Include a list of minimally
required data fields (e.g., Case ID, Case
Receipt Date, Case Resolution Date) for
each measure to ensure that the
appropriate data and level of detail is
being captured for accurate reporting.

No
change.

Page 58 of 58