CY 2023 DV_DV Procedure Manual_20221014

Medicare Part C and Part D Reporting
Requirements Data Validation
Procedure Manual

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of
information unless it displays a valid OMB control number. The valid OMB control number for this information
collection is 0938-1115 (Expires: 04/30/2023). The time required to complete this information collection is
estimated to average 30 hours per response, including the time to review instructions, search existing data
resources, and gather the data needed, and complete and review the information collection. If you have
comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write
to: CMS, 7500 Security Boulevard, Attn: PRA Reports clearance Officer, Mail Stop C4-26-05, Baltimore,
Maryland 21244-1850.
Prepared by:
Centers for Medicare & Medicaid Services
Center for Medicare
Medicare Drug Benefit and C&D Data Group
Last Updated: September 2022

1.
1.1.
1.2.
1.3.
1.4.
1.5.
2.
2.1.
2.2.
3.
3.1
3.1.1
3.1.2
3.1.3
3.2
3.2.1
3.2.2
3.3
3.4
3.4.1
3.4.2
3.4.3
4.
4.1
4.2
4.2.1.
4.2.2.
4.2.3.
4.3
4.3.1
4.3.2
4.3.3
4.4
4.4.1
4.4.2
4.4.3
4.4.4
4.4.5
4.5
5.
5.1
5.1.1.
5.1.2.
5.1.3.
5.1.4.
5.2
5.3

Table of Contents

INTRODUCTION ............................................................................................................................................................................................. 4
Data Validation Requirement ........................................................................................................................................................................... 4
Data Validation Scope ..................................................................................................................................................................................... 4
Types of Organizations Required to Undergo Data Validation ........................................................................................................................ 4
Required Data Validation Manual and Tools ................................................................................................................................................... 5
Organization of the Procedure Manual ............................................................................................................................................................ 5
PART C AND PART D REPORTING SECTIONS REQUIRING DATA VALIDATION IN 2023 ....................................................................... 7
Part C and Part D Reporting Sections Requiring Data Validation in 2023 ...................................................................................................... 7
Reporting Requirements Excluded from the Validation Requirement ............................................................................................................. 8
PLANNINGFOR DATA VALIDATIONACTIVITIES ............................................................................................................................................ 9
Select appropriate contractor based on Standards for Selecting a Data Validation Contractor ...................................................................... 9
Standards for Selecting a Data Validation Contractor ........................................................................................................................................ 9
Timing of Data Validation Contractor Selection ................................................................................................................................................. 9
Requesting a Contractor Change Mid-Review ................................................................................................................................................ 9
Notify CMS of DVC Selection / Request Access to Health Plan Management System (HPMS) Plan Reporting Data Validation Module
(PRDVM) ....................................................................................................................................................................................................... 10
Documentation of Data Validation Contractor Selection Process ..................................................................................................................... 10
Request Access to HPMS Plan Reporting Data Validation Module .............................................................................................................. 10
Complete the Web-based Data Validation Training....................................................................................................................................... 11
Review all Data Validation Documents .......................................................................................................................................................... 11
Introduction to the Data Validation Standards ................................................................................................................................................. 11
Data Validation Standards and Reporting Section Criteria ............................................................................................................................... 12
Reporting Section Criteria.............................................................................................................................................................................. 14
PERFORMING DATA VALIDATIONACTIVITIES ........................................................................................................................................... 17
Complete Organizational Assessment Instrument (OAI) and Provide Appropriate Documentation to Selected DVC per the OAI’s (Appendix
E’s) Documentation Request ......................................................................................................................................................................... 17
Analyze OAI (Appendix E) Responses .......................................................................................................................................................... 17
Perform OAI (Appendix E) Gap Analysis ....................................................................................................................................................... 17
Review Source Code and Other Documentation ........................................................................................................................................... 18
Prepare Interview DiscussionGuide............................................................................................................................................................... 18
Prepare for On-Site or VirtualVisit ................................................................................................................................................................. 18
Select Dates and Appropriate Location(s) of On-Site or Virtual Visit ................................................................................................................ 18
Develop Agenda for On-Site or Virtual Visit ................................................................................................................................................... 18
Prepare for Data Extraction and Sampling ...................................................................................................................................................... 18
Conduct On-Site or Virtual Visit...................................................................................................................................................................... 19
Conduct Entrance Conference ....................................................................................................................................................................... 19
Conduct Interviews with Organization Staff ..................................................................................................................................................... 19
Observe Reporting Processes ....................................................................................................................................................................... 19
Extract Census or Sample Data ...................................................................................................................................................................... 19
Conduct Exit Conference ............................................................................................................................................................................... 20
Request Additional Documents (If Required) ................................................................................................................................................. 21
ANALYZING RESULTS AND SUBMISSION OF FINDINGS ........................................................................................................................ 22
Determine Compliance with Data Validation Standards and Record Findings in the Findings Data Collection Form (FDCF) (Appendix J) to
Upload into the HPMS PRDVM ..................................................................................................................................................................... 22
Reporting Findings for Standards Using Binary Scale ..................................................................................................................................... 22
Reporting Findings for Standards Using Likert Scale ....................................................................................................................................... 23
Review of the Findings Data Collection Form .................................................................................................................................................. 24
Guidance for Interpreting Standards and Making a Findings Determination....................................................................................................... 31
Provide Draft Findings to Sponsoring Organization ......................................................................................................................................... 38
Review Draft Findings with Sponsoring Organization and Obtain Additional Documentation Necessary to Resolve Issues........................ 38

5.4.
5.4.1.
5.4.2.
6.
6.1
6.2
6.2.1
6.2.2
6.3

Submit Data Validation Review Findings via HPMS PRDVM ............................................................................................................................ 39
Data Validation Contractor’s Submission of Findings ...................................................................................................................................... 39
Sponsoring Organization Disagreement with Findings.................................................................................................................................... 39
POST- DATA VALIDATIONACTIVITIES ........................................................................................................................................................ 40
Compile Archive of Data Validation Work Papers ............................................................................................................................................ 40
Receive Pass or Not Pass Threshold Level and Assess Pass or Not Pass Determination based on Final Scores ...................................... 40
Pass/Not Pass Determination......................................................................................................................................................................... 40
Passing Data Validation – Minimum threshold .............................................................................................................................................. 40
Sponsoring Organization Appeal of Data Validation Determination .............................................................................................................. 40

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Appendices
Appendix A: Standards for Selecting a Data Validation Contractor
Appendix B: Part C and Part D Reporting Section Data Validation Standards
Appendix C: Model Language for Letter to Confirm Selection of Data Validation Contractor
Appendix D: Example Application for Access to CMS Computer Systems
Appendix E: Organizational Assessment Instrument
Appendix F: Interview Discussion Guide
Appendix G: Example Site Visit Agenda for On-site or Virtual Visits
Appendix H: Data Extraction and Sampling Instructions Appendix I: Example Data File Inventory Log
Appendix J: Findings Data Collection Form
Appendix K: Data Validation Pass/Not Pass Determination Methodology

3

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

1. INTRODUCTION
1.1. Data Validation Requirement
The Centers for Medicare & Medicaid Services (CMS) requires that sponsoring organizations (SOs) contracted
to offer Medicare Part C and/or Part D benefits be subject to an independent yearly review to validate data
reported to CMS. 1 The purpose of the independent data validation (DV) is to ensure that Part C and Part D
SOs are reporting health and drug plan data that are reliable, valid, complete, comparable, and timely.
The primary purpose of this Procedure Manual (Manual) is to provide SOs and the data validation contractors
(DVCs) they select to perform DV with information regarding the Part C and Part D Reporting Requirements
Data Validation program. The Manual provides background information and an overview of the DV program,
discusses the scope and timeframe required for the DV, and describes the tools and processes used for
conducting the DV.
The validated data improves reporting and assures that data are credible and consistently collected and
reported by Part C and Part D SOs. CMS uses these reported data to respond to inquiries from Congress,
oversight agencies, and the public about an SO’s performance using indicators such as operations, costs,
availability and use of services, provider network adequacy, and grievance rates. The validated data also
allows CMS to more effectively monitor and compare the performance of SOs over time. These data may be
used for Star Ratings, and other performance measures. Additionally, SOs can take advantage of the DV
process to more effectively assess their own performance and make improvements to their internal data,
systems, and reporting processes.

1.2. Data Validation Scope
CMS requires that the annual, DV be conducted once per year. For the 2023 DV cycle, the DV will take place
during the April 1, 2023 – June 30, 2023 timeframe and will incorporate all data submitted to CMS by March
31st based on the previous calendar years’ reporting requirements. Any data submitted or re- submitted by an
SO after March 31 cannot be used for purposes of the DV. The reviewer must submit findings from the annual
DV review to CMS by June 30, 2023.
The DV reviews will continue to be conducted at the contract level. CMS believes the contract is the most
appropriate unit of analysis in conducting this DV, given that the Part C/D data are generally available at the
contract level and that the contract is the basis of any legal and accountability issues concerning the rendering
of services.

1.3. Types of Organizations Required to Undergo Data Validation
All Part C and Part D SOs that report Part C and/or Part D data to CMS per the Reporting Requirements,
regardless of enrollment size, are required to undergo an annual DV review.
The only SOs exempt from participating in the data validation program are:

•
•
•

Program of All-Inclusive Care for the Elderly (PACE) SOs and Part C Health Care Prepayment Plans.
Organizations/sponsors that non-renew or terminate during the measurement year or reporting/data
validation year and are not included in the HPMS plan table.
Organizations/sponsors with contracts or Plan Benefit Packages (PBPs) that non-renewed or
terminated in 2022 or prior to July 1, 2023, are not required to report 2022 Part C/D reporting
requirements data (due in 1Q of 2023), or undergo Part C/D data validation of 2022 data (due June
2023).

Any SO that delegates the data collection, calculation, and/or reporting for any reporting section or data
element to a Pharmacy Benefit Manager (PBM) or other type of delegated entity must have the DVC it hires
include the data and reporting processes for which the PBM/delegated entity is responsible in its DV review for
each applicable contract. For example, all entities are required to provide applicable policies, procedures, and
source data to the DVC for validation if they submit data to an SO that is used for any reporting section.

1

See 42 CFR §422.516(g) and §423.514(j)

4

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

1.4. Required Data Validation Manual and Tools
This manual and its appendices must be used to conduct the annual data validation. The following documents
are included:
1. Standards for Selecting a Data Validation Contractor (Appendix A)
2. Data Validation Standards (Appendix B)
3. Model Language for Letter to Confirm Selection of Data Validation Contractor (Appendix C)
4. Example Application for Access to CMS Computer Systems (Appendix D)
5. Organizational Assessment Instrument (OAI) (Appendix E)
6. Interview Discussion Guide (IDG) (Appendix F)
7. Example Site Visit Agenda for On-site or Virtual Visits (Appendix G)
8. Data Extraction and Sampling Instructions (Appendix H)
9. Example Data File Inventory Log (Appendix I)
10. Findings Data Collection Form (FDCF) (Appendix J)
11. Pass/ Not Pass Determination Methodology (Appendix K)
The Data Validation Standards (Standards) and other documentation associated with the implementation of the
DV program assess an SO’s information systems capabilities and overall processes for collecting, storing,
compiling, and reporting the required Part C and Part D reporting sections. CMS updates and incorporates
periodic clarifications to this Manual and the DV tools contained in its appendices as needed; the most current
version is posted publicly at Medicare Part C and D Data Validation and within the Health Plan Management
System (HPMS) Plan Reporting Data Validation Module (PRDVM). Prior to each annual DV it is the
responsibility of all SOs and DVCs to confirm that they are using the most recent DV documentation available
on the CMS DV website.
In the event of a conflict between the Reporting Requirements and the Data Validation Standards reporting
section criteria, the Data Validation Standards supersede the Reporting Requirements. DVCs must use the
Data Validation Standards reporting section criteria to determine DV findings. CMS will take a conflict between
the Reporting Requirements and the Data Validation Standards into consideration when evaluating the results
of the DV review.

1.5. Organization of the Procedure Manual
Exhibit 1 below illustrates how the Manual is organized. The document’s content is structured according to the
four phases that comprise the DV process. The graphic presents the phases in the order in which the annual
DV cycle is conducted.
EXHIBIT 1. DATA VALIDATION PROGRAM PHASES

Planning for DV Activities => Performing DV Activities=> Analyzing Results and Submission of
Findings => Completing Post-DV Activities
Each phase of the DV review process contains several activities. Exhibit 2 displays the activities in the order in
which they are found in the document and the order in which they are conducted, beginning with the selection
of an appropriate DVC and ending with the appeal of DV determinations. The DV review process largely entails
a collaborative effort between the SO and its independent, external DVC in terms of information sharing up to
the point of the DVC’s final submission of DV review findings to CMS. Each of these steps is described in more
detail throughout the Manual.

5

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual
EXHIBIT 2. DATA VALIDATION PROGRAM ACTIVITIES

DV Phase

Step

Planning for DV 1
Activities
Planning for DV 2
Activities
Planning for DV 3
Activities
Planning for DV 4
Activities
Performing DV 5
Activities

Responsible
DV Activities
Party
SO
Select appropriate DVC based on Standards for Selecting
a Data Validation Contractor
DVC, SO
Notify CMS of DVC Selection / Request Access to Health
Plan Management System (HPMS) Plan Reporting Data
Validation Module (PRDVM)
DVC, SO
Complete the web-based Data Validation Training

Timeline
December*March
January-April
February-March

DVC, SO

Review all DV documents

January-March

SO

Complete Organizational Assessment Instrument (OAI),
also known as Appendix E, and provide appropriate
documentation to selected DVC per the OAI’s
documentation request
Analyze OAI (Appendix E) Responses

March 2 - April 1

Performing DV 6
Activities
Performing DV 7
Activities
Performing DV 8
Activities

DVC, SO

Performing DV
Activities
Analyzing
Results and
Submission of
Findings
Analyzing
Results and
Submission of
Findings
Analyzing
Results and
Submission of
Findings
Analyzing
Results and
Submission of
Findings
Completing
Post- Activities
Completing
Post- Activities

9

DVC

10

DVC

11

DVC

Provide draft findings to SO

12

DVC, SO

Review draft findings and obtain additional documentation June
necessary to resolve issues

13

DVC

Submit findings to CMS via HPMS PRDVM and receive
DV scores

No Later than
June 30

14

DVC, SO

Compile archive of DV work papers

July 31

15

SO

Receive Pass or Not Pass threshold level and assess
Summer/Fall
Pass or Not Pass determination based on final DV scores

Completing
16
Post- Activities

SO

Appeal DV determination(s) (if applicable)

DVC, SO
DVC, SO

March 2 or later

Prepare for on-site or virtual visit (site visit agenda,
Early April
resource needs, and logistics)
Conduct on-site or virtual review (convene entrance
Early April (allow
conference, conduct interviews with SO staff, observe
for up to 1 week)
SO’s reporting processes, and obtain census and/or
sample files)
Request additional documents following on-site or virtual Mid/Late April
visit (if applicable)
Determine compliance with Data Validation Standards and June
record findings in Excel-version of the Findings Data
Collection Form (FDCF), also known as Appendix J
June

By June 30th
deadline.

* References to December refer to the calendar year before the DV review; all other references to months refer to the
same calendar year as the DV review.

6

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

2. PART C AND PART D REPORTING SECTIONS
REQUIRING DATA VALIDATION IN 2023

This section provides an overview of the Part C and Part D reporting sections that will undergo validation. The DV
reporting section criteria that are included in the DV Standards are mapped specifically to these reporting sections.

2.1. Part C and Part D Reporting Sections Requiring Data Validation in 2023
Seven Part C and Part D reporting sections are included in Exhibit 3.
EXHIBIT 3. PART C AND PART D REPORTING SECTIONS REQUIRING DATA VALIDATION

2022 Reporting Section

Reporting Period(s)

Data Submission Due
Date(s) to CMS

DV Findings
Due to CMS

Part C

No data

No data

No data

Grievances

1/1 -3/31
4/1 -6/30
7/1 -9/30
10/1 -12/31

First Monday of February in
2023

6/30

Organization Determinations/
Reconsiderations

1/1 -3/31
4/1 -6/30
7/1 -9/30
10/1 -12/31

Last Monday of February in
2023

6/30

Special Needs Plans (SNPs) Care
Management

1/1 -12/31

Last Monday of February in
2023

6/30

Part D

No data

No data

No data

Medication Therapy Management
Programs

1/1 -12/31

Last Monday of February in
2023

6/30

Grievances

1/1 -3/31
4/1 -6/30
7/1 -9/30
10/1 -12/31

First Monday of February in
2023

6/30

Coverage Determinations and
Redeterminations

1/1 –3/31
4/1 -6/30
7/1 -9/30
10/1 -12/31

Last Monday of February in
2023

6/30

Last Monday of February in
2023

6/30

1/1 -3/31
1/1 -6/30
Improving Drug Utilization Controls
1/1 -9/30
1/1 -12/31

7

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

2.2. Reporting Requirements Excluded from the Validation Requirement
EXHIBIT 5. PART C AND PART D REPORTING SECTIONS EXCLUDED FROM DATA VALIDATION

•
•
•
•

Part C Reporting Sections

Part D Reporting Sections

Enrollment/ Disenrollment
Employer Group Plan Sponsors
Rewards and Incentives Programs
Payments to Providers

• Enrollment/ Disenrollment
• Employer/ Union- Sponsored Group
Health Plan Sponsors

8

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

3. PLANNING FOR DATA VALIDATION ACTIVITIES
3.1 Select appropriate contractor based on Standards for Selecting a Data
Validation Contractor
The DV must be conducted by an independent, external entity to ensure that the data used to develop plan
performance measures are credible to other stakeholders and that information used to respond to Congressional
and public inquiries are reliable for monitoring plans. The SO is responsible for acquiring the independent Data
Validation Contractor (DVC) and for all other costs associated with completing the independent DV and reporting
the results to CMS.

3.1.1 Standards for Selecting a Data Validation Contractor
CMS has provided a set of standards for an SO selecting a DVC (Appendix A) These standards describe the
minimum qualifications, credentials, and resources that the selected DVC must possess, as well as the conduct
that the DVC must exhibit. SOs must acquire one DVC to conduct the validation on reported data, and if
necessary, the DVC may subcontract to ensure that it has the expertise required for each DV area and to meet
the minimum standards. SOs may use their own staff only to assist the DVC in obtaining the information, data,
and documents needed to complete the DV review.
SOs may also permit a different DVC to perform mock audits, pre-assessments, and any other types of review
throughout the year. However, in order to meet CMS’ standards for organizational independence, an SO may
not use the same DVC who conducted these activities to conduct the subsequent DV review of those reported
data. More detailed information pertaining to organizational independence is included in Appendix A, Standards
for Selecting a Data Validation Contractor. While the DVC conducting the formal DV review may not participate
in mock audits, pre-assessments, or other types of reviews, the DVC can begin preparing for the DV review prior
to April 1. These types of preparation activities may include:

•
•

Meeting with the SO to discuss the validation process, resource needs, timeline, etc.
Providing the SO with a list of documents, data, and materials that are needed to complete the review.

Any specific questions about what types of activities are permitted prior to April 1 or regarding whether or not a
particular entity meets the organizational independence standard should be directed to
[email protected].
The Standards for Selecting a Data Validation Contractor also contains best practices that DVCs are expected
to adhere to throughout the review. The DVC should remain an objective, independent third party and avoid
acting in a consulting capacity. The DVC should remain impartial in all of its activities and focused on
determining if SOs’ systems, programs, data, etc., are accurate, reliable, valid, and complete based on
instructions and standards outlined in Appendix A and CMS’ policies. The DVC should provide general feedback
and specific information on deficiencies to help SOs improve and should maintain the confidentiality of SOs’
privileged information.

3.1.2 Timing of Data Validation Contractor Selection
An SO may select a DVC at any time, up to and during the April through June DV review period. SOs should
implement the contract to allow sufficient time for the DVC to perform all of the requirements of the review during
the required timeframe and submit findings to CMS via the PRDVM in HPMS by June 30.

3.1.3 Requesting a Contractor Change Mid-Review
An SO may not change its DVC during the formal review period (April-June) unless there are conditions that are
unrelated to DV findings, such as negligence or malfeasance on the part of the DVC. If a change in a contractor
is required, the new DVC is required to complete the DV review in its entirety [starting with the OAI (Appendix E)
analysis through the submission of findings to CMS] within the required April - June DV review timeline.
CMS will consider DVC change requests submitted mid-review on a case-by-case basis only. Requests must be
in writing and be submitted to CMS via the [email protected] resource mailbox.

9

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

3.2

Notify CMS of DVC Selection / Request Access to Health Plan
Management System (HPMS) Plan Reporting Data Validation Module
(PRDVM)

3.2.1 Documentation of Data Validation Contractor Selection Process
SOs must document their DVC selection process and be able to show, upon request by CMS, how their chosen
DVC meets the minimum qualifications, credentials, and resources set forth in the Standards for Selecting a Data
Validation Contractor. This includes maintaining a copy of the documentation that all DVC staff assigned to the
applicable DV review team completed the CMS Data Validation Training program (see Section 3.3). CMS
requires that the SO retain this documentation for the 10-year retention period per federal regulations. 2
If an SO selects the same DVC used for a previous year’s DV review, it must still document the selection
process as described above.
*Submission of this information to CMS is not required but should be available upon request.

3.2.2 Request Access to HPMS Plan Reporting Data Validation Module
Once the SO has selected a DVC, the next step is for the DVC to request staff access to the PRDVM in HPMS.
This module allows users to upload and review DV findings and submit them to CMS. The credentials assigned
to a user will allow that individual to access only the PRDVM and those SO(s)/contract(s) with which they are
associated. The DVC staff will use these credentials to access the appropriate screen(s) to upload DV findings
within the PRDVM starting no earlier than April 1 of the calendar year.

3.2.2.1 Access to HPMS requires a CMS user ID. Questions regarding the user ID process should be directed
to [email protected]. Additional information on obtaining HPMS access can be found at
the following link: https://www.cms.gov/Research-Statistics-Data-and-Systems/Computer-Dataand-Systems/HPMS/UserIDProcess.
3.2.2.2 Process for Sponsoring Organization
For access to the PRDVM the SO provides their DVC with an official letter in either hardcopy or an
emailed pdf format attachment containing the following information:
•
•
•
•
•
•

Acknowledgment that the SO has contracted with the selected DVC to complete the review,
Name of each individual who requires access (up to 5 individuals),
Type of functionality that each individual user requires,
Acknowledgment that the individuals have completed the web-based DV Training,
Contract number(s) the DVC will need access to, and
SO’s Chief Executive Officer’s (CEO) signature.

Model language for this letter can be found in the Model Language for Letter to Confirm Selection of Data
Validation Contractor (Appendix C).
If an SO uses the same DVC used for a previous year’s DV review, it must still provide the DVC with
this signed letter for the current year’s DV activities.

2

See 42 CFR §422.504(d) and § 423.505(d)
10

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

3.2.2.3 Process for Data Validation Contractor
DVC staff must obtain individual access to the HPMS PRDVM. If the designated user(s) from the DVC
does not have active access to HPMS download the Application for Access to CMS Computer Systems
at: Access to CMS Data & Application and follow the instructions provided in Example Application for
Access to CMS Computer Systems (Appendix D) for requesting reviewer access to the HPMS PRDVM.
CMS allows up to 5 individuals from each DVC to have access to this Module on behalf of each SO.
The DVC must create their user ID using EFI, then email the letter from each SO for which they are
under contract to complete the DV review. Since we will process the request for a new user ID first, the
user will only get the HPMS main home page until a DV letter is received. The letters may be sent as
email attachments to [email protected].
Users may follow the instructions for getting their user ID by clicking the Instructions for Requesting
Plan Access via EFI link under the Download section on the CMS HPMS website:
https://www.cms.gov/Research-Statistics-Data-and-Systems/Computer-Data-andSystems/HPMS/UserIDProcess.
The user must select Data Validation Consultant from the “I am a—” drop down. The system will
automatically fill out the company name and Plan number. The HPMS team has also created a video to
walk a user through getting a new CMS user ID: https://youtu.be/KAXwdnq1hKs
For individuals with an active CMS Enterprise User Administration (EUA) User ID and HPMS access, a
new Application for Access to CMS Computer Systems is unnecessary. But, their current credentials
must be modified to allow access to the PRDVM. For this access, individuals need to ensure that the
letter from each SO linking the DVC to the SO includes their current User ID and an explanation that the
user already has HPMS Access. This letter must be sent to CMS via email or traceable carrier to the
address indicated above. If a user had PRDVM access as a DVC previously, they do not need letters
re-sent on their behalf. Letters are only required for changes to a DVC user’s account.

3.3

CMS strongly recommends requests for HPMS PRDVM access be submitted by early April. Requests
received after this period will be processed on a rolling basis. It will take approximately four weeks for
the designated individuals to obtain the credentials (CMS EUA User IDs and passwords) to access the
PRDVM.

Complete the Web-based Data Validation Training

CMS developed a web-based DV Training for SOs and DVCs to learn more about the DV program and its
specific requirements. The training is on cms.gov on the MLN page and found at: https://www.cms.gov/Outreachand-Education/Medicare-Learning-Network-MLN/MLNProducts/WebBasedTraining
All SO staff involved in the DV preparation should complete the CMS web-based DV Training to familiarize
themselves with the DV process and requirements.
All DV staff are required to take the CMS web-based DV Training prior to working on the DV project. Once the
training is completed successfully a certificate of completion is generated. DVC staff should provide training
certificates to the SO before commencing work on the DV.
DVC staff that participated in a previous year’s DV must still take the current year’s CMS web-based DV
Training.

3.4

Review all Data Validation Documents
As noted in Section 1.4, there are 11 documents (including this Manual) that should be reviewed well in advance
of the DV period. This Manual describes these materials. This section will focus specifically on the DV standards,
which are further described in Data Validation Standards (Appendix B).

3.4.1 Introduction to the Data Validation Standards
The DV Standards include general standards and reporting section criteria that DV staff must use to determine
whether the data each SO reported are accurate, valid, timely, and reliable.
The standards assess an SO’s information systems capabilities and its processes for collecting, storing, compiling,
and reporting Part C and/or Part D data; whether SOs follow the applicable Reporting Requirements and Technical
Specifications to compile data, make appropriate data exclusions, verify calculations, computer code, and
algorithms.
11

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

In preparation for the DV process, both the SO and the DVC must review and learn the standards. Refer to
Appendix B for the complete set of Part C and Part D Reporting Section Data Validation Standards along with
guidance related to interpreting the standards.

3.4.2 Data Validation Standards and Reporting Section Criteria
3.4.2.1 Data Validation Standards Instructions
The DV Standards include identical instructions relating to the types of information that must be
reviewed for each reporting section, a set of validation standards, and reporting section criteria that are
based on the applicable Reporting Requirements and Technical Specifications.
The DVC uses these standards in conjunction with the Data Extraction and Sampling Instructions
(Appendix H) and the Excel version FDCF (Appendix J) to evaluate the SO’s processes for producing
and reporting the reporting sections. CMS strongly recommends that the DVC and the SO’s leadership
team and reporting section report owners/ data providers review the DV Standards documentation
before and during the review of each reporting section to ensure that they thoroughly understand the
standards and reporting section criteria. This will also help to ensure all applicable data fields are
extracted for each reporting section.
The top portion of each set of standards details the documents and reports that the DVC uses to
determine compliance with the standards for each specific reporting section. The documents and
reports are listed within the gray box underneath the name of the applicable reporting section and are
displayed in Exhibit 6.
EXHIBIT 6. GENERAL INSTRUCTIONS FOR DATA VALIDATION STANDARDS

[NAME OF REPORTING SECTION]

To determine compliance with the standards for Grievances (Part C), the data validation contractor will assess the
following information:
• Written response to OAI Sections 3 and 4, and
• Data file created for submission to CMS and copy of HPMS
documentation requested per OAI Sections 5 and 6
screen shots of data entered
• Outlier/data integrity notification(s)- See OAI 4.3.3 for
• Other relevant information provided by organization
instructions on how to retrieve notices
• Results of interviews with organization staff
• Census and/or sample data
Also contained within this section, if applicable, are notes to the DVC regarding a specific reporting
section and any nuances or differences that may be encountered during the review of that reporting
section. The second section of each set of standards is identical for all Part C and Part D reporting
sections.

3.4.2.2 Data Validation Standards 1 - 7
3.4.2.2.1 Standard 1
Standard 1 (see Exhibit 7) contains the general and specific criteria for validating source documentation
that the SO provides to the DVC.

12

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual
EXHIBIT 7. STANDARD 1: REQUIRED DATA FIELDS ARE ACCURATELY CAPTURED AND PROPERLY
DOCUMENTED

DATA VALIDATION STANDARD 1
1.

A review of source documents (e.g., programming code, spreadsheet formulas, analysis plans,
saved data queries, file layouts, process flows) indicates that all source documents accurately
capture required data fields and are properly documented.

Criteria for Validating Source Documents:
a. Source documents are properly secured so that source documents can be retrieved at any time to
validate the information submitted to CMS via HPMS.
b. Source documents create all required data fields for reporting requirements.
c. Source documents are error-free (e.g., programming code and spreadsheet formulas have no messages
or warnings indicating errors, use correct fields, have appropriate data selection, etc.).
d. All data fields have meaningful, consistent labels (e.g., label field for patient ID as Patient ID, rather than
Field1 and maintain the same field name across data sets).
e. Data file locations are referenced correctly.
f. If used, macros are properly documented.
g. Source documents are clearly and adequately documented.
h. Titles and footnotes on reports and tables are accurate.
i. Version control of source documents is appropriately applied.
3.4.2.2.2 Standard 2
Standard 2 (see Exhibit 8) instructs the DVC to validate the completeness of the underlying data and the
accuracy of each reported reporting section. Standard 2 provides an overview of reporting section criteria,
which must be met for each of the Part C and Part D reporting sections being reported and is further
detailed in section 4.2.3. For example, the reporting section criteria assess whether the appropriate date
ranges for the reporting period are captured by the data system, and whether the expected counts and
calculations are accurate and match the corresponding source code and analysis plan. The criteria are
also used to verify that the SO has properly interpreted and defined key terms used to determine which
data are applicable. For example, the SO must properly define the terms “Coverage Determinations and
Redeterminations” in accordance with CMS regulations, guidance, and the Reporting Requirements and
Technical Specifications in order to ensure the quality of the reported data for that reporting section.
Standard 2e is further broken down into additional criteria that map to the relevant technical specification
data elements.
EXHIBIT 8. STANDARD 2: DATA ELEMENTS ARE ACCURATELY IDENTIFIED, PROCESSED, AND CALCULATED

DATA VALIDATION STANDARD 2
2.

A review of source documents (e.g., programming code, spreadsheet formulas, analysis plans, saved data queries, file
layouts, process flows) and census or sample data, whichever is applicable, indicates that data elements for each
reporting section are accurately identified, processed, and calculated.

Criteria for Validating Reporting Section Criteria (Refer to reporting section criteria section below):
a. The appropriate date range(s) for the reporting period(s) is captured.
b. Data are assigned at the applicable level (e.g., plan benefit package or contract level).
c. Appropriate deadlines are met for reporting data.
d. Terms used are properly defined per CMS regulations, guidance, Reporting Requirements, and Technical
Specifications.
e. The number of expected counts (e.g., number of members, claims, grievances, procedures) are verified; ranges of
data fields are verified; all calculations (e.g., derived data fields) are verified; missing data has been properly
addressed; reporting output matches corresponding source documents (e.g., programming code, saved queries,
analysis plans); version control of reported data elements is appropriately applied; QA checks/thresholds are
applied to detect outlier or erroneous data prior to data submission.
3.4.2.2.3 Standard 3
Standard 3 (see Exhibit 9) is used to determine whether an SO implements policies and procedures for
each reporting section’s data submission. Not only should the DVC validate that the reported data were
correctly derived from the underlying database, but they should also verify that the data are accurately

13

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

uploaded and/or entered into HPMS. If a reporting section requires both a file upload and data entry, both
have to occur in order for an SO to meet Sub-Standard 3a.
EXHIBIT 9. STANDARD 3: DATA SUBMISSION

DATA VALIDATION STANDARD 3
3.

Organization implements policies and procedures for data submission, including the following:
a. Data elements are accurately entered/uploaded into HPMS and entries match corresponding source
documents.
b. All source, intermediate, and final stage data sets and other outputs relied upon to enter data into HPMS are
archived.
3.4.2.2.4 Standards 4 and 5
For Standards 4 and 5 (see Exhibit 10), the DVC must verify that the SO has and implements policies and
procedures for regular database updates, and for data archiving and restoration. This ensures that data
are kept up to date and that systems are in place for timely data submission or re-submission in the event
of data loss.
EXHIBIT 10. STANDARDS 4 AND 5: DATA SYSTEM UPDATES AND ARCHIVE/RESTORATION

DATA VALIDATION STANDARDS 4 & 5
4.

Organization implements policies and procedures for periodic data system updates (e.g., changes in enrollment,
provider/pharmacy status, and claims adjustments).

5.

Organization implements policies and procedures for archiving and restoring data in each data system (e.g., disaster
recovery plan).
3.4.2.2.5 Standards 6 and 7
Standards 6 and 7 (see Exhibit 11) are applicable only in certain situations. Standard 6 is applicable if an
SO’s data systems underwent any changes during the reporting period. If this occurred, the DVC must
examine the documentation of the changes to ensure there were no issues that adversely impacted the
reported data.
Standard 7 applies if any of the data collection or validation processes are outsourced to another entity.
This standard assesses whether the SO has policies and procedures in place that address routine
monitoring of the delegated entities work and whether those policies and procedures are implemented.
The DVC should mark “Not Applicable” in of FDCF (Appendix J) if Standard 6 or 7 is not applicable to the
reporting section or contract under review.
EXHIBIT 11. STANDARDS 6 AND 7: DATA SYSTEM CHANGES AND OVERSIGHT OF DELEGATED ENTITY REPORTING

DATA VALIDATION STANDARDS 6 AND 7
6.

If organization’s data systems underwent any changes during the reporting period (e.g., because of a merger,
acquisition, or upgrade): Organization provided documentation on the data system changes and, upon
review, there were no issues that adversely impacted data reported.

7.

If data collection and/or reporting for this reporting section is delegated to another entity: Organization regularly monitors
the quality and timeliness of the data collected and/or reported by the delegated entity or first tier/downstream
contractor.

3.4.3 Reporting Section Criteria
The reporting section criteria vary for each Part C and Part D reporting section. Reporting section criteria are
used in conjunction with Standard 2 to determine if data elements are accurately identified, processed, and
calculated. The first three reporting section criteria for each reporting section (see Exhibit 12 for a sample
from Part C Special Needs Plans Care Management) are used to validate whether the SO is utilizing the
appropriate reporting period, reporting level, and reporting deadline(s) per CMS requirements.

14

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual
EXHIBIT 12. REPORTING SECTION CRITERIA FOR APPROPRIATE REPORTING PERIOD, REPORTING LEVEL, AND
REPORTING DEADLINE IN PART C SPECIAL NEEDS PLANS (SNPs) CARE MANAGEMENT

REPORTING SECTION CRITERIA
1.

Organization reports data based on the required reporting period of 1/1 through 12/31.

2.

Organization properly assigns data to the applicable CMS plan benefit package.

3.

Organization meets deadline for reporting annual data to CMS by 2/27/2023.
Note to reviewer: If the organization has, for any reason, re-submitted its data to CMS for this reporting section, the reviewer
should verify that the organization’s original data submission met the CMS deadline in order to have a finding of “yes” for
this reporting section criterion. However, if the organization re-submits data for any reason and if the re-submission was
completed by 3/31 of the data validation year, the reviewer should use the organization’s corrected data submission for the
review of this reporting section.
Several of the reporting section standards contain a reporting section criterion to validate whether the SO
properly defined key terms that it used to compile reported data per CMS regulations, guidance, Reporting
Requirements and Technical Specifications. Exhibit 13 shows an example of this criterion for the Part D Coverage
Determinations and Redeterminations reporting section.
EXHIBIT 13. REPORTING SECTION CRITERION FOR DEFINING KEY TERMS IN PART D COVERRAGE DETERMINATIONS AND
REDETERMINATIONS SECTION

REPORTING SECTION CRITERIA
4.

Organization properly defines the term “Coverage Determinations” in accordance with 42 C.F.R. Part 423, Subpart M, and
the Parts C & D Enrollee Grievances, Organization/Coverage Determinations and Appeals Guidance. This includes
applying all relevant guidance properly when performing its calculations and categorizations.
Organization properly defines the term “Redetermination” in accordance with 42 C.F.R. Part 423, Subpart M, and the Parts
C & D Enrollee Grievances, Organization/Coverage Determinations and Appeals Guidance. This includes applying all
relevant guidance properly when performing its calculations and categorizations.
The other reporting section criteria reference the applicable data element from the Reporting Requirements
when possible and differ considerably depending on the reporting section and data element. Exhibit 14 shows
an example of selected reporting section criteria applicable to the Part C Grievances reporting section. The
exact criteria for each Part C and D reporting section are based on the Reporting Requirements and
Technical Specifications.

15

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual
EXHIBIT 14. REPORTING SECTION CRITERIA FOR SELECTED PART C GRIEVANCES DATA ELEMENTS

6.

REPORTING SECTION CRITERIA
Organization accurately calculates the total number of grievances, including the following criteria:
a. Includes all grievances that were completed (i.e., organization has notified member of its decision) during
the reporting period, regardless of when the grievance was received.
b. Includes all grievances reported by or on behalf of members who were previously eligible, regardless of
whether the member was eligible on the date that the grievance was reported to the organization.
c. If a grievance contains multiple issues filed under a single complainant, each issue is calculated as a
separate grievance.
d. If a member files a grievance and then files a subsequent grievance on the same issue prior to the
organization’s decision or the deadline for decision notification (whichever is earlier), then the issue is counted
as one grievance.
e. If a member files a grievance and then files a subsequent grievance on the same issue after the
organization’s decision or deadline for decision notification (whichever is earlier), then the issue is counted
as a separate grievance.
f. Includes all methods of grievance receipt (e.g., telephone, letter, fax, and in-person).
g. Includes all grievances regardless of who filed the grievance (e.g., member or appointed representative)
h. Includes only grievances that are filed directly with the organization (e.g., excludes all complaints that are
only forwarded to the organization from the CMS Complaint Tracking Module (CTM) and not filed directly
with the organization). If a member files the same complaint both directly with the organization and via the
CTM, the organization includes only the grievance that was filed directly with the organization and
excludes the identical CTM complaint.
i. For MA-PD contracts: Includes only grievances that apply to the Part C benefit (If a clear distinction
cannot be made for an MA-PD, cases are reported as Part C grievances).
j. Excludes withdrawn grievances.

[Data Elements A-E]
7. Organization accurately calculates the number of grievances for which it provided timely notification of the decision,
including the following criteria:
a. Includes only grievances for which the member is notified of decision according to the following timelines:
i. For standard grievances: no later than 30 days after receipt of grievance.
ii. For standard grievances with an extension taken: no later than 44 days after receipt of grievance.
iii. For expedited grievances: no later than 24 hours after receipt of grievance.
[Data Elements B]

16

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

4. PERFORMING DATA VALIDATION ACTIVITIES
4.1

Complete Organizational Assessment Instrument (OAI) and Provide
Appropriate Documentation to Selected DVC per the OAI’s (Appendix E’s)
Documentation Request
The Organizational Assessment Instrument (OAI) (Appendix E) focuses on how the SO collects, stores, and
reports data. Completing the OAI (Appendix E) is mandatory, and CMS highly recommends that SOs complete
this document in advance of the DV, as the DV review relies significantly on the information captured in this tool.
SOs should provide the completed OAI (Appendix E) to their selected DVC electronically. CMS estimates that the
OAI (Appendix E) should take a minimum of two weeks to complete and should be submitted to the DVC no later
than early April. SOs may not send their completed OAI (Appendix E) or source code, SOPs, etc. to their DVCs
prior to the start of the DV cycle on April 1.
Each SO must provide to its DVC the basic information regarding its Medicare contracts and which Part C and/or
Part D reporting sections each contract submits to CMS. SOs that hold more than one contract with CMS need
only complete one version of the OAI (Appendix E) that covers all of its contracts. If the information provided in
the OAI (Appendix E) varies by contract, the document allows for the flexibility to identify the differences for the
DVC in applicable sections.
All documentation and responses to questions in the OAI (Appendix E) should reflect the SO’s systems and
processes that were in place during the reporting period(s) undergoing the DV review. For example, if the data
being reviewed are for the 2022 reporting period, the SO should include only diagrams of the information systems
in place in 2022 or the programming code used in 2022 to calculate the reporting sections.
The SO and its DVC should work out mutually agreeable methods for sharing and protecting proprietary data,
such as that requested in the OAI (Appendix E), and protected health information. The Standards for Selecting a
Data Validation Contractor (Appendix A) includes minimum security requirements with which the DVC’s facility,
equipment, and processes must comply. The SO is responsible for ensuring that the DVC complies with all
Health Insurance Portability and Accountability Act (HIPAA) privacy and security requirements.
The SO must supply all the information required for the DV review; otherwise, it will be out of compliance with
CMS requirements and will be subject to compliance actions from CMS. If an SO contracts with delegated entities
(e.g., PBMs) that are not cooperative in supplying required information, the SO remains responsible for the
required information, and how to proceed. Additionally, if an SO or its delegated entity does not provide the
information required to determine if a standard or sub-standard has been met, the DVC is required to select “No”
or 0 in the FDCF (Appendix J) for that standard or sub-standard.

4.2

Analyze OAI (Appendix E) Responses
CMS recommends DVCs perform a preliminary review of the documentation submitted in the OAI (Appendix E) in
advance of each site visit (on-site or virtual) so that any follow-up regarding the documentation can be done
during the site visit (on-site or virtual). The documentation submitted by the SO when completing the OAI
(Appendix E) should be adequate and enabling of an effective review. The amount of detail provided in the
documentation will determine the ease of the review process, especially for the review of programming
code/source code.
Additionally, the OAI (Appendix E) provides supplemental questions to help the DVC better understand the
processes used by the SO to compile and submit its reporting sections. The SO’s responses to these questions will
provide insight as to who is responsible for the quality control and submission of the data, the processes for
incorporating CMS updates to the Reporting Requirements and/or Technical Specifications into the SO’s systems,
and descriptions of any past issues that may have occurred during the reporting process.

4.2.1. Perform OAI (Appendix E) Gap Analysis
Upon receiving the completed OAI (Appendix E), the DVC should review the document for completeness and
accuracy. Sections of the OAI (Appendix E) that are missing or incomplete should be noted, and the DVC should
follow-up with the SO to complete. It is up to the DVC to determine whether any identified gaps in the OAI
(Appendix E) responses require addressing prior to the site visit (on-site or virtual) or can be addressed during
the site visit (on-site or virtual) portion of the review.
17

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

4.2.2. Review Source Code and Other Documentation
Data dictionaries and source code are critical for allowing the DVCs to map ambiguous field names and internal
status codes to meaningful descriptions. Well organized and structured documentation of the reporting and data
extraction processes for the various reporting sections will assist the DVC in gaining a more thorough
understanding of the SO. The DVC should be familiar with data systems and processes detailed by the SO in the
OAI (Appendix E) to ensure thorough preparation for the site visit (on-site or virtual).

4.2.3. Prepare Interview Discussion Guide
The Interview Discussion Guide (IDG) (Appendix F) is intended to facilitate the discussion between the DVC and
the SO’s report owners and subject matter experts. The IDG is a dynamic tool containing both general and
reporting section questions that can guide an effective discussion regarding an SO’s underlying data systems
and reporting processes. If, during review of the documentation provided in response to the OAI (Appendix E),
the DVC discovers evidence that may indicate errors in the SO’s data or reporting processes, the DVC should
modify the IDG used for that SO with new questions that may identify any vulnerabilities or opportunities for
repeated errors with data collection or reporting. Additionally, the IDG should serve as a “guide” for the DVC; it is
up to the DVC’s discretion to include additional questions and/or detail to the document to discuss during on-site
or virtual interviews and to ensure the additional detail is documented accordingly.

4.3

Prepare for On-Site or Virtual Visit

4.3.1 Select Dates and Appropriate Location(s) of On-Site or Virtual Visit
During the site visit (on-site or virtual), the following DV review activities occur: (1) conduct interviews with SO
staff, (2) observe the SO’s reporting processes, and (3) obtain census and/or sample files to support the
validation of Part C and Part D reporting sections. SOs and DVCs are responsible for determining mutually
agreeable dates for performing the site visit or holding a virtual site meeting. It is estimated that the site visit (onsite or virtual) for a full Part C and Part D data validation review should take up to one week to complete.
It is up to the discretion of the DVC to determine the most appropriate location(s) of the site visit (e.g., virtual,
SO’s facility, PBM’s facility, other delegated entity’s facility). SOs and DVCs may elect to conduct a physical site
visit or use virtual meeting tool(s) or teleconference(s), if appropriate.

4.3.2 Develop Agenda for On-Site or Virtual Visit
The DVC and SO should work together to prepare a site visit (on-site or virtual) agenda. A sample agenda in
Appendix G can be used either for a virtual or on-site visit. The sample agenda is structured to include an
entrance and exit conference and interviews and demonstrations of data systems for each reporting section
included in the DV. It is also recommended that the DVC have a method to collect contact information for each
SO’s report owners and subject matter experts in case any follow-up is required.
The time required to complete an on-site or virtual visit may be contingent upon the size of the SO, the efficiency
of the SO’s operations, the level of reporting automation, and the scope of the DV review. A DVC session with
the SO’s report owner(s) for each reporting section must be scheduled to allow sufficient time for the SO to
provide an overview of each of the relevant data systems used in gathering data and producing reports, as well
as to complete the data extraction/sampling process (see Section 4.4.4 for more information). To ensure optimal
time and resource management during the on-site or virtual visit, a DVC may either conduct multiple sessions
concurrently, or structure the agenda to allow scheduling of interviews and demonstrations of reporting
processes by each report owner to reduce repetitive discussions and demonstrations, especially in cases where
one report owner oversees the processes for multiple reporting sections that use the same data system(s).

4.3.3 Prepare for Data Extraction and Sampling
In preparation for the data extraction and sampling during the site visit (on-site or virtual), the DVC should review
information provided in the completed OAI (Appendix E) and, if necessary, hold conference calls with the SO to
discuss the SO’s processes. Calls held specifically with each reporting section’s report owner can also provide
an opportunity for the DVC to review the Data Extraction and Sampling Instructions (Appendix H) in more detail
and for the report owners to seek clarification as needed. These discussions can also inform the DVC about the
SO’s data systems and sources from which the sample data would be pulled.
There are two methodologies that can be used to extract data for each reporting section. The first is to extract
the full census of data for a reporting section, meaning that every data record that is relevant to a reporting
section is extracted. When possible, DVCs should attempt to extract the full census. Extracting the census will
enable the DVC to determine with the greatest precision whether reporting sections were submitted accurately.
18

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

If the size or complexity of a database presents an unusual time burden on the DVC and/or SO, then the second
method, extraction of a random sample, which is a subset of the full census, can be used. DVCs should use
their best judgment to decide if extracting a full census is feasible, or if selecting a random sample will provide
the data necessary for the DV review. Refer to Appendix H for further details regarding these two
methodologies. In addition, DVCs should determine if the SO’s staff requires supervision during the actual data
extraction process, or if the SO’s staff are able to extract data without supervision. See Section 4.4.4 for
additional requirements if the DVC is unable to supervise the data extraction process.

4.4 Conduct On-Site or Virtual Visit
4.4.1 Conduct Entrance Conference
The entrance conference provides an opportunity for the DVC and the SO’s management and individual report
owners to introduce themselves and discuss expectations for the on-site or virtual visit. At the entrance conference,
the DVC should describe the objectives for the review and discuss any administrative needs of the team.
Optionally, the SO may provide a high-level overview of its organization, focusing on its operations with respect
to meeting the CMS reporting requirements. CMS recommends that the entire review team also meet briefly with
the SO’s management and individual report owners at the beginning of each day of the visit to go over
administrative needs and review the day’s agenda.

4.4.2 Conduct Interviews with Organization Staff
During the on-site or virtual visit, the DVC must conduct interviews with the subject matter experts and report
owners for each reporting section and reporting system. These interviews provide a first-hand opportunity for the
DVC to gain a thorough understanding of each SO’s data collection and reporting processes involved with meeting
CMS reporting requirements. The DVC should reference the IDG as needed to ensure that all key topics are
addressed during the interviews. In addition, any outstanding questions and follow-up items identified during the
analysis of OAI (Appendix E) responses should be addressed during the interviews.

4.4.3 Observe Reporting Processes
The on-site or virtual visit allows the opportunity for the SO to provide a significant amount of useful information to
the DVC. Designated SO staff (i.e., report owners) must provide visual demonstrations of the data systems and
reporting processes, including data extraction from originating data sources, data analysis, quality assurance
processes, and processes for entering or uploading final data into HPMS. If SOs and DVCs elect to conduct a
virtual visit, designated SO staff can use virtual meeting tool(s) or teleconference(s) to provide visual
demonstrations. The following is a sample list of the parts of the process that should be demonstrated:
•
•
•
•
•
•
•
•
•
•

Contact information and location of report owner and data providers
Function and location of all data warehouses
Types of data used (format, amount of tables)
Links and joins to other areas/ departments/data
Types of programming used to create the reports
Review processes and oversight
Timeframes for the process (amount of time it takes to run specific parts of the report)
Approximations of report volume
Updates to the process and system changes
Storage locations (e.g., building or server), security, and access constraints

The visual demonstrations provide a clear illustration of the reporting processes, provide the DVC with insight into
the SO’s ability to ensure accurate, valid, and timely data, and allow an opportunity to get immediate responses to
any questions or concerns about the reported data.

4.4.4 Extract Census or Sample Data
The next step is for the DVC to work with the report owners to draw a census or a random sample from each
reporting section’s final stage data set, following the Data Extraction and Sampling Instructions (Appendix H).
The document describes guidelines and methodologies for extracting SOs’ data. Two methodologies of
extraction are available to DVCs. The first method is referred to as the census. Extracting all records used in the
calculation of data elements for a specific reporting section would constitute extracting a census of data. When
possible, DVCs should attempt to extract the full census. Extracting the census enables the DVC to determine
with the greatest precision whether reporting sections were submitted accurately. The second method used for
data extraction is a random sample. The random sample is a subset of the census data. If extraction of the
19

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

census proves to be too burdensome due to the size or complexity of the data for a specific reporting section, a
sample of records should be extracted instead.
The random sampling process involves using built-in random number generators in the applications used to
display the data or perform the query (Microsoft Excel or SQL). Once a random number is assigned to each
unique record ID, the data owner can sort the data by the random number field and choose the statistically
appropriate number of records as the sample. A discussion of minimum sample sizes can be found in Data
Extraction and Sampling Instructions. The unique IDs from the random sample in the final stage data set are
then applied against the source data set to pull the corresponding source data records. The processes used to
draw the random data samples vary considerably, depending on the report owner and reporting section. For
example, some report owners may be able to easily draw the sample data for their reporting section without
having to manually clean or manipulate the data, while other report owners may have to perform more extensive
query programming and manual data cleaning in order to draw the sample data. During each of the sessions to
demonstrate reporting processes, the SO’s report owners should brief the review team on the processes used to
assemble the sample data files, including the source, intermediate, and final stage data sets. When uploading
the DV findings to CMS in the PRDVM, the DVC must report which data extraction method was used (full
census or random sample) to validate data for each standard. For both methods, DVCs must examine source
data as a means of verifying that the organization’s underlying data are correct: for example, reviewing customer
service call logs or member letters to verify that grievances were properly categorized as grievances. Source
data examples for each reporting section are provided in the Data Extraction and Sampling Instructions.
DVCs are expected to include the number and percentage of errors or variance from HPMS-filed data found
when examining the source data. For purposes of recording results in the FDCF (Appendix J), an error is any
discrepancy that either impacted the number of events reported or has the potential to impact the number of
events reported in future reporting periods. These errors must be reported in the “Review Results” area of the
FDCF (Appendix J) and include the sample size selected for the source data.
It is mandatory that DVCs follow the Data Extraction and Sampling Instructions. If the SO’s staff is extracting the
data, it is highly recommended that the DVC supervise the data extraction process to ensure these instructions
are followed correctly. If the DVC is unable to supervise the data extraction process, the DVC should obtain
documentation from the SO describing how the extraction process was performed. For example, if a random
sample is extracted, the DVC should request and validate the programming code used to extract the sample
data. If a full census is extracted, the DVC should validate that the record counts match between the census
extraction and the source and final stage data files.
CMS recommends that the DVC record details about each reporting section’s data set into a Data File Inventory
Log (Appendix I). Appendix I contains an example log that the DVC can use. It includes details such as the
reporting section name, report owner, data file name, type of data file (e.g., source, intermediate, or final stage
data file), number of rows or records, and a description of the file. By completing this log, the DVC will be able to
easily reference the data files during its post-site visit assessment of data.
The SO should write all data files to tab-delimited or comma-delimited text files with variable names in the first
row and transfer these files to the DVC’s secure storage device for each reporting section’s data. The SO must
also provide the DVC a file layout or data dictionary for the data files in either Word documents or Excel
spreadsheets on the same secure storage device. The SO and DVC must ensure that they have established
mutually agreeable methods for sharing protected health information and that the DVC complies with all HIPAA
privacy and security requirements.

4.4.5 Conduct Exit Conference
CMS recommends that the entire DV review team meet briefly with the SO’s management and individual report
owners at the end of each day of the on-site or virtual visit to go over any action items or outstanding
documentation needs. The on-site or virtual visit should conclude with an exit conference, where the DVC
should provide the SO with a summary of next steps and note any follow-up that may need to occur.

20

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

4.5 Request Additional Documents (If Required)
CMS recognizes that it may not be possible to obtain all of the required data and documentation
during the scheduled on-site or virtual visit, and follow-up conversations and requests may be
required. The DVC should make every attempt to gather all required data and documentation during
the on-site or virtual visit. In the event that not all information is available or follow-up is required after
the conclusion of the scheduled on-site or virtual visit, the DVC should have additional conversations
with the SO and/or make requests for documentation. DVCs and SOs should understand that the DV
is an iterative and collaborative effort, and SOs should be prepared to provide additional data and
documentation after the on-site or virtual visit has been held.

21

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

5. ANALYZING RESULTS AND SUBMISSION OF
FINDINGS
5.1 Determine Compliance with Data Validation Standards and Record
Findings in the Findings Data Collection Form (FDCF) (Appendix J)
to Upload into the HPMS PRDVM
Following the on-site or virtual visit, the DVC assesses the documentation and census/sample data
received from the SO, as well as the information gained during the interviews and demonstrations of
the SO’s reporting processes and information systems.
The DVC completes the FDCF (Appendix J) as it determines the findings for each contract included
in the scope of the review. DVCs use the FDCF (Appendix J) to record findings and then translate the
findings into the PRDVM in a data file upload. The FDCF (Appendix J) mirrors the content of the DV
Standards document but allows the DVC to record notes, data sources referenced, and findings for
the different standards and criteria specified for a given reporting section. The DVC will record
reporting section-level, and in some cases data element-level, findings for each reporting section.
Most DV standards (Standards 1, 4, 5, 6, and 7) are assessed at the reporting section-level, as they
assess SO processes that are not likely to vary at the data element-level. Once the DVC uploads the
FDCF (Appendix J) into PRDVM, the DVC may print the findings he/she uploaded into the PRDVM
and share them with the SO at any point during the review by accessing the HPMS report entitled
“Review Data Validation Findings Report.”
When using the FDCF (Appendix J), DVCs should only complete areas displayed in white for data
sources, review results, and findings. Areas displayed in gray are not applicable and should not be
completed. In the "Data Sources and Review Results:" column, the DVC should enter the data
sources used and review results for each standard or sub-standard. Next to this column, in the
"Findings" column, the DVC selects the appropriate choice based on whether or not the plan met the
requirement for the standard or sub-standard.

5.1.1. Reporting Findings for Standards Using Binary Scale
For all standards except 1.c, 1.d, 1.e, 1.g, 1.h, and 2.e, the findings are reported using binary scale.
The DVC selects "Y" if the requirements for the standard or sub-standard have been completely met. If
any requirement for the standard or sub-standard has not been met, the DVC must select "N." In
instances where a standard or sub-standard is not applicable, the DVC must select “N/A” and must
enter the reason for the “N/A” in the “Review Results” field.
CMS expects that there will be situations when the DVC finds that an SO is only in partial
compliance with specific DV standards. CMS has established a threshold whereby a minimum of
90% of records are accurate (e.g., sample or census records, source documents, policies and
procedures, data entry records) in order to record a “Yes” finding for any standard. Applying this
threshold to standards that require the review of policies and procedures should be done when it is
possible to readily quantify the adherence to or implementation of said policies and procedures (see
Exhibit 15). Exhibit 16 provides examples of how to calculate this minimum threshold specifically for
Standard 4, for which the DV involves samples or the complete census of records and/or data
values.
EXHIBIT 15. EXAMPLES OF CALCULATIONS TO DETERMINE MINIMUM THRESHOLD OF CORRECT SAMPLE/CENSUS
RECORDS FOR “YES” FINDING

Calculation for
Minimum Threshold

Sample/
Census
Size
150

0.90 x 150 = 135

205

0.90 x 205 = 184.5

Minimum Threshold of Correct Records for “Yes” Finding
At least 135 of the records are correct for the standard to be recorded as
“Yes”.
At least 185 of the records are correct for the standard to be recorded as
“Yes” (round 184.5 to 185).

22

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual
EXHIBIT 16. EXAMPLE OF HOW TO DETERMINE MINIMUM THRESHOLD OF IMPLEMENTED POLICIES OR
PROCEDURES FOR “YES” FINDING

Standard
4

Standard Description
Organization implements
policies and procedures for
periodic data system
updates (e.g., changes in
enrollment,
provider/pharmacy status,
and claims adjustments).

Minimum Threshold for a “Yes” Finding
SO has a policy in place for updating its enrollment system on a
monthly basis to ensure accurate information and protect the data
integrity.
Eleven out of the twelve months in the contract year, the SO
implemented the enrollment system update policy as it is written
(11/12 = 91.6%)

5.1.2. Reporting Findings for Standards Using Likert Scale
For standards 1.c, 1.d, 1.e, 1.g, 1.h, and 2.e, the scoring uses a five-point Likert-type scale. DVCs
review the percentage of records that meet the standards and enter a score based on the Likert- type
scale. The scale corresponds to the percentage of errors that are found in plan data as shown below:
1. Contract data has more than 20 percent error in records – Contract will receive a score of 1
for the given standard/ sub-standard.
2. Contract data has between 15.1 percent and 20 percent error in records – Contract will
receive a score of 2 for the given standard/ sub-standard.
3. Contract data has between 10.1 percent and 15 percent error in records – Contract will
receive a score of 3 for the given standard/ sub-standard.
4. Contract data has between 5.1 percent and 10 percent error in records – Contract will
receive a score of 4 for the given standard/ sub-standard.
5. Contract data has less than or equal to a 5% error in records – Contract will receive a score
of 5 for the given standard/ sub-standard.
Exhibit 17 provides examples of different scenarios a DVC might face and corresponding scores assigned to the
contract for the standards that may use either Likert or binary scoring.
EXHIBIT 17. EXAMPLE OF HOW TO DETERMINE MINIMUM THRESHOLD OF IMPLEMENTED POLICIES OR PROCEDURES FOR
FINDINGS USING BINARY AND LIKERT SCORE.

Standard

Percentage of Errors That are Found by the
DVC in Plan Data

DV Response (DV Findings
Reported on Column 'H' in
Appendix J)

1a, 1.b, 1.i, 2.a, 2.b, 2.c, 2.d,
3.a, 3.b, 4, 5, 6, 7

Fewer than 10 percent error in contract data
for the given reporting section/ data
element(s)

Yes

1a, 1.b, 1.i, 2.a, 2.b, 2.c, 2.d,
3.a, 3.b, 4, 5, 6, 8

More than 10 percent error in contract data for
the given reporting section/ data element(s)

No

1a, 1.b, 1.i, 2.a, 2.b, 2.c, 2.d,
3.a, 3.b, 4, 5, 6, 9

Standard not applicable

Leave blank

1.c, 1.d, 1.e, 1.g, 1.h, 2.e

If any of the listed standards is
not applicable
More than 20 percent error in contract data for the
given reporting section/ data element(s)

Leave blank

Between 15.1 percent and 20 percent error
in contract data for the given reporting
section/ data element(s)

2

1.c, 1.d, 1.e, 1.g, 1.h, 2.e
1.c, 1.d, 1.e, 1.g, 1.h, 2.e

1

23

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Percentage of Errors That are Found by the
DVC in Plan Data

Standard

DV Response (DV Findings
Reported on Column 'H' in
Appendix J)

1.c, 1.d, 1.e, 1.g, 1.h, 2.e

Between 10.1 percent and 15 percent error
in contract data for the given reporting
section/ data element(s)

3

1.c, 1.d, 1.e, 1.g, 1.h, 2.e

Between 5.1 percent and 10 percent error in
contract data for the given reporting section/
data element(s)

4

1.c, 1.d, 1.e, 1.g, 1.h, 2.e

5 percent error or less in contract data for
the given reporting section/ data element(s)

5

5.1.3. Review of the Findings Data Collection Form
Exhibit 18 illustrates an example of the FDCF (Appendix J) for Standard 1. The DVC will assess this
standard at the reporting section-level and determines a finding for each of the nine sub-standards
contained in Standard 1.
EXHIBIT 18. EXAMPLE ROWS FROM FDCF (APPENDIX J) FOR STANDARD 1

Standard/
Substandard
ID

Reporting
Section
Criteria ID

Standard/Sub-standard Description

Data Sources and
Data
Review Results:
Element Enter review results
and/or data sources

Enter ‘Findings’
using the applicable
choice in the
appropriate cells.
Cells marked with an
‘*’ should not be
edited.
*

1

No data

A review of source documents (e.g.,
programming code, spreadsheet formulas,
analysis plans, saved data queries, file
layouts, process flows) indicates that all
source documents accurately capture
required data fields and are properly
documented.

No data

Data Sources:

1.a

No data

Source documents are properly secured so
that source documents can be retrieved at
any time to validate the information
submitted to CMS via HPMS.

No data

Review Results:

No data

1.b

No data

Source documents create all required data
fields for reporting requirements.

No data

Review Results:

No data

1.c

No data

No data

Review Results:

No data

1.d

No data

No data

Review Results:

No data

1.e

No data

Source documents are error-free (e.g.,
programming code and spreadsheet
formulas have no messages or warnings
indicating errors, use correct fields, have
appropriate data selection, etc.).
All data fields have meaningful, consistent
labels (e.g., label field for patient ID as
Patient ID, rather than Field1 and maintain
the same field name across data sets).
Data file locations are referenced correctly.

No data

Review Results:

No data

1.f

No data

If used, macros are properly documented.

No data

Review Results:

No data

1.g

No data

Source documents are clearly and
adequately documented.

No data

Review Results:

No data
24

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standard/
Substandard
ID

Reporting
Section
Criteria ID

Standard/Sub-standard Description

1.h

No data

Titles and footnotes on reports and tables
are accurate.

1.i

No data

Version control of source documents is
appropriately applied.

Enter ‘Findings’
using the applicable
Data Sources and
choice in the
Data
Review Results:
appropriate cells.
Element Enter review results Cells marked with an
and/or data sources
‘*’ should not be
edited.
No data
Review Results: No data
No data

Review Results:

No data

Standard 2 requires the DVC to assess reporting section-level findings for Sub-Standards 2.a through
2.c, which are based on reporting section criteria 1 through 3 and, if applicable, Sub-Standard 2.d, which
is based on reporting section criterion 4. Exhibit 19 illustrates an example of the FDCF (Appendix J) for
Standard 2, Sub-Standards 2.a through 2.d. for the Part D Grievances reporting section.
EXHIBIT 19. EXAMPLE OF THE FDCF (APPENDIX J) FOR STANDARD 2, SUB-STANDARDS 2.A - 2.D. FOR THE PART D
GRIEVANCES REPORTING SECTION

Standar
d/ Substandar
d ID
2

2.a

Reporting
Section
Criteria ID

No data

RSC-1

Standard/Sub-standard Description

Data
Element

Data Sources
and Review
Results: Enter
review results
and/or data
sources

Enter 'Findings' using
the applicable choice
in the appropriate
cells. Cells marked
with
an '*' should not be
edited.

No data
A review of source documents (e.g.,
programming code, spreadsheet formulas,
analysis plans, saved data queries, file
layouts, process flows) and census or sample
data, whichever is applicable, indicates that
data elements for each reporting section are
accurately identified, processed, and
calculated.

Data Sources:

The appropriate date range(s) for the
reporting period(s) is captured.

No data

Review Results: No data

No data

Review Results: No data

*

Organization reports data based on the
periods of 1/1 through 3/31, 4/1 through 6/30,
7/1 through 9/30, and 10/1 through 12/31.
2.b

RSC-2

Data are assigned at the applicable level
(e.g., plan benefit package or contract level).
Organization properly assigns data
to the applicable CMS contract.

25

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standar
d/ Substandar
d ID

Reporting
Section
Criteria ID

2.c

RSC-3

Standard/Sub-standard Description

Appropriate deadlines are met for reporting
data (e.g., quarterly).

Data
Element

Data Sources
and Review
Results: Enter
review results
and/or data
sources

Enter 'Findings' using
the applicable choice
in the appropriate
cells. Cells marked
with
an '*' should not be
edited.

No data

Review Results: No data

No data

Review Results: No data

Organization meets deadlines for reporting
data to CMS by 2/6/2023. [Note to reviewer:
If the organization has, for any reason, resubmitted its data to CMS for this reporting
section, the reviewer should verify that the
organization’s original data submissions met
the CMS deadline in order to have a finding
of “yes” for this reporting section criterion.

2.d

RSC-4

However, if the organization re-submits data
for any reason and if the re-submission was
completed by 3/31 of the data validation
year, the reviewer should use the
organization’s corrected data submission for
the review of this reporting section.]
Terms used are properly defined per CMS
regulations, guidance, Reporting
Requirements and Technical Specifications.
Organization properly defines the term
“Grievance” in accordance with 42 CFR
§422.564 and the Parts C & D Enrollee
Grievances, Organization/Coverage
Determinations and Appeals Manual. This
includes applying all relevant guidance
properly when performing its calculations.

26

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

The DVC also determines data element-level findings for Sub-Standard 2.e, which examines each data element
for compliance with the applicable reporting section criteria that varies across the data elements reported by the
SO. For example, in Part D Grievances, RSC-5, RSC-6, and RSC-7 provide the calculations to determine the
data element findings. Exhibit 20 illustrates an example of the FDCF (Appendix J) for Standard 2, Sub-Standard
2.e, RSC-5 for the Part D Grievance reporting section.
EXHIBIT 20. EXAMPLE ROWS FROM FDCF (APPENDIX J) FOR STANDARD 2, SUB-STANDARD 2.E RSC-5 FOR PART D
GRIEVANCES REPORTING SECTION

Standard/
SubStandard
ID

Reporting
Section
Criteria ID

2.e

RSC-5

Standard/Sub-standard Description

The number of expected counts (e.g., number of
members, claims, grievances, procedures) are
verified; ranges of data fields are verified; all
calculations (e.g., derived data fields) are verified;
missing data has been properly addressed;
reporting output matches corresponding source
documents (e.g., programming code, saved
queries, analysis plans); version control of
reported data elements is appropriately applied;
QA checks/thresholds are applied to detect outlier
or erroneous data prior to data submission.

Data
Element

Data Sources
and Review
Results:
Enter review
results and/or
data sources

No Data

Data
Sources:

Data Element
B

Review
Results:

No Data

Data
Sources:

Data Element
D

Review
Results:

Enter
'Findings
' using
the
applicabl
e choice
in the
appropri
ate cells.
Cells
marked
with an '*'
should
not be
edited
*

RSC-5: Organization accurately reports data by
applying data integrity checks listed below and
uploads it into HPMS.

2.e

RSC-5.a

2.e

RSC-5.b

2.e

RSC-5.b

a: Number of total grievances in which timely
notification was given (Data Element B) does not
exceed number of total grievances (Data Element
A).

RSC-5: Organization accurately reports data by
applying data integrity checks listed below and
uploads it into HPMS.
b: Number of expedited grievances in which timely
notification was given (Data Element D) does not
exceed number of total grievances in which timely
notification was given (Data Element B).

No Data
*

No Data

27

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standard/
SubStandard
ID

Reporting
Section
Criteria ID

2.e

RSC-5.c

2.e

RSC-5.c

2.e

RSC-5.d

2.e

RSC-5.d

2.e

RSC-5.e

2.e

RSC-5.e

2.e

RSC-5.f

2.e

RSC-5.f

Standard/Sub-standard Description

RSC-5: Organization accurately reports data by
applying data integrity checks listed below and
uploads it into HPMS.
c: Number of expedited grievances (Data Element
C) does not exceed total grievances (Data
Element A).
No Data
RSC-5: Organization accurately reports data by
applying data integrity checks listed below and
uploads it into HPMS.
d: Number of expedited grievances in which timely
notification was given (Data Element D) does not
exceed total expedited grievances (Data Element
C).
No Data
RSC-5: Organization accurately reports data by
applying data integrity checks listed below and
uploads it into HPMS.
e: Number of dismissed grievances (Data Element
E) are excluded from the total.
No Data
RSC-5: Organization accurately reports data by
applying data integrity checks listed below and
uploads it into HPMS.
f: If the organization received a CMS outlier/data
integrity notice, validate whether or not an internal
procedure change was warranted or resubmission
through HPMS.
No Data

Data
Element

Data Sources
and Review
Results:
Enter review
results and/or
data sources

No Data

Data
Sources:

Data Element
C

Review
Results:

No Data

Data
Sources:

Data Element
D
No Data

Review
Results:
Data
Sources:

Data Element
E

Review
Results:

No Data

Data
Sources:

Data
Elements A-E

Review
Results:

Enter
'Findings
' using
the
applicabl
e choice
in the
appropri
ate cells.
Cells
marked
with an '*'
should
not be
edited
*

No Data
*

No Data
*

No Data
*

V

28

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standard 3 contains two Sub-Standards. Sub-Standard 3.a requires the DVC to assess data element-level
findings and Sub-Standard 3.b requires reporting section-level findings. Sub-Standard 3.a is assessed at the
data element-level for reporting sections that are manually entered into the HPMS Plan Reporting Module
because it confirms that there were no manual data entry errors for each data element, and for reporting
sections that are reported as file uploads, it confirms at the sub-standard level that the SO used the correct file
layout. Exhibit 21 illustrates an example of the FDCF (Appendix J) for Standard 3 for the Part D Grievances
reporting section.
EXHIBIT 21. EXAMPLE ROWS FROM FDCF (APPENDIX J) FOR STANDARD 3 FOR PART D GRIEVANCES REPORTING
SECTION

Standard/
Substandard ID

Reporting
Section
Criteria ID

Standard/Sub-standard
Description

Data
Element

Data Sources
and Review
Results: Enter
review results
and/or data
sources

Enter 'Findings' using the
applicable choice in the
appropriate cells. Cells
marked with an '*' should
not be edited.

Organization implements policies
and procedures for data
submission, including the
following:
Data elements are accurately
uploaded into HPMS and entries
match corresponding source
documents.

No data

Data Sources:

Data
Element A

Review Results: No Data

No data

No data

Data
Element B

Review Results: No data

3.a

No data

No data

Data
Review Results: No data
Element C

3.a

No data

No data

Data
Review Results: No data
Element D

3.a

No data

No data

Data
Element E

Review Results: No data

3.b

No data

All source, intermediate, and
final stage data sets and other
outputs relied upon to enter data
into HPMS are archived.

No data

Review Results: No Data

3.

No data

3.a

No Data

3.a

*

29

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standards 4 through 7 assess policies and procedures for periodic data system updates; an SO will most likely
have these policies and procedures in place for an entire reporting section, as opposed to having them in place
for only certain data elements. Exhibit 22 displays example rows from the FDCF (Appendix J) for Standards 4
through 7.
EXHIBIT 22. EXAMPLE ROWS FROM FDCF (APPENDIX J) FOR STANDARDS 4 THROUGH 7

4

No data

Organization implements policies and
procedures for periodic data system
updates (e.g., changes in enrollment,
provider/pharmacy status, and claims
adjustments).

No data

Enter 'Findings'
using the
applicable choice
in the appropriate
cells. Cells marked
with an '*' should
not be edited.
Review Results: No data

5

No data

Organization implements policies and
procedures for archiving and restoring
data in each data system (e.g.,
disaster recovery plan).

No data

Review Results: No data

6

No data

No data

Review Results: No data

7

No data

If organization’s data systems
underwent any changes during the
reporting period (e.g., because of a
merger, acquisition, or upgrade):
Organization provided documentation
on the data system changes and, upon
review, there were no issues that
adversely impacted data reported.
If data collection and/or reporting for
this reporting section is delegated to
another entity: Organization regularly
monitors the quality and timeliness of
the data collected and/or reported by
the delegated entity or first
tier/downstream contractor.

No data

Review Results: No data

Standard/
Substandard
ID

Reporting
Section
Criteria ID

Standard/Sub-standard Description

Data
Element

Data Sources
and Review
Results: Enter
review results
and/or data
sources

30

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

5.1.4. Guidance for Interpreting Standards and Making a Findings Determination
In order to ensure consistency with the review process, CMS has provided below a description of the data
sources and criteria that DVCs use to determine findings for each of the DV standards.
Standard 1
This validation standard is assessed at the reporting section-level and is used to determine that all source
documents accurately capture required data fields and are properly documented. The guidance for
evaluating Standard 1 is described below in Exhibit 23.
EXHIBIT 23. GUIDANCE FOR DATA VALIDATION STANDARD 1

Criteria

Guidance

A review of source documents (e.g., programming
code, spreadsheet formulas, analysis plans, saved
data queries, file layouts, process flows) indicates
that all source documents accurately capture required
data fields and are properly documented.

Determine if the SO’s source documents (e.g., programming
code, spreadsheet formulas, analysis plans, saved data
queries, file layouts, process flows) accurately capture the
data fields required for each reporting section under review
and are documented with the necessary detail and information
to create data file sets and other outputs.

Criteria for Validating Source Documents (SubStandards):
a) Source documents are properly secured so
that source documents can be retrieved at any
time to validate the information submitted to
CMS via HPMS.
b) Source documents create all
required data fields for reporting
requirements.
c) Source documents are error-free (e.g.,
programming code and spreadsheet
formulas have no messages or warnings
indicating errors, use correct fields, have
appropriate data selection, etc.).
d) All data fields have meaningful, consistent
labels (e.g., label field for patient ID as
Patient ID, rather than Field1 and maintain
the same field name across datasets).
e) Data file locations are referenced correctly
f) If used, macros are properly documented.
g) Source documents are clearly
and adequately documented.
h) Titles and footnotes on
reports and tables are
accurate.
i) Version control of source documents
is appropriately applied.

Ensure that all source documentation is legible, descriptive,
and understandable, including each of the following:
• Standard Operating Procedures (SOPs) include detailed
workflows and processes related to managing,
producing, and tracking source documents.
• Titles and footnotes used in programs and reported
output are legible and correspond to HPMS reports
and tables.
• SOPs, file-naming conventions, dates of source
documents, and output reports reflect application of
version control.
• Data file locations are referenced correctly within
source code (i.e., these files can be located using the
references that exist within the source code).
• Dated HPMS entries match the source
document(s) used to create the data entered into
HPMS.
Ensure that the data validation reviewer is using the
documentation that is current and relevant to the time period
of the reporting requirements.

Please note that Standards 1 and 2 should be addressed concurrently given that an evaluation of source
documents directly impacts the quality of the actual data and vice versa (that elements for each reporting
section are accurately identified, processed, and calculated). For example, the DVC should ensure that all
source documentation (file layouts, data dictionaries, programming code, work instructions, SOPs, etc.) is
available and allows for the complete validation for each reporting section’s validation.

31

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standard 2
This validation standard assesses whether the data elements for each reporting section are accurately identified,
processed, and calculated. Each DVC should ensure that it has staff fluent in the programming language (SQL,
SAS, Microsoft VBA) used by the SO. The guidance for evaluating Standard 2 is described below in Exhibit 24.
Since the DV reviews must be conducted at the contract level, for the reporting sections that require reporting at
the plan benefit package (PBP) level, if the DVC finds that the SO incorrectly identified, processed, or calculated
the data reported for any of the PBPs included under a contract, then the DVC must assign a “No” finding in the
FDCF (Appendix J) for the entire contract for the applicable sub-standard or data element (for Sub-Standard
2.e).
While careful inspection of the source code should detect most errors in the reported data, a careful review of
the census or sample data gathered from the SO will minimize the chance that a programming error was
undetected by the DVC. Many of the same items that will be checked in reviewing the source code can also be
checked by analyzing the extracted data sets.
EXHIBIT 24. GUIDANCE FOR DATA VALIDATION STANDARD 2

Criteria
A review of source documents (e.g.,
programming code, spreadsheet formulas,
analysis plans, saved data queries, file
layouts, process flows) and census or
sample data, whichever is applicable,
indicates that data elements for each
reporting section are accurately identified,
processed, and calculated.
Criteria for Validating Reporting Section
Criteria (Refer to reporting section
criteria section below):
a) The appropriate date range(s) for
the reporting period(s) is captured.
b) Data are assigned at the applicable
level (e.g., plan benefit package or
contract level).
c) Appropriate deadlines are met
for reporting data (e.g.,
quarterly).
d) Terms used are properly defined
per CMS regulations, guidance,
Reporting Requirements, and
Technical Specifications.

Guidance
(Sub-Standard 2a – 2d)
Assess the programming code to determine if the data was extracted from the
system properly and if the calculations used in reporting data to CMS are
accurate according to the reporting section criteria applicable to each reporting
section under review.
A thorough review of source code must examine every line of code to ensure the
following for each reporting section under review:
• Data is extracted from the appropriate source system: Verify that all data
sets found in the programming code can be traced back to the appropriate
source data sets.
• Data sets are filtered correctly: Verify that data inclusion and
exclusion criteria were applied according to the reporting section
criteria.
o For example, proper inclusion of records would ensure that source
code indicates that only those records falling within the reporting
period date range are included in the reported data. An example of
correct exclusion would document source code that indicates
beneficiaries are not eligible for a particular benefit (e.g.,
Medication Therapy Management Program).
• Individual data sets are joined or merged correctly (this is especially
important when moving data from source data sets to intermediate data
sets): Verify that the correct key data field was used to generate the new
data set and that the correct type of join (or data merge) was used to
avoid creating duplicate records or improperly combining records from
various data sets.
• Data set progression is accurate: Verify that required data fields in
both the source and final stage files allow for file comparison and
understanding of data production from source system through the
final stage file.
o If full census data is not extracted, verify that the sample size is
sufficient and representative of the population of interest.
o While the Data Extraction and Sampling Instructions provide
minimum sample sizes, reviewers often will need larger data sets
to check for errors that occur infrequently. Statisticians should rely
on standard statistical practices when determining the proper
sample sizes that any estimates generated are statistically
significant.
• All data elements are accurate: Verify that each data element
is consistent with the reporting section criteria.

32

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Criteria

Guidance
(Sub-Standard 2c)
Assess the Submission Activity Report from the HPMS Plan Reporting Module
to determine if appropriate deadlines were met for reporting data by performing
the following:
• Request a copy of the contract’s Submission Activity Report
from the SO: This report displays information about the original
submission and all subsequent resubmissions for a particular
contract or contracts. The report also displays Reporting Period,
Contract Number, Plan ID, Submission Version, Due Date, and
Date Submitted for each section.
• Determine if the SO has, for any reason, re-submitted its data
to CMS for a reporting section:
o The data validation reviewer should verify that the SO’s original
submission(s) met the CMS deadline.
o If the deadline was met, the reviewer must assess a “Yes” finding for
this reporting section criterion. However, if an SO re-submits data
for any reason and if the re-submission was completed by March 31
of the calendar year of the data validation review (i.e., immediately
prior to the data validation review timeframe), the data validation
reviewer should use the SO’s corrected data submission for
performing the validation, not the original data. The March 31st
deadline will give the reviewer enough time to include the corrected
data in the scope of its review of data and determination of findings.
o If the SO received CMS permission to submit data after the
reporting deadline (i.e., its first submission), the reviewer must
request that the SO show proof that it requested and was granted
an extension by CMS. If this proof is valid, then the reviewer should
consider the deadline as being met and assess a “Yes” finding for
this reporting section criterion.
o For either of the above scenarios, the reviewer must clearly
document the circumstances in the “Data Sources and Review
Results” section of the FDCF (Appendix J).

e)

The number of expected counts (e.g.,
number of members, claims,
grievances, procedures) are verified;
ranges of data fields are verified; all
calculations (e.g., derived data fields)
are verified; missing data has been
properly addressed; reporting output
matches corresponding source
documents (e.g., programming code,
saved queries, analysis plans); version
control of reported data elements is
appropriately applied; QA
checks/thresholds are applied to detect
outlier or erroneous data prior to data
submission.

(Sub-Standard 2e)
Assess the census/sample data provided by the SO to determine each of the
following for each reporting section under review:
• Data records are selected properly:
o Perform frequency calculations to list all unique occurrences of data
fields pertinent to the calculation of the reporting section to verify
they contain values within an acceptable range for the data field.
o Calculating the frequency of occurrence for certain data fields might
also alert the reviewer to obvious mistakes in the data extraction.
o Verify that data has been selected at the proper level (e.g., either
the contract or the plan benefit package level).
o Check date ranges, demographic information, and eligibility
information to examine proper data filtering.
• Individual data sets are joined or merged correctly:
o Sample a few records, when individual data sets are available (most
likely for intermediate data sets), from the individual data sets to
confirm that they were joined properly.
o Check for duplicate records and determine if record counts for the
component data sets agree with those found in the merged data set.
• All data elements are calculated accurately:
o Recalculate the data fields that the SO used to calculate the data
elements and refer to the reporting section criteria for each reporting
section.
o Calculate sums of the individual records within each reporting
section to ensure that they equal those reported to CMS.
Verify that the calculation of each of the data elements is consistent with the
reporting section criteria*.

33

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

* CMS has added a reporting section criterion (RSC #5) which will be used by the DVCs to confirm that the data
does not have any logical errors. RSC #5 includes data integrity checks that the DVC must verify at the data
element level. These data integrity checks include confirming that a data element does not include outlier
records.
Exhibit 24 provides several examples of how to review source code and evaluate the integrity of the data.
However, the DVC may use other methods of DV to ensure a comprehensive and complete review of the source
code and census/sample data. The DVC must clearly document all errors found in programming code, referring
to the program examined, the precise location in the program, the nature of the error, and the impact of the error
in the “Data Sources and Review Results” section of the FDCF (Appendix J). Likewise, any evidence from the
review of census/sample data that leads to a negative finding must be clearly documented in the applicable
section of the FDCF (Appendix J).

34

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standard 3
This validation standard assesses whether the SO implements policies and procedures for entering and/or
uploading each data submission to HPMS. The guidance for evaluating Standard 3 is described in Exhibit 25.
EXHIBIT 25. GUIDANCE FOR DATA VALIDATION STANDARD 3

Criteria
Organization implements policies and
procedures for data submission, including
the following:
a) Data elements are accurately entered /
uploaded into HPMS and entries match
corresponding source documents.

Guidance
(Sub-Standard 3a)
Determine who is responsible for entering/uploading data into HPMS
for each reporting section under review and if the SO has written work
instructions or policies and procedures for the entry or submission of
the Part C and Part D Reporting Requirements.
Evaluate Sub-Standard 3a by performing the following actions:
• Compare the data file created for submission to CMS with a copy of
the HPMS screenshots of data entered to confirm there were no
manual data entry errors.
• For file uploads, confirm that the data file adheres to the record
layout specified in the applicable Technical Specifications
document.
• For the reporting sections that require reporting at the plan benefit
package (PBP)-level, if the reviewer finds that the SO did not
accurately enter and/or upload data reported for any of the PBPs
included under a contract, then the reviewer must assign a “No”
finding in the FDCF (Appendix J) for the entire contract for the
applicable data element(s) for Sub- Standard 3a.
• If a reporting section requires both a file upload and data entry,
both have to occur in order for an SO to meet Sub- Standard 3a.

b) All source, intermediate, final stage data sets (Sub-Standard 3b)
Determine if the SO has a policy or procedure for archiving all source,
and other outputs relied upon to enter data
intermediate, and final stage data sets relied upon to enter data into
into HPMS are archived.
HPMS, and confirm that the SO implemented this policy for the reporting
section under review.

35

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standard 4
This validation standard is assessed at the reporting section-level and is used to assess whether the SO has and
implements policies and procedures for regular database updates. The data sources and criteria for evaluating
Standard 4 are described in Exhibit 26.
EXHIBIT 26. GUIDANCE FOR DATA VALIDATION STANDARD 4

Criteria
Organization implements policies and
procedures for periodic data system
updates (e.g., changes in enrollment,
provider/pharmacy status, and claims
adjustments).

Guidance
Determine if the SO has policies and procedures in place for
performing periodic updates for each data system used for the
reporting section under review that ensures reported data are
accurate and timely.
Determine if the SO implements and adheres to the policies and
procedures referenced above (i.e., was any data for the reporting
section under review negatively impacted by a failure to
implement or follow these policies and procedures?).

Standard 5
This validation standard is assessed at the reporting section-level and is used to assess whether the SO has
and implements policies and procedures for data archiving and restoration. The data sources and criteria for
evaluating Standard 5 are described in Exhibit 27.
EXHIBIT 27. GUIDANCE FOR DATA VALIDATION STANDARD 5

Criteria
Organization implements
policies and procedures for
archiving and restoring data in
each data system (e.g., disaster
recovery plan).

Guidance
Determine if the SO has policies and procedures in place for
archiving and restoring data in each data system used for the
reporting section under review that ensures timely data
submission or re-submission in the event of data loss.
Determine if the SO implements and adheres to the policies and
procedures referenced above (i.e., was any data for the reporting
section under review negatively impacted by a failure to implement
or follow these policies and procedures?).

36

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standard 6
This validation standard is assessed at the reporting section-level and is used to assess whether the validity of
the SO’s data was adversely impacted by any changes to data systems during the reporting period. The data
sources and criteria for evaluating Standard 6 are described in Exhibit 28.
Standard 6 applies if an SO’s data systems underwent any changes during the reporting period. The DVC
should mark “Not Applicable” in the FDCF (Appendix J) if Standard 6 is not applicable to the contract under
review.
EXHIBIT 28. GUIDANCE FOR DATA VALIDATION STANDARD 6

Criteria
If organization’s data systems underwent
any changes during the reporting period
(e.g., because of a merger, acquisition, or
upgrade): Organization provided
documentation on the data system changes
and, upon review, there were no issues that
adversely impacted data reported.

Guidance
Review documentation on data system changes and
determine if changes to an SO’s data system adversely
impacted data reported.by conducting the following
activities:
• Determine if there were any changes to data sources
used for data collection and storage, data processing,
analysis, and reporting for the reporting section under
review.
• Determine if data system changes were the root cause
of any outlier notices received from CMS for the
reporting section under review.
• Determine if the SO implemented any process or
quality improvement activities during the reporting
period specifically related to the data system change
for the reporting section under review.
Determine if the validity of the SO’s data was adversely
impacted by any changes to data systems during the reporting
period.

37

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Standard 7
This validation standard is assessed at the reporting section-level and is used to assess whether the SO
routinely monitors the quality of a delegated entity’s work and processes related to the reporting
requirements. The data sources and criteria for evaluating Standard 7 are described in Exhibit 29.
Standard 7 applies if any of the data collection or validation processes are outsourced to another entity. The
DVC should mark “Not Applicable” in the FDCF (Appendix J) if Standard 7 is not applicable to the reporting
section or contract under review.
EXHIBIT 29. GUIDANCE FOR DATA VALIDATION STANDARD 7

Criteria
If data collection and/or
reporting for this reporting
section is delegated to another
entity: Organization regularly
monitors the quality and
timeliness of the data collected
and/or reported by the
delegated entity or first
tier/downstream contractor.

Guidance
Assess the following if data collection and/or reporting for a
reporting section is delegated to another entity:
• Determine if the SO has policies and procedures in place
for overseeing the delegated entity’s reporting process /
results for the reporting section under review.
• Determine if the SO implements and adheres to the policies
and procedures referenced above (i.e., was any data for the
reporting section under review negatively impacted by a
failure to implement or follow these policies and
procedures?).
Plans are not expected to replicate the delegated entities
process and recalculate all of their numbers but are expected
to have policies and procedures in place for routine monitoring.
It is expected that these policies and procedures are
implemented as frequently as needed to verify the delegated
entities’ reporting.
SOs are responsible for delegated entities' calculations and
numbers and therefore if they are incorrect, the responsibility
ultimately falls on the SO.

5.2

Provide Draft Findings to Sponsoring Organization
Once the findings have been documented in the FDCF (Appendix J), the DVC must share the draft findings with the SO.
When the DVC uploads the Microsoft Excel version of the FDCF (Appendix J) into the PRDVM during its
review, they may print the findings uploaded into the PRDVM and share them with the SO at any point during
the review by accessing the PRDVM report entitled “Review Data Validation Findings Report.”

5.3

Review Draft Findings with Sponsoring Organization and Obtain
Additional Documentation Necessary to Resolve Issues
The SO and DVC should build time into the April-June DV schedule to allow sufficient review of the findings.
Any issues identified during this review must be resolved prior to the data validation contractor’s June 30
deadline for submitting findings to CMS.
Following any review of the draft findings with the SO, the DVC must update the FDCF (Appendix J) with any
necessary revisions. This final version will be used to report the results of the data validation review to CMS.

38

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

5.4. Submit Data Validation Review Findings via HPMS PRDVM
5.4.1. Data Validation Contractor’s Submission of Findings
Following the conclusion of the DV review and the finalization of findings, the DVC must report the findings by
uploading the FDCF (Appendix J) to CMS via the PRDVM in HPMS by June 30. Instructions for using this
module are contained in the PRDVM Quick Reference Guide, which is available in the PRDVM. The FDCF
(Appendix J) includes review results and/or data sources that were reviewed for each standard or substandard, as well as the Yes, No, or Not Applicable finding, or a 1-5 Likert scale. Associated with each standard
or substandard.
DVCs should also indicate which extraction method (full census or sample) was used for each standard.

5.4.2. Sponsoring Organization Disagreement with Findings
If the SO disagrees with any of the findings submitted by the DVC, it may submit information indicating this
disagreement to CMS by the June 30th data validation deadline. Submissions should be sent to CMS via the
[email protected] resource mailbox and should contain all of the following information
in order to be considered for review:
•
•

•

Email subject line must state: “Data Validation: Reported Findings Discrepancy”
Content of email must include the information below, in list format and in the following order:
o Name of SO
o CMS contract number(s)
o SO’s contact name, title, phone number, and email address
o Name of DVC organization
For each area of discrepancy, list the following information:
o Part C or Part D, name of reporting section
o Standard/ sub-standard ID, reporting section criteria ID
o Description of DVC’s finding
o Reason for disagreement with finding
o Steps that were taken to resolve the disagreement with the DVC prior to the submission of the
findings
o Outcome of discussions, areas of impasse, and any additional information CMS will review any
findings disagreements on a case-by-case basis.

39

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

6. POST- DATA VALIDATION ACTIVITIES
6.1

Compile Archive of Data Validation Work Papers
The DVC must prepare a complete archive of work papers associated with the annual DV and provide it to the
SO. At a minimum, this archive must contain the documentation described in Exhibit 30. The DVC should also
retain a complete copy of this archive in accordance with its contract with the SO.
When the SO receives the archive from the DVC, the SO must add the documentation of its DVC selection
process to the archive, including how its chosen DVC meets the minimum qualifications, credentials, and
resources set forth in the Standards for Selecting a Data Validation Contractor. The SO must retain this
complete archive for the 10-year retention period required per federal regulations and be prepared to provide the
archive to CMS upon request.
EXHIBIT 30. MINIMUM DOCUMENTATION REQUIRED FOR DATA VALIDATION ARCHIVE

•
•

•

•
•
•

Documentation of Data Validation
Contractor Selection Process
Documentation of completion of CMS
Data Validation Training for all staff
assigned to the data validation team
Completed OAI (Appendix E), including
all documentation provided in response
to OAI (Appendix E) Section5
Final Site Visit Agenda
Completed Sign-in Sheets from site
visit (if used)
Final IDG used during site visit

•
•
•
•
•
•

Copies of any formal presentations during site visit
Notes on staff interviews and demonstrations
during site visit
Census/sample data
Additional documentation provided by SO
during/after site visit
Draft findings in FDCF (Appendix J) Notes on
issues resulting in changes to draft findings
Final FDCF (Appendix J)

6.2

Receive Pass or Not Pass Threshold Level and Assess Pass or
Not Pass Determination based on Final Scores

6.2.1

Pass/Not Pass Determination
For each of the standards, sub-standards, and data elements, the reviewer must assess a “Yes/No” finding
or a score using a 1-5 Likert scale. Each finding is associated with CMS-assigned percentage points and
can vary depending on the sub-standard or data element being scored. A “No” or 1 finding, however, will
always result in a score of zero percentage points. The Data Validation Pass/Not Pass Determination
Methodology (Appendix K) identifies the individual score CMS has assigned to each standard and substandard for all reporting sections.
After all findings are submitted to CMS, CMS will calculate a percentage score for all Part C reporting
sections as a group, all Part D reporting sections as a group, and a combined Part C and Part D
determination for those contracts reporting both Part C and Part D data. CMS then establishes passing
thresholds for Part C, Part D, and an overall combined Part C/Part D score based on the distribution of
scores.
Contracts can view their data validation results in HPMS (https://hpms.cms.gov/). To access this page, from
the top menu select “Monitoring,” then “Plan Reporting Data Validation.” Select the appropriate contract
year. Select the PRDVM Reports. Select “Score Detail Report.” Select the applicable reporting section. If
you cannot see the Plan Reporting Data Validation module, contact [email protected].

6.2.2

6.3

Passing Data Validation – Minimum threshold
CMS has established 95% as the passing DV threshold for each reporting section, as well as for the
Part C, Part D, and combined scores. SOs may view their individual contracts’ validation results in HPMS.
CMS will send follow-up communication to active contracts scoring below 95% on the overall Part C, Part D, or
combined score.

Sponsoring Organization Appeal of Data Validation

40

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Determination
An SO has the right to appeal:
•

•
•
•

Reporting section score of less than 95%
Non-compliant data validation standards/sub-standards i.e., a "No" or a 1, 2, or 3 on the 5-point Likert
scale in the specific data element's data validation
Contracts that score < than 95% on either Part C and/or Part D overall
Contracts that score < than 95% in their combined Part C and Part D score

If the SO wishes to appeal, it must submit an appeal to CMS by June 30th deadline. Submissions must be
sent to CMS via the [email protected] resource mailbox and must contain all of
the following information in order to be considered.
•

•

•

Email subject line must state: “Data Validation: Appeal of ”
Content of email must include the information below, in list format and in the following order:
o Name of SO
o CMS contract number(s)
o SO’s contact name, title, phone number, and email address
o Name of DVC organization
For each appeal, list the following information:
o Justification for appeal
o Include as an attachment any documentation supporting the justification for the appeal. The
documentation must have been in existence at the time of the DV. For example, if after the DV,
the SO resubmits corrected data, revises a policy and procedure, or corrects a programming
code that caused it to improperly calculate reported data; the SO cannot submit documentation
of these corrections to appeal.

Once the appeal is received, CMS will carefully consider the justification and any supporting documentation
to determine if any changes should be made. CMS has not established a timeframe for the consideration of
SO appeals.

41