CMS-10305 Organizational Assessment Instrument

Medicare Part C and Part D Reporting Requirements
Data Validation Procedure Manual
Appendix E: Organizational Assessment Instrument

Prepared by:
Centers for Medicare & Medicaid Services
Center for Medicare
Medicare Drug Benefit and C & D Data Group
Last Updated: January 2018

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection
of information unless it displays a valid OMB control number. The valid OMB control number for this
information collection is 0938-1115 (Expires: XX/XX/20XX). The time required to complete this
information collection is estimated to average 44 hours per response, including the time to review
instructions, search existing data resources, gather the data needed, and complete and review the
information collection. If you have comments concerning the accuracy of the time estimate(s) or
suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports
clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.

i

Appendix E: Organizational Assessment Instrument

Table of Contents
1 OBJECTIVES ................................................................................................................................... 1
2 INSTRUCTIONS .............................................................................................................................. 1
2.1
2.2

Instructions for Data Validation Contractor .............................................................................. 1
Instructions for Organization ..................................................................................................... 1

3 GENERAL QUESTIONS ................................................................................................................ 2
3.1
3.2
3.3
4

Organization Information .......................................................................................................... 2
Contact Information .................................................................................................................. 3
Part C and Part D REPORTING SECTIONs Undergoing Validation ...................................... 3

UNDERLYING DATA SOURCES AND REPORTING PROCESSES....................................... 4
4.1
4.2
4.3

Underlying Data Sources ........................................................................................................... 4
Programming and Software ....................................................................................................... 6
Supplemental Questions Regarding Reporting Processes ......................................................... 7

5 DATA VALIDATION DOCUMENTATION REQUEST............................................................. 8
5.1
5.2
5.3
5.4

Request for Programming Code and Example Output .............................................................. 8
Request for Data Dictionary ...................................................................................................... 8
Request for Analysis Plan, Reporting Process Flows, and Diagrams ........................................ 9
Request for Standard Operating Documents: Standard Operating Procedures (SOPs),
Policies and Procedures, or Other Work Instructions ................................................................ 9

6 DATA VALIDATION DOCUMENT LOG .................................................................................... 9
APPENDIX A: DOCUMENT LOG TEMPLATE ................................................................................. 11
APPENDIX B: DATA DICTIONARY EXAMPLE............................................................................... 12

List of Tables
Table 1. Timeline of OAI Activities ............................................................................................................. 2
Table 2. Organization Information................................................................................................................ 3
Table 3. Part C Contact Information ............................................................................................................. 3
Table 4. Part D Contact Information ............................................................................................................. 3
Table 5. Reporting Sections Undergoing Validation .................................................................................... 4
Table 6. Underlying Data Sources ................................................................................................................ 5
Table 7. Programming Software Specifications ............................................................................................ 6

ii

Appendix E: Organizational Assessment Instrument

1 OBJECTIVES
CMS is providing this Organizational Assessment Instrument (OAI) as a tool for data validation
(DV) contractors to understand Part C and Part D sponsoring organizations’ (SOs’) reporting
processes more completely and to request documentation that will be evaluated during the
review process. The information collected in this OAI will help prepare DV contractors and will
reduce resources required for the site visit portion of the review. CMS requires that SOs
complete the OAI to add efficiencies to the review process.

2 INSTRUCTIONS
2.1

INSTRUCTIONS FOR DATA VALIDATION CONTRACTOR

Prior to the start of the data validation review period, the SO must begin completion of the OAI. It
is recommended that each section of the OAI be completed prior to the site visit; however, if
preferred, the DV contractor may determine an alternative sequence (e.g., the DV contractor may
request that an SO complete Sections 3 and 4 prior to the site visit, but complete Sections 5 and 6
following the site visit). The OAI is designed to capture preliminary information about the SO’s
processes for collecting and reporting data per the CMS reporting requirements. The DV
contractor must analyze the OAI prior to the site visit and follow-up on any incomplete or
ambiguous responses during the site visit portion of the review. The OAI must be electronically
distributed to the SO undergoing a review. Following the SO’s completion of this document, the
DV contractor must attach a completed copy of the OAI in a file of all data validation review
work papers that will be shared with the organization.

2.2

INSTRUCTIONS FOR ORGANIZATION

Organizations must complete each section of the OAI in advance of the data validation review
period, or according to the set timeline of the reviewer. The SO should complete the OAI and
provide documentation to the DV contractor as early as possible at the start of the data validation
review period so that data validation can begin on April 1. All documentation and responses to
questions should reflect the SO’s systems and processes that were in place during the reporting
period(s) undergoing the data validation review. SOs with multiple contracts should complete
only one OAI. If the information provided in the OAI varies by contract, the SO should specify
the differences within the OAI to allow the DV contractor to more easily identify differences that
may impact reporting section calculations or reporting for a given contract.
The SO must submit the OAI, documentation, and any additional information to the reviewer
electronically. The organization is responsible for ensuring that it has established mutually
agreeable methods for sharing proprietary and/or secure (PHI/PII) information with the reviewer
and that the reviewer complies with all HIPAA privacy and security requirements.
The completed OAI and any additional information provided as a result of this request will be
assessed by the reviewer. If an SO has any questions while completing the OAI, it should contact
the DV contractor. Each stage of the data validation review should entail a collaborative effort
between the SO and DV contractor. An overview of the timeline related to OAI activities is
outlined in Table 1.

1

Appendix E: Organizational Assessment Instrument
Table 1. Timeline of OAI Activities

Performing Data Validation Activities

DV
Phase

Step

Responsible
Party

Data Validation Activities

Timeline

1

SO

No earlier
Complete Organizational Assessment Instrument (OAI) and
provide appropriate documentation to selected reviewer per the than 30
OAI’s documentation request
days prior
to the start
of the DV
cycle on
April 1.

2

DVC, SO

Analyze OAI

Allow two
weeks after
receipt.

3

DVC, SO

Prepare for site visit (site visit agenda, resource needs,
and logistics)

Early April

4

DVC, SO

Conduct on-site review (convene entrance conference, conduct Early April
interviews with SO staff, observe SO’s reporting processes, and (allow for
obtain census and/or sample files)
up to 1
week)

5

DVC

Request additional documents following site visit (if applicable)

Mid/Late
April

3 GENERAL QUESTIONS
The information gathered below will provide a better understanding of the scope for the SO’s
data validation review, including which contract(s) will be reviewed and which Part C and/or
Part D reporting sections the SO is reporting for validation.

3.1

ORGANIZATION INFORMATION

Complete Table 2, indicating each Medicare contract that the SO held during the reporting
period(s) undergoing the data validation review. For the “Contract Type” field, select from the
following list:
•
•
•
•
•
•
•

CCPP
FFS
MSA
1876 Cost
Employer/Union Direct Contract
PDP
Demo

Also indicate whether the contract includes the Part C and/or Part D benefit and provide the
number of plan benefit packages (PBP) associated with each contract. Finally, indicate if any of
the PBPs associated with the contract are Special Needs Plans or Employer/Union “800 Series”
plans. The SO may add rows to this table as necessary, but should not manipulate the columns.

Appendix E: Organizational Assessment Instrument
Table 2. Organization
Information
Parent Organization Name:
CMS Contract Number

Contract
Type

Includes
Part C?
(Y/N)

Includes
Part D?
(Y/N)

No. of Plan
Benefit
Packages

Includes
SNP
PBP(s)?
(Y/.N)

Includes
Employer/Union
“800 Series”
PBP(s)?
(Y/N)

Example: Contract 123

PFFS

Y

Y

3

N

N

Example: Contract 123

CCP

Y

Y

1

Y

N

[add rows as required]

3.2

CONTACT INFORMATION

Complete Table 3 and Table 4 indicating the SO’s primary and secondary points of contact
responsible for the Part C and Part D reporting requirements data validation review for each
contract included in this OAI.
Table 3. Part C Contact Information
Primary Part C Point of Contact
Name:
Title:
Company:
Address:
City, State, Zip:
Telephone:
Fax:
Email:

Secondary Part C Point of Contact
Name:
Title:
Company:
Address:
City, State, Zip:
Telephone:
Fax:
Email:

Table 4. Part D Contact Information
Primary Part D Point of Contact
Name:
Title:
Company:
Address:
City, State, Zip:
Telephone:
Fax:
Email:

Secondary Part D Point of Contact
Name:
Title:
Company:
Address:
City, State, Zip:
Telephone:
Fax:
Email:

Appendix E: Organizational Assessment Instrument

3.3

PART C AND PART D REPORTING SECTIONS UNDERGOING
VALIDATION

Complete Table 5 for the contract(s) included in this OAI, indicating which of the Part C
and/or Part D reporting sections the SO has submitted for data validation review, the
applicable contract numbers (Column B), and whether the SO is able to report on all
required data elements per the CMS Part C and Part D Reporting Requirements Technical
Specifications (Column C).
Table 5. Reporting Sections Undergoing Validation
A. Reporting Section

B. CMS Contract
Number(s)

C. Are all required
data elements
captured by your
internal data
system(s)?
(Yes/No)

D. If the answer to
Column C. is no,
please indicate
which delegated
entities’ data
systems contain
the data elements

Part C:
Grievances
Organization Determinations/Reconsiderations
Special Needs Plans (SNPs) Care Management
Part D:
Medication Therapy Management Programs
Grievances
Coverage Determinations and Redeterminations
Improving Drug Utilization Review Controls

4 UNDERLYING DATA SOURCES AND REPORTING
PROCESSES
The questions below address the underlying data sources and reporting processes
used to produce the Part C and Part D reporting sections.

4.1

UNDERLYING DATA SOURCES

Complete Table 6 for the contract(s) included in this OAI, indicating the name of
the data source(s) used to generate each Part C and Part D reporting section
(Column B). If additional rows are required to list the data sources for a given
reporting section, insert new rows into the table.
Please indicate all underlying data sources involved in the reporting process, beginning with the
originating data systems (e.g., claims adjudication system, enrollment system) and including all
other data sources used for data collection and storage, data processing, analysis, and reporting.

Appendix E: Organizational Assessment Instrument
Table 6. Underlying Data Sources
A. Reporting Section
Part C:
Example Part C Reporting
Section

Grievances

Organization
Determinations /
Reconsiderations

Special Needs Plans
(SNPs) Care Management

B. Data Source Name
(e.g., Claims, Enrollment, Provider Information)
Claims Adjudication System ABC
Enrollment System DEF
Reporting Data Warehouse GHI
Reporting Data Warehouse JKL
Bob’s Individual Desktop Database MNO

Appendix E: Organizational Assessment Instrument
Part D:
Medication Therapy
Management Programs

Grievances

Coverage Determinations
and Redeterminations

Improving Drug Utilization
Review Controls

4.2

PROGRAMMING AND SOFTWARE

In Table 7 specify the programming languages and software used to generate the reporting
section data for reporting (e.g., MS Access, SAS, SQL, Crystal Reports, Cognos, SPSS) for
the contract(s) included in this OAI.
Table 7. Programming Software Specifications
A. Reporting Section
Part C:
Grievances
Organization Determinations/Reconsiderations
Special Needs Plans (SNPs) Care Management
Part D:
Medication Therapy Management Programs
Grievances
Coverage Determinations and Redeterminations
Improving Drug Utilization Review Controls

B. Programming Code/Software

Appendix E: Organizational Assessment Instrument

4.3

SUPPLEMENTAL QUESTIONS REGARDING
REPORTING PROCESSES

The questions below address additional information required to review the processes used to
compile and report the Part C and Part D reporting sections.
4.3.1

How does your organization ensure it meets the reporting requirements deadline for
the contract(s) included in this OAI? Who is responsible for submitting the data into
the HPMS Plan Reporting Module (i.e., responsible department, delegated entity or
first tier/downstream contractor)?

4.3.2

What is your organization’s process for correcting or revising data results that
have been returned/rejected by CMS for the contract(s) included in this OAI? Who
is responsible (i.e., responsible department, delegated entity, or first
tier/downstream contractor)?

4.3.3

Did your organization receive an outlier/data integrity notification for any of the
reporting sections that are currently undergoing data validation review (as identified
in Table 5) for the contract(s) included in this OAI? If so, the organization is
required to retrieve such notices via the Download Files page of the Monitoring Parts
C and D Reporting Website received for the reporting section and any corrective
actions taken to address the issue.

4.3.4

For the contract(s) included in this OAI, how does your organization track CMSissued changes to the Part C and/or Part D Reporting Requirements Technical
Specifications? Who is responsible (i.e., responsible department, delegated entity or
first tier/downstream contractor)? How are these changes incorporated into your
organization’s data collection and reporting systems?

4.3.5

Describe any process or quality improvement activities your organization has
implemented since the prior reporting year/period that may affect reporting section
results submitted to CMS (e.g., development of steering committees, identification
of inefficiencies) for the contract(s) included in this OAI.

Appendix E: Organizational Assessment Instrument

5 DATA VALIDATION DOCUMENTATION REQUEST
The purpose of the documentation request is to obtain documents that will assist the reviewer in
determining that data elements for each reporting section are accurately identified, calculated, and
documented. This request is applicable to all organizational processes used in creating the final
HPMS submission for the Part C and Part D reporting requirements.
The SO is responsible for ensuring that it has established mutually agreeable methods for sharing
proprietary and/or secure (PHI/PII) information with the reviewer and that the reviewer complies
with all HIPAA privacy and security requirements. Instructions for logging the information
provided by the SO are included in Section 6.
Please reference the Part C and/ or Part D Reporting Requirements Technical Specifications for
the data elements that will require supporting documentation.

5.1

REQUEST FOR PROGRAMMING CODE AND EXAMPLE OUTPUT

For the contract(s) included in this OAI, SOs should provide programming code/source code and
example output for computer programs used to calculate the data elements collected for each of
the CMS reporting sections that are currently undergoing data validation review (as identified in
Table 5). Such code may include the following:
•
•
•
•

Programming language for extracting data from the source (including any exclusion
criteria)
Joins between multiple data sources (including validation checks)
Data preparation (such as cleansing and missing data)
Manipulation to produce the final reports

The following are examples of the types of documents and files required:
•

•

•

If using SAS, SPSS, or similar software, provide the programming code, the log file that
shows the results of the compiled programming code, and the list file that shows the
output (e.g., tables and listings) generated by the programming code.
If using MS Access, SQL Server, Oracle, or other database systems, provide the code
used to generate the database query, results of the compiled query, and the output
generated by the query (e.g., saved data queries).
If using MS Excel or other spreadsheet programs, provide the Visual Basic code that
produced the spreadsheets (if applicable), and the actual workbooks with all formulas
used to calculate the values contained in each spreadsheet.

Submitted programming code should ideally be neatly structured and documented so that a third
party can easily read it and understand the programming logic. Best practice is to include
comments within the code; however, if not possible, provide documentation (e.g., work
instructions) that enables the reviewer to interpret the programming logic.

5.2

REQUEST FOR DATA DICTIONARY

Organizations should provide a data dictionary or any such documentation that provides file
layouts, field definitions, explanation of calculations, and other information about the underlying
data that are used in creating the data submission for the Part C and Part D reporting
requirements for the contract(s) included in this OAI. Appendix B of the OAI includes an
example data dictionary which should at a minimum include the field name, data type, field
description, and additional notes regarding the data field values.

5.3

REQUEST FOR ANALYSIS PLAN, REPORTING PROCESS
FLOWS, AND DIAGRAMS

Appendix E: Organizational Assessment Instrument

Organizations should provide a copy of their analysis plan, reporting process flows, diagrams,
and any other related documents. These documents should include a description or illustration of
the analysis requirements, analysis methods, and processes used for generating all reporting
section output reports for the Part C and Part D reporting requirements for the contract(s)
included in this OAI.

5.4

REQUEST FOR STANDARD OPERATING DOCUMENTS:
STANDARD OPERATING PROCEDURES (SOPS), POLICIES
AND PROCEDURES, OR OTHER WORK INSTRUCTIONS

Organizations should provide a copy of the documentation that describes their data and reporting
systems and processes for the contract(s) included in this OAI. Documents of interest include:
•
•
•
•

Work instructions, policies and procedures for the compilation, administration, and/or
submission of the Part C and Part D reporting requirements
Information Systems SOPs (e.g., system maintenance, upgrade, validation procedures)
Data Processing SOPs (e.g., data collection and storage process and frequency)
Data Archive/Restoration SOPS (e.g., disaster recovery plans)

Appendix E: Organizational Assessment Instrument

6 DATA VALIDATION DOCUMENT LOG
The Data Validation Document Log is intended to be used as inventory for all documents and
files provided by the SO as per Section 5. Organizations should complete the Document Log (see
Document Log Template in Appendix A of the OAI) in order to facilitate review of
documentation and files associated with the different stages of the reporting process.
•

•
•

•

•

Reporting Section: Reporting section for which document or file has been provided. For
example, if submitting programming code that generates the SNPs reporting section, then
indicate “SNPs” in this column. Otherwise, indicate “N/A” (note that IT system SOPs may
be N/A).
Document Name: Electronic file name of document.
Document Type: Type of document or file (e.g., work instruction, policy and procedure,
programming code, programming output/report, data dictionary/file layout, reporting
process diagram).
Reporting Stage: Stage in the reporting process to which the document applies. This usually
applies to programming code, data queries, and programming output and reports. Examples
of stages include, but are not limited to: data extract from adjudication system, data input
into internal database, output/report from internal database, data analysis to summarize data
for reporting, or final report for HPMS entry. Otherwise indicate “N/A” (note that IT SOPs
may be N/A).
Document Description: Work instructions, policies, and procedures are usually selfexplanatory. However, for programming code, SOs should include a description of the input
data sources, the applicable stage in the reporting process, the intended output, and name of
the output file. For data dictionaries/file layouts, indicate the name of the applicable
database and source tables containing the data fields. For screen shots, process flows, and
diagrams provide the relevant description of the indicated charts, diagrams, and process
flows.

APPENDIX A: DOCUMENT LOG TEMPLATE
To the extent possible, list the documents in a logically-ordered fashion so the reviewer can identify sets of documents relative to the reporting
process stage for each reporting section.
A. Reporting
Section

B. Document File Name

C. Document Type

D. Reporting Stage

E. Document Description

Example: Grievances

Grievances_SOP.doc

SOP

N/A

SOP documents the methods used to gather, analyze,
and report Grievances data according to CMS reporting
requirements

Example: Grievances

Grievances_Data_Load.SAS

SAS Program Code

Load grievance data into
Data Warehouse

SAS program extracts data from adjudication
system and loads into internal data warehouse

Example: Grievances

Grievances_Summary.SAS

SAS Program Code

Summarizes data for HPMS
reporting

SAS program cleans and summarizes data for entry
into HPMS

Example: Grievances

Grievances Summary Report.xls Example Report Output

Summarized data report
example for HPMS entry

Example of report output from the Grievances
Summary. SAS program for HPMS entry

APPENDIX B: DATA DICTIONARY EXAMPLE
Please refer to the following as an example of the information required in a Data Dictionary.
A. Field Name

B. Type

C. Description

D. Additional Notes

Example: Redetermination ID

Long Integer

Unique ID for each case

Example: Date Created

Date

Date the case was created

Example: Redetermination Status

Integer

Status of the case

1=Open; 2=Closed

Example: Redetermination Outcome

Integer

Outcome of the case

1=Overturned; 2=Withdrawn; 3=Dismissed;
4=Upheld