CMS-10305 Appendix 2: Organizational Assessment Instrument

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual Appendix 2: Organizational Assessment Instrument Version 7.0

Prepared by:

Centers for Medicare & Medicaid Services Center for Medicare

Medicare Drug Benefit and C & D Data Group Last Updated: October 2016

Table of Contents

1. OBJECTIVES 1

2. INSTRUCTIONS 1

   1. Instructions for Data Validation Contractor 1

   2. Instructions for Organization 1

3. GENERAL QUESTIONS 2

   1. Organization Information 2

   2. Contact Information 3

   3. Part C and Part D REPORTING SECTIONs Undergoing Validation 3

4. UNDERLYING DATA SOURCES AND REPORTING PROCESSES 4

   1. Underlying Data Sources 4

   2. Programming and Software 6

   3. Supplemental Questions Regarding Reporting Processes 7

5. DATA VALIDATION DOCUMENTATION REQUEST 8

   1. Request for Programming Code and Example Output 8

   2. Request for Data Dictionary 8

   3. Request for Analysis Plan, Reporting Process Flows, and Diagrams 9

   4. Request for Standard Operating Documents: Standard Operating Procedures (SOPs),

Policies and Procedures, or Other Work Instructions 9

6. DATA VALIDATION DOCUMENT LOG 9

APPENDIX A: DOCUMENT LOG TEMPLATE 11

APPENDIX B: DATA DICTIONARY EXAMPLE 12

List of Tables

Table 1. Timeline of OAI Activities 2

Table 2. Organization Information 3

Table 3. Part C Contact Information 3

Table 4. Part D Contact Information 3

Table 5. Reporting Sections Undergoing Validation 4

Table 6. Underlying Data Sources 5

Table 7. Programming Software Specifications 6

1. OBJECTIVES

CMS is providing this Organizational Assessment Instrument (OAI) as a tool for data validation contractors (reviewers) to understand organizations’ reporting processes more completely and to request documentation that will be evaluated during the review process. The information collected in this OAI will help prepare reviewers and will reduce resources required for the site visit portion of the review. CMS requires that organizations complete the OAI to add efficiencies to the review process.

2. INSTRUCTIONS

   1. INSTRUCTIONS FOR DATA VALIDATION CONTRACTOR

Prior to the start of the data validation review period, the organization must begin completion of the OAI. It is recommended that each section of the OAI be completed prior to the site visit; however, if preferred, the reviewer may determine an alternative sequence (e.g., the reviewer may request that an organization complete Sections 3 and 4 prior to the site visit, but complete Sections 5 and 6 following the site visit). The OAI is designed to capture preliminary information about the organization’s processes for collecting and reporting data per the CMS reporting requirements. The reviewer must analyze the OAI prior to the site visit and follow-up on any incomplete or ambiguous responses during the site visit portion of the review. The OAI must be electronically distributed to the organization undergoing a review. Following the organization’s completion of this document, the reviewer must attach a completed copy of the OAI in a file of all data validation review work papers that will be shared with the organization.

2. INSTRUCTIONS FOR ORGANIZATION

Organizations must complete each section of the OAI in advance of the data validation review period, or according to the set timeline of the reviewer. The organization should complete the OAI and provide documentation to the reviewer as early as possible at the start of the data validation review period so that the reviewer can begin data validation on April 1. All documentation and responses to questions should reflect the organization’s systems and processes that were in place during the reporting period(s) undergoing the data validation review. For organizations with multiple contracts, the organizations should complete only one OAI. If the information provided in the OAI varies by contract, the organization should specify the differences within the OAI to allow the reviewer to more easily identify differences that may impact reporting section calculations or reporting for a given contract.

The organization must submit the OAI, documentation, and any additional information to the reviewer electronically. The organization is responsible for ensuring that it has established mutually agreeable methods for sharing proprietary and/or secure (PHI/PII) information with the reviewer and that the reviewer complies with all HIPAA privacy and security requirements.

The completed OAI and any additional information provided as a result of this request will be assessed by the reviewer. If your organization has any questions while completing the OAI, contact the reviewer. Each stage of the data validation review should entail a collaborative effort between the organization and reviewer. An overview of the timeline related to OAI activities is outlined in Table 1.

Table 1. Timeline of OAI Activities

DV Phase	Step	Responsible Party	Data Validation Activities	Timeline
Performing Data Validation Activities	1	SO	Complete Organizational Assessment Instrument (OAI) and provide appropriate documentation to selected reviewer per the OAI’s documentation request	No earlier than 30 days prior to the start of the DV cycle on April 1.
	2	DVC, SO	Analyze OAI	Allow two weeks after receipt.
	3	DVC, SO	Prepare for site visit (site visit agenda, resource needs, and logistics)	Early April
	4	DVC, SO	Conduct on-site review (convene entrance conference, conduct interviews with SO staff, observe SO’s reporting processes, and obtain census and/or sample files)	Early April (allow for up to 1 week)
	5	DVC	Request additional documents following site visit (if applicable)	Mid/Late April

3. GENERAL QUESTIONS

The information gathered below will provide a better understanding of the scope for the organization’s data validation review, including which contract(s) will be reviewed and which Part C and/or Part D reporting sections the organization is reporting for validation.

1. ORGANIZATION INFORMATION

Complete Table 2, indicating each Medicare contract that your organization held during the reporting period(s) undergoing the data validation review. For the “Contract Type” field, select from the following list:

• CCPP

• FFS

• MSA

• 1876 Cost

• Employer/Union Direct Contract

• PDP

• Demo

Also indicate whether the contract includes the Part C and/or Part D benefit and provide the number of plan benefit packages (PBP) associated with each contract. Finally, indicate if any of the PBPs associated with the contract are Special Needs Plans or Employer/Union “800 Series” plans. The organization may add rows to this table as necessary, but should not manipulate the columns.

Table 2. Organization Information

Parent Organization Name:
CMS Contract Number	Contract Type	Includes Part C? (Y/N)	Includes Part D? (Y/N)	No. of Plan Benefit Packages	Includes SNP PBP(s)? (Y/.N)	Includes Employer/Union “800 Series” PBP(s)? (Y/N)
Example: Contract 123	PFFS	Y	Y	3	N	N
Example: Contract 123	CCP	Y	Y	1	Y	N
[add rows as required]

2. CONTACT INFORMATION

Complete Table 3and Table 4 indicating your organizations primary and secondary points of contact responsible for the Part C and Part D reporting requirements data validation review for each contract included in this OAI.

Table 3. Part C Contact Information

Primary Part C Point of Contact	Secondary Part C Point of Contact
Name:	Name:
Title:	Title:
Company:	Company:
Address:	Address:
City, State, Zip:	City, State, Zip:
Telephone:	Telephone:
Fax:	Fax:
Email:	Email:

Table 4. Part D Contact Information

Primary Part D Point of Contact	Secondary Part D Point of Contact
Name:	Name:
Title:	Title:
Company:	Company:
Address:	Address:
City, State, Zip:	City, State, Zip:
Telephone:	Telephone:
Fax:	Fax:
Email:	Email:

3. PART C AND PART D REPORTING SECTIONS UNDERGOING VALIDATION

Complete Table 5 for the contract(s) included in this OAI, indicating which of the Part C and/or Part D reporting sections your organization has submitted for data validation review, the applicable contract numbers (Column B), and whether your organization is able to report on all required data elements per the CMS Part C and Part D Reporting Requirements Technical Specifications (Column C).

Table 5. Reporting Sections Undergoing Validation

A. Reporting Section	B. CMS Contract Number(s)	C. Are all required data elements captured by your internal data system(s)? (Yes/No)	D. If the answer to Column C. is no, please indicate which delegated entities’ data systems contain the data elements
Part C1:
Grievances
Organization Determinations/Reconsiderations
Plan Oversight of Agents
Special Needs Plans (SNPs) Care Management
Part D2:
Medication Therapy Management Programs
Grievances
Coverage Determinations and Redeterminations
Plan Oversight of Agents

¹ The following Part C reporting sections are required for CMS reporting in 2014 but are not included in the 2015 data validation review: Enrollment/Disenrollment, Employer Group Plan Sponsors, PFFS Plan Enrollment Verification Calls, and PFFS Provider Payment Dispute Resolution Process.

²The following Part D reporting sections are required for CMS reporting in 2014 but are not included in the 2014 data validation review: Enrollment/Disenrollment, Retail, Home Infusion, and Long-Term Care Pharmacy Access, Prompt Payment by Part D Sponsors, Fraud, Waste, and Abuse Compliance Programs, and Employer/ Union- Sponsored Group Health Plan Sponsors.

4. UNDERLYING DATA SOURCES AND REPORTING PROCESSES

The questions below address the underlying data sources and reporting processes used to produce the Part C and Part D reporting sections.

1. UNDERLYING DATA SOURCES

Complete Table 6 for the contract(s) included in this OAI, indicating the name of the data source(s) used to generate each Part C and Part D reporting section (Column B). If additional rows are required to list the data sources for a given reporting section, insert new rows into the table.

Please indicate all underlying data sources involved in the reporting process, beginning with the originating data systems (e.g., claims adjudication system, enrollment system) and including all other data sources used for data collection and storage, data processing, analysis, and reporting.

Table 6. Underlying Data Sources

A. Reporting Section	B. Data Source Name (e.g., Claims, Enrollment, Provider Information)
Part C:
Example Part C Reporting Section	Claims Adjudication System ABC
	Enrollment System DEF
	Reporting Data Warehouse GHI
	Reporting Data Warehouse JKL
	Bob’s Individual Desktop Database MNO
Grievances
Organization Determinations / Reconsiderations
Plan Oversight Of Agents- Part C
Special Needs Plans (SNPs) Care Management

Part D:
Medication Therapy Management Programs
Grievances
Coverage Determinations and Redeterminations (1.A- 1.K)
Coverage Determinations and Redeterminations (1.L1.S, 2.A-2.G, 3.A-3.B)
Plan Oversight of Agents

2. PROGRAMMING AND SOFTWARE

In Table 7 specify the programming languages and software used to generate the reporting section data for reporting (e.g., MS Access, SAS, SQL, Crystal Reports, Cognos, SPSS) for the contract(s) included in this OAI.

Table 7. Programming Software Specifications

A. Reporting Section	B. Programming Code/Software
Part C:
Grievances
Organization Determinations/Reconsiderations
Plan Oversight of Agents
Special Needs Plans (SNPs) Care Management

Part D:
Medication Therapy Management Programs
Grievances
Coverage Determinations and Redeterminations
Plan Oversight of Agents

3. SUPPLEMENTAL QUESTIONS REGARDING REPORTING PROCESSES

The questions below address additional information required to review the processes used to compile and report the Part C and Part D reporting sections.

1. How does your organization ensure it meets the reporting requirements deadline for the contract(s) included in this OAI? Who is responsible for submitting the data into the HPMS Plan Reporting Module (i.e., responsible department, delegated entity or first tier/downstream contractor)?

2. What is your organization’s process for correcting or revising data results that have been returned/rejected by CMS for the contract(s) included in this OAI? Who is responsible (i.e., responsible department, delegated entity, or first tier/downstream contractor)?

3. Did your organization receive any outlier notifications from CMS in the previous reporting year for any of the reporting sections that are currently undergoing data validation review (as identified in Table 5) for the contract(s) included in this OAI? If so, please describe any such notices received for the reporting section and any corrective actions taken to address the issue.

4. For the contract(s) included in this OAI, how does your organization track CMS-issued changes to the Part C and/or Part D Reporting Requirements Technical Specifications? Who is responsible (i.e., responsible department, delegated entity or first tier/downstream contractor)? How are these changes incorporated into your organization’s data collection and reporting systems?

5. Describe any process or quality improvement activities your organization has implemented since the prior reporting year/period that may affect reporting section results submitted to CMS (e.g., development of steering committees, identification of inefficiencies) for the contract(s) included in this OAI.

5. DATA VALIDATION DOCUMENTATION REQUEST

The purpose of the documentation request is to obtain documents that will assist the reviewer in determining that data elements for each reporting section are accurately identified, calculated, and documented. This request is applicable to all organizational processes used in creating the final HPMS submission for the Part C and Part D reporting requirements.

The organization is responsible for ensuring that it has established mutually agreeable methods for sharing proprietary and/or secure (PHI/PII) information with the reviewer and that the reviewer complies with all HIPAA privacy and security requirements. Instructions for logging the information provided by the organization are included in Section 6.

Please reference the Part C and/ or Part D Reporting Requirements Technical Specifications for the data elements that will require supporting documentation.

1. REQUEST FOR PROGRAMMING CODE AND EXAMPLE OUTPUT

For the contract(s) included in this OAI, organizations should provide programming code/source code and example output for computer programs used to calculate the data elements collected for each of the CMS reporting sections that are currently undergoing data validation review (as identified in Table 5). Such code may include the following:

• Programming language for extracting data from the source (including any exclusion criteria)

• Joins between multiple data sources (including validation checks)

• Data preparation (such as cleansing and missing data)

• Manipulation to produce the final reports

The following are examples of the types of documents and files required:

• If using SAS, SPSS, or similar software, provide the programming code, the log file that shows the results of the compiled programming code, and the list file that shows the output (e.g., tables and listings) generated by the programming code.

• If using MS Access, SQL Server, Oracle, or other database systems, provide the code used to generate the database query, results of the compiled query, and the output generated by the query (e.g., saved data queries).

• If using MS Excel or other spreadsheet programs, provide the Visual Basic code that produced the spreadsheets (if applicable), and the actual workbooks with all formulas used to calculate the values contained in each spreadsheet.

Submitted programming code should ideally be neatly structured and documented so that a third party can easily read it and understand the programming logic. Best practice is to include comments within the code; however, if not possible, provide documentation (e.g., work instructions) that enables the reviewer to interpret the programming logic.

2. REQUEST FOR DATA DICTIONARY

Organizations should provide a data dictionary or any such documentation that provides file layouts, field definitions, explanation of calculations, and other information about the underlying data that are used in creating the data submission for the Part C and Part D reporting requirements for the contract(s) included in this OAI. Appendix B of the OAI includes an example data dictionary which should at a minimum include the field name, data type, field description, and additional notes regarding the data field values.

3. REQUEST FOR ANALYSIS PLAN, REPORTING PROCESS FLOWS, AND DIAGRAMS

Organizations should provide a copy of their analysis plan, reporting process flows, diagrams, and any other related documents. These documents should include a description or illustration of the analysis requirements, analysis methods, and processes used for generating all reporting section output reports for the Part C and Part D reporting requirements for the contract(s) included in this OAI.

4. REQUEST FOR STANDARD OPERATING DOCUMENTS:

STANDARD OPERATING PROCEDURES (SOPS), POLICIES AND PROCEDURES, OR OTHER WORK INSTRUCTIONS

Organizations should provide a copy of the documentation that describes their data and reporting systems and processes for the contract(s) included in this OAI. Documents of interest include:

• Work instructions, policies and procedures for the compilation, administration, and/or submission of the Part C and Part D reporting requirements

• Information Systems SOPs (e.g., system maintenance, upgrade, validation procedures)

• Data Processing SOPs (e.g., data collection and storage process and frequency)

• Data Archive/Restoration SOPS (e.g., disaster recovery plans)

6. DATA VALIDATION DOCUMENT LOG

The Data Validation Document Log is intended to be used as inventory for all documents and files provided by the organization as per Section 5. Organizations should complete the Document Log (see Document Log Template in Appendix A of the OAI) in order to facilitate review of documentation and files associated with the different stages of the reporting process.

• Reporting Section: Reporting section for which document or file has been provided. For example, if submitting programming code that generates the SRAEs reporting section, then indicate “SRAEs” in this column. Otherwise, indicate “N/A” (note that IT system SOPs may be N/A).

• Document Name: Electronic file name of document.

• Document Type: Type of document or file (e.g., work instruction, policy and procedure, programming code, programming output/report, data dictionary/file layout, reporting process diagram).

• Reporting Stage: Stage in the reporting process to which the document applies. This usually applies to programming code, data queries, and programming output and reports. Examples of stages include, but are not limited to: data extract from adjudication system, data input into internal database, output/report from internal database, data analysis to summarize data for reporting, or final report for HPMS entry. Otherwise indicate “N/A” (note that IT SOPs may be N/A).

• Document Description: Work instructions, policies, and procedures are usually self- explanatory. However, for programming code, organizations should include a description of the input data sources, the applicable stage in the reporting process, the intended output, and name of the output file. For data dictionaries/file layouts, indicate the name of the applicable database and source tables containing the data fields. For screen shots, process flows, and diagrams provide the relevant description of the indicated charts, diagrams, and process flows.

APPENDIX A: DOCUMENT LOG TEMPLATE

To the extent possible, list the documents in a logically-ordered fashion so the reviewer can identify sets of documents relative to the reporting process stage for each reporting section.

A. Reporting Section	B. Document File Name	C. Document Type	D. Reporting Stage	E. Document Description
Example: POA	POA_SOP.doc	SOP	N/A	SOP documents the methods used to gather, analyze, and report POA data according to CMS reporting requirements
Example: POA	POA_Data_Load.SAS	SAS Program Code	Load SRAE data into Data Warehouse	SAS program extracts data from adjudication system and loads into internal data warehouse
Example: POA	POA_Summary.SAS	SAS Program Code	Summarizes data for HPMS reporting	SAS program cleans and summarizes data for entry into HPMS
Example: POA	POA Summary Report.xls	Example Report Output	Summarized data report example for HPMS entry	Example of report output from the POA_Summary.SAS program for HPMS entry

APPENDIX B: DATA DICTIONARY EXAMPLE

Please refer to the following as an example of the information required in a Data Dictionary.

A. Field Name	B. Type	C. Description	D. Additional Notes
Example: RedeterminationID	Long Integer	Unique ID for each case
Example: DateCreated	Date	Date the case was created
Example: RedeterminationStatus	Integer	Status of the case	1=Open; 2=Closed
Example: Redetermination Outcome	Integer	Outcome of the case	1=Overturned; 2=Withdrawn; 3=Dismissed; 4=Upheld

1

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0938-1115. The time required to complete this information collection is estimated to average 327.5 hours per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.

Expiration date 12/31/2018