Analysis of Alternatives

PERS Analysis of Alternatives

Analysis of Alternatives

Program Evaluation Reporting system

PERS

Karna, LLC.

11030 Jones Bridge Road, #202

Alpharetta, GA 30022

VERSION 1.0

APPROVED: 10/30/13

Table of Contents

Introduction 2

PERS High-Level Requirements 2

Evaluation Criteria and Weighting 2

Grading 3

Alternatives 4

Alternative 1 - Program Solutions 4

Alternative 2 - Evaluation Web 5

Alternative 3 - WISEWOMAN 5

Evaluation Matrix 6

Conclusions 6

Sponsor Acceptance 7

Introduction

The Analysis of Alternatives (AoA) is an analytic comparison different systems/applications that could be used in lieu of developing an new Performance Evaluation and Reporting System for the Division of Adolescents and School Health. The AoA will document each alternative’s current capabilities, its availability, and the readiness of the Development/Maintenance team to support DASH and its grantees.

In addition, in the AoA, a comparative analysis based of DASH’s requirements will be performed. DASH’s leadership will provide evaluation criteria and weight for each based on their level of importance.

PERS High-Level Requirements

PERS is a Program evaluation and monitoring system that will be used by DASH to organize, plan, and track grantees (funded organizations) activities and events organized to meet performance and process measures established by CDC. PERS will host grantees records, their affiliated State Education Agencies (SEA), Local Education Agencies (LEA), and Non-Governmental Organization (NGO), which school districts they support, which activities they perform and which event they host. PERS will serve as a real time tracking system where those organizations can enter reporting information that will be readily available to DASH. DASH will use the data on an ongoing basis for reporting and to provide support and feedback to the funded organizations.

The goal of the PERS is to aggregate data at a high level from data entered in different manners and at different levels. Some data must follow a chronologic logic (Post, Delay-post), some should be collected at the event level, at the Organization level, at the child Organization level,…

Evaluation Criteria and Weighting

This list of the Evaluation Criteria is defined based on the most important PERS requirements. They are categorized as PERS System Functions, System availability, and System and User Support.

PERS System Functions

Can the system accommodate NGO, SEA and LEA structures based on DASH needs?

Can the system accommodate creation of Process Measures based on DASH needs?

Can the system accommodate creation of Performance Measures based on DASH needs?

Does the system allow Organizations to report data based on their structures and their related measures?

Does the system allow creation of different roles based on organizations?

Does the system provide reporting options to aggregate data at the organization level based on performance and process measures?

Does the system allow for generation of SAS Data Sets for analysis?

System Availability

Will the system be fully operational as of 08/01/2014 including ALL DASH's requirements?

Does the program have the bandwidth to support DASH needs?

System and User Support

Will the Program provide full support for operation and maintenance to DASH?

Will the Program provide full support for to all DASH CDC and Non-CDC users ?

The Weighting is defined as follow:

Criteria Importance	Weight	Comments
Critical	Yes/No	If an Alternative does not meet a critical criterion, it will automatically be disqualified.
Very Important	20
Important	15
Somehow Important	10

The Weighted Evaluation Criteria is defined as follow:

PERS System Functions	Weighting	Criteria importance
Can the system accommodate NGO, SEA and LEA structures based on DASH needs?	20	Very Important
Can the system accommodate creation of Process Measures based on DASH needs?	20	Very Important
Can the system accommodate creation of Performance Measures based on DASH needs?	20	Very Important
Does the system allow Organizations to report data based on their structures and their related measures?	20	Very Important
Does the system allow creation of different roles based on organizations?	15	Important
Does the system provide reporting options to aggregate data at the organization level based on performance and process measures?	15	Important
Does the system allow for generation of SAS Data Sets for analysis?	10	Somewhat Important
System Availability
Will the system be fully operational as of 08/01/2014 including ALL DASH's requirements?	Yes/No	Critical
Does the program have the bandwidth to support DASH needs?	Yes/No	Critical
System and User Support
Will the Program provide full support for operation and maintenance to DASH?	10	Somewhat Important
Will the Program provide full support for to all DASH CDC and Non-CDC users?	10	Somewhat Important

Grading

For each criteria, each alternative will be reviewed and graded 0 (Zero), 5 (five), or 10 (ten).

Grade	Capability
0	Alternative could be modified to include the function. Application recoding is required to accommodate the requirement as needed by DASH.
5	Alternative could be configured to include the function. The Alternative can be modified to accommodate the requirement as needed by DASH.
10	Alternative has the function readily available to be used. The Alternative does accommodate the requirement out of the box as needed by DASH.

Alternatives

Alternative 1 - Program Solutions

Program Solutions is a system under development by the HHS Center for Excellence. The project has been delayed multiple times and more recently after the Government shutdown. The Center for Excellence could evaluate what can be done for DASH only after a complete requirements analysis is completed. It’s very uncertain that Programs Solutions will be ready for use by DASH by August 1^st, 2014.

Based on information received from Gideon Slifkin, a contractor working on the Program Solutions integration, here is the status of Program Solutions modules development:

1. OLDC (the current system for online surveys) – available now

2. Basic New Forms Framework (enables simple surveys) –available in the next couple of months

3. Advanced New Forms Framework (enables more complex surveys) – supposed to be available next March, but might change date

4. Advanced Reporting (using IBM COGNOS Reporting Tools) – available now

5. Program Management Modules (Workplan, Budget, Review Tools, Progress Report Tools, Technical Assistance Tracking Tools etc) -2015/2016 (exact dates TBD)

The modules that are available may be good. However, based on PERS requirements, the needed module will be available only in 2015-2016 (“Program Management Modules (Workplan, Budget, Review Tools, Progress Report Tools, Technical Assistance Tracking Tools etc.) -2015/2016 (exact dates TBD)”).

• Work Plan: For PERS, planning for Process and Performance Measures per Organization and the needed follow up as they progress during a year would correspond to Program Solutions’ Workplan.

• The PERS CDC users’ module would be the “Review Tools”, it will help DASH access up-to-date data entered by grantees for review.

• The reports envisioned in PERS will help DASH pull information by Organization, State, District… this is “Progress Report Tools” in Program Solutions.

• For Technical Assistance, PERS will help provide information uploaded by the grantees and our resources will help with the analysis; in the Program Solutions, it’s the TA Tracking Tools.

All these tools are not yet available in Program Solutions as they are being developed. They will not be completed until 2015 or 2016 timeframe.

Alternative 2 - Evaluation Web

Evaluation Web is proprietary system that DHAP leases from Luther Consulting. Evaluation Web can extract data from Partner Services Web another Luther Consulting Application that some Health Departments use.

This system is built to capture data on HIV Testing and Behavioral Risk information.

Although there are “Sites” (Testing locations) in Evaluation Web, PERS description of sites is totally different and the type of relationships defined between sites does not exist in EW. They associate Sites to Agencies. They also have Programs that are associated to sites and users. In the documentation they talk about adding Intervention to Agencies as well.

Evaluation Web has a data submission function through file upload. Template files are available online; they are tailored for HIV testing needs.

When it comes to Measures, Evaluation Web does not provide measures evaluation for sites; this is the main function of the PERS.

Evaluation Web allows management of Agency funding which PERS will not include.

If we were to use EW, there will be a lot of programming and a lost of customization to accommodate PERS requirements.

Alternative 3 - WISEWOMAN

This Well Integrated Screening and Evaluation for Women Across the Nation (WISEWOMAN) Program funds 21 grantees across the United States to improve cardiovascular health among low-income, underinsured, and uninsured women ages 40 to 64. Grantees are required to collect and report MDEs as part of standardized data reporting for the WISEWOMAN Program. MDEs are used by CDC and its grantees to describe, monitor, and assess progress and performance.

The application integrates 5 Performance Goals. Submission of data is done via Template files (ASCII Text, Excel, Word) available for download from the site. When the data is uploaded, the application parses through the files and saves the data in SQL. The uploaded files are stored on the Server. Reports are accessible based on user roles. SAS data analysis is done from raw data; the application does not generate SAS datasets.

The system is developed and maintained by SciMetrika (Contractor). The application is hosted on SciMetrika secured server. It has been in production for 4 years.

Evaluation Matrix

Conclusions

Based on the Evaluation Matrix, none of the three alternatives will be able to accommodate DASH’s requirements:

• Program Solutions was disqualified based on its availability. DASH requires that the system be up and running, usable by the Funded Agencies on August 1^st, 2014. Program Solutions is under development and will not be delivered by that date.

• There are several disadvantages to using Evaluation Web for DASH Evaluation program needs including that the system is proprietary, the users of the system are health departments and CBOs rather than education agencies, and there would need to be significant new development to meet our needs because of the significant differences in the kinds of information being collected.

• The WISEWOMAN application has some similarities with PERS. However, it is very limited in available functions that could be re-used in the context of PERS. PERS has a more complex hierarchy of Organizations with different relationships between them. If the WISEWOMAN application was to be used, at least 80% of programming will be necessary to meet PERS requirements.

Sponsor Acceptance

PERS Project Sponsor	PERS Business Steward
Leah Robin Lead Health Scientist NCHHSTP/DASH	Elana Morris Health Scientist NCHHSTP/DASH
Date:	Date:
Signature:	Signature:

2