Analysis of Alternatives

“Promoting Adolescent Health through School-Based HIV Prevention”

OMB #0920-1275 Extension

Attachment 10 Analysis of Alternatives

INTRODUCTION

The Analysis of Alternatives (AoA) is an analytic comparison of different systems/applications that could be used in lieu of enhancing the existing Performance Evaluation and Reporting System for the Division of Adolescents and School Health (PERS). The AoA documents each alternative’s current capabilities, its availability, and the readiness of the Development/Maintenance team to support DASH and its grantees.

In addition, the AoA includes a comparative analysis based on DASH’s requirements. DASH’s leadership will provide evaluation criteria and weight for each based on their level of importance.

PERS HIGH-LEVEL REQUIREMENTS

PERS is a program evaluation and monitoring system that uses web-based questionnaires to collect, organize, and track DASH activities implemented by funded Local Education Agencies (LEAs), commonly known as school districts, and their priority schools. The activities being tracked support the achievement of progress and outcome measures established by CDC. LEAs will enter LEA-level data and data about the priority schools in their district. PERS will host LEA records, including records on the activities they perform. PERS will serve as a real time tracking system where LEAs can enter reporting information that will be readily available to DASH. DASH will use the data on an ongoing basis for reporting and to provide support and feedback to the funded organizations.

The goal of PERS is to aggregate data at a high level from data entered in different formats and at different levels (LEA and Priority School). Some data must follow a chronological logic (Post, Delay-post), some should be collected at the event level, at the Organization level, at the child Organization level.

EVALUATION CRITERIA

This list of the Evaluation Criteria is defined based on the most important PERS requirements. They are categorized as PERS System Functions, System Availability, and System and User Support.

PERS System Functions

• Can the system accommodate LEA structures based on DASH needs?

• Can the system accommodate creation of Process Measures based on DASH needs?

• Can the system accommodate creation of Outcome Measures based on DASH needs?

• Does the system allow LEAs to report data based on their structures and their related measures?

• Does the system allow creation of different roles based on organizations?

• Does the system provide reporting options to aggregate data at the organization level based on process and outcome measures?

• Does the system allow for generation of SAS Data Sets for analysis?

System Availability

• Does the program have the bandwidth to support DASH needs?

System and User Support

• Will the Program provide full support for operation and maintenance to DASH?

• Will the Program provide full support for all DASH CDC and Non-CDC users?

The Weighting is defined as follow:

Criteria Importance	Weight	Comments
Critical	Yes/No	If an Alternative does not meet a critical criterion, it will automatically be disqualified.
Very Important	20
Important	15
Somehow Important	10

The Weighted Evaluation Criteria is defined as follow:

PERS System Functions	Weighting	Criteria importance
Can the system accommodate LEA structures based on DASH needs?	20	Very Important
Can the system accommodate creation of Process Measures based on DASH needs?	20	Very Important
Can the system accommodate creation of Outcome Measures based on DASH needs?	20	Very Important
Does the system allow LEAs to report data based on their structures and their related measures?	20	Very Important
Does the system allow creation of different roles based on organizations?	15	Important
Does the system provide reporting options to aggregate data at the organization level based on performance and process measures?	15	Important
Does the system allow for generation of SAS Data Sets for analysis?	10	Somewhat Important
System Availability
Will the system be fully operational as of 11/01/2019 including ALL DASH's requirements?	Yes/No	Critical
Does the program have the bandwidth to support DASH needs?	Yes/No	Critical
System and User Support
Will the Program provide full support for operation and maintenance to DASH?	10	Somewhat Important
Will the Program provide full support for to all DASH CDC and Non-CDC users?	10	Somewhat Important

Grading

For each criteria, each alternative will be reviewed and graded 0 (Zero), 5 (five), or 10 (ten).

Grade	Capability
0	Alternative could be modified to include the function. Application recoding is required to accommodate the requirement as needed by DASH.
5	Alternative could be configured to include the function. The Alternative can be modified to accommodate the requirement as needed by DASH.
10	Alternative has the function readily available to be used. The Alternative does accommodate the requirement out of the box as needed by DASH.

ALTERNATIVES

Alternative 1 – WISEWOMAN

This Well Integrated Screening and Evaluation for Women Across the Nation (WISEWOMAN) Program funds 21 grantees across the United States to improve cardiovascular health among low-income, underinsured, and uninsured women ages 40 to 64. Grantees are required to collect and report MDEs as part of standardized data reporting for the WISEWOMAN Program. MDEs are used by CDC and its grantees to describe, monitor, and assess progress and performance.

The application integrates 5 Performance Goals. Submission of data is done via Template files (ASCII Text, Excel, Word) available for download from the site. When the data is uploaded, the application parses through the files and saves the data in SQL. The uploaded files are stored on the Server. Reports are accessible based on user roles. SAS data analysis is done from raw data; the application does not generate SAS datasets.

The system is developed and maintained by SciMetrika (Contractor). The application is hosted on SciMetrika secured server. It has been in production for 7 years.

Alternative 2 – REDCap

REDCap is a browser-based web application for building and managing online surveys and databases, specifically geared to support data capture for research studies. It is widely used in the academic research community: the REDCap Consortium, which distributes the software, is a collaborative, international network of more than 2400 institutional partners in over 115 countries.

REDCap requires that its “data collection instruments” (questionnaires) be built using the tools on its website. With REDCap, one or more of these instruments can be combined into projects, which are web pages for entering data. Each instrument has its own access controls, allowing the access to be tightly controlled by the administrator.

Reports (data queries) can be built using the report designer; they can be saved in the project, and the results can be exported in different formats, including Excel and SAS.

E
VALUATION MATRIX

CONCLUSIONS

Based on the Evaluation Matrix, neither of the alternatives will be able to accommodate DASH’s requirements:

• The WISEWOMAN application has some similarities with PERS. However, it is very limited in available functions that could be re-used in the context of PERS. PERS has a more complex hierarchy of Organizations with different relationships between them. If the WISEWOMAN application was to be used, at least 80% of the programming will be necessary to meet PERS requirements.

• The REDCap application is a general purpose survey tool commonly used for clinical surveys. While flexible and relatively easy to set up for simple projects, those that are more complex, such as those required for PERS, can quickly become difficult to maintain and test. The complex hierarchy of Organizations and relationships needed by PERS would also be difficult to implement in REDCap and would require additional work. In addition, the underlying response data is not intended to be directly accessible; instead queries are used to extract the results. This would make the data cleaning that PERS requires much more difficult.