Training Module 3

Module 3: Red Carpet Entry (RCE)—Measuring
Success

Module Overview
1. Learning Objectives
2. Monitoring and
Evaluation
3. RCE Monitoring
4. Outcomes Evaluation
5. The RCE Report Card
6. Next Steps

Learning Objectives

After taking this training module, you should be
able to…
1. Understand the difference between monitoring and evaluation
2. Identify the key monitoring and evaluation metrics for RCE
3. Understand the types of informal client feedback that can help
quality improvement
4. Identify the evaluation outcomes data to be collected from your
electronic health record (EHR) system(s)
5. Understand the RCE Report Card metrics and how to use the Report
Card for quality improvement
VIDEO TRAINING MODULES

4

Who Should Take This Training Module?

RCE
Champion

RCE
Concierge

Clinic Data
Manager

VIDEO TRAINING MODULES

5

Key Staff

Key Staff for Monitoring and Evaluation
Data Manager

Clinic Champion

• Extracts data for consenting
patients from the clinic EHR
through queries or manually
input into the RCE Access
Database

• Oversees preparation

• Completes quality
assurance checks on RCE
and clinical data

• Supports RCE staff

• Sends data to RTI according
to schedule
• Works with RTI to resolve
any data issues identified by
RTI

• Secures leadership buy-in
• Reviews RCE procedures
• Monitors RCE fidelity
• Troubleshoots problems
with the RCE team

RCE Concierge
• Receives referrals from CTR
Counselor and self-referrals
• Facilitates access to medical
care and wraparound
services through RCE Visit
• Conducts outreach to
clients who missed their RCE
appointment

VIDEO TRAINING MODULES

7

Key Terms

RCE Visit
During a visit to the RCE clinic, the client receives the following services
within 72 hours of referral:
• Introduction to the RCE Concierge
• Brief intake
• Assessment for wraparound services
• Financial assistance/benefits discussion
• Introduction to their HIV Specialist
• Laboratory tests
• Visit wrap up
VIDEO TRAINING MODULES

9

RCE Outreach and Reengagement
A formal structure for outreach and engagement—initiated following a
client’s loss of contact or missed RCE Visit:
• The Concierge makes two attempts to warmly engage the client,
address barriers to care, and schedule or reschedule the RCE Visit.
• If attempts are unsuccessful, the Concierge contacts the referring
CTR Counselor—except in cases of self-referral—to inform them of
the client’s loss of contact/absence.
• The CTR Counselor makes at least two attempts to warmly engage
the client, address barriers to care, and schedule or reschedule the
RCE Visit.
VIDEO TRAINING MODULES

10

Monitoring and Evaluation

Measuring Success Helps You Answer
Important Questions
1. Is RCE being implemented as intended?
2. Is RCE meeting clients’ needs?
3. Is RCE affecting client outcomes?
4. Are there areas for improvement?

VIDEO TRAINING MODULES

12

Monitoring and Evaluation
Monitoring is the regular collection of information about project activities
that allows project managers to:
• Identify whether things are going as planned
• Identify and solve problems quickly
Evaluation asks whether the objectives, aims, and goals of the project are
being achieved.

Source: https://www.who.int/hiv/topics/vct/sw_toolkit/monitoring_and_evaluation/en/

VIDEO TRAINING MODULES

13

Pop Quiz
What key questions does monitoring and evaluation help you answer?
Write down your answer.

1.

3.

2.

4.

VIDEO TRAINING MODULES

14

Check Your Answer
What key questions does monitoring and evaluation help you answer?
Answers
1. Is RCE being implemented as
intended?

3. Is RCE affecting client outcomes?

2. Is RCE meeting clients’ needs?

4. Are there areas for improvement?

VIDEO TRAINING MODULES

15

RCE Monitoring

Monitoring Activities
The RCE Concierge will record all activities
related to RCE in the RCE Access Database,
including:
1. All referrals
2. RCE Visits
3. RCE Outreach and Reengagement
attempts
VIDEO TRAINING MODULES

17

RCE Monitoring Key Metrics
1.
2.
3.
4.
5.

Date, time, and source of the referral
Date and time of the scheduled RCE Visit
Client’s attendance at the RCE Visit
RCE core components received at the RCE Visit
Second appointment scheduled

VIDEO TRAINING MODULES

18

RCE Client Input
It may be valuable to obtain informal feedback from your
clients, such as
• Their experience with HIV and their challenges, concerns,
fears, and needs
• Their experience going through RCE, including what they
liked or what they wish was different
This information can inform quality improvement and help
identify additional partners to engage for wraparound services
and referrals.
VIDEO TRAINING MODULES

19

Pop Quiz
Which of the following is NOT a key RCE monitoring metric?
Select all that apply.
A. Client’s attendance at the RCE
Visit

D. Date and time of the scheduled
RCE Visit

B. Date, time and source of referral

E. RCE core components received at
the RCE Visit

C. Reasons for a missed RCE Visit

F. Second appointment scheduled

VIDEO TRAINING MODULES

20

Check Your Answer
Which of the following is NOT a key RCE monitoring metric?
Answer

C. Reasons for a missed RCE Visit

VIDEO TRAINING MODULES

21

RCE Monitoring Reports
Weekly Report
• Aggregated data only
• Generated by and
exported from the RCE
Access Database
• Sent to RTI weekly

RCE Data
• Individual-level data
• Exported from the RCE Access
Database
• Sent to RTI monthly

VIDEO TRAINING MODULES

22

Outcomes Evaluation

Evaluation Data Sources
1. RCE data collected in the RCE Access Database
2. Appointment data, date of diagnosis, and demographic
data extracted from your clinic’s EHR into the RCE Access
Database

VIDEO TRAINING MODULES

24

EHR Data Collection
1. Appointment Data
Includes the following data for all HIV primary care
appointments at your clinic in the 4 months following the
client’s RCE Visit:
• Appointment date
• Appointment time
• Attendance status
VIDEO TRAINING MODULES

25

EHR Data Collection (cont.)
2. Date of HIV diagnosis
3. Demographic data
•
•
•
•

Age
Sex
Gender
Race

• Ethnicity
• Insurance status
• HIV risk factors

VIDEO TRAINING MODULES

26

Pop Quiz
Which data should be collected from your clinic’s EHR?
Select all that apply.
A. Date of Diagnosis

D. CD4 Counts

B. Appointment Data

E. Demographic Data

C. Viral Load

F. Provider Visit Notes

VIDEO TRAINING MODULES

27

Check Your Answer
Which data should be collected from your clinic’s EHR?
Answers
A. Date of Diagnosis

E. Demographic Data

B. Appointment Data

VIDEO TRAINING MODULES

28

Key Evaluation Metrics
• Linkage to care
• Client linked to care within 72 hours of referral
• Retention in care
• Client attended a second primary care appointment 3
months after the RCE Visit

VIDEO TRAINING MODULES

29

Evaluation Reporting
EHR Data
• Individual-level data for
consenting clients only
• Extracted from clinic EHR
• Sent to RTI every 2 months

RCE Data
• Individual-level data
• Exported from the Access
Database
• Sent to RTI monthly

VIDEO TRAINING MODULES

30

EHR Data Extraction
• Data collected for all enrolled clients (RCE clients who have
consented to share their EHR data)
• Data should be extracted between the data collection dates
for each client:
• Start date – day of consent
• End date – up to 4 months after day of consent OR day of
withdrawal, if client has withdrawn from the study

VIDEO TRAINING MODULES

31

The RCE Report Card

RCE Report Card Process

1

2

3

4

Site sends
RCE data to
RTI

RTI generates
RCE Report
Card every
2 months

RTI reviews
RCE Report
Card with site
on technical
assistance
call

Site staff
review
internally and
set objectives

VIDEO TRAINING MODULES

33

RCE
Report
Card

VIDEO TRAINING MODULES

34

RCE Report Card Metrics
Monitoring
1. Participant characteristics
2. Number of clients referred, by source
3. Number of clients receiving RCE Visit within 0 to 24 hours or 25
to 72 hours of referral
4. Number of clients receiving each RCE Visit component
Evaluation
1. Number of clients attending second appointment
VIDEO TRAINING MODULES

35

RCE Report Card Metrics (cont.)
Narrative Report
1. What went well during the reporting period?
2. What didn’t go well during the reporting period? What could be
improved?
3. What actionable steps can your clinic take in the next 2 months to:
•
•
•
•
•
•

Improve the number of eligible clients who are reached through RCE?
See more clients within 72 hours of referral?
Make referrals to wraparound services at the RCE Visit?
Improve the client’s overall experience?
Improve the intake process?
Ensure clients receive all indicated labs?

If your clinic
collects informal
feedback from
clients, use key
learnings to
inform your
narrative report.

VIDEO TRAINING MODULES

36

Using the RCE Report Card for Quality
Improvement
The RCE Report Card can be used to facilitate quality
improvement by…
• Using metrics to garner support from stakeholders
• Ensuring fidelity to the RCE core components
• Identifying aspects of RCE that need additional resources
(e.g., staff or time)
• Initiating conversations with the internal team about
implementation progress
• Tracking trends over time
VIDEO TRAINING MODULES

37

Reflect

1. Has your organization been engaged in quality improvement
initiatives previously?
2. What lessons can you apply to quality improvement for RCE?

VIDEO TRAINING MODULES

38

Summary

Monitoring and Evaluation
Monitoring

Evaluation

• Is RCE being implemented as
intended?
• Is RCE meetings clients’ needs?
• Are there areas for improvement?

• Is RCE affecting client outcomes?

VIDEO TRAINING MODULES

40

Key Monitoring and Evaluation Metrics
Monitoring
• Date, time, and source of referral
• Date and time of scheduled RCE
Visit

Evaluation
• Linkage to care
• Retention in care

• Client’s attendance at the RCE Visit
• RCE core components received at
the RCE Visit
• Second appointment scheduled
VIDEO TRAINING MODULES

41

Informal Client Feedback
• Feedback can be obtained through informal conversations
with RCE clients
• Client input can help inform quality improvement
• Informal feedback includes information:

• The client’s experience with HIV and their challenges, concerns, fears,
and needs
• The client’s experience going through RCE, including what they liked
or what they wish was different
VIDEO TRAINING MODULES

42

Evaluation Outcomes Data from the EHR
1. Appointment data, including date, time, and attendance
status
2. Date of HIV diagnosis
3. Demographic data
•
•
•
•

Age
Sex
Gender
Race

• Ethnicity
• Insurance status
• HIV risk factors

VIDEO TRAINING MODULES

43

RCE Report Card
The RCE Report Card includes the following:
• Monitoring metrics, including number of clients served and
time between referral and RCE Visit
• Evaluation metrics, including number of clients attending a
second appointment
• Narrative report
This information can be used to garner support from others in
your organization or to identify areas for improvement.
VIDEO TRAINING MODULES

44

Next Steps

More Information
For more information about monitoring and evaluation of RCE,
refer to Chapter 4 of the RCE Implementation Manual.

VIDEO TRAINING MODULES

46

Questions