Matrix of Evaluation Questions, Indicators, and Data Sources

Attachment 3 - CRCCP Evaluation Matrix

Evaluation Questions	Example Indicators/Measures		Data Source
Establish partnerships with health systems/clinics
What are the characteristics of recipients’ partner health systems and clinics?	#/% health system types (e.g., FQHCs) #/% by clinic size (patient population) #/% clinic locations (e.g., rural/urban) #/% clinics terminated #/% health systems/clinic with agreement, by type		Clinic data
What is the annual and overall reach of the CRCCP?	# health systems recruited # active health systems # clinics recruited # active clinics recruited Avg. # (and range) of clinics recruited per recipient # clinic patients age 50-75 Avg. # (and range) of patients age 50-75 recruited per recipient # primary care providers within clinics Geographic location of clinics (mapped) Geographic location of clinics mapped with population density overlay		Clinic data External data sources (TBD)
To what extent do recipients provide clinic partners with financial support?	#/% of clinics with agreement, by agreement type #/% clinics receiving CDC funds Total CDC funding distributed to clinics Avg. amount (and range) of CDC funds received by clinics		Clinic data Cost study
What are the characteristics of clinics’ CRC screening efforts?	#/% primary CRC screening test used, by type #/% clinics providing mailed FIT kit program #/% of clinics with screening champions #/% of clinics with screening policies #/% of clinics that distribute free FIT kits		Clinic data
Are recipients partnering with clinics serving populations of need with low CRC screening rates?	#/% clinics located in high burden areas #/% of clinics located in counties with high CRC incidence and death rates #/% clinics that are FQHCs #/% clinics with uninsured populations >20% Avg. clinic baseline screening rate #/% of clinics in different baseline rate categories (e.g. <20%, 20%-30%, …)		External data sources (TBD) Clinic data
To what extent are partner clinics terminated over time? What are the characteristics of terminated clinics? Why are clinics terminated?	#/% of clinics terminated, in aggregate and by recipient #/% clinics terminated within a given health system, by recipient Avg. time (in years) of clinic participation before termination Characteristics of terminated clinics (e.g., size, type, location) Baseline and annual screening rates of terminated clinics terminated Reasons for termination		Clinic data Case studies (if indicated)
Establish partnerships to support implementation of EBIs
With what implementation partners are recipients partnering?	#/% of implementation partners, by type Avg. # (and range) implementation partners per recipient		Annual Recipient survey
To what extent do recipients provide implementation partners with financial support?	#/% of partners with MOU or contract #/% partners receiving CDC funds Total CDC funding distributed to partners Avg. amount (and range) of CDC funds received by partners		Annual Recipient survey Budget data
What kinds of implementation support are partners providing to clinics?	Types of implementation support provided #/% recipients that use an established approach to deliver TA to clinics Characteristics of recipient implementation plans Utility and appropriateness of implementation support provided to clinics		Annual Recipient survey Case studies Clinic Implementation Planning Summaries
How often is implementation support being delivered to clinics?	Frequency of implementation support provided to clinics over time, in aggregate and by clinic type		Clinic data
Conduct implementation readiness assessments of partner primary care clinics
To what extent have recipients conducted formal assessments of clinics’ readiness to implement the CRCCP?	#/% of recipients with a standard approach to readiness assessment #/% of recipients conducting specified areas of assessment (e.g., assess EHR capacity)		Recipient survey Clinic Assessments
Implement EBIs
What EBIs are recipients implementing in clinics? What EBIs are in place at baseline? What EBIs are planned, newly implemented, or enhanced annually? Are CDC resources used to support EBI planning or implementation? Are clinics implementing multiple EBIs?	#/% clinics implementing each EBI #/Types of EBIs planned, newly implemented, enhanced, by clinic Types of EBIs supported with CDC funds, by clinic #/% clinics implementing new EBIs #/% clinics implementing 3-4 EBIs Spending on EBIs implemented/enhanced by recipients.		Clinic data Cost study
How are EBIs delivered within clinics?	Descriptions of EBI delivery protocols Avg. # ways patient reminders sent per clinic Avg. # ways provider reminders sent per clinic Avg. frequency of provider assessment and feedback per clinic #/% clinics reducing structural barriers in more than one way		Case studies Clinic data
Are the EBIs sustainable? How long does it take for a newly implemented EBI to become sustainable? How do recipients and/or clinics support sustainability? Which EBIs are more/less sustainable than others, and why?	#/% of clinics with at least 1 sustainable EBI #/% of clinics with sustainable EBIs, by EBI type #/% of sustained EBIs by clinic type Avg. # years from newly implemented EBI to sustained, by EBI type Strategies to support sustainability of EBIs		Clinic data Case studies
What are the costs and cost effectiveness of the EBIs being implemented by CRCCP recipients?	Amount of CDC funds used to support EBI implementation activities Average spending for EBIs implemented/enhanced by recipient		Cost studies
To what extent are PN and small media implemented? How is PN implemented? Are PN and small media sustainable?	#/% of clinics implementing PN #/% of clinics implementing small media Avg amount of time patient received PN #/% of clinics where PN is sustainable #/% of clinics where small media is sustainable		Clinic data Recipient survey
Facilitate patients’ linkages to follow up colonoscopy
To what extent did health systems/clinics utilize CDC funding to support follow up colonoscopy completion?	Amount of CDC funds awarded to health systems/clinics to support follow up colonoscopies # patients receiving CDC funded follow-up colonoscopies %/# clinics who received CDC funds to support follow up colonoscopy		Recipient survey Clinic data
What are the screening results among patients who received follow-up colonoscopy paid for with CDC funds?	#/% patients with adenomatous polyps removed who received a CDC-funded follow-up colonoscopy #/% patients with other abnormal results who received CDC-funded follow up colonoscopy #/% patients diagnosed with cancer who received CDC-funded follow up colonoscopy		Clinic data
Data Quality, Program Monitoring, and Evaluation
To what extent are recipients developing evaluation plans consistent with CDC requirements as stated in the NOFO?	#/% recipients with evaluation plans submitted within 6 months of award #/% evaluation plans that meet basic CDC requirements		Evaluation plans
To what extent are clinic data complete and high quality?	#/% clinics with no missing baseline data records, by recipient #/% clinics with no missing annual records, by recipient Avg clinic data error rates, by recipient #/% clinics with decreased error rates over time, by recipient #/% clinics with low/medium/high confidence in EHR-generated screening rate, by recipient #/% clinics that conduct screening validation through chart review within first two years of participation #/% of clinics that change EHR vendors over time Type of staff collecting clinic data Activities taken to ensure high quality data		Clinic data Recipient survey
What quality assurance mechanisms are in place within clinics?	#/% clinics with access to HCCN #/% clinics with screening rates monitored at least quarterly #/% clinics that conduct screening validation within one year of clinic enrollment #/% recipients participating in annual CDC-led data quality review process #/% clinics with QI processes in place #/% clinics using HIT tools for program monitoring #/% clinics that change EHR vendors across years		Clinic data
Increased CRC screening
To what extent are screenings completed among patients who receive a screening referral?	Annual clinic-level FIT kit return rates Annual clinic-level colonoscopy completion rate		Clinic data
To what extent are clinics meeting their screening targets?	#/% of clinics that set appropriate screening targets #/% clinics that meet their annual screening rate target		Clinic data
To what extent have clinic screening rates changed over time?	Avg weighted change in percentage points of screening rate, by recipient and for CRCCP Avg weighted change in percentage points of screening rate for CRCCP by clinic characteristics, EBI implementation and other relevant groups.		Clinic data
To what extent have the number of CRC screenings changed over time?	Number of screening tests per year, in aggregate and by grantee Avg number (and range) of screening tests completed per year Percent increase in the number of screening tests completed year to year	Clinic data

Other Evaluation Questions		Data Source
What are significant predictors of greater screening rate increases?		Clinic data
What are significant predictors of the implementation of specific EBIs?		Clinic data
How do we characterize the implementation of EBIs in multi-level analysis?		Clinic data
How do we characterize the longitudinal trajectory of clinic screening growth?		Clinic data
What is the cost effectiveness of specific intervention under CRCCP?		Clinic data Cost study
What are the characteristics of clinics with the highest and lowest screening rate increases?		Clinic data
What does de-implementation of CRCCP look like within partner clinics?		Special study
Are screening rates sustained once the CRCCP stops actively working with clinics?		Clinic data Special study
What are the characteristics of clinics and CRCCP implementation where screening rate changes are sustained once the CRCCP stops actively working with clinics?		Clinic data Special study
What are the significant predictors of EBI sustainability?		Clinic data
What is the long-term impact of the CRCCP on lives saved?		Special study
Program Management Question	Indicators/Measures	Data Source
What are the characteristics of the recipients funded under 20-2002?	#/% of recipients by type (e.g., health dept, university) Avg award, range in CDC funding by year, by recipient type	Budget data
What CDC TA resources have been most useful for recipients?	#/% of resources used, by utility (i.e., helpfulness)	Recipient Survey
What non-CDC financial resources do recipients have to support their CRCCP?	Funding amounts by type (e.g., State) Total non-CDC funds supporting CRCCP #/% recipients receiving non-CDC support	Recipient Survey
How many and what type of staff are recipients using to manage and implement the CRCCP over time?	#/% of recipient staff, by type Change in recipient staff type over time	Budget data
What challenges do recipients encounter when implementing the CRCCP?	#/% of recipient challenges, by type	Quarterly Program Updates
What successes have been realized by recipients?	Description of successes #/% of recipients reporting successes annually	Quarterly Program Updates
To what extent do recipients encounter staff vacancies?	# staff vacancies Types of vacant staff positions	Quarterly Program Updates
What TA needs do recipients experience?	Description of TA needs	Quarterly Program Updates

Page 4 of 4