Evaluation Questions and Indicators

ATTACHMENT 3

Evaluation Questions and Indicators

Evaluation Questions & Constructs	Data Sources
Question 1: In comparison to the MDPH established decision aid and the NCI PDQ®, how effective is Talk to Nathan About Prostate Cancer Screening in improving outcomes, including improving prostate cancer–related knowledge; informed decision-making self-efficacy; technology use self-efficacy; intention to engage in informed decision-making with a healthcare provider; overcoming health literacy barriers; and resolving decisional conflict?
Decisional conflict	Pre-exposure survey; post-exposure survey; post-clinic visit survey
Prostate cancer knowledge
Autonomous decision-making	Pre-exposure survey; post-exposure survey
Decision self-efficacy	Post-exposure survey
Preparation for decision-making
Exposure to assigned materials
Shared decision-making	Post-clinic visit survey
Time spent with provider discussing PSA test
Prostate cancer screening practices	Provider survey
Attitudes towards prostate cancer screening
Question 2: How does Nathan help men make decisions about the harms and benefits of prostate cancer screening that are in line with their individual values and preferences?
Decisional conflict	Pre-exposure survey; post-exposure survey; post-clinic visit survey
Decision self-efficacy	Post-exposure survey
Nathan dosage	Usability survey; user experience interviews; usage data gathered through the Nathan platform
Question 3: What is the impact of Nathan on the quality of prostate cancer screening decisions, engagement in shared decision-making, and time spent with providers discussing screening?
Quality of screening decision (from the Preparation for Decision-Making Scale)	Post-exposure survey
Shared decision-making	Post-clinic visit survey
Time spent with provider discussing PSA test
Question 4: What are the barriers and best practices for incorporating Nathan into the flow of primary care practice?
Barriers to incorporating Nathan into workflow	Clinic coordinator interviews; monthly data collection progress reports
Facilitators to incorporating Nathan into workflow
Best practices in incorporating Nathan into workflow
COVID-19 impact and telemedicine	Usability survey
Questions 5 and 6: Is Nathan accessible and usable? Specifically, is it accessible and usable from a health and digital literacy perspective?
Acceptability of Nathan	Usability survey; user experience interviews
Perceived fit of Nathan
Usability of Nathan
Recommendations for Nathan improvements
Barriers to Nathan use	User experience interviews
Facilitators to Nathan use
Technology acceptance	Usability survey
Help needed to review Nathan
Digital literacy	Pre-exposure survey; user experience interviews
Health literacy	Pre-exposure survey
Help needed to review assigned materials
Nathan dosage	Usability survey; user experience interviews; usage data gathered through the Nathan platform
Question 7. Does Nathan help people clarify their values and preferences?
Decisional conflict (values clarity subscale)	Pre-exposure survey; post-exposure survey; post-clinic visit survey
Nathan dosage	Usability survey; user experience interviews; usage data gathered through the Nathan platform
Questions 8 and 9: Are there disparities in decision-making among different groups (e.g., by income, by racial/ethnic group)? Is the impact of Nathan different for high-risk groups (e.g., Black men, individuals with a family history of prostate cancer) or based on age?
Decisional conflict	Pre-exposure survey; post-exposure survey; post-clinic visit survey
Prostate cancer knowledge
Autonomous decision-making	Pre-exposure survey; post-exposure survey
Health literacy	Pre-exposure survey
Prostate cancer experience
Decision self-efficacy	Post-exposure survey
Preparation for decision-making
Screening behavioral intent	Post-clinic visit survey; EHR review
Shared decision-making	Post-clinic visit survey
Time spent with provider discussing PSA test
Demographic characteristics	Pre-exposure survey; provider survey
Digital literacy	Pre-exposure survey; user experience interviews
Question 10. Does Nathan change screening behavior (decision to get tested with a prostate-specific antigen [PSA] test or not)?
Screening behavioral intent	Post-clinic visit survey EHR review
Screening behavior
Nathan dosage	Usability survey; user experience interviews; usage data gathered through the Nathan platform
Previous exposure to informational materials about prostate cancer screening	Pre-exposure survey
Contamination	Post-exposure survey
Informational materials used in making screening decision	Post-clinic visit survey