DNP 661 Module 3 Ethical Reasoning and Cognitive Pitfalls in the Age of AI Example

A Second Opinion That Sees Less: Ethical Reasoning, Automation Bias and a Skin Lesion App in Rural Primary Care

Student Name

Edson College of Nursing and Health Innovation, Arizona State University

DNP 661: Ethics and Advanced Nursing Practice

Instructor Name

Month Day, Year

A Second Opinion That Sees Less: Ethical Reasoning, Automation Bias and a Skin Lesion App in Rural Primary Care

Introduction

My clinic in Safford serves ranchers, mine workers and families from the San Carlos Apache Reservation, many of whom wait months for a dermatology appointment in Tucson. This fall the clinic began piloting a smartphone app that photographs a skin lesion and returns a risk rating for skin cancer. The appeal is obvious: faster triage for patients with little access to specialists. The ethical risk is less obvious. A tool that is right most of the time can make clinicians stop thinking, and a tool trained mostly on light skin can be wrong most often for the patients who already wait longest. This paper analyzes that risk through the course's two lenses, how clinicians reason and which ethical theories should guide them.

How Clinicians Reason

Diagnostic reasoning runs on two systems. The first is fast, intuitive and based on pattern recognition; the second is slow, analytic and deliberate (Croskerry, 2009). Experienced clinicians rely on the fast system most of the time, and it usually serves them well. Its weakness is that it is prone to predictable errors, such as anchoring on the first impression, premature closure once a plausible answer appears and overconfidence (Croskerry, 2003). The slow system can catch those errors, but only if something prompts the clinician to use it.

An AI rating enters this process at the worst possible moment: right after the first glance. If the app says "low risk," it confirms a fast impression and removes the prompt that might have triggered a slower look. If it says "high risk" for a lesion the clinician thought benign, it can override good judgment. Either way, the tool shapes which system the clinician uses.

Automation Bias

Automation bias is the tendency to over-rely on automated advice, accepting it without enough scrutiny, including when it is wrong. A systematic review of clinical decision support found that automation bias was common, that it led clinicians to reverse correct decisions after incorrect advice, and that it was more likely when tasks were complex, workloads high and users confident in the system (Goddard et al., 2012). Each of those conditions fits a rural clinic with a full schedule and a new tool that staff have been told is accurate.

The review also identified mitigations: training that explains the system's limits, displaying the system's confidence, and requiring the clinician to record an independent judgment before seeing the advice (Goddard et al., 2012). These matter for the safeguards proposed below.

The Problem of Uneven Accuracy

The app's vendor reports strong overall accuracy, but overall accuracy can hide uneven performance. When researchers tested dermatology AI models on a curated set of clinical images that included many darker skin tones, the models performed substantially worse on darker skin than on lighter skin, and the gap narrowed when models were retrained with diverse images (Daneshjou et al., 2022). The pattern is not limited to dermatology. An algorithm widely used to allocate extra care to high-risk patients rated Black patients as needing less help than equally ill white patients, because it was trained to predict spending, and spending tracks access rather than illness (Obermeyer et al., 2019).

For my clinic, this means that the patients least able to travel to a specialist may be the ones for whom the app is least reliable. A "low risk" rating that sends such a patient home could delay the diagnosis of a melanoma, which in people with darker skin is already more often found late.

Weighing the App Through Ethical Theories

Principlism offers four lenses (Beauchamp & Childress, 2019). Beneficence favors the app if it shortens the path to diagnosis for patients who would otherwise wait months. Nonmaleficence warns against false reassurance, especially where accuracy is lowest. Respect for autonomy requires that patients know a machine helped rate their lesion and how much weight it carried. Justice is the sharpest concern: a tool that works best for patients who already have the most access widens the gap it was meant to close.

A virtue lens asks a different question: what kind of clinician does the tool make me? Prudence, the practical wisdom to judge particular cases well, is a central clinical virtue. Leaning on an app that confirms first impressions could, over months, erode the habit of careful looking that prudence depends on. A virtuous use of the tool would treat its rating as one more piece of evidence, never as permission to stop examining. The principlist analysis tells us what is at stake; the virtue analysis tells us what habits to protect.

Safeguards for Using the App

These safeguards do not reject the app. They place it after the clinician's own reasoning, which is where the mitigation research suggests it does least harm.

Conclusion

Artificial intelligence does not remove the ethical work of clinical reasoning; it moves it. The app can help patients who wait too long, but only if the nurse practitioner keeps doing the slow thinking the app makes it easy to skip, and only if the clinic checks whether the tool serves all its patients equally. An advanced practice nurse's obligation is to use such tools the way a careful clinician uses any test: as evidence to weigh, not as an answer to accept.

References

Beauchamp, T. L., & Childress, J. F. (2019). Principles of biomedical ethics (8th ed.). Oxford University Press.

Croskerry, P. (2003). The importance of cognitive errors in diagnosis and strategies to minimize them. Academic Medicine, 78(8), 775-780. https://doi.org/10.1097/00001888-200308000-00003

Croskerry, P. (2009). A universal model of diagnostic reasoning. Academic Medicine, 84(8), 1022-1028. https://doi.org/10.1097/ACM.0b013e3181ace703

Daneshjou, R., Vodrahalli, K., Novoa, R. A., Jenkins, M., Liang, W., Rotemberg, V., Ko, J., Swetter, S. M., Bailey, E. E., Gevaert, O., Mukherjee, P., Phung, M., Yekrang, K., Fong, B., Sahasrabudhe, R., Allerup, J. A. C., Okata-Karigane, U., Zou, J., & Chiou, A. S. (2022). Disparities in dermatology AI performance on a diverse, curated clinical image set. Science Advances, 8(32), eabq6147. https://doi.org/10.1126/sciadv.abq6147

Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121-127. https://doi.org/10.1136/amiajnl-2011-000089

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342