CAI1020 Week Seven: AI in Healthcare

  • 230 million people globally ask health questions on ChatGPT each week
  • GPT-5.5 Instant responses were rated higher than physician-written responses
  • 71% reduction in factual issues compared to earlier versions.
  • AI has no way to examine the patient physically, cannot order tests or prescribe treatments, and is not accountable to any regulatory body

    • 1. The Scale and Stakes of AI in Healthcare

  • access to medical knowledge is not equally distributed.
  • AI can give anyone access to specialized medical information
  • An AI system that systematically gives bad advice to 230 million weekly users causes a catastrophe.

    • The evidence for AI outperforming humans is strongest in specific, well-defined tasks:

  • Medical imaging interpretation and pathology.
  • Differential diagnosis assistance.
  • Drug interaction identification.
  • Symptom triage. (decide who gets help first)

    • Current Limitations

    Distributional shift- AI trained on a specific population may not be proficient with others

    Privacy- patients may be harmed if sensitive health information is used in ways they did not understand or consent to.

    Hallucinations- LLMs produce confident, plausible-sounding information that is wrong.

    The diagnosis-to-treatment gap.- AI can diagnose but can't treat.

    The Accountability Problem

  • Not licensed
  • Not liable
  • Not subject to professional boards
  • No ongoing relationship with users

    • 2. Informed Consent and the Black Box

    patients should receive enough information about any procedure to make a voluntary and informed decision.

    AI should disclose-
    that the recommendation came from an AI system system’s known limitations and failure modes what recourse exists if the recommendation is wrong the ability to request human clinical review.

    AI can generate recommendations without giving patients or clinicians a clear explanation of why that recommendation was produced

    AI may provide a persuasive explanation that sounds medical while functioning more like a post-hoc justification than a transparent account

    problems:
  • patients may not be able to evaluate AI-generated recommendations;
  • clinicians may not be able to fully scrutinize the recommendation before relying on it
  • errors may be difficult to trace
  • biased or unsafe patterns may be hard to detect
  • consent may be incomplete if the patient cannot understand the basis of the recommendation.

    • Central question: Can a patient give informed consent when the recommendation comes from a system whose reasoning cannot be fully explained, challenged, or held accountable?

    3. Psychiatry and Mental Health

  • Psychiatric diagnoses are more complex, more contested, and more value-laden than most medical diagnoses.
  • The risk that AI therapy substitutes for human therapy is a serious concern.

    • 4. Toward a Framework for Healthcare AI

  • 1. What is the decision? High-stakes decisions warrant greater human involvement
  • 2. Is there meaningful consent?
  • 3. Is there accountability? If the AI is wrong and harm results, is there clear accountability?
  • 4. Is the AI interpretable? Can the reasoning behind AI recommendations be explained in ways that allow patients and clinicians to evaluate it?