Beyond the book | The Red Line

Beyond the book

The book is finished. The decisions it describes are not. They are being made every day, in hospitals, in insurance offices and in the companies building these systems.

This page is for what happens next: questions hospital leaders and patients can ask now about systems already in use, and notes on new developments and what they could mean for patients, medicine and society.

For hospitals and hospital leaders

I do not want this to remain an interesting argument about AI ethics. I want something to happen on Monday morning.

If you work in a hospital and have responsibility for AI, whether clinical, administrative, technical, safety, compliance, leadership, or especially procurement, start with what is already deployed. Not what the strategy document says.

What is actually running? What is writing notes? What is scoring risk? What is prioritizing patients? What is influencing diagnosis, triage, treatment, coverage, access, or allocation?

Then ask five questions about every consequential model.

  1. What does it optimize?
  2. On what population of patients was it validated?
  3. Can a clinician override it without being penalized for doing so?
  4. Who monitors it after deployment, and what triggers a pause?
  5. Who owns its clinical performance, by name?

Every hospital deploying consequential AI needs a named governance function with enough authority to monitor these systems and stop them when necessary. That may mean a dedicated Clinical AI Governance and Accountability Office. In other hospitals, it may mean giving an existing function real authority. The name matters less than whether someone actually owns the responsibility.

And procurement has to be part of it, because before an AI system becomes a clinical workflow, somebody signs a contract. Those five questions should be answered before the next contract is signed, not after a patient has been harmed.

If a hospital cannot answer those five questions about a system it is already using, then it is not really governing that system. It is just running software.

For patients and families

You do not need to understand how an AI system works to ask how it was used in your care. Whether you are asking for yourself or someone you love, here are a few questions to begin with.

  1. Was an AI tool used in my care, and what was it used for?
  2. Who reviewed what it produced before it was used to make a decision about me?
  3. How does this recommendation take account of my symptoms, my medical history, and what matters to me?
  4. If something in my information is wrong or missing, how can I have it corrected and the decision reconsidered?
  5. Who is responsible for this decision, and who can I speak to if I disagree?

These questions can also be asked when a decision concerns insurance coverage or access to treatment, rather than a medical recommendation.

You should not have to investigate a system to receive safe care. The responsibility belongs to the people and institutions using it. These questions are a way to ask for an explanation, not a transfer of that responsibility to you.

Notes on AI and medicine

The questions above are a starting point. These notes look at new developments and ask what they could mean for patients, for hospitals and for the people responsible for these systems.

Safety before the race

What does the Hugging Face incident have to do with medicine?

Imagine a hypothetical hospital scenario. A doctor is preparing to discharge a patient because the record shows that her medication review is complete. But nobody has actually done the review.

The doctor is still making the final decision. The problem is that the information behind that decision has already been changed.

Read the note →

The White House executive order renaming AI

On 29 September, the President of the United States signed an executive order titled Inaugurating the Era of Super Intelligence. Federal departments and agencies are now directed to use “Super Intelligence” and “SI” where they once used “Artificial Intelligence” and “AI” in their official communications and documents. For now, the new term covers the technologies already defined in law as artificial intelligence. The order also calls for proposed legislation within 60 days that would establish a federal definition of Super Intelligence.

Before we rename AI, we should say what we mean. The note also separates three kinds of AI that now share one name, and sets out what government and companies should do.

Read the note →

Do not confuse a system’s ability to produce an answer with its right to close the question. In many fields, an answer can be enough. In medicine, the next question may be the diagnosis.