28 April 20269 min readAI Research

Why Explainability Matters More Than Accuracy in Medical AI

90% accuracy sounds impressive until a radiologist asks ‘where did the model look?’ Here’s what building Brain Tumor AI with Grad-CAM and MC Dropout taught me about trust in clinical systems.

From the project: Brain Tumor AI →
TUMOR 0.92

90% accuracy sounds impressive until a radiologist asks: “where exactly did the model look?” Building Brain Tumor AI forced me to understand why explainability is not a nice-to-have in clinical settings — it is the only path to trust.

The Problem With Headline Accuracy

When I started building Brain Tumor AI, the natural instinct was to optimize for validation accuracy. EfficientNet-B0 reached 90.56% peak validation accuracy. The test summary reported around 93% overall. On paper, this looks clean. In practice, a radiologist reviewing this system would ask an immediately harder question: “What region of the MRI drove that classification?”

Without an answer to that question, the model is a black box. And black boxes do not belong in clinical pipelines — even as research demos.

What Grad-CAM Actually Does

Gradient-weighted Class Activation Mapping (Grad-CAM) works by computing the gradient of the class score with respect to the final convolutional feature map. The result is a heatmap that highlights which spatial regions the model weighted most heavily when reaching its decision.

In Brain Tumor AI, Grad-CAM runs against the EfficientNet-B0 backbone after every prediction. The overlay is rendered directly on the original MRI so the spatial relationship is immediately legible — not buried in a separate chart.

// Design decision

“The Grad-CAM heatmap and the original MRI are shown side by side, not sequentially. A clinician reviewing uncertainty needs both images in the same frame to make a judgment.”

Why Monte Carlo Dropout Changes the Game

Standard neural networks return a single deterministic prediction. If a glioma MRI produces 78% confidence on one forward pass, you have no way of knowing whether that confidence is stable or brittle.

Monte Carlo Dropout solves this by keeping dropout active at inference time and running 20 stochastic forward passes. The variance across those passes becomes an uncertainty estimate. A prediction with low variance is stable. A prediction with high variance is a signal to escalate review — not an error to hide.

The 90.56% vs 93% Discrepancy — Explained

Both numbers are real and refer to different evaluation windows:

90.56% is the peak validation accuracy recorded in the training history — the best the model achieved on the held-out validation split before early stopping triggered. This is the most rigorous per-epoch figure.

~93% is the overall test summary reported across the full test set after training completed. Test sets tend to be easier than the noisiest validation batches the model saw during training, which is why the two numbers differ. Both are honest. Neither is inflated.

What I Learned

The lesson from this project is not about accuracy. It is about the difference between a model and a system. A model predicts. A system earns trust — through uncertainty quantification, visual evidence, and honest framing of what the numbers actually mean.

Explainability is not a feature you add after training. It is a design requirement you build the architecture around from the beginning.

DON'T TAKE MY WORD FOR IT

Try it, live.

Everything this essay argues for, running: the trained model, twenty dropout passes and the Grad-CAM map, in your browser.