Lorem ipsum: your baseline model is lying to you politely

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Every model report I’ve ever reviewed opens with an AUC, and almost none of them open with a calibration curve. Sed do eiusmod tempor incididunt ut labore — which is a shame, because the second number is the one your stakeholders are implicitly betting on.

Ut enim ad minim veniam, quis nostrud exercitation. Here’s the check I run before anything ships:

from sklearn.calibration import calibration_curve

prob_true, prob_pred = calibration_curve(
    y_test, y_prob, n_bins=10, strategy="quantile"
)
# if this isn't roughly y = x, your AUC is decorative

Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore. The point isn’t that AUC is useless — it’s that ranking quality and probability quality are different promises, and only one of them survives contact with a pricing model.