The most important number an agent produces isn’t its answer — it’s how sure it is. Compose that number from independent signals, check it’s calibrated, grade the high-stakes calls with a second model, and route the rest to humans well.

This is Level 3 of a six-level maturity model for running LLM systems in production. Levels 1 and 2 got you to where the system works and you can see it working. Level 3 is confidence: the system acts on its own only when its calibrated confidence is high, grades the decisions that matter with an independent judge, and routes everything it’s unsure about to a human in a way that actually catches errors.

These three things are one mechanism. Confidence is the dial that decides what runs automatically. A judge is one of the signals that dial is built from — and the trigger that pulls a decision off the automated path. The human handoff is where the dial sends everything below the line. Get all three right and you have a system that knows when it doesn’t know, and can therefore be trusted with more.

Engineering a Confidence Score

Ask most LLM systems “how sure are you?” and you get nothing useful — either no number, or the model’s self-reported confidence, which is notoriously miscalibrated (models are cheerfully certain when wrong). So you have to engineer a confidence signal. A production score is composed from independent signals, never lifted from the model’s own claim:

@dataclass
class Signals:
    model_self: float            # the model's own score — useful but weighted DOWN (overconfident)
    verification: float          # fraction of deterministic checks passed (0..1)
    judge_agreed: bool | None    # True/False if this decision was judged; None if not sampled (most aren't)
    historical: float            # this slice's past accuracy (e.g. 0.99 vs 0.80)

WEIGHTS = {"model_self": 0.25, "verification": 0.35, "judge": 0.25, "historical": 0.15}

def compose(s: Signals) -> float:
    # only ~8% of decisions are judged. When judge_agreed is None, DROP the judge term and
    # renormalize the rest — an unjudged decision must not be penalized (False) or faked (True).
    parts = {"model_self": s.model_self, "verification": s.verification, "historical": s.historical}
    if s.judge_agreed is not None:
        parts["judge"] = 1.0 if s.judge_agreed else 0.0
    return sum(WEIGHTS[k] * v for k, v in parts.items()) / sum(WEIGHTS[k] for k in parts)

The exact weights matter less than the principle: a single source of confidence is a single point of failure. Note model_self is deliberately the smallest weight — a model that’s confidently wrong gets dragged down by failed verification or a disagreeing judge, so “confidently wrong” can’t on its own clear the bar. The three other signals are also the ones you can check: deterministic verification either passed or didn’t, the judge either agreed or didn’t, and historical accuracy is a measured fact about this slice. The model’s opinion of itself is the one input you can’t independently verify, which is exactly why it gets the least say.

Calibration

A score is only useful if it’s calibrated — if “0.9” means “right ~90% of the time.” Check it: bucket decisions by predicted confidence and measure real accuracy per bucket (a reliability diagram).

predicted bucket  mean predicted  actual accuracy  verdict
----------------  --------------  ---------------  ---------------------------------------
0.90 – 1.00       0.95            0.71             ⚠ overconfident — recalibrate / raise T
0.80 – 0.90       0.85            0.84             ✓ well-calibrated
0.70 – 0.80       0.75            0.77             ✓
< 0.70            0.55            0.52             ✓ (correctly unsure)

If your top bucket is right 71% of the time, your automation threshold is too loose. The binning that produces that table (and the gap to alert on):

def reliability(samples, bins=10):           # samples: list[(confidence, was_correct: 0|1)]
    rows, ece, n = [], 0.0, len(samples)
    for b in range(bins):
        lo, hi = b/bins, (b+1)/bins
        last = (b == bins - 1)
        bucket = [s for s in samples if lo <= s[0] < hi or (last and s[0] == 1.0)]  # top bin includes 1.0
        if not bucket: continue
        conf = sum(c for c, _ in bucket) / len(bucket)
        acc  = sum(ok for _, ok in bucket) / len(bucket)
        rows.append((round(lo, 1), round(conf, 3), round(acc, 3), len(bucket)))
        ece += (len(bucket) / n) * abs(conf - acc)
    return rows, ece

A weighted sum is not calibrated by construction — compose() returns a number in [0,1], not a probability. Close the loop: fit a monotonic map from raw composed score → empirical accuracy and route on that.

from sklearn.isotonic import IsotonicRegression
calibrate = IsotonicRegression(out_of_bounds="clip").fit(raw_scores, correct)
p_correct = calibrate.predict([composed])[0]     # THIS is what the threshold compares against

Recompute on a rolling window — calibration drifts with the model and inputs. Prefer Brier score or the signed per-bucket gap over ECE for alerting (ECE is binning-sensitive and can read ~0 for a miscalibrated model). And mind the independence caveat: weighting historical slice-accuracy into compose() and then calibrating per slice double-counts — encode slice reliability in one place, not both.

Routing and Thresholds

Once composed and calibrated, automation is a threshold — and the right T is per slice, set from the calibration data:

def route(conf: float, slice_key: str, thresholds: dict[str, float]) -> str:
    T = thresholds.get(slice_key, 0.95)        # default conservative
    if conf < ABSTAIN_FLOOR:  return "abstain"  # too unsure to even recommend
    return "auto" if conf >= T else "hitl_recommended"

Routing here is a slimmed view of Level 1’s four-way vocabulary: abstain is a new floor below the human-review states (for inputs too uncertain to even pre-fill a draft), while Level 1’s reject (failed verification) and hitl_required (very low confidence) still apply. Start T high (almost everything to a human), then lower it for a slice only once its calibration proves the band is safe. You can defend every automated band by pointing at the accuracy data that justified it.

The highest form of this is abstention — declining to decide. An agent that says “I’m not confident; a human should look” is more trustworthy than one that always answers. Make it a first-class outcome, not a failure path. It’s also your best defense against unknown unknowns: you can’t enumerate every weird input production will send, but a calibrated score plus an abstain floor means the weird ones fall through to a human automatically.

ABSTAIN_FLOOR = 0.40    # below this: don't even propose; escalate as "needs human"

The whole system tunes around three levers, and calibration data tells you which way to move each:

lever                raise it                                    lower it
-------------------  ------------------------------------------  ---------------------------------------
threshold `T`        fewer auto-errors, more human load          more automation, more risk
`model_self` weight  trusts the model more (risky)               leans on verification/judge
abstain floor        fewer bad auto-proposals, more escalations  fewer escalations, more noise to humans

A few anti-patterns reliably wreck this layer: using raw model confidence as the dial (miscalibrated — compose instead); a single global threshold (different slices have wildly different accuracy); never re-checking calibration (it rots — a quarter-old threshold is a quarter-old risk model); and treating abstention as an error (it’s the system correctly recognizing its competence edge — measure it, don’t suppress it).

The Judge

One of the signals in compose() was judge_agreed. Here’s where it comes from — and it earns its own section, because for high-stakes decisions it’s the difference between a confident guess and a checked one.

A single model pass is a single point of failure. It’s confident when it’s wrong, it has characteristic blind spots, and you can’t tell a good answer from a plausible one by looking at the output. The cheap, effective insurance is a judge: a second, independent model that evaluates the first’s decision before you act on it.

@dataclass
class Verdict:
    agrees: bool
    confidence: float
    failure_mode: str | None    # if it disagrees, why

def judged_decision(inputs, primary_out, judge) -> Verdict:
    return judge.evaluate(inputs=inputs, proposed=primary_out)   # narrow question: is this right?

Three design choices make it work.

Make the judge independent — and skeptical

If the judge is the same model with the same prompt, it shares the blind spots and rubber-stamps. Use a different model family where you can, a different framing, and prompt it to refute, not confirm.

You are a strict reviewer. You will be given INPUTS and a PROPOSED DECISION made by another system.
Your job is to find the strongest reason the PROPOSED DECISION is WRONG, unsafe, or unsupported by
the inputs. Do not be agreeable. If, after genuinely trying to refute it, you cannot, then agree.

Return JSON: {"agrees": bool, "confidence": 0..1, "failure_mode": string|null}

INPUTS: {{inputs}}
PROPOSED DECISION: {{proposed}}

A judge told to refute catches far more than one told to approve — the framing does real work here, because an agreeable reviewer asked “is this right?” will find a way to say yes, while an adversarial one asked “where is this wrong?” surfaces the failure mode you’d otherwise only discover in production.

Sample — don’t judge everything

Judging doubles model cost. Spend it where stakes or uncertainty are high:

decision class                          judge policy
--------------------------------------  ------------------------------------------------------
highest-impact / irreversible           100% judged
borderline confidence (near threshold)  always judged
everything else                         random sample, e.g. 5–10% (a continuous quality probe)
def should_judge(d, thresholds, rate=0.08) -> bool:
    if d.stakes == "high":                              return True
    if abs(d.confidence - thresholds.get(d.slice, 0.95)) < 0.05:  return True   # borderline (per-slice T, safe default)
    return deterministic_hash(d.decision_id) % 10000 < rate * 10000   # deterministic + sub-1% safe

Treat disagreement as a routing signal — and a metric

When the judge disagrees, that decision goes to a human, every time. The rate of disagreement is also one of your best system health signals — a spike indicates an attack, a bad deploy, or a model regression, and should trigger an alert before you discover the problem through downstream errors.

Together, the confidence layer — composed score, calibration, per-slice thresholds, abstention floor, and an adversarial judge on high-stakes decisions — gives you a system that earns automation incrementally and fails safely when it reaches the edge of its competence.