Accuracy, false positives, and how we report error

We publish two numbers for our classifier and we would rather explain their limits than let them do work they cannot do. This page states what they were measured on, why accuracy is a poor single metric for a problem with this class balance, how we distinguish a genuine false positive from a disagreement about scope, and the points in the workflow where a person must look at a finding before it goes anywhere.

#The two numbers we publish

Classification accuracy

95.2% — the proportion of assets in our labelled evaluation set assigned the correct vulnerability classification.

Macro-F1

0.94 — the unweighted mean of the per-class F1 scores on the same set, so each class counts equally regardless of how many examples it has.

Evaluation set size

[PLACEHOLDER: labelled evaluation set size]

Composition

[PLACEHOLDER: evaluation set composition by asset class, protocol family and sector]

Labelling

[PLACEHOLDER: labelling date, reviewer count and adjudication process]

Method version under test

[PLACEHOLDER: method version the evaluation was run against]

Both figures come from an evaluation set we assembled and labelled ourselves. It is not a public benchmark, it was not audited by a third party, and it reflects the kinds of estate we have worked on rather than a representative sample of enterprise cryptography in general. We state plainly that we do not claim these figures generalise to every estate. If your environment is unlike the ones in that set — an unusual protocol family, a heavily bespoke stack, a large population of embedded devices — our error rate on your estate may be higher, and the honest position is that we do not know by how much until we have run.

#Why accuracy is the wrong single metric here

Accuracy is the proportion of decisions that were right. In a problem where the classes are badly unbalanced, that quantity can be high while the model is useless, because a classifier that always predicts the majority class inherits the majority class’s share as its accuracy for free.

Our pilot estate makes the point with real figures. Discovery found 42 assets and classified 36 of them vulnerable. Consider a classifier that does no work at all and labels every asset vulnerable.

do-nothing baseline on the pilot estate composition

  predict "vulnerable" for all 42 assets

  correct   = 36   (the vulnerable ones)
  incorrect =  6   (the sound ones, all wrong)

  accuracy  = 36 / 42 = 0.857 = 85.7%

  This is not a model result. It is the arithmetic of a
  constant prediction on a 36-of-42 split, and it is the
  floor any real accuracy figure has to be read against.

A trivial predictor reaches 85.7% on that composition while getting every sound asset wrong, which in practice means recommending migration work on cryptography that does not need it. Read against that floor, our 95.2% is a smaller claim than it looks at first glance. That is exactly why we publish the second number.

#What macro-F1 adds, and what it still does not tell you

F1 is the harmonic mean of precision and recall, so it drops when either one drops. Macro-F1 computes F1 separately for each class and then takes an unweighted average, which means the rarest class carries the same influence as the most common one. A model that ignores a small class cannot hide behind the large one. Our 0.94 macro-F1 is the number that actually constrains us.

• It does tell you that performance is not concentrated in one easy class, because a collapsed class would pull the unweighted mean down sharply.
• It does not tell you which class is weakest, because averaging discards that. We know internally which one it is; we have not published a per-class figure and will not until it is stable.
• It does not tell you the cost of an error. A missed signing key and a misread cipher suite in a test namespace are both one error to the metric and are not remotely equivalent to you.
• It does not measure the score. Both figures describe the vulnerability classification, not whether the exposure score or the tier was right. There is no ground truth for a score, so no accuracy figure for one is meaningful.

That last point is the one most often misread. The classifier decides whether an asset is quantum-vulnerable. The score and tier are then produced by published arithmetic over rubric anchors, as described in How the exposure score is built. A tier you disagree with is usually a rubric disagreement or an input error, not a model error, and it is resolved differently.

#False positive, or scope disagreement?

Most reported false positives are not classification errors. They are cases where we correctly identified the cryptography and you correctly believe it does not matter. Both parties are right about different questions, and calling that a model failure would send us chasing the wrong fix.

Only the first row counts against the classifier, and only the first row changes our metrics. We track the other four separately because conflating them would let us report an improving error rate by loosening our scope, which is the wrong incentive. Every route above is recorded through the override mechanism in Why a specific asset was flagged.

#The error classes we track

Four distinct failures can occur, and they need different responses. Grouping them under one error rate would hide the one that matters most.

1. 01Missed asset. Cryptography exists and discovery never saw it. This is the most serious class, because a missing finding produces no alert, and it is why we publish a scope-and-coverage statement instead of implying completeness. See What discovery will not do.
2. 02Misclassified primitive. The asset was found and the algorithm or parameters were read wrongly. This is the class the published metrics measure.
3. 03Wrong input. The primitive is right and a dimension value is wrong, usually a declared horizon or an assumed reachability. Corrected by override, not by retraining.
4. 04Stale finding. The finding was right when it was made and the estate has since changed. Managed with the first-seen and last-confirmed dates rather than treated as an error.

The manual comparison behind our pilot took 22 hours of reviewer time against 14 minutes for AutoPQC on the same estate. That is a throughput measurement, not an accuracy one. Speed is what makes a re-run cheap enough to do often, and frequent re-running is the practical answer to stale findings. It is not evidence that the classifier is right.

#Where a human review is mandatory

Some findings never reach a customer decision on the strength of an automated result alone. These gates are enforced in the workflow rather than left to reviewer discretion.

• Any finding tiered Acute before it enters a remediation plan.
• Any finding where the dimension that decided the tier has provenance class Assumed.
• Any override that lowers a tier, including every compensating control.
• Any finding touching key management, a hardware security module, or a signing or certificate trust anchor.
• Any finding that will be cited as audit evidence or included in a Posture Attestation.
• Any asset where the detector reported a primitive our algorithm register does not recognise.

When the evaluation set changes or the metrics move, it is recorded against a method version and announced. The archive of past versions and their published figures is described in How the method is versioned, and each change appears in the Signal Log.