Accuracy, false positives, and how we report error

The classifier figures previously quoted came from a synthetic evaluation. They are not evidence of accuracy on your estate. This page explains the limits of aggregate metrics, separates classification errors from scope or input disagreements, and describes how to review a finding against its evidence.

#What an accuracy claim needs

Earlier classifier evaluation

Synthetic; not evidence of customer accuracy

Customer accuracy

Not established by the synthetic evaluation

Evaluation set size

[PLACEHOLDER: labelled evaluation set size]

Composition

[PLACEHOLDER: evaluation set composition by asset class, protocol family and sector]

Labelling

[PLACEHOLDER: labelling date, reviewer count and adjudication process]

Method version under test

[PLACEHOLDER: method version the evaluation was run against]

A synthetic dataset reflects the cases chosen by its authors. It can help test behaviour on those cases, but it does not establish an error rate for a different estate. Dataset composition, labels, protocol and per-class results are needed before a reader can judge what an evaluation supports.

#Why accuracy is the wrong single metric here

Accuracy is the proportion of decisions that were right. In a problem where the classes are badly unbalanced, that quantity can be high while the model is useless, because a classifier that always predicts the majority class inherits the majority class’s share as its accuracy for free.

An illustrative split of 42 assets, with 36 labelled vulnerable, shows the problem. Consider a classifier that labels every asset vulnerable. This is a teaching example, not a customer performance result.

do-nothing baseline on the illustrative class composition

  predict "vulnerable" for all 42 assets

  correct   = 36   (the vulnerable ones)
  incorrect =  6   (the sound ones, all wrong)

  accuracy  = 36 / 42 = 0.857 = 85.7%

  This is not a model result. This illustrates class imbalance.
  It is not a baseline for a result measured on a different dataset.

A constant predictor can achieve a high accuracy on an imbalanced dataset while missing every example of a smaller class. That illustrates the limitation of accuracy alone; it is not a baseline against which to compare a result from a different dataset.

#What macro-F1 adds, and what it still does not tell you

F1 combines precision and recall. Macro-F1 averages the per-class F1 scores with equal weight for each class, so a common class does not receive more weight merely because it has more examples. It still hides variation between classes and does not establish performance on another dataset.

• Equal weighting: each class contributes equally to the average
• Limits: the average does not identify the weakest class; per-class results are needed
• Error cost: the metric does not say how consequential a missed or misclassified asset is
• Different questions: classification metrics do not validate the published exposure-score arithmetic

An observation, its classification and the score computed from published rules are different parts of a finding. A disputed tier may arise from a disputed input or rubric interpretation. Trace the disagreement to that specific step; see How the exposure score is built.

#False positive, or scope disagreement?

Most reported false positives are not classification errors. They are cases where we correctly identified the cryptography and you correctly believe it does not matter. Both parties are right about different questions, and calling that a model failure would send us chasing the wrong fix.

Only the first row counts against the classifier, and only the first row changes our metrics. We track the other four separately because conflating them would let us report an improving error rate by loosening our scope, which is the wrong incentive. Every route above is recorded through the override mechanism in Why a specific asset was flagged.

#The error classes we track

Four distinct failures can occur, and they need different responses. Grouping them under one error rate would hide the one that matters most.

1. 01Missed asset. Cryptography exists and discovery never saw it. This is the most serious class, because a missing finding produces no alert, and it is why we publish a scope-and-coverage statement instead of implying completeness. See What discovery will not do.
2. 02Misclassified primitive. The asset was found but the algorithm or parameters were read incorrectly. Check the evidence for that finding; an aggregate synthetic result cannot settle it.
3. 03Wrong input. The primitive is right and a dimension value is wrong, usually a declared horizon or an assumed reachability. Corrected by override, not by retraining.
4. 04Stale finding. The finding was right when it was made and the estate has since changed. Managed with the first-seen and last-confirmed dates rather than treated as an error.

Review speed and accuracy require separate evidence. A timing comparison needs a common scope, a documented manual protocol, run logs, repeated runs and checks of output quality. No review-speed claim is substantiated by the synthetic scoring example.

#Where a human review is mandatory

Some findings never reach a customer decision on the strength of an automated result alone. These gates are enforced in the workflow rather than left to reviewer discretion.

• Any finding tiered Acute before it enters a remediation plan.
• Any finding where the dimension that decided the tier has provenance class Assumed.
• Any override that lowers a tier, including every compensating control.
• Any finding touching key management, a hardware security module, or a signing or certificate trust anchor.
• Any finding that will be cited as audit evidence or included in a Posture Attestation.
• Any asset where the detector reported a primitive our algorithm register does not recognise.

When the evaluation set changes or the metrics move, it is recorded against a method version and announced. The archive of past versions and their published figures is described in How the method is versioned, and each change appears in the Signal Log.