Reproducing our numbers

Six numbers appear across this site: 42 assets, 36 of them vulnerable, an aggregate of 75/100, 22 hours against 14 minutes, 95.2% accuracy and 0.94 macro-F1. They come from one pilot run on one estate. This page states precisely what each one measured, the unit it is in, the conditions it holds under, and what would make it useless as a prediction about your infrastructure. Where the material an independent party would need is not published, we say so and mark the gap.

#What reproduction can mean here, and what it cannot

There are three different things people mean by reproducing a vendor's figures, and they are not equally available.

• Recomputation. Take our published inputs and rules, redo the arithmetic, and confirm the number follows. You can do this today from /methodology and the figures below.
• Replication. Run the same procedure on the same estate with the same tooling and get the same result. Available to the pilot organisation and to us. Not available to a reader, because the estate is not ours to publish.
• Independent reproduction. A third party, holding the labelled data and the evaluation harness, reproducing the model figures from scratch. Not available today. What it would take is listed further down this page.

We would rather draw that line sharply than describe recomputation as though it were independent reproduction. Most published security metrics are recomputation at best, including ours.

#The six figures, precisely stated

#The estate figures: 42, 36 and 75/100

The count of 42 is a count after normalisation, not a count of raw observations. The same private key seen through a certificate store, a configuration file and a dependency manifest is one asset, not three. This is why the number is smaller than a naive scan would report, and why the definition in What counts as an asset has to be read alongside it. Change the asset definition and the count changes without a single byte of the estate changing.

The 36 is the subset classified as quantum-vulnerable: assets whose security rests on primitives that Shor's algorithm defeats, principally RSA, ECDSA, ECDH and DSA. It is a classification outcome, not a severity judgement. Several of the 36 are low-priority because they protect data with a short confidentiality life, and the ranking is what separates them.

The two counts divide out to a vulnerable share of roughly 0.857. We do not publish that ratio as a headline claim, because it is an arithmetic artefact of one estate's composition and it would be read as an industry statistic, which it is not.

The 75 is the criticality-weighted aggregate of the per-asset scores. One property is worth checking for yourself against the tier table on /methodology: 75 sits inside the Elevated band, ten points above its floor and ten below Acute. That is a genuine fact about the pilot estate's arithmetic rather than a rounded headline, and we have left it exact rather than presenting it as roughly three quarters.

Boundary — scope

Only systems inside the signed scope record were counted. Out-of-scope cryptography is absent, not zero

Boundary — point in time

A single assessment. No re-scan interval is implied

Boundary — method version

theos-method-v1.0. A later version could return a different count and a different aggregate

Estate profile

[PLACEHOLDER: pilot estate sector, size and infrastructure profile]

Assessment date

[PLACEHOLDER: pilot assessment date]

#The time figures: 22 hours against 14 minutes

This pair is the most easily abused number on the site, so here is what it is. Both figures are elapsed wall-clock time spent producing a comparable cryptographic inventory of the same estate under the same scope. The 22 hours is a manual baseline: engineers working through configuration, certificate stores and code by hand. The 14 minutes is an AutoPQC collection and classification run over the same sources.

What sits inside the measurement matters more than the ratio. Excluded from both figures: scope negotiation, access provisioning, and the reading and decision-making that happens once an inventory exists. Included in both: collection, deduplication and classification, up to the point where a countable inventory exists.

The two figures divide out to roughly 94 times faster. We do not lead with that multiple. It is the ratio of one manual effort on one estate to one automated run, and it is dominated by how much of that estate happened to be reachable by automated collection. On an estate with more undocumented systems, more human interviewing and less machine-readable configuration, the manual baseline rises and the automated run rises far less, which flatters the ratio for reasons that have nothing to do with the product being better.

Manual baseline staffing

[PLACEHOLDER: number and seniority of engineers in the manual baseline]

Manual baseline tooling

[PLACEHOLDER: tooling and scripts permitted in the manual baseline]

Automated run environment

[PLACEHOLDER: collector configuration and infrastructure for the 14-minute run]

Repetitions

One of each. No variance figure is available, and we do not quote one

#The model figures: 95.2% and 0.94

Accuracy of 95.2% is the proportion of assets in our labelled evaluation set assigned the correct category by the classifier. Macro-F1 of 0.94 is the unweighted mean of the per-class F1 scores over the same set. We report both deliberately: accuracy alone can look strong on an imbalanced set by favouring the common classes, and macro-F1 penalises exactly that. The gap between the two figures is small here, which is the reassuring direction, but a pair of aggregate figures still hides which classes are weak.

The evaluation set is ours. We built it, we labelled it, and we chose its composition. That is a real limitation and no quantity of decimal places fixes it. A model evaluated on data curated by the same organisation that trained it is being measured against its author's idea of what the world looks like.

How we treat error inside an engagement, including what happens to a low-confidence asset and how false positives are surfaced rather than suppressed, is in Accuracy, false positives, and how we report error.

#Where these numbers stop transferring

We do not claim any of these figures generalise. Stated plainly: this was one estate, assessed once, under one method version, with an evaluation set we built ourselves. If you are using our numbers to forecast your own results, here is what will break the forecast.

• Estate composition. A mainframe-heavy bank and a container-native SaaS platform have almost nothing in common in the shape of their cryptography. Counts and aggregates do not carry across.
• Scope width. Our 42 reflects the scope that was signed. A wider scope finds more, a narrower one finds less, and comparing counts across different scopes compares paperwork.
• Machine-readable coverage. The time comparison depends on how much of the estate a collector can read. Undocumented and physically isolated systems shift the balance back towards human work.
• Data longevity profile. Because Confidentiality Horizon carries 25 of the 100 points, an estate holding thirty-year records scores higher than an otherwise identical estate holding session data.
• Class distribution. If your estate is dense in categories thinly represented in our evaluation set, the accuracy you experience can be below 95.2%. Macro-F1 is a partial guard against that, not a guarantee.
• Method version. Every figure here is pinned to theos-method-v1.0. Read How the method is versioned before comparing across versions.

#What full independent reproduction would require

The gap between what is published and what a reviewer would need is not mysterious. It is a list, and we would rather write it down than let the absence pass unnoticed.

1. 01The labelled evaluation set, or a held-out sample of it, released on terms that permit re-evaluation.
2. 02The evaluation harness and the exact model version, pinned, so that a run is repeatable rather than approximately repeatable.
3. 03Per-class metrics and a confusion matrix, so a reviewer can see which categories carry the error instead of inferring it from an average.
4. 04A synthetic or anonymised reference estate whose composition is published, so the estate figures can be recomputed end to end by somebody outside the company.
5. 05A repeated-measures design for the time comparison, with more than one run of each arm and a stated variance.
6. 06An external party's assessment of all of the above, with their report published whether or not it agrees with ours.

None of that exists today. There is no download, no API endpoint and no dataset request form for it, and we will not pretend otherwise: [PLACEHOLDER: target date for publishing evaluation artefacts]. If you need reproducible evidence to satisfy an internal model-risk or procurement process, tell us which of the six items above is blocking and write to us through /contact.

The alternative is better than any of it. Our numbers describe our pilot estate. Yours would describe yours, and yours are the only ones that can inform your migration. A first assessment produces your counts, your aggregate, your tier distribution and your own elapsed time, stamped with the method version that produced them, and you can hold every figure against the boundary conditions on this page. How to read your first exposure report covers what arrives.