Proving the performance impact
Measure the cost of a cryptographic change on a representative system before you commit to it, then report the comparison honestly
On this page — 6 sections
The most common reason a post-quantum change gets reverted is not a cryptographic problem. It is a performance surprise. Handshakes grow, connection setup behaves differently under load, an appliance in the path stops cooperating, and the change goes back out because nobody can say whether the regression is acceptable. Measuring first is what turns that argument into a decision.
Proving the performance impact before anything ships is the part of AutoPQC we consider our differentiator. This article covers what we measure, why key and signature sizes matter more than raw speed, how to build a test that resembles your system, and how we report the comparison.
Measure before you commit, not after you deploy
Cryptographic changes have an unusual property: they touch every request rather than one feature, and the cost lands on connection setup rather than steady-state throughput. That makes them easy to under-test. A change that adds nothing measurable to a warm connection can still change the behaviour of a service that opens many short-lived connections under bursty load.
So the sequence is: measure the current system, apply the change in a representative environment, measure again with the same method, and compare. The comparison, not the absolute number, is the deliverable. Absolute numbers vary with hardware and load. A well-controlled delta survives those differences and is something a change advisory board can act on.
There is an organisational benefit as well. A measured before-and-after removes the loudest objection to post-quantum work, which is the unfalsifiable claim that it will be too slow. Once there is a number, the conversation moves from fear to capacity planning.
What to measure
Six families of metric cover almost every regression we have seen. Collect all six before the change and after it, from the same harness, on the same infrastructure.
| Metric | Why it matters | How to capture it |
|---|---|---|
| Handshake latency | Where post-quantum key establishment lands, and the first thing users feel on new connections | Time to completed handshake, reported at median, p95 and p99 |
| Throughput | Confirms the bulk data path is unaffected, which it usually is once keys are established | Sustained transfer rate on established connections, plus requests per second |
| CPU and memory | Determines whether you need more capacity, and whether front doors become the bottleneck | Per-process utilisation under a fixed offered load, not under best effort |
| Payload and certificate size | Larger keys and signatures inflate handshakes and chains, with knock-on effects on packets and storage | Bytes on the wire per handshake, and full chain size in bytes |
| Connection setup under load | Regressions frequently appear only at high concurrent-handshake rates | New connections per second, held at a target rate until behaviour stabilises |
| Tail latency | Averages conceal the failures that generate tickets; the tail is where retries and timeouts live | p99 and p99.9, alongside handshake failure and retry counts |
Averages will mislead you here
Handshake cost is not evenly distributed. A change can leave the median untouched and still move p99 enough to cross a client timeout, at which point the visible symptom is retries rather than slowness. Report distributions, and always report failure counts next to latency.
Why size matters more than raw speed
For the post-quantum algorithms in current use, computation is not the interesting cost. ML-KEM operations are fast on ordinary server hardware. The cost that reshapes system behaviour is size. Public keys, ciphertexts and signatures are substantially larger than their elliptic-curve equivalents, and size interacts with the network in ways computation does not.
- Packets and round trips. Once a handshake exceeds what fits in the initial exchange, you may pay an extra round trip. On a high-latency path that dominates any CPU difference.
- Loss amplification. More packets per handshake means more chances for one to be dropped, and a dropped handshake packet costs a retransmission timeout, not a microsecond.
- Middlebox limits. Appliances that assume a maximum handshake size, or that parse the handshake, fail on larger messages. That presents as an outage, not as a slowdown.
- Storage and MTU. Larger certificates strain constrained devices, small MTUs, embedded flash, and anything with a hard field-length limit.
- Signature growth is the sharper problem. ML-DSA signatures under FIPS 204 and hash-based signatures under FIPS 205 are considerably larger than ECDSA, which is why chain size gets its own line in the report.
The practical consequence is that a CPU-only benchmark will tell you post-quantum cryptography is cheap, and your production estate may disagree. Any credible test has to put real network conditions and real middleboxes in the path. Exact per-parameter-set sizes belong in your own report, cited to the standard: [PLACEHOLDER: key, ciphertext and signature sizes per parameter set, cited to FIPS 203/204/205].
Build a representative test, not a synthetic benchmark
A synthetic benchmark measures an algorithm. A representative test measures your system running that algorithm. The difference is usually the difference between a reassuring number and a useful one.
- 1
Start from a real traffic profile
Take connection rates, connection lifetimes, request sizes, session-resumption rates and the concurrency distribution from production telemetry. Connection lifetime matters most: an estate of long-lived connections barely notices handshake cost, while an estate of short-lived ones notices immediately.
- 2
Keep the full path in scope
Include the load balancer, the TLS terminator, any inspection appliance and the actual network path. A test that runs between two processes on one host removes precisely the components most likely to break.
- 3
Change one variable at a time
Measure classical, then hybrid, then post-quantum only if relevant, holding hardware, load generator, kernel and library versions constant. Mixed changes produce results nobody can attribute.
- 4
Hold load, then push past it
Run at your normal peak until behaviour is stable, then increase until something degrades. You need both the cost at your operating point and the amount of headroom left above it.
- 5
Include the failure cases
Test against a peer that does not support the new construction, an appliance with a restrictive size limit, and a lossy path. Graceful degradation is a result worth recording.
- 6
Record the environment with the numbers
Library versions, parameter sets, hardware, kernel, load-generator configuration and the exact test window. A measurement without its environment cannot be reproduced, and an unreproducible measurement is an anecdote.
How we report a before-and-after comparison
Every AutoPQC performance report pairs each metric with its baseline, states the delta in both absolute and relative terms, and carries the test environment alongside. We report distributions rather than single figures, and we place failure counts next to latency so a change that trades errors for speed cannot look like an improvement.
We also state what the test did not cover. Coverage is part of the result. A report that measured one service on one path should say so plainly, rather than implying estate-wide validity. Accuracy, false positives, and how we report error describes the same discipline applied to classification.
- Baseline
- Classical configuration, same harness, same window [PLACEHOLDER: baseline run identifier]
- Candidate
- Hybrid or post-quantum configuration, single variable changed [PLACEHOLDER: candidate run identifier]
- Reported statistics
- Median, p95, p99, p99.9, plus handshake failure and retry counts
- Environment
- [PLACEHOLDER: library versions, parameter sets, hardware profile, kernel version]
- Coverage statement
- [PLACEHOLDER: services, paths and appliances actually exercised]
- Verdict
- Proceed, proceed with a capacity change, or do not proceed, with the threshold that decided it
Illustrative figures are labelled as such
Any latency or size figure in a template, a worked example or a conversation with us is marked illustrative. Only figures produced from your estate, with the environment recorded, belong in a decision document. If you find an unlabelled number in something we sent you, treat it as an error and tell us.
The one platform figure we quote, and what it is not
On our pilot estate, review work that took a team 22 hours by hand took 14 minutes with AutoPQC. That is the single effort figure we publish, and it is worth being precise about what it describes.
It is analyst effort: the time to discover cryptographic assets, classify them, map them to the relevant standards, and produce a reviewable exposure report. It is not a cryptographic benchmark. It says nothing about handshake latency, key-exchange throughput, or the runtime cost of ML-KEM on your hardware. Anyone quoting it as a performance number, including us, is misusing it.
The distinction matters because the two questions have different answers. Review effort is where automation compounds. Runtime performance is where packet sizes and physics apply, and no platform makes a larger key smaller. The remaining pilot figures, including the asset counts and the classification accuracy, are set out in Reproducing our numbers.
We measure. We do not tune your systems or certify a result.
AutoPQC produces the comparison and the evidence trail. We do not optimise your servers, resize your fleet, replace your appliances, or issue a performance certification. We cannot measure a path we are not given access to, and we will not extrapolate from one service to an estate. Where coverage is partial, the report states which parts were tested and which were not, rather than presenting a partial result as a whole one.