Latency benchmarks
Measured, not asserted.
Every latency number on this site comes from the harness described below. It ships in the repository, it runs with one command, and the payloads behind it are fixed and checked in — a published figure means nothing if you can’t reproduce the request that produced it.
Free-text note, model response and retrieved RAG context. The heaviest realistic payload class.
Structured numeric parameters only. The security stages have no free text to scan.
Through the full ASGI stack including validation, auth, proof construction and the audit write. Excludes network transit.
End to end, by payload class
The pipeline itself, measured in-process. Latency is dominated by how much free text a decision carries, not by how many rules it evaluates — the security stages scan text, the rule engine compares numbers.
| Payload class | p50 | p95 | p99 | What it exercises |
|---|---|---|---|---|
numeric_minimal | 0.051 | 0.077 | 0.132 | Four numeric fields. Rule engine only. |
numeric_wide | 0.158 | 0.197 | 0.272 | Thirty numeric and categorical fields. |
text_note | 1.135 | 1.378 | 2.282 | Adds a free-text underwriting note. PII, toxicity and injection scanning engage. |
text_with_rag | 1.268 | 1.976 | 5.541 | Adds a model response and retrieved policy context. Full pipeline including groundedness scoring. |
injection_attempt | 0.218 | 0.247 | 0.311 | Trips the injection scanner and returns at stage 5. Faster than a full pass by design. |
Cold start — the first evaluation in a fresh process, including lazily-imported modules — measured 0.165 ms. That is a single sample, because you only get one first call.
Where the time goes
Each stage measured on its own entry point, with the same payload. PII detection is the dominant cost: regex across 24 pattern families over every parameter, twice — once to scan, once to redact.
| Stage | p50 | Execution type |
|---|---|---|
| 2 — PII scan | 0.355 | Deterministic · regex |
| 2 — PII redact | 0.356 | Deterministic · regex |
| 5 — Injection scan | 0.304 | Deterministic · pattern matching, authored confidences |
| 4 — Safety scoring | 0.253 | Deterministic · keyword and regex, authored weights |
| 3 — Toxicity filter | 0.052 | Deterministic · keyword matching |
| 6 — Hallucination check | 0.004 | Deterministic · consistency rules |
| 8 — Data quality | 0.003 | Deterministic · arithmetic over counts |
These do not add up to the end-to-end figure
They are measured out-of-band, by calling each stage’s entry point directly, so the production decision path carries no instrumentation overhead to produce a marketing number. The consequence is that per-stage timings apportion cost — they show you which stages are expensive — but they do not decompose a specific request. If you add this column up and compare it to 1.27 ms, the numbers will disagree. That is expected.
Every stage above is deterministic: pattern matching and arithmetic over hand-authored constants, with no fitted parameters and no model call anywhere in the decision path. One component in the wider platform is genuinely statistical — the anomaly scorer, which maintains a running mean and variance — and it can be disabled, which is what makes the SR 26-2 deterministic-mode attestation possible. The classification is verified against the code by an AST scan, not maintained by hand.
How to reproduce this
git clone <repo> && cd integrity-stack python -m venv venv && source venv/bin/activate pip install -e ".[dev]" make benchmark
Results land in benchmarks/results/latest.json — diff that against ours — and benchmarks/results/latest.md. Run with other applications closed; percentiles are sensitive to CPU contention. Expect p50 within tens of microseconds on the numeric cases and a few tenths of a millisecond on the text cases. p99 is noisier and will vary more.
The trap we hit, so you don’t have to
Rate limiting is stage 1 and it returns early. A benchmark that fires requests at a single agent trips the configured limit and then measures the rejection path for every remaining iteration — roughly 0.085 ms, about a fifteenth of a real text evaluation, and completely meaningless as a verdict latency.
Our harness raises the limit for the measurement and reports the rejection path separately, as its own line, because it is a real code path with a real cost that should never be mistaken for a decision. If you benchmark this system yourself and get a suspiciously good number, check whether you are being rate limited.
This is also how we found that the configured rate limit was being accepted by the API and never actually read. It is fixed. The benchmark is why we noticed.
What these numbers do not tell you
- In-memory stores only. Figures with PostgreSQL persistence enabled will be different and are not measured here.
- No network. The HTTP figures use an in-process client. Add real transit for a deployed number.
- Single request, no concurrency. These are latencies, not throughput under load. We have not published a load profile and will not imply one.
- One machine. The hardware is stated above. Numbers from different hardware are not comparable, including ours to yours.
- No LLM in the decision path. That is what makes these numbers achievable, and it is an architectural property rather than an optimisation. It also means the pipeline detects known attack families rather than reasoning about novel ones.
If you run this and get materially different numbers, and you think ours are wrong, we would like to hear about it.
Design partners
Test it against your own workload.
The benchmark uses our payloads. Yours will be different. We will run it against a representative sample of your traffic and show you the numbers either way.