The full method behind every number — open, so you can verify it.

1 · The four failure modes

What we measure — and the ground truth each one requires

Each stratum measures one failure mode and must have machine-checkable expected behaviour. The dangerous failures hide in what the model does when the answer is not in the text.

2 · Building the test set for a new rule-set

Construction, corpus by corpus

1. Corpus: the client’s rule-set, unchanged (that IS the deployment form). Segment it into units.
2. Seed questions per unit, in every relevant language — one per failure mode: direct (covered), scenario-shaped (looks uncovered), neighbouring topic (related but uncovered), and entirely outside the corpus.
3. Machine-checkable key facts (regex over the answer) for the answerable types, so correctness is scored without a subjective judge and can be audited.
4. No teaching to the test: eval questions must not share content with any training or installation data.

3 · Power — enough questions for the number to mean something

At least 30–40 per type per language · always with a confidence interval

Twelve questions per type is under-powered: “42 %” could in reality lie anywhere between 19 % and 68 %. Hence: at least 30–40 per stratum per language, and every rate is always reported with its 95 % Wilson confidence interval — never a bare number. We expand each seed with paraphrases in the same language (preserving the underlying case), and report both the number of cases and the number of questions, so the figure covers robustness to phrasing, not just how many cases there were.

4 · We challenge our own measurement

Adversarial validation of the method — documented

A weak rate must not be caused by a mislabelled question. So we validate our own test set:

1. Retrieval integrity (always): the answerable types MUST retrieve their section in the top 3; the neighbouring topic must retrieve a related section; the entirely uncovered one must not. Otherwise the question measures a search failure, not the model’s behaviour — and it is fixed or dropped.
2. An independent, stronger model as judge — blind to our label, it decides from the retrieved passages whether the question is answerable. If it disagrees with our expectation, the question is a fault in OUR eval → it is dropped from the powered set and logged.
3. The drop log is the evidence: how many were dropped, by reason. The check routinely catches our own hand-made questions too — that is what makes it real.

5 · The misuse arm

Two numbers, not one

The input layer that refuses misuse (instruction override, system-prompt leakage, handing over other people’s data, help with circumvention, prohibited content, off-scope) is measured on two numbers, because either one without the other is worthless: recall (share of misuse correctly refused) AND a precision check (share of legitimate, sensitive “may I …?” questions that are NOT wrongly refused). A gate that refuses everything scores perfectly on the first and uselessly on the second. The categories are grounded in the OWASP Top 10 for LLM applications.

6 · The knobs we calibrate

The three load-bearing knobs

Each knob is treated as a calibration parameter per instance, not as a fixed default, and its value and the basis for it are documented in the audit pack. On top of that come retrieval and prompt parameters (search weights, k, chunking, prompt templates); they are not hidden, but documented per run in the same way, so the whole setup can be verified.

7 · Instrument check before use

We check that the test can measure anything at all

8 · Self-service — an auto-generated test set from any corpus

The same validation, whoever built the questions

The test set can be generated automatically from an uploaded rule-set (the four types per unit). The auto-generated questions are not exempt from the validation in point 4 — they go through exactly the same adversarial check, and the drop rate is reported. Uploaded content is treated as data, never as instructions: every unit is screened for instruction-like patterns and flagged for human review. That is a first filter, not a guarantee against every attack, so a flagged corpus is reviewed by hand before deployment.

9 · Re-validation and reporting

When something changes, we measure again

The test set is rebuilt and re-validated when the corpus or the rules change, on a new language, or on a model change (continuous risk management, Art. 9). The test set is versioned like any other data. Every rate is accompanied by its count + 95 % CI and a note that the question types are adversarial diagnostic arms, not representative traffic — over-refusal on the hardest arm is not the same as over-refusal on all traffic, and both denominators are stated.

Note: technical method and measurement, not legal advice. The numbers support declaration under Regulation (EU) 2024/1689 Art. 15 (accuracy/robustness) and Art. 9 (continuous risk management); the legal classification is confirmed by your own compliance function.