Whitepaper · choosing a model for rule-following

What do you get from a big model — and what does it cost?

A guide to choosing and hosting a language model that has to answer from your own rule-set. Built on measurements and established research, not promises.

“How big a model can we afford?” is the wrong first question. Size buys language and breadth, but not trustworthy retrieval of your rules, and a model that is too big carries its own costs. This guide takes five steps, from what size actually buys to how you can verify the reliability yourselves.

Contents

We measure what your task actually requires, and pick the smallest model that handles it with margin. The whole method behind the number is open, so you can verify it yourselves.

Part 1

“Big” is three things, not one

A language model’s abilities sit on three independent axes. Customers, and quite a few vendors, treat them as one scale from small to big. The difference between the three is the whole difference between a system that sounds convincing and one that can be trusted.

Axis 1

Training volume

→ what it knows about the world

How much and how broadly the model was trained decides its general knowledge. That knowledge is stored as a fixed, data-determined capacity2 and covers the rare questions poorly3.

Does not buy: knowledge of your corpus. The model has never seen your handbook. Citing your rule is retrieval’s job, not size’s.

Axis 2

Parameters

→ how fluently it writes

The number of parameters decides how well the model writes, and how complex a task it can keep track of in a single answer1. A larger model also tolerates more competing requirements before adherence falls.

Does not buy: better retrieval. A big model writes a convincing answer even when it is wrong.

Axis 3

Architecture

→ safety + correct retrieval

Making the model retrieve the right rule, stick to it and abstain when the answer is missing is built in the layer around the model: scoping (RAG), grounding checks and a calibrated threshold.

This is where safety is built. It comes from the layer around the model, not from its size. The architecture is built to make the model abstain rather than guess, and to catch fabrication when it happens. No size does that on its own.

Axes 1 and 2 rise with size. Axis 3 does not; it is built, whatever the model’s size. Precisely where general knowledge fails on the rare question, good retrieval finds the answer better than the big model’s parametric memory4.

A larger model buys broader world knowledge and better prose. It does not buy better retrieval of your rules. That guarantee comes from the architecture.

Part 2

Bigger is better — but rarely worth the money

The bigger the model, the better it gets. But at some point it handles your rule-set reliably, and from there extra size buys no more usable compliance. It just costs more: the biggest model cannot run EU-sovereign or on-premise, it costs more than the value you get from it, and it draws more power and CO2. The right choice is the smallest model that reliably handles your task. Where that boundary sits does not depend on the number of rules, but on how many requirements compete for the model’s attention in the same answer — and that is what we measure.

Value for money against model size
TOO SMALL cannot apply the rule TOO BIG not EU-sovereign · expensive · more CO2 THE RIGHT CHOICE smallest model that handles your task MODEL SIZE → VALUE FOR MONEY

Beyond price and latency: here is what size actually means for compliance — from the research and from our own measurements:

Over-refusal is a small-model problem

Safety training can make a model refuse legitimate questions that merely look like something prohibited89 — but it only bites on safety-adjacent content. In a rules and HR domain, over-refusal is on the contrary worst on the small models (capacity), and is handled by the gate layer. That argues against choosing too small — not against choosing big.

No more faithful in RAG

Even the largest models hallucinate against the text they are handed121314. More parameters do not solve the grounding problem.

Can be less truthful

In some cases the largest models reproduce common misconceptions more convincingly and are therefore less truthful5. The effect is often U-shaped and no law6.

Sycophancy

More aligned models tend to agree with the user and adopt the user’s framing rather than correct it7. That is a problem when the answer should be no.

The commonest misconception: “big models know better when they are wrong”

That holds for raw, pre-trained models11. But the alignment training that makes a model helpful typically degrades calibration again and makes it overconfident10. Calibration is an effect of how the model was post-trained, not of how many parameters it has. That is why we measure and calibrate it per model, language and corpus.

Part 3

Where should the model run?

There is no one right place to host a compliance assistant. It depends on how data-conscious and jurisdiction-sensitive your organisation is. The vendors are named to show the landscape, not as recommendations.

Two axes that get confused

Origin

Who made the model’s weights. EU origin (EuroLLM, Mistral) strengthens a “purely European” story; American or Chinese origin (Llama, Qwen, GLM) can still be hosted in the EU.

Hosting

Where and by whom the model runs. It is hosting, not the origin of the weights, that decides where personal data physically sits. GDPR follows hosting and data.

OptionData locationJurisdiction / ownerUS legal exposureGPU & priceOperational burden
Own hardware / on-premyour own server roomYour own premisesYour own — full controlNoneYou buy the GPUs yourselves (capex)High
EU-owned cloudHetzner · Scaleway · OVHcloudEU data centreEU companyNoneGood; often cheap (Hetzner)Medium
US hyperscaler, EU regionAWS · Azure · Google CloudEU region (data in the EU)US parent companyYes — CLOUD Act / FISAWidest choice + managedLow
Managed EU inferenceScaleway · Mistral-APIEUEU companyNonePay per call; no GPU operationsLow

Own hardware / on-prem

self-hosted · your own server room or your own GPU

Pros

  • Data never leaves the building — the strongest sovereignty and audit defensibility.
  • No third party, no foreign legal exposure.

Cons

  • A high up-front investment in GPU hardware you have to source yourselves.
  • The whole operation sits with you, and it is hard to scale on demand.

Typically suits: heavily regulated or very data-conscious organisations with their own IT operations.

EU-owned cloud

Hetzner (DE) · Scaleway (FR) · OVHcloud (FR)

Pros

  • EU data centre and EU ownership, with no US parent and no CLOUD Act exposure.
  • A purely European story, which many public-sector and data-conscious customers prefer.
  • Often competitive or low pricing.

Cons

  • The choice of GPUs is not always as wide as at the hyperscalers.
  • You still operate the model itself.

Typically suits: municipalities and data-conscious companies that want everything in the EU without the on-prem burden.

US hyperscaler in an EU region

AWS · Microsoft Azure · Google Cloud

Pros

  • The widest GPU choice and fully managed inference — the lowest operational burden.
  • Data can be kept in an EU region, and there is a growing range of “sovereign” offerings.

Cons

  • The parent company is subject to US law (CLOUD Act, FISA 702), which can in principle compel disclosure, including for data in an EU region.
  • Requires a valid transfer basis (Data Privacy Framework / standard contractual clauses) to be defensible under GDPR.
  • Many public authorities rule it out on sovereignty grounds, even where it is formally lawful.

Typically suits: commercial companies without a hard sovereignty requirement, prioritising capacity and low operations.

Managed EU inference (API)

Scaleway Generative APIs · Mistral La Plateforme

Pros

  • No GPU operations — you pay per call and get started quickly.
  • An EU provider means EU jurisdiction with no US legal exposure.

Cons

  • A smaller selection of models; you are tied to the provider’s catalogue.
  • The running per-call price can exceed self-operation at very high load.

Typically suits: smaller setups and pilots that want to start quickly and keep everything in the EU.

The hosting choice does not change the model’s reliability

Where the model runs decides jurisdiction, price and operations, but not whether it retrieves your rule correctly or abstains when the answer is missing. That reliability is a property of the architecture and is measured per model, language and corpus, whichever of the four options you choose.

The third bottom line: CO2

Model choice and hosting also carry a climate footprint. A right-sized (smaller) model typically draws less energy per answer, and EU hosting on a French low-carbon grid (Scaleway, for instance) lowers the footprint further. Scaleway supplies a compliance-ready CO2 and water account (kgCO₂e, scope 1–3, ADEME method, via dashboard or API) that you can feed straight into your CSRD/ ESG reporting. “Pick the smallest model that handles the task” is therefore a green decision too, not only an economic one.

Note: a technical overview, not legal advice. The lawfulness of a given transfer or provider is confirmed by your own data protection / legal function.

Part 4

What weighs heaviest for you?

We always confirm it by measuring on your own rule-set; the guide points, it does not decide.

If this weighs heaviest

Sovereignty and auditability

Often where you are a public authority, work under good administrative practice, or may be asked to account for a decision to a regulator.

Then it points to the smallest possible model, hosted on-prem or in an EU-owned cloud, where auditability weighs more than the last percent of coverage.

The measurement gives you a full trace per turn (decision, source and grounding), so a decision can be accounted for to a regulator. Danish is measured separately, and the system refers to a caseworker rather than making an automatic decision.

If this weighs heaviest

Documentable data protection

Often where you process personal data, want to be able to show it stays in the EU, and would rather see evidence than promises.

Then it points to a mid-sized model that handles your C with margin, hosted in an EU-owned cloud or on managed EU inference.

The measurement gives you fabrication resistance and abstention rates measured per language, documentation that data stays in the EU, and evidence that the model abstains on coverage gaps rather than inventing, ready for your own internal control under Art. 15 and 9.

If this weighs heaviest

Usability, speed and price

Often where you first and foremost want it to work, to keep the operational burden down, and where the hosting location matters less as long as it is lawful.

Then it points to a size chosen to maximise coverage on your C, hosted wherever it is easiest and cheapest — including a US hyperscaler in an EU region, if the personal-data exposure is low.

The measurement gives you coverage and over-refusal measured together, so the system does not irritate users with false refusals, and a threshold calibrated to your pain point.

Few weigh only one thing. If two concerns weigh equally, the choice becomes a trade-off, and we measure our way to it rather than guessing.

Part 5

How we measure — the whole recipe behind every number

A reliability figure is only worth trusting if you can see how it came about. The full, followable protocol is in the appendix.

What we measure

We do not just test “does it answer correctly?”. We split the questions into four types, each of which uncovers its own failure mode, because the dangerous failures hide in what the model does when the answer is not in the text.

Question typeExpectedMeasures
Covered questionthe answer is in a policyANSWER + factsCoverage — can it answer when the answer exists?
Looks uncoveredanswerable, but obliquely phrasedANSWER + factsOver-refusal — does it refuse legitimate questions?
Neighbouring topica related policy, the answer not in itABSTAINFabrication from a related-but-insufficient rule
Entirely uncoverednothing in the corpus covers the topicABSTAINFabrication when there is nothing to guess from

How we make the number trustworthy

  1. Enough questions. At least 30–40 per type per language, and every number is always accompanied by its 95 % confidence interval.
  2. We challenge our own questions. A stronger, independent model, blind to our answer key, checks whether each question really is answerable from the retrieved text. If it disagrees with our label, the question is discarded and logged — including our own hand-made ones.
  3. Retrieval integrity first. Before a question counts, we verify that retrieval finds the right policy, so we are not measuring a search failure.

The three load-bearing calibration knobs

CONFIDENCE threshold

How uncertain the model may be before it abstains.

per model × corpus

Coverage threshold

How well retrieval has to hit before there is enough to answer from.

per corpus

Grounding strictness

How firmly the answer has to be anchored in the retrieved passages.

per model

On top of that come retrieval and prompt parameters (search weights, k, chunking, prompt templates). They are not hidden: every value is documented per run, so the setup can be verified.

Instrument check before use

Spread: at least 30 percentage points between the easiest and hardest type on the model, otherwise the test is “masked”. Checked per model.
Format match: we profile in the same answer format we evaluate in.
Per language: a curve measured in one language does not transfer to another. Each language is measured separately.
Full data record per answer: decision, reasoning, source, grounding and raw data are kept, so any metric can be recomputed at audit.

Misuse is measured too

The input layer that refuses misuse is measured on two numbers, because either one without the other is worthless: what share of misuse it catches (recall), and how many legitimate sensitive questions it wrongly refuses (a precision check). A gate that refuses everything scores perfectly on the first and uselessly on the second.

Why show the whole recipe?

Because a reliability figure you cannot verify is not worth having. Everything above you can run again on your own rule-set. Few do, but the possibility is the whole difference between a claim and a proof.

Appendix

The full measurement protocol

The appendix is the precise, followable version of the method in Part 5, which you can follow and verify step by step. (Reproducing the numbers themselves exactly also requires the same frozen model versions and configurations we run in a given measurement — we state those per run.) We publish the method because the value lies in running it properly, calibrating it per model, language and corpus, and delivering a dated proof, not in keeping the method secret.

Construction for a new rule-set

  1. Corpus: the client’s rule-set, unchanged. Segment it into units.
  2. Seed questions per unit, in every relevant language with one per failure mode.
  3. Machine-checkable key facts (regex over the answer) for the answerable types, so correctness is scored without a subjective judge.
  4. No teaching to the test: eval questions share no content with any training or installation data.

Self-service — an auto-generated test set from any corpus

The test set can be generated automatically from an uploaded rule-set. The auto-generated questions go through exactly the same adversarial check as the hand-built ones, and the drop rate is reported. Uploaded content is treated as data, never as instructions: every unit is screened for instruction-like patterns and flagged for human review before anything is deployed. That is a first filter, not a guarantee against every attack, so a flagged corpus is reviewed by hand.

Re-validation and reporting

The test set is rebuilt and re-validated when the corpus or the rules change, on a new language, or on a model change (continuous risk management, Art. 9), and is versioned like any other data. Every rate is accompanied by its count + 95 % CI and a note that the question types are adversarial diagnostic arms, not representative traffic, and both denominators are stated.

Research basis

  1. Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. doi:10.48550/arxiv.2001.08361
  2. Allen-Zhu, Z., & Li, Y. (2024). Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. doi:10.48550/arxiv.2404.05405
  3. Kandpal, N., et al. (2023). Large Language Models Struggle to Learn Long-Tail Knowledge. doi:10.48550/arxiv.2211.08411
  4. Mallen, A., et al. (2023). When Not to Trust Language Models. ACL 2023. doi:10.18653/v1/2023.acl-long.546
  5. Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA. ACL 2022. doi:10.18653/v1/2022.acl-long.229
  6. McKenzie, I. R., et al. (2023). Inverse Scaling: When Bigger Isn't Better. doi:10.48550/arxiv.2306.09479
  7. Sharma, M., et al. (2023). Towards Understanding Sycophancy in Language Models. doi:10.48550/arxiv.2310.13548
  8. Cui, J., et al. (2024). OR-Bench: An Over-Refusal Benchmark. doi:10.48550/arxiv.2405.20947
  9. Röttger, P., et al. (2024). XSTest. NAACL 2024. doi:10.18653/v1/2024.naacl-long.301
  10. Tian, K., et al. (2023). Just Ask for Calibration. EMNLP 2023. doi:10.18653/v1/2023.emnlp-main.330
  11. Kadavath, S., et al. (2022). Language Models (Mostly) Know What They Know. doi:10.48550/arxiv.2207.05221
  12. Niu, C., et al. (2024). RAGTruth. ACL 2024. doi:10.18653/v1/2024.acl-long.585
  13. Chen, J., et al. (2024). Benchmarking LLMs in Retrieval-Augmented Generation (RGB). AAAI 38(16). doi:10.1609/aaai.v38i16.29728
  14. Ming, Y., et al. (2024). FaithEval. ICLR 2025. doi:10.48550/arxiv.2410.03727

Disclaimer: technical guidance and measurement, not legal advice. The numbers support declaration under Regulation (EU) 2024/1689 Art. 15 (accuracy/robustness) and Art. 9 (continuous risk management); the legal classification is confirmed by your own compliance function. The vendors named are examples, not recommendations.