← Build, buy and runFree decision tool · Build, buy and run

Fine-tune it, or gate it?

Seven inputs covering sensitivity, revocability, regulation, and the value this data adds. A weighted score returns a clear call, with the risk flags a "yes" answer still needs.

Listen to this briefing

Should You Fine-Tune, or Use Retrieval?

0:00

How the call is made

Each input is placed on a 0 to 1 scale and weighted. Model value is the only input that raises the fine-tune suitability score; the other six lower it. The score lands in one of three bands, and the bar for fine-tuning sits at 70 because a fine-tune cannot be undone.

How the call is made

Seven weighted inputs and the three bands of the suitability score

Model value×2.5
Blast radius if leaked×2.2
Revocability need×1.8
Regulatory obligation×1.5
Restriction level today×1.3
PII density×1.2
Update frequency×1
A higher answer raises the scoreA higher answer lowers the score
Suitability score out of 100
0 to 49Retrieval-only, gatedServe it through permission-gated retrieval
50 to 69RAG + targeted fine-tuneGate the facts, fine-tune behaviour on de-identified data
70 to 100Fine-tune safeSafe to bake into the model

The fine-tune bar sits at 70 instead of 50 because fine-tuning is irreversible. A dataset at the extreme on both blast radius and regulatory obligation is flagged for legal review before it touches any AI system, gated or not.

Weights and thresholds as used by this tool's scoring model. Each answer is placed on a 0 to 1 scale before weighting.

Your dataset

0 of 7 inputs set. Drag or tap each slider, including any you would leave at 3.

Not set

How restricted is access to this data today, before any AI touches it?

Not set

If this data needed to be deleted or revoked later, how important is that?

Not set

Is this data subject to specific retention, deletion, or processing rules?

Not set

How much would training directly on this data improve the model's usefulness?

Not set

How much personal or identifiable information does this data contain?

Not set

How often does this data change, get corrected, or need updating?

Not set

If this surfaced to the wrong person, how severe would the harm be?

Your call

Answer to start. Set any input and the suitability score appears here. The call firms up once all seven are set.

The possible calls

Fine-tune safe

When it applies

The data adds real value to the model, and you would be comfortable handing it to every employee who has access to the model. Low sensitivity, no meaningful revocability need, and nothing that a regulator would ask you to trace or erase later.

What to still do
  • De-identify any residual PII before training, even at low density
  • Document the training set for EU AI Act traceability requirements, and confirm the process satisfies GDPR's right-to-erasure obligations for any personal data involved
  • Re-check this classification if the dataset's contents change materially

Retrieval-only, gated

When it applies

The data is sensitive, regulated, changes often, or you need the ability to revoke access later. A model can't selectively forget what it memorised during training, so this data stays in a document store where every query is filtered against the requester's clearance before anything reaches the model.

What to still do
  • Gate at query time with row-level security, not after the model has already generated an answer
  • Redact structured sensitive fields at ingestion, before they reach the screen
  • Log every retrieval with an identity trail for audit and incident response

RAG + targeted fine-tune

When it applies

The score is borderline: the data is neither clearly safe to bake into a model nor clearly sensitive enough to rule out fine-tuning entirely. Current guidance treats this as a false binary and recommends a split: keep the fact-bearing or sensitive core in gated retrieval, and fine-tune separately, on de-identified or synthetic data, for stable behaviours like output format, tone, or refusal patterns.

What to still do
  • Never fine-tune the behaviour layer on the raw sensitive or fact-bearing data, use a de-identified or synthetic set built for that purpose
  • Test the fine-tuned layer with extraction prompts before deployment to confirm it does not leak facts from the gated retrieval layer
  • Classify the retrieval corpus and the fine-tune set separately at the next review; they often land on different sides of the threshold

If it's still not comfortable

When it applies

A small category of data should not enter an AI system at all, gated or otherwise: material under legal hold, privileged legal communications, or data with an explicit contractual or regulatory bar on automated processing. This tool scores fine-tune against retrieval and does not decide that third question.

What to still do
  • Route anything scoring high on both Regulatory Obligation and Blast Radius to legal review before it touches any AI system
  • Treat "gated retrieval" and "not connected to AI at all" as different answers
  • Re-run this check after a legal or compliance review changes the facts
Terence Kok
Before You Go

I built this after one too many meetings where someone asked, can we just fine-tune on this data, and nobody had a clean way to answer. I set the fine-tune-safe threshold at 70 out of 100 instead of the usual 50, because fine-tuning is permanent: once a fact is baked into a model's weights, you cannot selectively delete it later. Scores between 50 and 69 now get a third answer instead of a forced coin flip: keep the sensitive core in gated retrieval, and fine-tune separately, on de-identified data, for the parts of the job that are about behaviour instead of facts. If your dataset lands in that band, treat the discomfort as useful information about the dataset.

Terence Kok

Last updated 29 Sept 2026

More in Build, buy and run

Next in this group · Decision toolAI Incident Response PlannerIf one of our AI systems fails tomorrow, who switches it off, and which regulator clock has started?Build the playbook →
← All 8 in Build, buy and run