Proving your AI system still behaves the way you said it did.
The hard question is not “does it work?” but “how do you know it still works, and who signed off on the change?”. We put the versioning, evaluation, monitoring and evidence in place to answer that without a fire drill.
It passed review in March. It is now September.
Between then and now the prompt was edited twice, the provider moved to a new model version, the knowledge base grew by forty per cent and two guardrails were relaxed to fix a complaint. Each change was reasonable. Nobody can say what the system's behaviour is now, or demonstrate that it has not drifted somewhere it should not have gone.
No regression suite
Quality is assessed by whoever is nearest, trying a few prompts they remember. Real regressions are found by customers.
Silent model updates
A provider deprecates a version and behaviour shifts underneath a system that had no pinned version and no alert.
Approval that leaves no trace
Someone senior said yes in a meeting. There is no record of what was approved, on what evidence, or what has changed since.
The operational discipline that makes an AI system boring.
Versioned everything
Prompts, model identifiers, retrieval configuration, guardrail policy and tool definitions held as versioned artefacts, so “what is in production?” has one answer.
Evaluation sets & gates
Golden sets, adversarial cases and refusal tests run automatically on every change, with thresholds that block a deployment rather than producing a report nobody reads.
Drift & quality monitoring
Sampling of live traffic, scoring against the same criteria, alerting on shifts in refusal rate, latency, cost per interaction and answer quality before they become complaints.
Approval workflow
Who reviewed, what evidence they saw, what they approved and when — recorded as part of the deployment, not as a separate document that goes stale immediately.
Incident & feedback loop
A path from a reported bad answer to a reproducible trace to a new test case, so the same failure cannot reappear quietly.
Evidence pack
A standing set of artefacts mapped to the controls you are subject to, generated from the system rather than assembled by hand the week before an audit.
Standards & ISO readinessFour to eight weeks, then it is part of how you ship.
Baseline
Establish what is actually in production today, which is usually the most uncomfortable week. Pin versions, capture current behaviour, record the gaps.
Measure
Build the evaluation sets with the people who own the outcome, including the adversarial and refusal cases, and agree the thresholds that would block a release.
Automate
Wire evaluation and approval into the deployment pipeline, and monitoring and alerting into the running system.
Operate
Review cadence, drift reports, incident path and the evidence pack, handed to a named owner on your side.
Write the controls into the architecture now.
The EU AI Act phases obligations in over several years, and the detail that applies to any given system depends on how it is classified. We are engineers rather than your legal advisers, so we do not tell you what your obligations are — we make sure the system can produce the evidence when your advisers tell you what to produce. Logging, versioning, human oversight points and traceability are cheap to design in and expensive to retrofit.
- Traceability from an output back to its inputs, retrieved context and model version.
- Documented human oversight points, with the record of what a human actually reviewed.
- Data provenance and retention that match the policy you have published.
- Evaluation evidence over time, not a single point-in-time report.
- A change history that shows what moved, when, and on whose approval.
Questions we get asked
Is this only relevant if we are regulated?▼
No. Every organisation that puts an AI system in front of customers or staff eventually gets asked why it did something. Regulation makes the answer mandatory; everyone else still needs it the first time something goes wrong in public.
Can you do this on a system somebody else built?▼
Yes, and it is a common request. We start with a short assessment of what is running, then add versioning, evaluation and monitoring around it. We do not need to have written the original code, and we will not recommend a rewrite unless the alternative is worse.
Does an evaluation gate slow releases down?▼
It adds minutes to a pipeline and removes days of unplanned work. Teams that put a gate in typically ship more often afterwards, because a change stops being frightening.
Who owns the evaluation sets?▼
You do. They live in your repositories from the first day, and they are usually the most durable asset produced by the whole engagement — models change, the definition of a good answer does not.
Do you provide legal advice on the EU AI Act?▼
No. We build the technical evidence and controls, and we work alongside whoever advises you on classification and obligations. Where a control is ambiguous, we implement the stricter reading and flag it for your advisers.
One free hour. No pitch, no obligation.
Bring the problem you are stuck on — an integration that keeps breaking, a cloud bill nobody can explain, an AI project that is all demo and no product, or a platform you are about to invest in. You will leave with a straight answer and a written summary, whether or not you ever work with us.
- Architecture and integration review
- AI feasibility — what will actually work, and what will not
- AWS, Azure and Google Cloud cost and design
- Business process audit and ISO readiness
- Technical due diligence before you invest or acquire