Two and a half years of shipping a government chatbot slowly

Case in one sentence

The UK Government Digital Service spent two and a half years building, testing, and publicly reporting on an AI assistant for GOV.UK, ran it past more than ten thousand members of the public in two pilots, and improved its accuracy from a self-assessed 76% to a self-assessed 90% — a figure nobody outside the team has ever checked.

Executive summary

Most cases in this library exist because something went wrong or a regulator intervened. This one exists because a deployment simply ran, and the team running it published what happened.

GOV.UK Chat began as a prototype in July 2023, built with Langchain and Gradio.¹ It is a retrieval-augmented generation system: it searches GOV.UK’s published guidance, retrieves the relevant pages, and generates an answer grounded in them.²

Testing was staged deliberately. Three internal phases including cross-government red teaming, then a dozen users, then a thousand invited users on a live private pilot in late 2023.² The January 2024 write-up of that pilot reported that nearly 70% of surveyed users found responses useful and just under 65% were satisfied — and, in the same post, that answers “did not reach the highest level of accuracy demanded for a site like GOV.UK”, that hallucinations occurred, and that some users underestimated the risk of inaccuracy precisely because the GOV.UK brand made them trust it.²

A second pilot followed in November 2024, aimed at people running businesses, with accuracy scored alongside HMRC subject matter experts.³

By March 2026 the team reported the totals: two public pilots over 18 months, 10,136 users and 23,838 questions on the web, 641 users and 2,670 questions in the GOV.UK app. Accuracy up from 76% to 90%. An 88% answer rate for in-scope questions. All 508 jailbreak attempts blocked. An average of 10.7 seconds per answer. And, in the app follow-up survey, 73% finding it useful and 64% satisfied.⁴

Those last two figures are worth holding against the first pilot’s “nearly 70%” useful and “just under 65%” satisfied.² Two years and fourteen points of accuracy later, user satisfaction is where it started.

On 26 March 2026 the assistant went live in the GOV.UK app with no announcement, and on 14 May it was announced. In between, more than 7,800 people asked more than 15,000 questions, and the team reported that it was seeing “the same level of performance that we achieved in our recent pilot” — a sentence containing no number.⁵

Every number in this case comes from the team that built the system. The National Audit Office examined government AI adoption in the same period and does not mention GOV.UK Chat once.⁶

Research question

What does a carefully run public-sector AI deployment look like from the inside — and how much can be concluded from even the most detailed self-report?

Organization and operating context

The Government Digital Service runs GOV.UK, the single domain for UK government information and services, which the team describes as having more than 700,000 pages.³ The GOV.UK AI team is multidisciplinary: data scientists, developers, user researchers, content designers, and interaction designers.³¹

Two features of the context shape everything else. First, the subject matter is tax, benefits, and visas — questions where a wrong answer has consequences and where the correct answer is often conditional on circumstances the site cannot see. Second, GOV.UK is trusted, and the team knows it, describing the site as “a highly trusted (according to internal polling) information source and a ubiquitously recognised symbol in the UK.”² That trust is the asset the deployment risks.

The governance environment around it was, by the independent auditor’s account, unsettled. The NAO reported in March 2024 that government “lacked a coherent plan to support adoption of AI in the public sector”, that responsibility was split between DSIT and the Cabinet Office with no stated overall ownership, and that the draft adoption strategy set out no implementation plan, performance metrics, or funding.⁶ GOV.UK Chat is not mentioned in that report.

The situation before AI

The problem predates the technology. The team frames it as “a problem that’s as old as GOV.UK itself — how we can save people time and make their interactions with government simpler, faster and easier.”² The existing answer is search plus navigation across hundreds of thousands of pages, and for people who cannot find what they need, a departmental call centre.

No baseline is published. There is no stated figure for how long users currently take to find an answer, how often they succeed, or what a call to HMRC costs — which means every claim about time saved in this case is a user’s impression rather than a measurement against a starting point.

The AI intervention or event

Architecture. A conversational RAG system combining semantic search with generation. GOV.UK content is filtered by metadata to prioritise authoritative, current sources and to exclude pages containing personally identifiable information, then split by hierarchical semantic chunking that preserves the HTML header structure. Retrieval uses semantic search with a metadata-based re-ranking layer built with content-design and information-architecture colleagues, on Amazon’s Titan embeddings. A tool-calling router classifies user intent — greeting, request for advice, genuine question — and picks a response strategy. Generation uses chain-of-thought and goal-oriented prompting with structured JSON input and output, instructions to answer only from GOV.UK guidance and in GOV.UK style, and citation of the content used. Two guardrail layers sit either side of generation, one blocking inappropriate queries and one validating responses.¹

The stack moved from Langchain and Gradio to a Ruby application on AWS, using Anthropic’s Claude models via Amazon Bedrock, built so models can be swapped as new ones appear.¹⁴

Evaluation framework. Three pillars: automated evaluation using LLM-as-a-judge and custom metrics against ground-truth datasets; manual expert review of conversation logs with content designers and subject matter experts, plus red teaming with the Incubator for Artificial Intelligence and the AI Security Institute; and automated live monitoring flagging safety and quality issues for human review. Six criteria: groundedness, relevance and answer rate, factual accuracy, factual completeness, reliability, and reputational safety — with each iteration required to improve at least one without degrading the others.¹

Privacy. A data protection impact assessment was conducted, safeguards were put in place to stop users submitting personal data, and GOV.UK pages containing personal data were removed from the system so they could not be sent to the model.²

The pilots. Late 2023: 1,000 invited users. November 2024: a private beta linked from selected business pages, using a waiting list, estimated at four weeks.³ Then the app pilot via iOS TestFlight. Cumulatively, more than 10,000 users and 26,000 questions.⁴

The launch. GOV.UK Chat went live in the GOV.UK app quietly on 26 March 2026, with no proactive communications, so that the team could watch it under real conditions; it was announced publicly on 14 May 2026. In the weeks between, more than 7,800 people asked more than 15,000 questions. The team calls this the biggest change to how people interact with government content since GOV.UK launched in 2012, and says safety checks were carried out before release with the AI Security Institute.⁵

The launch post’s only statement about quality is that the team is “seeing the same level of performance that we achieved in our recent pilot.”⁵ Which level, measured how, against which of the six criteria, is not said. The service left pilot on a claim with no number in it.

Outcomes and economics

All figures are the team’s own.²⁴

The research effort behind them is substantial and itemised: 99 timed usability tests, a 15-person diary study with 30 hours of video, 576 surveys covering 2,135 questions, manual scoring of 419 conversations and 1,084 question-answer pairs, path analysis of 4,764 journeys, and detailed qualitative analysis of 29 conversations on the web pilot; then answer-quality review of 1,338 in-scope questions, a 144-participant answer-speed study across four speeds and three response modes, 12 observed users across three segments, and 45 survey responses in the app study.⁴

On economics there is nothing. No development cost, no running cost, no inference spend, no staff cost, and no estimate of the call-centre contacts avoided — the one saving the case repeatedly gestures at, in the form of users saying an answer saved them “a phone call to a department’s call centre.”⁴

Causal assessment and competing explanations

Labelled descriptive, and firmly so. Nothing here is designed to establish that GOV.UK Chat caused any outcome. There is no control arm, no comparison against GOV.UK search, and no before-and-after on any task metric. Users report saving time; nobody measured time saved.

The accuracy movement from 76% to 90% has a second explanation the team states itself: it credits both “the work of our data scientists” and “advances in the underlying AI models.”⁴ Since the system moved to newer Anthropic models over the same period, the improvement cannot be attributed to the team’s engineering, to model progress, or to the two in any particular proportion. The posts do not attempt to separate them, and neither does this case.

The satisfaction figures deserve a competing explanation too, because the obvious reading is uncomfortable. Useful went from nearly 70% to 73%; satisfied went from just under 65% to 64%.²⁴ The innocent explanations are real: the populations differ (1,000 invited users versus iOS TestFlight participants), the samples are small and differently constructed (n=157 versus n=45), and the second study deliberately focused on areas that had changed. But the flat result across a fourteen-point accuracy gain is at least consistent with satisfaction being driven by something other than factual accuracy — speed, for instance, which the team identifies as the thing users wanted most.⁴

Failures, limitations, and governance

• The headline metric is self-graded against an unpublished rubric. Accuracy is assessed by the team’s own subject matter experts and by automated tools including LLM-as-a-judge, with the ground-truth dataset and the scoring standard described but not released.⁴¹
• Ten per cent inaccuracy is characterised, not sized. The team says inaccuracy is “often” a matter of answering some but not all of a question.⁴ How often, and what the remainder consists of, is not stated — so a reader cannot tell how many wholly wrong answers about tax or visas the system produces.
• No error budget was ever published. The first pilot found accuracy below “the highest level of accuracy demanded for a site like GOV.UK”.² What that level is, numerically, is never given, so there is no stated threshold that 90% either meets or misses.
• The comparative claim carries no evidence. That GOV.UK Chat “scores higher than widely-used consumer AI assistants” on government questions is asserted with no benchmark, no assistants named, and no figures.⁴
• The brand-trust finding was reported once and never followed up. In January 2024 the team found users underestimating inaccuracy risk because of the GOV.UK brand.² The 2026 posts report that users appreciate the source-checking features, which is not the same thing, and offer no measurement of whether misplaced confidence persists.
• Sample sizes shrink as the claims grow. Ten thousand users in the headline; 45 survey responses behind the satisfaction figure.⁴
• “All 508 jailbreak attempts prevented” is self-adjudicated. What counted as an attempt, and who decided each was prevented, is not stated.⁴
• Independent scrutiny of government AI existed and did not reach here. The NAO audited AI adoption across government in March 2024 without mentioning this deployment.⁶

What this case demonstrates

1. A public-sector team can run a generative AI deployment slowly and publish against itself. The first published finding was that the system was not accurate enough, and it was published anyway.²
2. Staged exposure works as a governance mechanism: three internal phases, a dozen users, a thousand, then two public pilots, with a decision point at each.²
3. Institutional trust is a deployment risk, not only an asset. Users discounted the inaccuracy warnings because the answers came from GOV.UK.²
4. Accuracy and satisfaction can move independently. Fourteen points of measured accuracy bought no measurable satisfaction, and the team’s own reading is that users now want speed.⁴
5. Speed and accuracy trade against each other in current systems, and the trade is a design decision someone has to own. GDS chose accuracy and accepted 10.7 seconds.⁴
6. An evaluation framework can be specified in advance and made public — six named criteria, three measurement pillars, a rule that no iteration may degrade another criterion.¹
7. Detailed methods reporting is not the same as verifiable evidence. This is the most methodologically transparent deployment in this library and still the least independently checked.
8. Two and a half years from prototype to widening access is what “not moving fast and breaking things” costs, and the team said that was the point.²

What this case does not demonstrate

• It does not establish that the system is 90% accurate. It establishes that its builders assess it at 90% against a rubric they have described but not published.
• It does not show that users got better outcomes. No source measures whether anyone completed a task correctly, filed the right form, or avoided an error.
• It does not show the deployment saved money or staff time. No cost or call-deflection figure appears anywhere.
• It does not establish that the improvement came from the team’s engineering rather than from better underlying models. The team credits both without separating them.⁴
• It does not show the system is safe against jailbreaking in general — only that the team judged 508 specific attempts to have been blocked.
• It does not tell you that GOV.UK Chat is generally available across GOV.UK. It launched in the GOV.UK app on 14 May 2026, and the team’s stated intention to test it on the website is unconfirmed in anything read here, which is why the deployment stage is limited-production rather than scaled-production.⁵
• It does not establish that performance held at launch. The launch post says the team is seeing the same level of performance as in the pilot and gives no figure, no criterion, and no method.⁵
• It does not report any independent assessment, because none exists in the record.
• It does not describe harm. No source read here reports a user acting on a wrong answer, and the absence of such a report is not evidence that none occurred.

Evidence assessment

Grade C — the most transparent self-report in this library, and still a self-report.

This grade is a judgment call and worth defending, because on effort and candour the record is excellent. The team published eleven named research methods with sample sizes for each.⁴ It published its evaluation framework and criteria before claiming success against them.¹ It published its worst finding first — that accuracy fell short of what GOV.UK requires, that the system hallucinated, and that the institution’s own credibility was causing users to over-trust it.² Authors are named with job titles, posts are dated, and the lead product manager answers public questions in the comments. Very little in this library is documented this well.

But the library’s grade measures the evidence, not the good faith. Every figure here — accuracy, satisfaction, answer rate, response time, jailbreaks blocked — was produced, scored, and published by the team whose work it evaluates. The accuracy rubric is not public. The ground-truth dataset is not public. The comparative claim against consumer assistants has no data behind it at all. Under the library’s own definitions that is “largely self-reported… or only partly corroborated”, which is grade C.

The one genuinely independent body in this record establishes the absence rather than filling it. The National Audit Office examined government AI adoption in March 2024 and found strategy and governance wanting across the board; searching the report finds no mention of GOV.UK Chat, two of the Government Digital Service, and one of “chatbot”.⁶ It is cited here as context and explicitly not as corroboration, and its presence in the front matter should not be read as a second evidentiary chain — this case rests on one.

Two further limits belong on the record. The four GDS posts span two and a half years and describe different pilots, but they are one team on one blog, and treating them as separate chains would be the same category error this library declined to make in its METR case. And the comparison this case draws between the 2023 and 2026 satisfaction figures joins two studies with different populations, different recruitment, and samples of 157 and 45 — it is offered as an observation worth explaining, not as a measured non-improvement.

One small discrepancy is recorded rather than resolved: the January 2024 post gives the first-pilot survey as n=157, while the March 2026 post describes 576 surveys analysed for the web pilot.²⁴ These are almost certainly different surveys from different phases rather than a contradiction, but no source states the relationship, so the figures are reported separately and not combined.

Material claims

Direct quotations

“Overall, answers did not reach the highest level of accuracy demanded for a site like GOV.UK, where factual accuracy is crucial. We also observed a few cases of hallucination - where the system generated responses containing incorrect information presented as fact - mostly in response to users’ ambiguous or inappropriate queries.”

— Gregory, Tosi, McDonald, and Sewell, reporting on their own first pilot² · locator: “Our early findings”, fourth paragraph

“We also found that some users underestimated or dismissed the inaccuracy risks with GOV.UK Chat, because of the credibility and duty of care associated with the GOV.UK brand.”

— Gregory, Tosi, McDonald, and Sewell² · locator: “Our early findings”, final paragraph

“These findings validate why we’re taking a balanced, measured and data driven approach to this technology — we’re not moving fast and breaking things.”

— Gregory, Tosi, McDonald, and Sewell² · locator: “What’s next”, first paragraph

“Industry-wide, no one has been able to reach 100% accuracy when generating answers. This means managing the risk of ‘hallucinations’ – where the system generates responses containing incorrect information presented as fact – is part of working with this technology right now.”

— Sam Dub and Josh Davey, announcing the second pilot³ · locator: “Making improvements and addressing risks”, second paragraph

“In cases of ‘inaccuracy’, it’s often that GOV.UK Chat is able to answer some but not all aspects of a user’s question.”

— Sam Dub and Sharon McDonald, characterising the residual 10%⁴ · locator: headline finding 2, “We’ve made strong progress on increasing accuracy and answer quality”

“For us, accuracy is the most important thing, and consequently GOV.UK Chat responses are slower than we’d ideally like; an average 10.7 seconds for an answer.”

— Sam Dub and Sharon McDonald⁴ · locator: headline finding 5, “Users want answers even faster”

“Defining clear success criteria, measuring every change, and being willing to abandon what does not work ensures AI applications evolve responsibly and effectively. Looking closely at outputs, not just metrics, is what drives meaningful iteration.”

— Alessia Tosi and Nick Lange¹ · locator: “Key lessons and next steps”, first lesson

“The government lacked a coherent plan to support adoption of AI in the public sector as part of its 2021 National AI Strategy.”

— National Audit Office, in an audit that does not mention this deployment⁶ · locator: Summary, Key findings, paragraph 8

Revision notes

• 2026-09-13 — Source review, and a correction. This case recorded at publication that “nothing read here confirms a public launch”, and set deployment_stage: pilot on that basis. GOV.UK Chat had in fact launched: quietly in the GOV.UK app on 26 March 2026 and publicly on 14 May 2026, announced on the GDS blog rather than the Inside GOV.UK blog the case’s other sources come from, which is why the earlier sweep missed it. Stage corrected to limited-production — live to everyone using the GOV.UK app, not on the GOV.UK website. Added the launch post as a sixth source in the existing gds-govuk-chat family, a new claim, and a new section. outcome stays inconclusive: shipping is a decision, not a measurement, and the launch post’s only quality claim is that the team is seeing “the same level of performance that we achieved in our recent pilot”, with no figure, criterion, or method attached — which is the case’s own subject appearing once more. The upgrade path is narrowed rather than met: a search in September 2026 found no National Audit Office or Public Accounts Committee work naming this deployment, and the AI Security Institute’s pre-release testing, now named in two GDS posts, has produced nothing public. That last one is the most specific outside assessment known to exist, and it is the one to ask for.

• 2026-09-13 — Initial publication at Grade C, and the grade is the main editorial call in this case. The record is the most methodologically transparent in the library — eleven named research methods with sample sizes, a published evaluation framework, and the worst findings published first — but every figure was produced and scored by the team whose work it evaluates, the accuracy rubric and ground-truth dataset are unpublished, and no independent party has measured the system. Under the library’s own definitions that is C, not B, and the case says so rather than rewarding good faith with a grade the evidence does not carry. The NAO report is cited for governance context and explicitly not as a second evidentiary chain; the four GDS posts are one family, consistent with the reasoning applied to METR in AAI-2026-008. outcome is inconclusive because the deployment reports positive indicators and no outcome measure for citizens; deployment_stage is pilot because no source confirms general availability. The observation that satisfaction did not move while accuracy rose fourteen points is labelled Inference and its confounds are stated. First deployment case in the library, which had covered incidents, evaluations, regulatory actions, and economic arguments but never a deployment that simply ran.