Saturn’s analysis of 18 AI models across 121 financial questions found wrong answers the majority of the time
Popular AI models give inaccurate financial advice more than half the time, according to research commissioned by financial technology firm Saturn, which is calling on the FCA to regulate AI financial advice before consumers face significant losses.
Saturn’s report, Artificial Authority: Should you trust AI to deliver financial advice?, tested 18 AI models against 121 financial questions, repeating each question five times to assess consistency. Across more than 10,000 questions in total, the models gave wrong answers 57% of the time. On harder questions, the error rate rose to 88% on average, with some models giving incorrect answers to complex queries 99% of the time.
The research was conducted by Saturn and reflects its own methodology and scoring criteria. The firm has a commercial interest in the outcome of FCA deliberations on AI advice regulation.
Free models performed worse than paid alternatives. Free AI tools gave incorrect answers in 63% of cases against 49% for paid models, with free models failing 93% of the most difficult questions.
Among the models assessed, the worst performer was Claude Haiku 4.5, which made mistakes in 82% of answers. Google’s Gemini 3.1 Pro was second-worst, failing 73% of tests. xAI’s Grok 4.5 made mistakes 59% of the time, and ChatGPT 5.6 Luna 58%.
The best-performing model overall was Claude Opus 5 (reasoning), which made mistakes in 39% of answers.
Saturn scored each model against specific criteria for every question, marking answers as a fail where they contained factual errors, missed key points or omitted important warnings.
The research identified several specific errors with potentially serious financial consequences. One mistake on pension tax rules could have resulted in a consumer facing a £17,500 HMRC charge. In debt advice scenarios, models recommended paying off highest-interest debts first rather than priority bills such as rent and council tax, which could expose someone already in debt to eviction, bailiff action or legal proceedings.
One model invented a rule stating that a graduate could halt student loan repayments by moving abroad, which would in practice have led to higher monthly repayments. A Gemini model also incorrectly told a borrower that taking a mortgage payment holiday would not affect their credit score.
The FCA has separately published The Mills Review, which found that 26% of consumers trust general-purpose AI tools for financial advice. The regulator has warned of the risks facing consumers who take financial advice from AI without the protections that regulated advice provides, and is considering whether to regulate AI-generated financial advice.
Separate FCA research cited by Saturn, referenced by footnote in the report but not reproduced in the source material provided, suggests consumers are almost three times as likely to turn to AI tools for financial advice as they are to seek advice from a qualified adviser. This figure could not be independently verified from the materials supplied.
Amal Jolly, chief executive of Saturn, said: “The low quality of financial advice from mainstream AI models risks leading to widespread consumer harm. Millions of people are trusting the AI models for money advice, but they are getting wrong answers that can lose them money.
Jolly continued: “AI tools are fast becoming the first port-of-call for many consumers with financial questions. But our research reveals evidence that it is far too early to put so much trust in AI chatbots. We found some of the most widely used models gave the wrong answer to financial questions the majority of the time. In the worst examples, relying on AI’s answers to tax questions could cost families tens or hundreds of thousands of pounds.”
He added: “AI financial advice is currently unregulated, leaving consumers with none of the protections, including compensation, that they would get if they went to a human adviser. The FCA has started to think about this, but it needs to act fast to protect people. The FCA should regulate AI to ensure consumers are protected.”