Review into the long-term impact of AI on retail financial services (The Mills Review)
Our role as a frontline organisation means that we are already seeing early evidence of how AI is shaping people’s experiences. We also provide regulated debt advice, offering a unique view on the impact of AI on retail financial services.
Our response to the Mills Review focuses on:
AI Advice: We see in our data that more and more consumers are using and trusting Large Language Models (LLMs) for financial advice. Where this advice is accurate and robust this could provide benefits to consumers. But, we’re seeing a lack of nuance in the advice offered and instances of hallucinations and fictional cases quoted. We’re concerned that inaccurate advice from LLMs may undermine genuine regulated advice.
Our advisers report difficulties with clients not believing their advice when it conflicts with the advice from an LLM. We’ve also seen clients act on bad advice from LLMs, leading to negative outcomes
Agentic AI: We’re concerned that consumers may not always understand the degree of agency they’re giving away through agentic AI. We see a lack of clarity in the liability of decision making from agentic AI, and no enforced regulation on the warnings of what agentic AI might do. We also foresee market shifts in response to agentic AI, and are concerned this could disproportionately affect certain groups.
Risks to consumers: We’re concerned consumers will receive poorer quality customer support, through chatbots and other AI technologies. AI driven fraud is increasing in sophistication and scale. Consumers need transparency to make informed choices about how their data is used, and manage the associated privacy and security risks. Consumers in vulnerable circumstances are often less able to reap the benefits of competitive consumer markets, and we’re concerned by the asymmetries that AI may exacerbate.
Digital exclusion: The acceleration of AI adoption will likely widen the digital divide. In the UK, there are 1.6m adults who don’t have a phone, tablet or laptop and 1.9m households who struggle to afford their mobile contract. The opportunities of AI will not be readily available to these people.
Rights to redress: We already see algorithmic decision-making can result in bias and discrimination against minority groups. We are concerned that due to both training data and design, predictive models will embed systemic discrimination in decision making. We are concerned that there are currently limited avenues by which consumers can seek the right to redress due to algorithmic decision-making.
We recommend the FCA should:
Introduce mandatory labelling of AI products linked to financial decision making, with a focus on transparency of agentic AI risk.
Enforce clear and upfront warning about liability about agentic AI decisions. Introduce regulation to ensure that consumers have access to readable records of decision making made by AI, especially agentic AI.
We support calls for a statutory duty to provide a right to explanation in AI driven decision making in financial services.
Set clear parameters around its expectations of what certain outcomes held in the Consumer Duty require in the context of AI deployment in financial services.
Sets out clear rules that establish where the use of AI by financial services firms would not be acceptable and would lead to an unacceptable level of risk or consumer harm.
Sets out clear rules on where particular standards or processes must be in place in order to safely deploy AI systems or use AI within processes, for example in advice or decisions on credit worthiness.
Proactively monitors changes in the financial services market, to identify where new adoption of AI processes and systems may be transforming consumer outcomes for better or worse.