The risks of AI — hallucinations, data privacy, and what should never go into a chatbot

For nine weeks this column has argued that AI belongs in the Jamaican finance office: forecasting cash flow, cleaning a year of receipts, turning spreadsheets into slides, reading 200-page agreements before anybody signs them. Nearly every one of those columns closed with the same two warnings — verify everything, and keep confidential material on a business account. This week those warnings get the full column, because the tools are spreading through Jamaican businesses faster than the habits that make them safe.

The two risk families from this week’s column: what AI can do to information, and what canhappen to the information you give it. (Branded graphic by PGH Consulting, LLC) The risks fall into two families. The first concerns what AI does to information: it can invent it, fluently and with total composure.

The second concerns what happens to information you give it: it can travel further than you intended. Neither risk is a reason to sit the technology out. Both are reasons to use it with rules, and the rules fit on one page.

Hallucination is not a malfunction A large language model works by predicting what words plausibly come next, and most of the time plausibility and truth line up. When they do not, the machine does not stop or stumble; it produces something that sounds right, in exactly the tone it uses when it is right. The industry calls these failures hallucinations.

In a finance setting they look like this: a total that was never in the spreadsheet, a contract “summary” that includes a clause the contract does not contain, or a Jamaican tax threshold that is two years stale or entirely imagined, delivered with the same calm authority either way. From source files to a vague prompt to a confident answer — and the two habits that catchmost of it. (Branded graphic by PGH Consulting, LLC) The newest models hallucinate less than their predecessors, but less is not never, and the failure has an unkind shape: it shows up most where checking is hardest — precise figures, obscure facts, citations and references — and the confidence of the answer tells you nothing about its accuracy.

A chatbot is never more dangerous than when it is specific, fluent, and wrong. The two habits that catch most of it The first habit is the rule this series repeats every week: AI reads figures, it never generates them. If a number was not in the document, spreadsheet, or …