Hallucination

Language models calculate which sequence of words is likely to fit. They do not check by themselves whether a statement is true. If a piece of information is missing, the model sometimes fills the gap with something plausible: an opening time, a price, a commitment. On the phone this is delicate, because callers take the information as binding.

The risk can be reduced with several measures: allow answers only from stored knowledge, permit the assistant to say that it does not know something, maintain binding details such as prices and times in structured form, and verify actions such as bookings on the server. Errors cannot be ruled out completely. Spot checks in transcripts and a clear handover to people are therefore part of running such a system.