Measuring the Impact of Hallucinations on Human Reliance in LLM Applications

Abstract

Modern large language models generate outputs that often exhibit unexpected or fabricated details, commonly referred to as hallucinations, influencing how humans interpret and rely upon these systems. Behavioral experiments show that users sometimes defer to system outputs, assuming correctness in contexts where thorough verification may not be feasible. Recent studies highlight that such misplaced reliance can manifest in high-stakes domains, including medical triage, legal documentation, and policy recommendations, where the costs of erroneous information are severe. Quantitative assessments typically gauge hallucinations in terms of factual inconsistencies, yet the downstream human impact remains less systematically investigated. This paper develops an evaluation pipeline that measures the extent to which hallucinations shape user decisions and reliability judgments. By integrating controlled prompts with varied levels of fidelity, the approach isolates the effects of erroneous content from user-specific biases. Empirical results present evidence that even low-frequency hallucinations can erode trust and lead to suboptimal task performance in collaborative human–machine settings. This finding shows the importance of accurate metrics that capture not only the presence of factual deviations but also their effects on human behavior. A deeper understanding of these behavioral dimensions can inform the design of guidelines and protocols aimed at maintaining user engagement without inflating unwarranted trust in generative outputs.