Ally Financial, headquartered in Detroit, Michigan, is a technology-driven leader in the U.S. financial services industry. As one of the nation’s largest online-only banks, Ally provides a wide range of banking and financial services to over 11 million members.

Recognizing the potential of large language models for widescale data analysis, Ally Financial is exploring their use for improving business operations and decision-making. To support reliable implementation in a highly regulated environment like the banking industry, LLM-generated information must be accurate, verifiable, and grounded in relevant evidence.

Our LLM Hallucination Detection Framework validates responses produced by language models against relevant source information. Each response is decomposed into factual claims, which are then compared against evidence retrieved from source documents. Each claim is then labeled based on whether it is supported, contradicted or lacks enough information, and assigned a risk score.

Users are first greeted with a dashboard encompassing evaluation activity, flagged responses, analyzed claims, average evaluation time, and recent results. The evaluation page displays the original question and response, extracted claims, risk scores, and source evidence. Past evaluations are searchable for user review and comparison through the evaluation history page.

Our platform helps users understand why a response may be reliable or risky. By showing the evidence behind each decision, the system supports more careful use of language-model generated information in business settings.

The front end is developed using React, while the back end is implemented in Python using Flask. PostgreSQL and ChromaDB store application data and source information, while LangChain and Hugging Face technologies support document retrieval, claim analysis, and response evaluation.