Overcoming Hallucinations with Trustworthy Language Models

Large language models (LLMs) like GPT-4 have shown remarkable capabilities, but their tendency to "hallucinate" or generate incorrect information has been a major barrier to enterprise adoption.

Cleanlab has launched the Trustworthy Language Model (TLM) to address this key challenge of reliability in generative AI.

TLM augments existing LLMs by adding a trustworthiness score to every output, quantifying both the known uncertainties (aleatoric) that models are aware of as well as the unknown uncertainties (epistemic) that arise from lack of training data.

This allows organizations to contain and manage hallucinations, enabling new use cases previously unsuitable for unreliable LLMs.More Accurate Outputs and Better Cost Savings

Through rigorous benchmarking against GPT-4, Cleanlab has shown that TLM produces more accurate outputs overall. Crucially, TLM's trustworthiness scores are better calibrated than just using an LLM's self-evaluated confidence or output probabilities.

This enables greater cost and time savings by prioritizing human review of low-scoring outputs.For applications with a required error rate tolerance, using TLM's trustworthiness scores to triage outputs for review catches more hallucinations under a fixed review budget compared to existing approaches.

This unlocks new production use cases across medicine, law, finance and more.Avoiding Catastrophic Hallucinations

The consequences of unchecked LLM hallucinations can be severe, as some organizations have already experienced. From airlines being forced to refund customers to law firms facing fines over fabricated citations, the risks of deploying unreliable LLMs are real.

With TLM, teams can finally get the benefits of generative AI's capabilities while managing the reliability risks. By adding trustworthiness scoring, TLM is a key step towards responsibly deploying LLMs in the enterprise.I'm excited to see how TLM enables new generative AI applications!

You can try out the TLM API for free or experiment in their interactive demo.

https://cleanlab.ai/tlm/

Let me know what you think in the comments.

1 comment