Cách đặt tên con hợp tuổi bố mẹ chuẩn phong thủy, mang lại may mắn
Vectara Unveils Open-Source Hallucination Evaluation Model To Detect and Quantify Hallucinations in Top Large Language Models
SANTA CLARA, Calif., Nov. 06, 2023 (GLOBE NEWSWIRE) -- Large Language Model (LLM) builder Vectara, the trusted Generative AI (GenAI) platform, released its open-source Hallucination Evaluation Model. This is a first-of-its-kind initiative to proffer a commercially available and open-source model that addresses the accuracy and level of hallucination in LLMs, paired with a publicly available and regularly updated leaderboard, while inviting other model builders like OpenAI, Cohere, Google, and Anthropic to participate in defining an open and free industry-standard in support of self-governance and responsible AI.
By launching its Hallucination Evaluation Model, Vectara is increasing transparency and objectively quantifying hallucination risks in leading GenAI tools, a critical step toward removing barriers to enterprise adoption, stemming dangers like misinformation, and enacting effective regulation. The model is designed to quantify how much an LLM strays from facts while synthesizing a summary related to previously provided reference materials.
'In order to realize the true promise of Generative AI, we first have to tackle the challenge of hallucinations,” said Matei Zaharia, CTO and Co-Founder of Databricks. “The launch of the Hallucination Evaluation Model to the Hugging Face community encourages industry co-innovation and accountability through a powerful measurement tool accessible for all LLM builders.'
The Hallucination Evaluation Model launch includes releasing Vectara’s measurement code base as an open-source model on Hugging Face as well as a publicly accessible Leaderboard available from Vectara. The Leaderboard serves as a quality metric for LLM factual accuracy, similar to how credit ratings or FICO scores function for financial risk, giving businesses and developers insight into the realities of different GenAI tools before implementing them.
“For organizations to effectively implement Generative AI solutions including chatbots, they need a clear view of the risks and potential downsides,' said Simon Hughes, AI researcher and ML engineer at Vectara. 'For the first time, Vectara’s Hallucination Evaluation Model allows anyone to measure hallucinations produced by different LLMs. As a part of Vectara’s commitment to industry transparency, we’re releasing this model as open source, with a publicly accessible Leaderboard, so that anyone can contribute to this important conversation.”
Key Features of Vectara’s Hallucination Evaluation Model:
Objective Measurement: This model provides much-needed visibility into the LLMs' ability to synthesize data without introducing hallucinations. Many LLM vendors make claims about their capabilities to mitigate the impact of hallucinations, but until now, there have been no objectively verifiable methods for detecting and quantifying instances of irrelevant or incorrect data in model outputs. For the model, Vecatara built a machine-learning model, tuned for real world performance and using the latest advancements in hallucination research, to evaluate LLM summarizations without requiring objective scoring or influence.
Transparency Through Open Source: The Hallucination Evaluation Model is available for developers and industry stakeholders to integrate into their own pipelines through an Apache 2.0 License on Hugging Face. Developers can also use the open-source evaluation model to verify the accuracy of Vectara’s platform.
Dynamic Leaderboard: Vectara’s AI researchers and ML engineers (in collaboration with the open source community) will maintain and continually update the Leaderboard, showcasing the hallucination impact of different LLMs and offering a clear comparative perspective as new models emerge. The Leaderboard lists the accuracy and hallucination rates for each model tested in response to the same set of prompts.
The Leaderboard shows that OpenAI’s models have the strongest performance, followed by the Llama 2 models, Cohere and Anthropic. Google’s Palm models scored lower on the Leaderboard.
“Hallucination is one of the most serious issues to consider when deploying production LLMs. Having an open source benchmark model that can evaluate factual accuracy in a quantifiable way will allow developers to directly address the problems,” said Waleed Kadous, Chief Scientist at Anyscale. “Vectara’s new model sets the industry standard for measuring the extent to which LLMs hallucinate, and we’re excited to work with them as a launch partner.”
Vectara has led industry efforts to address hallucinations as a critical barrier to the safe, effective, and accurate use of GenAI. The model doesn’t solve hallucinations directly but rather enables more informed adoption and better decision-making by measuring the frequency and severity of this phenomena. Greater transparency into the quality of LLM-produced summarizations allows LLM users to evaluate GenAI solutions according to the risk profile of the intended use case.
GenAI adoption in highly regulated industries like legal, healthcare, finance, energy, and government will hinge upon vendors' ability to provide solutions with low to nearly zero risk of factual inaccuracies. Hallucinations have already been raised by stakeholders in these sectors as a serious issue. Until now, however, there has been no way to objectively compare the performance of available models outside of academic benchmarks, which don’t always translate to real-world settings.
Hallucinations also factor heavily in ongoing dialogue about GenAI regulation. Effective government oversight requires measurement tools universally recognized as transparent and objective. Vectara’s open-source model serves as an industry standard, providing the missing link to legislation that virtually all industry leaders agree is needed. With concerns around misinformation and other AI risks rising ahead of the U.S. presidential election and other geopolitical events, the Hallucination Evaluation Model and Leaderboard provide a tangible step toward data-driven and accessible oversight mechanisms.
About Vectara
Vectara is an end-to-end platform that empowers product builders to embed powerful Generative AI features into their applications with extraordinary results. Built on a solid hybrid-search core, Vectara delivers the shortest path to an answer or action through a safe, secure, and trusted entry point. Vectara is built for product managers and developers with an easily leveraged API that gives full access to the platform's powerful features. Vectara’s Retrieval Augmented (Grounded) Generation allows businesses to quickly, safely, and affordably integrate best-in-class conversational AI and question-answering into their application with zero-shot precision. Vectara never trains their models on customer data, allowing businesses to embed generative AI capabilities without the risk of data or privacy violations. To learn more about Vectara, visit www.vectara.com.
Media Contact
Carly Bourne
carly@bulleitgroup.com
423-443-0449

TIN LIÊN QUAN
Freshworks Unveils New Generative AI Enhancements Across Product Lines to Power Greater Business Efficiency
[img]http://phpstack-911252-3168991.cloudwaysapps.com/api/ImageRender/DownloadFile?resourceId=70f65da2-de89-4f8e-9b4e-ae2e5c2b78df&size=3[/img]
THỦ THUẬT HAY
Cách tải ảnh từ Instagram đơn giản và tiện lợi nhất cho Android
Có nhiều cách tải ảnh từ Instagram nhưng đều phải thực hiện qua nhiều bước rất không tiện lợi. Hôm nay chúng tôi sẽ giới thiệu cho bạn cách tải ảnh trực tiếp ngay bên trong ứng dụng Instagram cực kì đơn giản.
12 ứng dụng iOS trị giá 23 USD đang được miễn phí
Tiếp tục cập nhật danh sách 'Free App' hằng ngày dành cho iDevices, TCN đã tổng hợp thêm khá nhiều ứng dụng và game trả phí đang được miễn phí trong thời gian ngắn. Nếu bạn quan tâm hãy nhanh tay tải về trước khi bị
Hướng dẫn viết hồ sơ online theo mẫu trên ứng dụng VietnamWorks
Ứng dụng VietnamWorks sẽ cung cấp cho các bạn những mẫu CV xin việc miễn phí, vậy để viết một bộ hồ sơ xin việc online chuẩn để gửi đến nhà tuyển dụng sẽ như thế nào, bạn cần phải lưu ý những gì khi viết hồ sơ.
Thủ thuật sắp xếp lại các cửa sổ xếp chồng trên Windows Taskbar
Có rất nhiều tính năng kèm theo phiên bản Windows 10 nhưng đôi khi chúng không có sẵn. Đây cũng là cơ hội cho các ứng dụng của bên thứ ba lên ngôi. Trong bài viết này, chúng tối sẽ hướng dẫn bạn thủ thuật đơn giản giúp
Cách đánh số thứ tự cột tự động trên Excel
Trên Excel có tính năng đánh số thứ tự cột tự động, người dùng sẽ không cần thực hiện thao tác kéo ô thứ tự như cách làm thông thường nữa.
ĐÁNH GIÁ NHANH
Đánh giá Ford EcoSport 2018 sau gần nửa năm sở hữu
Với số tiền khoảng 500 – 600 triệu đồng, người tiêu dùng Việt Nam hiện nay có quá nhiều lựa chọn trên thị trường ô tô. Những cái tên có thể nghĩ tới như Ford EcoSport, Ford Fiesta, Hyundai Accent, Toyota Vios, Honda
Đánh giá chi tiết AMD APU Ryzen 5 2400G giá 4,35 triệu đồng
Xét về sự hấp dẫn thì đây là những mẫu APU đầu tiên của AMD được sản xuất theo quy trình 14nm+ đồng thời tích hợp nhân đồ họa Vega mới. Điều này giúp cải thiện đáng kể sức mạnh của chip so với thế hệ Excavator trước
Subaru Impreza 2019 phiên bản mới vừa ra mắt có những gì nổi bật?
So với thế hệ trước đó thì Subaru Impreza 2019 sẽ có kích thước rộng hơn 35 mm, chiều dài cơ sở tăng 25 mm còn chiều cao thì giảm 10 mm. Khoang hành lý của xe cũng được mở rộng với dung tích lên đến 1.280 lít. Nhờ