AI still has a hallucination problem: How MongoDB aims to solve it with advanced rerankers and embedding models

To get the best possible result from an AI query, organizations need the best possible data.

The answer that many organizations have had to overcome that challenge is retrieval-augmented generation (RAG). With RAG, results are grounded in data from a database. As it turns out, though, not all RAG is the same, and actually optimizing a database for the best possible results can be challenging.

Database vendor MongoDB is no stranger to the world of AI or RAG. The company’s namesake database is already being used for RAG, and MongoDB has also launched AI applications development initiatives. While the company and its users — such a medical giant Novo Nordisk — have had success with gen AI, there is still more to be done.

In particular, hallucination and accuracy continues to be an issue holding some organizations back from getting gen AI into production. To that end, MongoDB today announced the acquisition of privately-held Voyage AI, which develops advanced embedding and retrieval models. Voyage raised $20 million in funding in Oct. 2024 in a round supported by cloud data giant Snowflake. The acquisition will bring Voyage AI’s expertise in embedding generation and reranking — critical components for AI-powered search and retrieval — directly into MongoDB’s database platform.

“Over the last year, and especially as organizations have tried to think about how they could build AI powered applications, it became increasingly clear that the quality and trust of the applications they build, or the lack thereof, was becoming one of the barriers for applying AI to mission critical use cases,” MongoDB CPO Sahir Azam told VentureBeat.

What are the challenges of hallucination? Doesn’t RAG solve them?

The basic idea behind RAG is that, instead of simply relying on a knowledge base from trained data, the gen AI engine can get grounded data from a database.

Creating highly accurate RAG is quite complex, and there is still a potential risk for hallucinations — a challenge faced by MongoDB and its users. While Azam declined to provide any specific example or incident where gen AI RAG failed a user, he did note that accuracy is always a concern.

Improving accuracy and reducing hallucination involves multiple steps. The first is to improve the quality of retrieval (the ‘R’ in RAG).

“In many cases, the retrieval quality is not good enough,” Tengyu Ma, founder and CEO of Voyage AI, told VentureBeat. “In the retrieval step, if they are not retrieving relevant information, then the retrieval is not very useful, and the large language model (LLM) hallucinates because it has to guess some context.”

The Voyage AI models now part of MongoDB help improve RAG in a few key ways:

• Domain-specific models and re-rankers: These are trained on large amounts of unstructured data from specific verticals, allowing them to better understand the terminology and semantics of those domains.
• Customization and fine-tuning:  Users can fine tune the retrieval mechanism for unique datasets and use cases.

MongoDB’s competition

MongoDB isn’t the first or only vendor to recognize the need for and value of having highly optimized embedding and re-ranker technology. After all, that’s one of the reasons Snowflake invested in Voyage AI and is using the company’s models. 

It’s important to note that, even after being acquired by MongoDB, Voyage AI’s models will still be available to Snowflake and to Voyage AI’s other users. The big difference is that Voyage AI will now be increasingly integrated into MongoDB’s database platforms. 

Directly integrating advanced embedding models in a database is an approach taken by other rival database vendors, as well. Back in June 2024, DataStax announced its own RAGStack technology that combines advanced embedding and retrieval models.

Azam argued that MongoDB is a bit different, though. For one, it is an operational database, as opposed to an analytical database. Also, as opposed to just providing insights and analysis, MongoDB helps power transactions and real-world operations. MongoDB is also what is known as a “document model database,” which has a different structure than a traditional relational database. That structure does not rely on columns and tables, which are not particularly good at representing information about unstructured data (a critical element for AI applications).

“We’re the only database technology that combines the management of metadata about a customer’s information, the operations and transactions, which is the heartbeat of what’s happening in the business, as well as the foundation for retrieval — all with a single system,” said Azam.

Why Voyage AI matters for agentic AI workflows

The need for highly accurate embedding and retrieval models is being further accelerated by agentic AI.

“Agentic AI still needs retrieval methods, because an agent cannot make decisions out of context,” said Ma. “Sometimes, actually multiple retrieval components are used in even one decision.”

Ma noted that Voyage AI is currently working on specific models that are highly customized for agentic AI use cases. He explained that agentic AI can use different types of queries that can still benefit from more optimization.

As gen AI increasingly moves into operational use cases, the need to remove the risk of hallucinations is clearly paramount. While MongoDB has had success with gen AI, Azam expects the integration of Voyage AI to open new mission critical use cases.

“If we can now say, ‘Hey, we can give you well north of 90% accuracy for your applications that today may only, in some cases, get to 30 or 60% accuracy for the results,’ the aperture widens in terms of the types of opportunities people can apply AI to in their software applications,” said Azam.