Retrieval-Augmented Generation (RAG) and Contextual Document Embeddings
Retrieval-augmented generation (RAG) has emerged as a prominent method for enhancing large language models (LLMs) with external knowledge. RAG systems utilize an embedding model to encode documents from a knowledge corpus and identify those most relevant to user queries.
Standard retrieval methods often overlook context-specific details, which can significantly impact performance in application-specific datasets. Researchers at Cornell University address this limitation with "contextual document embeddings," a technique that equips embedding models with contextual awareness during the retrieval process.
The Limitations of Bi-Encoders
Bi-encoders, a common choice for document retrieval in RAG, create fixed representations of documents stored in a vector database. During inference, they compare the query embedding with these stored embeddings to identify relevant documents. Although bi-encoders are efficient and scalable, they frequently struggle with nuanced, application-specific datasets because they are typically trained on generalized data. In specialized knowledge corpora, they may even underperform compared to traditional statistical methods like BM25.
“Our project started with the study of BM25, an established algorithm for text retrieval,” explained John (Jack) Morris, a doctoral student at Cornell Tech and co-author of the study. “We found that the more out-of-domain the dataset, the more BM25 outperforms neural networks.”
BM25 adapts flexibly by calculating the weight of each word relative to the corpus being indexed. Words that appear frequently across many documents are assigned lower weights, adjusting for their importance in different contexts. In contrast, traditional neural network-based dense retrieval models set weights statically based on training data.
Introducing Contextual Document Embeddings
The Cornell researchers propose two complementary methods to enhance bi-encoder performance by integrating context into document embeddings.
“If we view retrieval as a ‘competition’ between documents for relevance to a query, we leverage ‘context’ to guide the encoder regarding the other documents involved,” Morris noted.
1. Training Process Modification: The first method alters how the embedding model is trained. The researchers cluster similar documents before training and employ contrastive learning to help the model distinguish between documents within each cluster. This unsupervised technique enhances the model’s sensitivity to subtle differences crucial in specific contexts.
2. Augmented Bi-Encoder Architecture: The second method enhances the bi-encoder architecture, allowing it to access the corpus during the embedding process. This enables the encoder to incorporate the context of the document, producing more nuanced embeddings.
In this two-stage approach, a shared embedding for the document cluster is calculated first, then combined with the document’s unique features to create a contextualized embedding. The output maintains the same size as standard bi-encoder embeddings, ensuring compatibility with existing retrieval processes.
Performance Impact and Applications
The researchers evaluated their methods against various benchmarks, consistently outperforming standard bi-encoders, particularly in out-of-domain scenarios where training and test datasets significantly differ.
“Our model is advantageous for domains that differ markedly from the training data and serves as a cost-effective alternative to fine-tuning domain-specific embedding models,” Morris stated.
Contextual embeddings offer improved performance for RAG systems across diverse domains. For instance, when documents share a common structure, traditional embedding models may waste space by storing redundant information. In contrast, contextual embeddings efficiently discard non-essential shared information, optimizing their representations.
The researchers have released a compact version of their contextual document embedding model (cde-small-v1), designed as a drop-in replacement for popular open-source tools like HuggingFace and SentenceTransformers for creating customized embeddings.
Morris emphasizes that the potential for contextual embeddings extends beyond text-based models and can be applied to other modalities, such as text-to-image architectures. There is also significant scope for enhancing these embeddings through advanced clustering algorithms and testing their effectiveness at larger scales.
Ranking
