What Is Cross-Encoder Reranking?
A cross-encoder is a neural network that takes a query and a document as a single concatenated input, applies full transformer attention across all tokens in both texts, and outputs a single relevance score. This is distinct from a bi-encoder, which encodes the query and document independently into separate vectors.
Cross-encoder reranking is the practice of using a cross-encoder as a second stage in a retrieval pipeline, applied to a small candidate set retrieved by a fast first-stage system (typically a bi-encoder or BM25). The cross-encoder reranks the candidates by their precise relevance scores, surfacing the most relevant documents to the top of the final result set.
Cross-encoders consistently achieve higher relevance accuracy than bi-encoders because full attention allows query tokens to directly interact with document tokens during scoring. A bi-encoder compresses each text into a single vector, losing the fine-grained token-level relationships that determine relevance for many queries. The cross-encoder has no such bottleneck.
How Cross-Encoder Reranking Affects AI Search Precision
Cross-encoder reranking is the final, most precise filtering step before content is passed to a language model for answer generation. Understanding how cross-encoders score relevance reveals what content characteristics are most valued at this stage.
Example: Direct Answer vs Buried Answer at the Reranker
Query: "what is cross-encoder reranking and how does it improve retrieval?"
Buried answer (low reranker score)
"RAG systems have multiple components. First-stage retrieval narrows the corpus to candidates. Various methods exist for candidate selection. After retrieval, results can be refined. Cross-encoders are one way to do this. They process query and document together."
Cross-encoder reranker score: ~0.41. The answer appears late in a long preamble. The relevant tokens are diluted by surrounding context. Not selected for the answer window.
Direct answer (high reranker score)
"Cross-encoder reranking is a second-stage retrieval step that scores query-document pairs jointly, allowing it to model fine-grained relevance. Unlike bi-encoders, the cross-encoder attends to interactions between query tokens and document tokens simultaneously, producing significantly more accurate relevance scores."
Cross-encoder reranker score: ~0.94. Definition in the first sentence. Every token in the passage contributes to the query-document interaction. Selected and cited.
- Direct answer presence is the strongest signal: Cross-encoders reward documents that directly address the query. A document that contains the answer in clear, unambiguous prose will score higher than one that contains related information but requires inference to reach the answer. Answer-first writing style aligns with this scoring behaviour.
- Query-document token interaction rewards specificity: Cross-encoder attention identifies matches between specific query terms and document passages. Content that uses precise, specific language corresponding to how queries are phrased will activate more high-similarity attention patterns.
- Context window relevance: Cross-encoders process the full candidate chunk in a single pass. Chunks with focused, on-topic content produce cleaner attention patterns than chunks with mixed topics. This reinforces the value of semantic chunking: focused chunks score better at the reranking stage.
- Quality over length: A concise, directly relevant paragraph will outscore a longer document with the same information buried in surrounding context, because the relevant tokens have a larger proportional presence in the cross-encoder's attention.
- AI-optimised content must pass the reranker: Our AI SEO approach specifically accounts for how content will be evaluated at the reranking stage, not just the first-stage retrieval stage.
How Cross-Encoders Work
A cross-encoder is a standard BERT-like transformer with a classification head. The input is a concatenation of the query and document, separated by special tokens: [CLS] query [SEP] document [SEP]. All layers of the transformer attend over all tokens in this combined input. The [CLS] token's final hidden state is passed through a linear layer to produce a scalar relevance score.
The key mechanism is the full attention matrix: every query token attends to every document token and vice versa. This enables the model to identify that the query term "capital" matches the document phrase "seat of government" through contextual reasoning, not just string matching. A bi-encoder cannot do this because the query and document tokens are never in the same attention context.
Cross-encoders are trained on pairs of (query, document) with relevance labels. MS MARCO, a large dataset of query-passage pairs from Bing search logs, is the primary training dataset for most open-source cross-encoders. Models fine-tuned on MS MARCO generalise well to other retrieval domains without further training.
The cost of full attention is that a cross-encoder cannot pre-encode documents offline. Every document must be processed together with the query at retrieval time. For 10 million documents, this would require 10 million forward passes per query, which is far too slow. This is why cross-encoders are only applied to a small candidate set of 50 to 200 documents returned by a faster first-stage retriever.
Two-Stage Retrieval Pipeline
The two-stage retrieval architecture is the standard deployment pattern for cross-encoder reranking. In the first stage, a fast retriever (bi-encoder ANN search, BM25, or hybrid RRF fusion) retrieves the top 100 to 200 candidate documents. This stage prioritises recall over precision: it may include some irrelevant documents, but should not miss relevant ones.
In the second stage, the cross-encoder scores every candidate against the query with a full forward pass. The candidates are re-sorted by their cross-encoder scores. The top 5 to 20 from this re-sorted list are passed to the language model for answer generation.
Running 100 to 200 cross-encoder forward passes adds 100 to 500 milliseconds of latency with a standard BERT-base cross-encoder on GPU. This is acceptable for most applications. For latency-sensitive systems, smaller distilled cross-encoders (MiniLM-based) or ColBERT late interaction rerankers can be used to reduce the cost.
The combination of hybrid first-stage retrieval and cross-encoder reranking represents the current state of the art for maximum retrieval precision in production RAG systems. It is used by production AI search systems at Cohere (Rerank API), Jina AI, and Voyage AI.
Cross-Encoder Models in Use
The sentence-transformers library provides the easiest access to cross-encoder models for reranking:
- cross-encoder/ms-marco-MiniLM-L-6-v2: The most widely used cross-encoder reranker. A 6-layer MiniLM distilled model trained on MS MARCO. Good balance of speed and accuracy for general-domain reranking.
- cross-encoder/ms-marco-MiniLM-L-12-v2: 12-layer version. More accurate than L-6 at roughly 2x the latency. Preferred when latency allows.
- BAAI/bge-reranker-v2-m3: A multilingual cross-encoder from BAAI that outperforms the MiniLM models on BEIR benchmarks while supporting over 100 languages.
- Cohere Rerank API: A hosted cross-encoder reranking service that takes a query and a list of documents and returns relevance scores. Model details are not disclosed but performance is competitive with the best open-source models.
- Jina Reranker: Jina AI's hosted and open-source reranker, supporting long documents up to 8192 tokens, useful for legal and technical document reranking where documents exceed standard BERT context limits.
Cross-Encoder vs ColBERT Late Interaction
Both cross-encoders and ColBERT late interaction improve over bi-encoder retrieval by preserving more token-level information. They differ in where this interaction occurs:
- Cross-encoder: Full attention at scoring time. Query and document tokens are processed together. Most accurate. Requires a forward pass per candidate; no offline pre-computation of document representations.
- ColBERT: Token vectors computed offline at indexing time. Query token vectors computed at query time. Interaction via MaxSim across pre-computed token vectors. Faster than cross-encoder at the cost of some accuracy. Index is much larger than bi-encoder index.
In practice, the choice depends on the candidate set size and latency budget. For 50-100 candidates, a cross-encoder adds 50-200ms and provides the highest precision. For 1,000-10,000 candidates, ColBERT's MaxSim is more practical. For first-stage retrieval over millions of documents, bi-encoder ANN or BM25 is the only viable option.
Frequently Asked Questions
Can I use a cross-encoder for first-stage retrieval?
No, in practice. A cross-encoder requires a forward pass per query-document pair, so retrieving the top 10 from 10 million documents would require 10 million forward passes per query, which would take hours on modern hardware. First-stage retrieval requires methods that can search the full corpus in milliseconds: bi-encoder ANN search, BM25, or ColBERT with PLAID.
How many candidates should I pass to the cross-encoder?
50 to 200 is the typical range. Below 50, the first-stage recall may be insufficient and relevant documents may be missing. Above 200, cross-encoder latency increases proportionally and accuracy gains diminish. The optimal number depends on the first-stage retrieval quality (higher recall requires fewer candidates) and your latency budget.
Do cross-encoder scores need to be normalised?
For reranking purposes, normalisation is not required. The cross-encoder scores are used only to sort the candidate set, so their absolute values do not matter. If you need to combine cross-encoder scores with other signals (e.g., freshness, authority), then normalisation (min-max scaling or sigmoid transformation) is needed to make the scores comparable to the other signals.
Is a cross-encoder the same as a large language model?
No. Cross-encoders used for reranking are typically smaller BERT-based models (66M to 340M parameters) fine-tuned for binary or pairwise relevance classification. Large language models (GPT-4, Claude, Llama) are generative models with billions of parameters. However, some systems use LLMs as zero-shot rerankers by prompting them to judge relevance, which is slower but can achieve high accuracy without fine-tuning.
Does adding reranking always improve end-to-end answer quality?
In most cases yes, particularly for complex queries where the first-stage retriever returns some irrelevant candidates. However, reranking adds latency and cost. If the first-stage retriever already produces high-quality results for your query distribution (e.g., simple factual lookups), the quality improvement from reranking may be marginal. Measure quality with and without reranking on a representative query sample before committing to the architecture.
Tharindu Gunawardana
Founder and Director of SearchMinistry
Tharindu Gunawardana is the Founder of SearchMinistry Media and a search strategist with 17 years of experience across Sri Lanka, Singapore, and Australia. A former Agency SEO Director, he specialises in helping brands transition from traditional SEO to AI-driven discovery.