09. IR - Neural Re-Ranking

Core part of re-ranking models is a matching module operating on a word interaction level

Training process:

1. Training is independent of the search engine’s retrieval stage. (Can be repeated to account for temporal shift in the data)

2. Uses triples → (query, relevant document, non-relevant document).

3. End-to-end (e2e) training for all components.

Evaluation process:

1. Compute a score for each (query, document) tuple.

2. Sort tuples based on scores.

3. Use ranking metrics (e.g., MRR@10) to measure effectiveness.

Mismatch: You cannot really compare training loss and IR evaluation metric

• Starting point for text processing in nn models

• Word token (id / piece / char based) to dense representation -> Having word boundaries is important in IR

• The core of many early neural IR models

• Matrix of similarities of individual word combinations

• Only a transformation – not parameterized by itself

• Measures direction of vectors, but not he magnitude

• Not a distance – but equivalent to Euclidian distance of unit (length = 1) vectors

BERT_cat (monoBERT) re-ranks documents by processing the query and passage together. ✅ Steps:

1. Concatenates inputs → [CLS] query [SEP] passage.

2. Pools [CLS] token representation.

3. Uses a linear layer to predict ranking scores.

Problem: BERT is limited to 512 tokens (query + document).

Solutions:

1. Cap document to fit within 512-token limit.

2. Sliding window approach → Use overlapping windows, take max score.

Three efficiency techniques:

1. Reduce model size → Smaller models run faster, but quality reduces drastically after certain threshold

2. Precompute passage representations → Store embeddings, avoid repeated calculations. -> Move computation away from query time

3. Reduce query latency -> Lifecycle efficiency includes training, indexing and retrieval steps

-> Concatenate the 2 sequences to fit BERT’s workflow:

• [CLS] query [SEP] passage

• Pool [CLS] token

• Predict the score with a single linear layer

• Works awesome out of the box

  • Concatenating the 2 sequences to fit BERT’s workflow

  • As long as you can have time or enough compute it trains easily

• Major jumps in effectiveness across collections and domains

  • But, of course, at the cost of performance and virtually no interpretability

  • Larger BERT models roughly translate to slight effectiveness gains at high efficiency cost

    • Problem: We need to repeat the inference by the re-ranking depth

• PreTTR

• ColBERT

PreTTR splits BERT across layers:

• The first N layers are precomputed and stored.

• Remaining layers are computed during inference.

• n is a hyperparameter

Benefits:

• Maintains BERT_cat quality.

Limitations:

• Still requires significant storage

• Still low query latency

1. Create match-matrix of BERT term-representations

2. Use simple max-pooling for the document-dimension & sum for the query dimensions

Benefits:

• Much faster query latency.

Drawback:

• Requires huge storage space for passage term vecotors.

Three main approaches:

1. MatchPyramid → Uses word similarity and CNNs.

2. BERT_cat → Powerful but slow due to repeated inference.

3. Efficiency-focused models → PreTTR, ColBERT.

Two key trends:

1. IR for LLMs → Retrieval-Augmented Generation (RAG).

2. LLM for IR → Query expansion and reformulation using LLMs.

Challenges:

• Context awareness → Making IR models smarter.

• Explainability → Improving transparency of ranking decisions.

• Domain generalization → Adapting to unseen datasets.