ADR-005: Hybrid Search Weight Configuration

Status: Accepted
Date: 2026-02-16
Decision Makers: Brian Moore, AI Team

Context

We needed to determine the optimal balance between vector (semantic) search and BM25 (keyword) search for code retrieval. The hypothesis was that a "BM25-first" approach (heavier BM25 weighting for keyword queries) might improve results for exact function name lookups.

Decision

Keep current hybrid weights (60% vector, 40% BM25) or consider vector-heavy (80% vector, 20% BM25).

BM25-first and BM25-heavy strategies underperform compared to vector-centric approaches.

Evaluation Results

Strategy	Vector Weight	BM25 Weight	P@5	MRR	Recall
Vector-Heavy	0.8	0.2	0.9828	0.9296	0.9626
Hybrid (current)	0.6	0.4	0.9828	0.8871	0.9626
BM25-First	adaptive	adaptive	0.8966	0.8356	0.8980
BM25-Heavy	0.2	0.8	0.8448	0.7934	0.8549

Dataset: 116 queries (golden-dataset-cortex-api-only.json)
Collection: cortex-api codebase (54 Go files, 769 chunks)
Embedding Model: Voyage AI voyage-code-3 (1024 dims)

Rationale

1. Voyage AI embeddings are highly effective for code: The voyage-code-3 model is specifically trained for code, producing embeddings that capture semantic meaning better than keyword matching.

2. BM25 adds value but shouldn't dominate: BM25 helps with exact term matches (function names, variable names) but overweighting it degrades overall performance.

3. Vector-heavy achieves best MRR: For user-facing search, higher MRR means the most relevant result appears earlier, improving user experience.

Configuration

# configs/default.yaml
search:
  bm25_weight: 0.4    # Keep current OR reduce to 0.2
  vector_weight: 0.6  # Keep current OR increase to 0.8

Consequences

Positive

• Maintains excellent P@5 (98.28%)
• Best-in-class MRR with vector-heavy (0.9296)
• Simpler implementation (no query classification needed)

Negative

• BM25-only queries (exact symbol lookup) may rank slightly lower
• Requires Voyage AI API for all queries (cost consideration)

Related

• evaluation/bm25_first_eval.py - Evaluation script
• evaluation/bm25-first-results.json - Detailed results
• cortex-api/hybrid_search.go - Implementation