Retrieval engineering · 4 min read
Hybrid search over tickets and design documents
I started two local search tools for a problem I kept running into, finding past engineering discussions: one over about 97,000 public Apache Jira tickets and their comments, and another over an approved set of internal design documents and runbooks. I designed their architecture and evaluation, and AI coding agents implemented them. Both combine exact-term and semantic retrieval, then require cited source passages in every answer. Evaluation rejected several plausible changes and showed where a cross-encoder reranker was worth its latency.
Problem
Engineers often remember that a problem was discussed without remembering the ticket number or the wording needed to find it. Keyword search needs that wording. Semantic (vector) search doesn't, but it always returns its nearest match, even for questions the corpus can't answer.
Context
Both tools run locally on SQLite, with no GPU and no service: FTS5 for keyword search with BM25 ranking, and sqlite-vec for vector search over embeddings from all-MiniLM-L6-v2, a small sentence-embedding model. Keyword search is weighted so that a title match counts for more than body text.
The two kinds of score can't be compared directly, so the result lists are merged with Reciprocal Rank Fusion (RRF). RRF ignores the scores and uses only each document's rank in each list: every list gives a document a vote that shrinks slowly as its rank falls, and the votes are added. In practice, a document that both retrievers found almost always beats one that only a single retriever ranked first.
A cross-encoder can then rerank the top results. It reads the query and each candidate together, which makes it the most accurate of the three, but nothing can be computed ahead of time, so every candidate costs a full model pass at query time.
Approach
Jira search: comments as their own list
The ticket index covers HBase, Phoenix, Hadoop, HDFS, MapReduce, YARN and ZooKeeper, with comments indexed separately: about 315,000 for HBase alone.
The first version added comment matches after the ticket matches, so a ticket that was only discussed in its comments could never outrank a weak title match. I split it into three independent lists, ticket keywords, comment keywords and vector similarity, and fused them with equal weights.
To check the change I generated 90 test queries from HBase tickets, built from titles, descriptions and comments, each labelled with the ticket it came from. The overall score barely moved (MRR@10 from 0.659 to 0.650), and the share of queries that found their ticket in the top ten didn't change. That turned out to say more about the benchmark than the change: with one label per query, a newly found ticket that is genuinely relevant can only push the labelled one down. The benchmark catches regressions, but tuning the weights needs a human-graded set.
Document search: chunks and a reranker
Design documents are split into chunks along their headings, and each chunk keeps its heading path and nearest anchor so an answer can cite it. Re-syncing a changed document re-embeds only the chunks that changed.
I froze two hand-labelled question sets before tuning, 27 for tuning and 15 held out. When the corpus grew from ten documents to several hundred, vector search started returning similar but wrong chunks. Capping chunks per document and reweighting the lists didn't fix it. Reranking the top ten fused results with a cross-encoder did:
| Mode | Tuning MRR@5 | Held-out MRR@5 |
|---|---|---|
| BM25 | 0.830 | 0.950 |
| Vector | 0.581 | 0.467 |
| Hybrid RRF | 0.835 | 0.967 |
| Hybrid + top-10 cross-encoder | 0.954 | 1.000 |
Reranking more than ten candidates added latency without improving quality. Even at ten, it took the median query from about 17 ms to about 274 ms, so it stays optional.
Runbook sections that are mostly screenshots have useful headings but almost no text, so neither retriever found them reliably. A third, low-weight list matches the question's words against headings, only for those image-heavy chunks. It can lift a chunk another retriever found, but it can't outvote them.
Knowing when not to answer
Vector search always returns something, so every mode that used it returned candidates for all 12 questions I wrote that the corpus can't answer. A score cut-off looked like a fix on the tuning set, but it turned out to measure whether the two retrievers agreed, which says nothing about whether the corpus holds an answer. On the held-out set it rejected 3 of 15 answerable questions, so I dropped it.
Instead, the answer layer decides. The agent must search before answering, every claim must quote a retrieved chunk, and when nothing retrieved answers the question it returns a fixed out-of-corpus response. A small set of general-knowledge prompts checks that it doesn't answer from model memory.
Result
Both tools fuse keyword and vector results with RRF and return ranked results, each citing its source: the ticket in Jira search, and the document, heading and anchor in document search.
In daily use, the semantic side is what I value most. I can describe a problem in my own words and still get back the ticket or design section that discussed it, which keyword search only finds when I happen to remember the original wording. The benchmarks above put that in context: on its own, vector search scored well below BM25 on the labelled document questions, and it pays off as one of the lists that RRF fuses.