🔗 pg_textsearch: How we built a BM25 search engine on Postgres pages

Building a BM25 search engine on Postgres required addressing significant limitations of the built-in ts_rank. As I learned, the lack of inverse document frequency and term frequency saturation affected the relevance and scalability of searches. By creating pg_textsearch, they implemented real BM25 scoring via a native indexing solution in C integrated with Postgres.

The design included a hybrid architecture with a write-optimized memtable and immutable disk segments. Notably, they achieved fast query performance through optimizations like Block-Max WAND, allowing for much quicker retrieval of top results without scoring every match. With extensive benchmarks, they showed pg_textsearch can outperform existing tools like ParadeDB, achieving up to 6.5x speed increases on 138 million document queries. This combination of efficiency and native integration makes it a compelling option for teams needing robust search capabilities within Postgres.