Announcing Hybrid Search Capability for Locusive's Chatbot And API
We are excited to announce that we have recently rolled out a new hybrid search capability to further improve Locusive's semantic search functionality.
Hybrid search combines the strengths of semantic search using vector embeddings with more traditional keyword search. This allows us to handle situations where content in our knowledge base might use similar but semantically different words, as well as short 1-2 word search queries.
Implementing hybrid search required finding a way to generate both dense and sparse vectors. Dense vectors excel at representing the semantic meaning of text numerically, enabling powerful capabilities like finding similar content. However, they sometimes fall short matching exact keywords. That's where sparse vectors come in - they focus on tracking the presence and weighting of specific keywords.
By combining dense and sparse vectors, we get the best of both worlds. But we found there were no off-the-shelf solutions providing sparse vector APIs, so, we've built our own microservice to do it ourselves.
We trained a BM25 model on a large English dataset to produce sparse vectors for search queries. These sparse vectors are then used alongside dense vectors to query our vector database system Pinecone. An optimization we are working on is dynamically reweighting the contribution of sparse vs dense vectors based on search query length.
Rolling out hybrid search also required migrating our vector indices to use a dot product similarity matrix, which supports the technique unlike cosine similarity matrices.
Early results are promising. The most visible impact is improved performance of our Slackbot's "/find" command for knowledge base search. But we expect hybrid search to enhance all capabilities relying on understanding user queries in context.
We look forward to continuing innovating with latest semantic search methods to further improve Locusive's conversational AI platform. Please reach out if you have any other questions!
