Finding more than a keyword.

The question behind it

I wanted to explore search that could work with both text and audio. A keyword search is useful when the wording matches, but it does not always help when someone remembers the idea instead of the exact words.

The project became a local FastAPI proof of concept with semantic and keyword search modes. I’m especially interested in how this could help with music and other collections I already work with.

How the prototype works

I worked with Jina’s omni small retrieval embeddings to represent entries and queries in a shared space. The application supports text and audio entries, batch uploads, and cosine similarity results.

Longer audio is split into chunks, so a result can point to a relevant portion of a recording with a timestamp. That makes an audio match more useful than returning a whole file without showing where the relevant part is.

What I pay attention to

I want the interface to make it clear when it is doing semantic retrieval and when it is doing keyword matching. I also want the similarity score to be understood correctly: cosine similarity is a measure used to rank matches, not a probability that the answer is right.

I have been working on local execution on Apple Silicon, with MPS where appropriate, and optional CUDA support. Retrieval quality and practical runtime behavior both matter to me.

Where it stands

This is an evolving proof of concept. I’m improving batch search and result quality, and exploring what text and audio retrieval can add to an actual song search workflow.