↓Skip to main content
  1. Posts/

Building a miniature Perplexity

·4 mins
Llm RAG Opensource Ai

For those who don’t know, Perplexity is an AI search engine that gives answers instead of straight up links. Unlike traditional AI models, where there is a training cut-off, perplexity always grounds its answer from the latest information and uses LLM only for its ability to generate the output from the resulting links.

To understand how it might be doing it in the backend, I tried building a miniature version of the perplexity. If you’d rather read the source code, it’s available on GitHub

–

At its core, perplexity is doing Retrieval Augmented Generation (RAG). RAG is nothing but grounding answers of an LLM to a particular set of data. Using this along with the right prompt and context, you can reduce LLMs from answering from their own dataset about the requested information. This greatly reduces the chances of LLMs hallucinating or providing outdated information. Companies use RAG to ground LLMs into their own datasets and in their own language.

The miniature version of perplexity has the following steps, which at a high-level so does the real version of Perplexity:

When a question is entered, as a first step perplexity decomposes the query and adds relevant domain information. Perplexity does this using an LLM and generates 3 subqueries. For example, “Why did Silicon Valley Bank collapse in 2023?” generates the following search queries:

  • Silicon Valley Bank collapse 2023 causes
  • SVB failure interest rate risk bond portfolio
  • SVB bank run March 2023 timeline deposit outflows

I’ve skipped this part for the demo as Tavily (the search API) handled semantic queries well. A search is performed using a search index API. For demo purpose, I’ve used Tavily, which fetches the results and also returns raw text content. Saves us a hassle of crawling the URLs and fetching their content.

The fetched text now has to be sent to LLM to compose the final answer. But, given the size of text, we should send only the relevant pieces of text lest we overblow the model’s context or get inaccurate results. Enter embedding.

Before getting the embeddings, we need to split the documents into multiple pieces. This is called chunking. There are different strategies to chunk the document as detailed here. The reason for chunking is the same reason as above. The model has a certain limit for embedding and sending large amount of text loses its meaning or straight up fails.

For the chunking itself, we need to decide on what the “boundary” of the text is going to be, as in how we would like to split the text. If we had crawled and fetched the content ourselves, we could’ve used the HTML boundary based chunking, splitting the text by their HTML tags. But, because we’re using Tavily’s raw content, which returns plain text, I’ve used just a simple character based split with 1000-char making up one chunk.

Once the document is split into chunks, we then convert the text into vectors using an embedding model. For this demo purpose, I’ve used OpenAI’s text-embedding-3-small. You can use any embedding model as long as you embed both the chunks and the query with the same embedding model. Along with the chunks, we also convert the original search query into vectors. The generation LLM is separate and can be from any provider; it only ever receives plain text.

After we have all the vectors, we can fetch the closest chunks from the source list based on the search query’s vector. This “closest chunk” is nothing but fetching items using the cosine distance between the two sequence of numbers. We fetch the top 5 chunks here, but it could be anything based on the need, the chunk size, the model size. For the demo purpose, I’ve used a Vector DB (chroma) to store the vectors and to query the relevant chunks. But for this size of payload, we can do the query & filtering in memory itself using SciPy or other libraries.

Once the chunks are fetched, we send the relevant text, along with a SYSTEM PROMPT to compose the final output with source attributions. The model will generate the requisite text from the provided text and the final results are streamed. That’s about it. For demo purpose I haven’t built a frontend or images support, but those are pretty much extending from this.

–

Obviously this is a very simplified implementation of what Perplexity does. The real engine does lot more stuff for performance, cost, and accuracy. Some things which I believe they should be doing are having their own search index, specialised crawlers, strong caching across the stack, custom/in-house models for latency. The core thesis is the same though.

Feel free to fork and extend as you’d like to play around with it more. The source code is available here - avinoth/minilexity