A Guide to Understanding AI Search

In the old days, site search was basically a keyword-based view into your site’s content. You might get fancy with fuzzy matching, or stemming, or even logical operators available for your power users. But ultimately, site search was fundamentally based on text matching with some algorithmic ranking layered on top. In the age of AI, site search can be so much more, often barely recognizable as a search function, but search still underpins the whole experience.

First, a clarification: I am writing about search that exists on your site to search only your content. This isn’t about getting mentioned by ChatGPT or Google’s AI search overview. Those are important topics, about which we have written extensively, but not the focus here.

AI Site Search can be difficult to discuss and understand because it encompasses a large swath of technologies, which create something of a spectrum of capabilities. On one end of the spectrum you have clearly an identifiable site search which is largely traditional, but with results enhanced by AI. On the other end you find chatbots which provide answers powered by an internal search never directly seen by the end user.

But why bother with a site search? Obviously a site search surfaces your content and connects your users with the information they need. A site search also provides you with information about what your users want and are looking for. A chatbot can tell you what they’re literally asking for. And as search evolves from listings of results to direct answers, a well-built chat integration can even provide answers from documentation that isn’t publicly available.

AI search terminology.

Before digging too deeply into the types of AI search available, it’s helpful to get a quick primer on the technologies that power this kind of search.

A vector search (sometimes called semantic search) searches words by approximate meaning. Take for example the words: king, queen, mother. We intuitively know that king and queen are related because they're signifiers of royalty. Mother and queen are associated with gender. You can think of these associations as “dimensions.” A prince would score high on the “royalty” dimension, but a car would not. Scoring words on enough of these dimensions gives us the ability to start to clump words together roughly around their meaning. These dimensions and how words score on them are called embeddings. For example, a larger, popular OpenAI embedding scores words on just over 3,000 dimensions. By comparing how words score along these dimensions, we can find words “near” each other. Vector search uses this to match meaning rather than exact wording.

A Large Language Model (LLM) is fundamentally a prediction engine. You feed it text (in short chunks called tokens), and it uses a massive statistical model to predict what text comes next. This of course is a massive simplification, but gets to the core idea. This text prediction is what gives us the chat responses, tool calls, and reasoning.

The model has something like attention and can look back on the preceding input while determining the next bit of text, but the attention is a bit limited in how much text it can actually consider. This is the context window. Because the context window is finite, we can’t necessarily just hand an LLM your entire corpus of text in a single document.

Retrieval Augmented Generation (RAG) is to some extent at the heart of most AI search systems. It’s a process to allow an LLM to effectively search and report on a large corpus of text. All of the content needed for search is split into manageable chunks and indexed into a vector database. When the LLM needs to find something, it first searches the index, reviews the chunks of content, and processes the result from there. Advanced setups allow the LLM to determine which terms to search, read the result snippets, and request full documents from the snippets to fill out full context.

The AI search spectrum.

From these concepts we can piece together the spectrum of AI site search possibilities.

First, there’s semantic search. This is the closest we get to traditional site search. It’s essentially a normal site search, but with vector search instead of text matching. Think of it as a slightly smarter keyword search. This type of search works well if you have a highly technical audience searching for very specific terms and needing direct access to real documents, not an LLM’s reinterpretation of those documents. Because no LLM is involved, it never hallucinates and only returns results that actually exist in the corpus of data. It pairs well with faceted search capabilities or other more advanced options. It may feel old-school, but still has its place.

In the middle of the spectrum, there’s RAG-enhanced search. In this scenario, we take the results of a semantic search and process them through an LLM for possible enhancement and re-ranking. This can help remove irrelevant results that technically match the search, for example. For the most part, results are still strictly constrained to the corpus of data, though the more involved the LLM is in tweaking results or providing additional context, the greater the possibility of a hallucination or misinterpretation. A RAG-enhanced search attempts to strike a balance between full-on chatbot and straightforward search.

At the far end of the AI search spectrum is a full-blown chatbot. In this scenario, a chatbot makes use of some kind of RAG-enhanced search mechanism internally, then gathers the results and presents an answer and additional information to the end user. In some cases, the bot may include citations and links to the content the answer comes from. In other cases, the user may not even realize a search is being performed at all. Here, the LLM has the most leeway to interpret and contextualize results, which also increases the risk of hallucinations and mistakes. It’s important to have strong guardrails and to check the bot’s performance frequently. This style of search works well for users who just want an answer or are not sure of the best terms to search for. In advanced scenarios, the LLM may even be provided with tools to pull data from outside sources, execute workflows, or to present forms, calculators, or other interactive tools to the user. This is the most powerful option for interactive, personalized search.

AI search considerations.

No matter where your specific AI search solution lands on the spectrum, all approaches demand some similar considerations.

• Any search works best with a solid corpus of text. As I’m fond of saying, “garbage in, garbage out.” The quality of search results will depend heavily on the quality of indexed documents.
• The index must be kept up-to-date. Stale or contradictory data leads to poor results. This was true in the days of traditional search and remains just as true today.
• Even the best frontier LLMs make mistakes, and there is always a risk that any LLM may respond incorrectly. Guardrails, guidance, and frequent tests are key here.
• Inference costs vary, but typically the costs increase the closer you get to a full-blown chatbot. Costs are typically based on usage, so it’s worth monitoring for any runaway costs or abuse of your search and/or chatbot system.
• “I don’t know” is a valid answer, and is a better answer than making something up. It’s almost always better for a user to get no results than a misleading or false result.

Ultimately, the AI search strategy must meet the needs of the audience and the goals of the organization. If you’re considering an AI site search solution and are looking for guidance, we’d love to chat.