Decision Making: Asking The Right Questions About AI Search Visibility

AI search visibility is a big talking point in the search world.

Last week I mentioned how measuring these new search systems powered by AI & LLMs are running into classical limitations — classical in the sense that one-off snapshots of positioning just aren’t enough anymore, as results and responses have become more and more user-dependent.

That is, responses can change dramatically depending on the particular users, the query (prompt) and other context around the moment of search (or chat session).

As mentioned last week, this coupling between users and associated responses has been happening for quite some time. One of the most obvious examples is in local search, where an inference in location is made about the user, so search engines present results from businesses and websites located closer to that individual user.

User location is one of many context-influencing elements that can affect potential results and responses – which are realized in the moment of search.

With the new AI/LLM responses, attempting to measure this moment of search (or prompt) means an act of creation happens – with potentially new or different results each time you search; this “measurement that creates” characteristic is what can throw off any classical measurements. A single request only tells you what that response looks like in that particular moment, with that particular context, for that specific query or prompt.

Subtle changes in the prompt or any related context (inferred or explicitly known) can alter the outcome (sometimes dramatically).

Boiling down this query/prompt -> result/response process – however you slice it – comes down to a decision made by the particular surface ( search, AI/LLMs, etc. ) in those moments.

Search Is Ultimately About Decision Making

One can go down many rabbit holes when explaining how traditional search works.

For major search engines, the amount of engineering involved in building systems to present relevant search results is immense (and incredibly fascinating).

Digging into search engineering is not something I want to do on this blog (yet), but when one steps back a bit, it’s really just a very sophisticated decision making process (retrieval is ultimately a subset of decision making).

Whenever a query or prompt is entered into one of these surfaces, a decision is made: what is the most relevant resource (or set of resources) for this particular user, in this particular moment?

Without getting into the thick of search engineering, scores are typically assigned to the most relevant resources, with the most relevant positioned closest to the customer (at the top of the search results page).

In the world of search engine optimization (SEO), the goal is simple: working on becoming the most relevant resource in the search space containing relevant resources for users searching in that space, in the right places, at the right time.

The fine art of SEO (the “how”) is too vast to go into here, but that’s really the working goal for any SEO in the world.

When you become the most relevant resource, you increase the odds of being picked (first, or “next”) when the decision is made by the search engine for that search.

The same idea applies to the AI/LLM spaces.

Decision Making In AI/LLM Spaces

Going into the exact engineering behind AI/LLM spaces – like search – is a topic for another day, but the goal of these systems is the same: deciding on what’s the most relevant in the moment.

Instead of resources ( business listings, links to web pages, ads, etc. ) being evaluated for relevance, AI/LLM spaces are deciding on more granular objects – called “tokens”.  (Often these tokens are words, but tokens can represent more than just words in many cases).

This decision making process is done in rapid succession: choosing the “next best” token and iterating that process until a coherent response is produced. Generation of responses can include business listings and links to web pages as well, like search.

The “next best” word is chosen (mostly) based on probabilities – each word is assigned (roughly through linear algebra and some additional mathematical machinery I’ll likely discuss in a future post) a probability to come next in a particular sentence or passage and the one that works best in the moment is chosen. These probabilities are “learned” during training and inferred in the moment of responding, given a particular context (context, again).

I’m overly simplifying this process (much like search engineering, AI engineering is a much more intricate task), but the essence is that decisions are made by these AI/LLM surfaces in the moment to present the most relevant information, given a particular context.

In search spaces, these decisions lead to results of resources – in AI/LLMs these decisions lead to conversational responses; both ultimately shaped by the context in which those decisions were made.

How the SEO world works on creating enough value to become the most relevant element in the “resource” set in search moments works exactly the same in the AI/LLM space: becoming the most relevant in the “token” set in those moments.

If you imagine those relevance scores assigned to resources in search as probabilities for “next” tokens in AI/LLM spaces – you have an analogous model for decision making. Of course relevance scores and probabilities are not assigned the same way, but they are generally analogous in how they’re used to produce results/responses.

Context Dependency And Asking The Right Questions About AI Search Visibility

General decision making processes feel easy in both search and AI/LLM spaces: simply pick the resource or token with the best relevance score or highest probability of being “next”.

However, “best relevance score” or “highest probability” ultimately changes depending on context – separating the response from the individual user becomes almost impossible, thus the “observer-dependent” nature I’ve been describing in this blog.

It’s this contextuality that is ultimately hindering our ability to measure these spaces – a single snapshot from a generic context setting won’t tell the whole story.

Understanding who is entering the query or prompt, what they are entering, when they are entering it, where they are entering it from, how they are entering it and why they are entering it are the better questions to ask when trying to measure it.

Who. What. When. Where. How. Why. The context settings.

The answers to these questions can ultimately change the decision making process in each moment (search or prompt).

It’s the great entanglement between users, their prompts or queries, search/AI/LLM spaces and the responses they generate.

Without evaluating (or at least considering) how these questions shape and affect responses, it’s going to be incredibly difficult to measure any of these systems with any degree of accuracy.

In The Meantime

Going back to last week’s post I mentioned how these new spaces exhibit the same kind of contextuality that quantum systems do. There is parallel research that dates back decades on how decision making processes (by humans) also exhibit this kind of contextuality, so you can kind of see where all this is going, potentially.

Next week I’ll explore some of the ways measurements are done in those spaces and how they can help shape the future of measurement in AI search.

In the meantime, if you’re deciding on options for improvement in search & LLM spaces, many of the same search fundamentals will still hold true — working on becoming the most relevant resource, for the right users that are using relevant prompts/queries, at the right time and in the right places.

(These things look similar to the context settings described above… don’t they?)