Measurement Operators On AI/LLM Search Spaces: Finding A New Basis For AI Search Visibility

The past few weeks have been a lot if you’ve been following along here.

To catch up folks that might have missed a few bits (and refresh those that have been reading), I started by walking through the little things when it comes to AI/LLM visibility, to how tools are reaching their classical limitations, to last week on how search & LLM spaces essentially breakdown into decision surfaces and the connection to parallel quantum-like decision modeling & research.

In not so many words, the overarching theme is that measurement tools (as far as measuring AI search visibility) need a new approach to properly capture (or at least consider) the different kinds of context (contextuality) that shape and influence responses – that are actualized in the moment, rather than pre-defined.

Subtle changes on the input side ( prompts/queries + surrounding context, like location and other items either inferred or explicitly known ) can change – often dramatically – what gets generated on the output (response) side.

On top of this, changes in the internal behavior of these spaces (model updates, retrieval augmented generation – otherwise known as RAG – improvements, among others) can also result in behavior that can change responses, even for the exact same inputs and context.

Together, all of these add up to outcomes (or observables) that are now probabilistic and more observer-dependent, rather than the more deterministic surfaces – with pre-defined lists and positioning/visiblity that could be tracked more readily – that we’ve come to know in the search world.

Observer-dependent measurements, measurements that create (rather than just record) on probabilistic spaces – things that classical measurements can have difficulties keeping up with, but in the non-classical world (quantum-like systems), they’re an intrinsic part of the field.

Waving the “quantum” word around shouldn’t be done lightly, however.

To clarify a few things, again, I’m not saying these spaces are physically quantum systems – they simply exhibit many of the same fundamental behaviors that quantum-like systems do (with supporting research and methods to make these observations more concrete), so the mathematical machinery used to measure these spaces should provide us with a better tools to do just that.

The exact mathematical machinery will be explored in future posts, but before doing so I’d like to walk through the steps & concepts that can be used in measuring these new spaces, now defined by probabilistic outputs.

Initial State & Context Settings Of The AI/Search Surfaces

To really understand how these surfaces respond under different circumstances you have to start with its initial state.

If you pull up one of these surfaces in a browser or app – starting with a fresh thread (no history, no user account logged in), the initial state is in a more general state – based primarily on the underlying model’s basic settings.

It may infer some things based on your location and other settings, but this is the most general/outer surface state it can be in.

As you start moving deeper into the context settings, that’s where settings become more personalized – things like being logged-in, chat history on, current chat context on top of any inferred location or explicitly known items – the state (and shape) of the system becomes more unique.

If you flip between different versions or specialized flavors of each model within these context settings, it adds even more to the complexity or shape of the space.

These unique states and different space shapes can and will influence the decisions made in generating the response — each one primed under different initial conditions, so understanding this initial state is important (vital) in understanding any measurements on it.

Observer-Dependent Outcomes & Measurement Bases

Understanding (and perhaps generalizing) how these spaces are primed or prepared becomes as vital for measuring as knowing what they’re actually prompting about.

In the quantum world (wildly simplifying this, but it gets us started), defining a state means defining a basis (basis has a more technical definition, but here it just means dimensions that can describe the space in total) – in this case the basis is the “primed” state that a user’s surface may be in before prompting.

Measuring (probing) these primed spaces with prompts/queries (operators) and observing the responses (observables) should give a more accurate view of how these spaces project into subspaces defined by the basis.

Prompts/Queries As Measurement Operators & Subspaces

Probes.

Every prompt, every query and other inputs can effectively act as measurement “probes” on the initial state (and future states) of these spaces.

In quantum terminology, these are called “operators”.

I’ll go into operators in another post, but all you need to know now, is that these measurement operators or “probes” effectively project the initial state into a subspace of that initial state, which is shaped by the makeup of the operator. This new subspace becomes the new “state” of the system for future prompts or queries (and so on).

As an example, say that the prompt for “best action movies of 2019” is input into a surface.

This effectively would project that initial state/space into a subspace, generating a response including specific action movies from 2019. This space is now primed for follow up prompts/queries related to those movies, which further shapes this subspace.

Each prompt or query effectively acts as a measurement – projecting the space further along the dimensions of the prompt and whatever state the system is currently in.

Since users can ask any questions about any topics in this new “action movies 2019” subspace, future unrelated (orthogonal even) questions can be influenced by the shape of this subspace (contextuality).

This is how things become more and more user-dependent (observer-dependent) — if users are prompting in the same subspace (same chat session) that they have been prompting other topics in, you can see how this additional context can ultimately affect the internal decisions made in generating responses (and any potential measurements on those responses).

Order of prompting (measurement) matters, essentially.

A Quick Break & Example

If you’re overwhelmed with the material above, no worries — it’s essentially the who, when, where, how and why context settings I described last week.

With the exception of the “what” (which are the prompts or measurement operators), each can help define the basis of the state (surface) before measuring (with the actual prompts).

For example, let’s define the basis of a state as:

Female, 40s, Real Estate Agent, Homeowner (Who)

1PM Pacific Time (When)

Seattle, Washington (Where)

In-App, Mobile Device, On Public WiFi, Logged-In, Chat History including research on Mexican restaurants in the area (How)

Looking For New Recipes For Dinner Tonight (Why)

You can define a basis any number of ways — but we’ll use this in our hypothetical.

So, the state of the system (surface) can be loosely defined by some combination of this basis — a prompt (measurement) on this system can result in a response that can be shaped using this combination – each which may carry it’s own weight on the response.

A prompt for “Find me new recipes that I can try out tonight” may be influenced by when ( perhaps pulling newer recipes or ones recently published ), where ( finding recipes by local influencers ), who ( recipes that are shared across other similar users ), how ( being on public WiFi may indicate being away from home, so grocery stores can be found for the trip home or knowing a preference for Mexican cuisine, pulling Mexican dishes ).

Knowing how each context setting can “pull” or have weight in each response (projecting into a subspace) is a story for another day, but without considering and defining a basis using them, it’s likely that a measurement will be further away from “real-world” results.

Along with this basis, defining how measurement operators (prompts) are constructed on the front side, can give us an even sharper view of things moving forward, as well. ( A lot more on operators in future posts. )

Moving From Static Positioning To Stochastic Inclusion

In more classical search settings, we can measure and yield (mostly) certain, static positioning. As mentioned in an earlier post, even that static positioning can change depending on context (like location), but for the most part we have a more clear understanding of positioning in “vanilla” search.

With the now non-classical search settings (ones that draw comparisons to quantum systems), we ultimately are moving towards more stochastic (probabilistic) inclusion.

Under certain, specific context settings (basis) and associated measurements (prompts/queries) – what is the probability of a brand, entity, website, etc. being included in the response?

This isn’t really breaking new grounds (many measurement platforms and other folks have recognized this and have starting making adjustments), but not many are seeing the underlying reason — without it, measurements can still be flawed.

Exact methods and calculations for these probabilities will be explored, but ultimately static metrics under generic context settings are increasingly unlikely to give us a full understanding of visibility.

Leveraging these already well-known non-classical measurement methods, starting with the concepts above should help us chart a better, more clear path to visibility measurements in AI search spaces.

The Road Ahead

Next week I’ll be digging into more reasons why these spaces may be exhibiting the non-classical behavior – a closer look at the internal workings of the transformer and attention mechanism – which will only reinforce our need for higher level measurement methods in our world, and tie-in how it relates to SEO and how our SEO fundamentals will be even more important moving forward.