Rigor: The Implications Of Contextuality & Observer-Dependent Outcomes In AI/LLM Search Spaces

Take a deep breath.

If you’ve been following along on this blog, the past few weeks have been a bit more heavy on technical & mathematical side of these new AI/LLM search surfaces, so this week I wanted to take a step back (another step back) and ask a very simple question:

What does it all mean?

Most of the posts here have been more diagnostic in nature – helping fill in the gaps of why these spaces are behaving more and more non-classic over time.

I’ll let you thumb through the past posts more in detail, but the TL;DR version is this: between the construction of the inherent representative vector spaces, coupled with the internal mechanisms of the transformer and attention mechanism (creating non-local bindings), responses/outputs from these surfaces are increasingly observer-dependent.

That is, it’s getting increasingly difficult to separate the responses from the individual users – and surrounding context at the time of interaction during that response.

Who enters the prompt (and any related attributes – inferred or explicitly known), where they enter it from, when they enter it, subtle changes in prompts, the order of the prompts, why they’re entering it, the state (or shape) of the surface when prompting, the model version of the platform and any specialized flavor of those models — all of these can potentially change the shape or rotate a potential response for each user.

These response-altering elements aren’t new – vanilla search has been using these for years (just ask anyone that has worked in local search the past decade or so), but with the more granular, hyper-sensitive spaces we’re now working in, each element can dramatically shift or rotate (rotate here means something else, but that will be included in a future post) those responses.

The intricate contextuality phenomenon we’re now facing has a few implications – a few that are known already, and a few that are obvious, logical consequences that everyone working in this world should already be working on (the good news).

Measurement Will Continue To Be Difficult

I’m driving this point home hard in this blog, for obvious reasons.

Single-snapshot realities of current measurement tooling is increasingly difficult to draw definite conclusions from.

There could be opportunities to find proxies for “downstream” measurements in AI/LLM responses, but getting the raw, static positioning picture is going to require a different approach, in my opinion.

While many of these platforms are starting to roll out more internal metrics on inclusion, citations and other items related to AI search, they will ultimately come with caveats — inclusions at one point in time, and under certain context & environmental conditions won’t necessarily allow you to extrapolate that to future inclusions & references.

Perhaps the most overlooked element is the constant internal model updates – this obfuscates things further even if you’re able to incorporate and capture the raw context & environmental conditions in a measurement setup.

These surfaces are still very much “settling”, so patience – and not straying away from fundamentals, will be key to success, I believe.

Fundamental SEO Becomes More Valuable (Context & Fidelity Management)

Last week I went into detail on why SEO becomes even more important in the AI/LLM search age.

As these spaces become more sensitive to context (both within their internal representation and input prompts/probes), being exceptionally diligent about intrinsic and extrinsic context – and managing fidelity between the two will be vital to cut through some of the new layers we’re working with.

As it turns out, in the search world we already do much of this heavy lifting — from on-site work to off-site work, there is little that SEO doesn’t already have a “finger on the pulse” with.

If anything, we have to be even sharper with this work – within these processes – not trying to reinvent the wheel, so to speak.

Observer-Dependence Means Knowing The Observer Takes Precedence

This is the section that will help ground things even further for us.

The technical and mathematical walk-through of this series may look complicated – and our obstacles still remain – but if you look at things holistically, the conclusion is pretty simple:

If responses are increasingly shaped by individual users, their activity and associated context settings, then those that know and understand those users best will be the best prepared for the road ahead.

And if that sounds familiar, it’s because it is: it’s digital marketing.

Mathematical machinery and technical engineering combined elegantly to create the conditions that digital marketing was meant for: lighting a pathway to reaching a target set of users through digital surfaces – including these new search surfaces.

It’s beautiful, in many ways – all of the little things adding up to a very logical, reasonable conclusion: observer dependency means getting sharper at understanding the observer (your target audience/users) – it’s brilliant.

What To Look Forward To

Despite this rather simple conclusion, there is an ocean of mathematical and technical details I plan to write about in future posts — things that should help us get some clarity and insights into both measurement and applications to SEO and digital marketing at large.

We have to kind of remember that these new surfaces are very new – and largely untested at the levels of scale that we’re seeing these days, which means it’s likely we’ll be running into new “seams” in the surfaces that will have be smoothed out – and, perhaps better explained through non-classical mathematical rigor.

And the more rigor (more notes next week) we can identify and find footing with, the better equipped we’ll be for understanding, measuring and predicting behavior as these new surfaces continue to evolve.

Take another deep breath – and keep moving forward.