The Little Things: Why You’re Having Such A Difficult Time Measuring AI Search Visibility

There’s been something bothering me in the search world.

I’ve been (quietly) working in search & digital marketing for nearly 2 decades and the past 7 or so years I’ve been doing everything I can to hold back my notes and thoughts on how things have evolved in recent years.

In my world (working largely with small businesses) we value quietly getting the job done – the little things that keep the world running while others chase glory, fame and recognition.

There’s nothing wrong with chasing those things (many are well deserving of recognition), but I’ve always found the true heroes of the world didn’t chase titles or followers – they just did the right things even if it meant going against the tide or sacrificing short-term rewards for long-term stability.

The little things.

My story in search isn’t really for the faint of heart (more on that below) but the quiet, little sacrifices all of those years didn’t seem to make a lot of sense – until recent years.

The things that have been bothering the past few years started finally reconciling, and perhaps this will be one of those little things that will keep our digital world running.

Particle Physics

My origin in search truly began in a tiny coffee shop while I was in college.

I didn’t know at the time I was going to have a career in search; in all honesty I didn’t really like computers all that much back then.

They were useful to be sure, but at that time they weren’t the omnipresent thing in our life – it was an optional resource that few really understood how to get the most out of.

I found more delight in a textbook than anything I could find online back then – including Mathematics (which eventually became my major course of study).

Mathematics wasn’t the real goal back then either, however – it was particle physics.

Shortly after my father’s passing in college was a strange time; my original pursuit of electrical engineering didn’t feel right or favorable anymore for some reason.

My interests turned more… philosophical – asking questions about existence (as one often does while coping with death I’ve found) – and the existence of a hidden world underneath ours that I could almost feel my entire life – something we couldn’t see or touch or feel (at least directly).

That’s when I came across particle physics – the study of the basic building blocks of the universe.

Surely if there were answers, I would find them there.

The little things.

Back to the coffee shop – it was there I was having a coffee with a friend and we began discussing particle physics.

I hadn’t studied it enough to really understand and explain it yet (I was still in electrical engineering at the time), but I could draw the diagrams of how antimatter could emerge from bombarding atoms together (that still absolutely fascinates me).

So on the glass window – in the condensation that had built up – I sketched out a very crude representation of how antimatter is created.

It fascinated the person I was with – her face lit up; it was like showing someone one of your favorite movies and they love it as well – all from a few little swipes on a glass window.

That little moment tipped me over the edge and I officially changed my major – not to physics, but to Mathematics.

Mathematics & Quantum Mechanics

Looking at the journey one takes to becoming a particle physicist, it was much different than that of an engineer.

The overlaps were there in Math and Physics – both subjects that I have a deep love for and were requirements for engineering as well, so that gave me the groundwork.

I had to go much, much deeper into both in order to properly pursue particle physics, so I made Mathematics my official undergrad major as it gave me a bit stronger foundation than a pure physics play.

The deeper I went into Math, the more things started to resonate and I found myself taking a course in Quantum Mechanics a few years later.

I remember having goosebumps in that class – the material was thick (it felt almost alien at certain points), but I felt like I was truly looking “under the hood” of this hidden world I was desperately seeking at the time.

What made it more fascinating was the probabilistic nature of this hidden world – and the fact that experiments change depending the observer – remember that for later: “observer-dependent outcomes”. Little changes could mean much different outcomes of an experiment.

The little things.

Economic Issues And A Path Diverged

Gliding through the Mathematics undergrad courses (with a healthy dose of Physics), I was all prepped and ready for attending grad school for Particle Physics.

The economy started to slump, however, and the whispers were growing into shouting and finally a choice was made: I would head to greener pastures where I could get a “bridge” job to make some money until grad school became more viable (a young man’s knee-jerk reaction in an uncertain world in hindsight).

There wasn’t much available for someone with an undergrad degree in Mathematics, I quickly found out though.

Through some connections I found myself working for an internet marketing sales office as an assistant of sort (there was no real title but I was desperate at the time).

Fresh out of college, my young idealist mind felt uncomfortable in the business world, but it was there I was introduced to the world of search.

Ideas came quickly, I rose fast (multiple times) – fell (incredibly) hard (multiple times).

Bruised, bloodied, wounded and scarred (literally and figuratively) – tales for another time – the journey hasn’t been easy and didn’t really make a lot of sense for years.

But here I am nearly 20 years later finding myself at the emergence of LLMs and AI from the search world – suddenly the lost particle physics path, the hard years, the battles won and lost, unhealed wounds and sacrifices are making more sense every day.

And it’s because of… the little things.

Harnessing Uncertainty (Observer-Dependent Realities Of Search, LLMs & AI)

There’s a major shift happening in the world of search – it’s actually been shifting for quite some time.

The tools that the search world uses to measure performance are becoming less reliable.

The shift from deterministic search results (with lists of things) to the probabilistic behavior of LLM/AI generated responses means measuring is no longer just recording things – every measurement is actively creating potentially new responses.

It’s not that our search practices have to change (I will go into this more in future posts) – every fundamental that we’ve come to know over the years are still absolutely relevant – it’s that measurement has become a moving target because measuring is creating, not just recording.

Little changes in inputs (queries, prompts, etc.) can yield dramatically different outputs (results, responses, etc.).

And little changes in context around those same inputs – can also yield unexpected results.

The little things.

What It All Means & What’s Next

For the longest time (especially in recent years above) I’ve had this intuition that we were working on something much larger than search.

All of the words were there.

The transformer architecture, the “non-local” behavior of the attention mechanism, probabilistic outcomes – all of it right in plain view – hidden beneath the surface of everything we’ve been doing for years.

While there are many theories in search, it’s increasingly clear what mental model we may need to use in order to study and understand what’s currently happening in our world – and measure it properly.

Recent research on these spaces I’ll post next week will make it even more clear – and all but confirms this mental model.

In fact, almost every bit of research lately has been leaving little clues to confirm the thing that’s been bothering me for so long.

Little clues – the little things.