Lighting The Way: Prisms, Spectral Theory & Toward Better Measurement In AI/LLM Search

Lighting the way with prisms.

In the 19th century “prism lights” were used to light underground spaces – under vaulted sidewalks, beneath buildings, storage spaces and other places where natural light was difficult to reach.

Prisms were more effective than ordinary glass – they disperse light – or cast a spectrum of light – rather than just letting light pass through; they can be adjusted or aimed in particular directions in order to get precise illumination patterns – lighting the way, so to speak.

In my post last week I discussed operators – maps or transformations from one vector space to another and how abstracting and modeling our input/prompt/query space as operator spaces allow us to leverage the mathematical machinery borrowed from operator theory.

Among many opportunities this exposes for us is the application of spectral theory – the study of the natural tendencies or “modes” of operators.

Prompts/queries are getting increasingly more complex — old school SEO mapped from keywords, modern SEO can loosely be mapped from topics – and in the age of AI where prompts and queries act more as probes on the space. This probing process perturbs the space as it retrieves and assembles responses, that come with more advanced, nuanced directives.

From non-classical physics context, probes are mapped to operators — different operator classes yield different observables on a quantum system, essentially. Studying the natural tendencies – through spectral theory – of each operator yields allow us to predict outcomes from what would seem impossibly unpredictable spaces.

They light the way in the darkness of unpredictable “black box” quantum spaces.

As you’ll see below, our operators are our “prism lights” as we chart a course toward measurement (from the outside looking in, that is).

Quick aside: Operator theory and spectral theory are incredibly dense subjects, so this won’t be an intricate rundown – simply an exploration – we’ll exploit the rigor in future posts.

Spectral Theory Intuition Building

The intro above is a pretty nice intuition primer, but let’s take an even closer at a few analogies.

Returning to the optical prism a bit more closely, it refracts light into its individual components.

The white light’s spectrum is the rainbow of colors it refracts as light passes through it — different wavelengths of light bend at different angles from violet to red.

It’s a decomposition of white light into its “natural” visible components.

Another quick analogy is from the music world — when you strike a chord on an instrument, it’s not a single note – its spectrum is the decomposition into individual, natural notes.

In short, spectral theory helps us decompose seemingly complex things (operators) into their most basic, natural components. In doing this, we can measure the individual components (and their contribution to the whole) instead of trying to create more complex measurements (and many situations where direct measurement is impossible).

Graph Theory Tie-In, Eigenvalues and Eigenvectors

Continuing the mental model from previous posts here, evaluating responses as walk trajectories through complex, dynamically generated graphs (governed by a transition matrix at each step) for some input operator – or composition of operators.

This is how we come across “Markovian ghosts” – relics or memories of natural transition tendencies between certain words, concepts, entities, et al, carried over from our traditional web structures for those input operators.

What these ghost remnants – in essence – reveal is the spectrum of those operators.

The individual components being frequently exposed words, passages, entities, concepts or other token/response units that the operator response trajectory can be decomposed into (and naturally pass through in its trajectory for any given operator).

In spectral theory, these would be related to the operator’s eigenvalues/eigenvectors — things that remain stable through transformation.

The mapping isn’t perfect, admittedly, as responses aren’t as nicely arranged like our light spectrum (and some hardening of the rigor of the operator side), but when you step back and itemize the response pieces (omitting common/connector words, etc.) and aggregate – (or integrated) – response trajectories, it’s the handle we’ve needed so desperately in the measurement world.

Towards Better (And Generalized) Measurement In AI/LLM Search

Although modern AI/LLM search platforms are mostly based on the transformer mechanisms and “attention is all you need” framework, each model is going to have its own primed state — between training and specialized/proprietary/unique engineering intricacies, meaning our traditional measurement tools likely won’t be overly useful (for long) the more these platforms diverge.

Generalizing our input/prompt/query space into operator theory and leveraging spectral theory for predictions and measurement essentially allow us to develop more… generalized instrumentation.

Even with different flavors of models and constant updates rolling out, we still have our operator space in place (that’s independently constructed from our prompt operator space) – and our spectral analysis to measure outcomes. What happens in between those two mechanisms becomes largely irrelevant (or, perhaps, neutralized), despite being “black boxed”.

A universal prediction and measurement mechanism, so to speak.

Lighting the way with prisms.

Much work to do, but stay tuned.