Precision In Abstraction: Leveraging Operator Theory In AI/LLM Search

Precision in abstraction.

Much of the attention in the AI/LLM search space these days (and, perhaps, since they’ve rolled out) is on outputs – what gets retrieved/generated as an outcome of different models/search engines, et al.

This is, of course, because the output of any system is ultimately the realization of some initial goal or expectation – it’s what gets observed, consumed and/or used when all is said and done.

I’ve written much on the internals of AI/LLM search spaces – and the mathematical fabric that binds them together – as well as the output and the challenges of measuring and predicting those outputs (from the outside looking in, that is).

The difficulties in measurement stem from many different areas – primarily the contextuality effect that is present in high dimensional spaces and processes like these, but one area that is all too overlooked are the inputs (prompts/queries) and the associated input spaces they form.

My last post here on priming, I mentioned how these inputs can and should be thought of as operators (and ultimately form their own aggregate operator spaces that includes user-dependent elements and context around the moment of search).

Operators.

That term by itself can likely take on many meanings depending on your background (or… primed state) – perhaps you think of old school telephone operators or if you have a search background, search operators (this is where my thoughts naturally flow) or, perhaps if you have some mathematics background, you might think of operators like addition, subtraction, multiplication or division.

Stepping back in abstraction a bit, when I mention that prompts/queries are operators, I’m pulling from operator theory – the study of transformation, essentially.

Strictly speaking operator theory studies maps – operators acting as maps from one vector space to another – and the characteristics of the transformation that takes place through that mapping.

Building Intuition: Artists As Operators

Artists.

An artist can take in some object or scene of the real world and map it down into a sketch/drawing, painting or whatever the medium happens to be.

It’s a transformation of the “real world” down into the “representational” depiction, but the essence is that a transformation took place – input to output through an operation.

Different kinds of artists may give you different transformations.

A photorealistic sketch artist may give you a photorealistic transformation.

A caricature artist may give you a skewed transformation.

An impressionist may give you a “fuzzy” transformation… (and so on.).

Different flavors of artists, different flavors of operators, essentially.

Operator Properties, Classes & Topologies (Heavy Metaphors Here)

As a post that’s meant more for a light introduction to operator theory I won’t be going into extreme detail here, but here are some common types and categories of operators (and some very loose intuition on each – this is not meant to be rigorous, simply an introduction to operator flavors):

Bounded operators: nicely behaved, well defined and predictable transformations.

Think of these as a volume knob – when you turn it up, you get a nice, predictable result (sound level increases). A volume knob also has limits – even if some “go to 11”.

Unbounded operators: often messy, unpredictable transformations that can go off into infinity.

This would be similar to an avalanche – a very small snowball rolling down a hill that keeps growing and gets larger, uncontrolled. On an infinite mountain, it grows infinitely.

Unitary operators: size preserving transformations.

Think rotations — spinning a basketball on your finger doesn’t change the size of the basketball.

Compact operators: stable transformations that map larger (often infinite) dimensional space down to more manageable, finite dimensional spaces.

Think summaries of books – the summary operator compresses the book down into a smaller paragraph, that still captures what the book is about, leaving out unimportant details.

In addition to these common classes and properties of operators, there are also two important concepts: strong and weak operator topologies.

Strong operator topology: This is metaphorically like observing more deterministic outputs for the same input (same trajectories and more exploitative in nature that precisely lands on the same output).  Think precise question-answer prompts.

Weak operator topology: This is metaphorically like observing more probabilistic outputs for the same few inputs (different trajectories and more explorative in nature with a slightly different way of interpreting the “goal” of the output).  Think more ambiguous prompts where direction matters more than precision.

Prompt Engineering As Operator Design

Again, mentioned in my post on projection, I’ve looked at prompt engineering as simply operator design.

What is the transformation you want to achieve?

As you construct a prompt or query, you’re looking to achieve different goals that can be aligned with different operator types.

Summarizing a book is designing a compact operator.

Designing a highly specific, domain-oriented prompt/query is designing for strong operator topologies.

Creating a vague/loose prompt/query that allows the space to explore different paths is designing for weak operator topologies.

(If you dig further into the internals of the attention mechanism you can find many instances of unitary transformations – simple rotations, et al, by the way.)

Including specific context in with the prompt implies that you want some combination of different operator types or categories.

Mixing and matching different operator types allow you to achieve the unique transformations you’re aiming at.

Prompts As Operators: What This Allows Us To Do

In old school search we had a fairly well defined input space — keywords/keyphrase lists that could reliably be identified, categorized and measured (addressable search results).  As the space became more fluid and associative it evolved into a more topical basis – less granular, but still fairly well defined.

As the space continues to evolve, become a bit more unpredictable in different contexts with different users and as users get more fluent with different types of prompts/queries, we also have to evolve how we think about that input space in order to properly measure output trajectories.

From keywords to topics to… operators.

When we abstract prompts/queries into operators, it allows us to leverage the well-known and precise mathematical machinery of operator theory ( specifically when applied to high dimensional Hilbert spaces, similar to what we see in AI/LLM spaces ).

Precision in abstraction, essentially.

Next week I’ll go more in depth on what we can do with this leverage, more on projections, notation and measurement in non-classical spaces.