Decoding The Decoder: Search, Encapsulated & Associative vs Addressable Memory In AI/LLM Spaces

Input. Output.

A wild simplification to any system or function is that it’s a “box” of sorts that take in an input, performs some kind of operation (or operations) and returns back some output.

The past few weeks I went over some of the more rigorous mathematical machinery that help give us a foothold in understanding the interpretation of input spaces inside AI/LLM search.

From topology, to metric spaces, vectors spaces and inner product spaces — each giving us a bit more structure and tooling along the journey from your raw content to ultimately being digested – one way or another – within the representational space in modern search spaces.

Creating a high fidelity representation and managing context across your website – and others’ websites – then, becomes critical for settling into relevant, relative areas and positions within those spaces.

Ultimately this is part of the encoding process within search — encoding raw information (input) into a representational space/surface that can be used during the decoding process — the step that yields results/responses (output).

While this is playing loose with the actual engineering and fine-grain nuances of how many of these spaces function (we’re setting aside Retrieval Augmented Generation and other tooling for now), this is the idea: encoding inputs and decoding outputs.

Input. Output.

As mentioned in my previous post on Query, Key and Values and the Attention Mechanism – you’ll find the heartbeat of search almost everywhere you look inside AI/LLM spaces.

Taking a closer look at the decoding process, it’ll be even more clear how search ultimately inspired – and forms the basis for – the AI/LLM world.

The Purpose Of The Decoder

Given some input ( query, prompt, etc. ) and context, the decoder’s fundamental purpose is to retrieve and assemble a set of results or a response from a representational candidate space that is relevant to the input/context.

In traditional search, the candidate space can be filled with web pages, business listings, passages/snippets, et. al. that populate a search result page.

In modern search and AI/LLM search spaces, the representational candidate space is a bit more granular – defined by tokens (words or subwords) that are recursively retrieved (and chosen), word-by-word (or sub-word by sub-word) to create a coherent, conversational response.

Abstracting away some of the finer details of the two spaces (traditional vs modern/AI/LLM), the decoder’s function is exactly the same: given a query/prompt (and context), what is the most relevant response?

We didn’t necessarily call this “decoding” in traditional search, but replacing tokens with web pages in the representational space, it’s clear that we’ve been staring at the decoder in search for a very long time.

Addressable Spaces vs Associative Spaces

In traditional search, we had more deterministic results — that is, the representational candidate space (we would call the index) was addressable.

Addressable here means that the exact location of a web page (or set of web pages) was relatively stable — there was a location within the space that made it easy to look up (and address book, of sorts).

It made search results mostly predictable.

As more features and different signals were added over the years, and the encoding in the candidate space became more granular and dynamic — words/passage tokenization within (loosely speaking) vector spaces et. al., we slowly moved toward more of an associative search space.

The “look up” function became context-dependent – where results depended not just on the incoming query/prompt, but also the associated context and other signals at the moment of search.

This associative search space means the decoder is doing more of a “soft” lookup across a more granular, fluid representational space (surface) that can change depending on context (where we find topology as a useful reference).

The Familiar Decoder Terminology

I won’t be getting into the exact engineering of the decoder (perhaps in a future post), but if you have some time, dig into some of the terminology used.

In your research you’ll likely find words like “greedy search” (choosing the most relevant candidate), “beam search” (and expansion of greedy search), top-k search (choosing top “k” number of most relevant candidates in a search space), top-p/nucleus search (choosing top “p” number of relevant candidates based on some collective probability).

Those aren’t the only decoding strategies or tools, but looking at the heart of the function here you can clearly see the pattern: retrieval, combined with a decision mechanism over a representational candidate space.

Decoders Ultimately Shape Responses/Results (For Better Or Worse)

Decoders are ultimately the “oracle” (wink) of sorts that shapes the outputs you see from AI/LLM search spaces.

Small changes and choices in decoding strategies can drastically change the landscape of the collective output space (surface).

As mentioned briefly above, decoding can pull from a model’s internal, parametric memory surface (our representational space) and also from external resources (known as RAG) to produce different responses over time, that can also dramatically change the shape of the output surface and associated responses.

What We Can Control vs. What We Can’t Control (Or Measure)

The contextuality phenomenon that stems from both the encoding and decoding process means there are pieces of the AI/LLM search process we’ll never have full resolution on.

To be frank, there were always elements of traditional search that even the best SEOs never quite had all the pieces for, but with the additional complexity of the encoding process and more fluid representational space and more sophisticated, granular decoding process, the handle can feel even more loose at times.

The modern nuances of the decoding process also mean that measurement can ultimately be obfuscated (I write about this in many of the earliest posts here), so it can compound the challenges we face even more (a challenge I have many notes on, coming soon).

While this all may feel overwhelming, focusing on what you can control is often the best route (something that was true even in traditional search).

SEOs and marketers control the intrinsic context (onsite work) and extrinsic context (offsite work) — that hasn’t changed since day 1 of search marketing (in many ways we were the attention mechanism before the modern transformer attention mechanism, but I digress).

Understanding prompts/queries and the accompanying attention mechanisms they pass through are important, but focusing on shaping – and improving – your representation within candidate spaces is ultimately still the heart of sound, smart SEO.

More notes on an extension of mathematical machinery beyond the AI/LLM search space (often overlooked) in coming weeks.