Everything Is A Math Wrapper, Part III: Vector Spaces And Norms In AI/LLM Search

If you work online, in search or digital marketing in recent years it’s likely you’ve seen or heard the word “vector” or “vector space” thrown around a lot in AI/LLM search media and thought leadership.

Those terms aren’t overly familiar or natural to the marketing world (mostly). Perhaps in high school or college studies you’ve come across those terms before, so you might have some intuition.

If you’ve come across my previous posts on topology, topological spaces and metric spaces, you’ve seen that those terms and concepts come with a bit stronger, strict definition from the mathematics world than most might know or understand.

Vectors and vector spaces are no different — they’re kind of thrown around a bit haphazardly sometimes in AI/LLM discourse (and engineering in general), to be honest.

Like the topological and metric spaces mentioned above, these AI/LLM spaces behave in ways close enough for them to be called vector spaces, though the thoughtful, hard definitions don’t always quite match (as you’ll see below).

Despite these inconsistencies (mostly caused indirectly by engineering, training and other nuances inside the AI/LLM search processes), I believe it’s useful to understand the rigor behind them – to solidify your understanding and intuition, if anything (and to know that there are more pure mathematical states inside these spaces possible, perhaps).

Following along with this recent series of blog posts from topological spaces, to metric spaces, vector spaces (together with a norm – more on that below) lay in a (loosely speaking) smaller subset of those two previously covered spaces.

That is, a normed vector space is a metric space, and every metric space is a topological space. So you can see how things fit together – and how structure starts to resolve and feel familiar as we look closer inside these spaces.

An Informal Intro To Vector Spaces

(x, y)

If you’ve taken even the most basic mathematics courses, you’ve likely run into this familiar pair (x,y).

An ordered set that describes a point in space that is “x” units along an x-axis and “y” units along a y-axis. This point in space represents some distance away from an origin (0,0) known as a magnitude – drawing a line from that origin to that point gives you a sense of direction.

The combined traits of having a magnitude and direction, make (x,y) a vector. That’s it, really. Magnitude and direction.

You can extend this idea to 3 dimensions – (x,y,z) – 4 dimensions, 5 dimensions — up to any arbitrary number of dimensions; there’s a magnitude and a direction, thus, vectors.

Take all of the possible points of (x,y) and collectively they form a vector space (over some field – something beyond the scope of this post).

A (Semi) Formal Intro To Vector Spaces

Stepping into a more formal setting, vector spaces – also called linear spaces – require some specific properties to be true vector spaces.

For a space to be called a vector space – call it V – the following must hold:

For all elements x, y, z in V:

(x + y) + z = x + (y +z) (Associativity)

x + y = y + x (Commutativity)

0 + x = x + 0 for some 0 in V (Additive Identity)

x’ + x = 0 = x + x’ for some x’ in V (Additive Inverse)

Collectively these form vector addition.

For all x in V and some scalars (numbers) a & b:

a · (B · x) = (aB) · x (Associativity)

1 · x = x (Multiplicative Identity)

Collectively these form scalar products.

For all x, y in V and scalars a & b,

a · (x + y) = a · x + a · y (scalar product distributes over vector addition)

(a + b) · x = a · x + b · x (scalar product distributes over scalar addition)

If these hold true, elements of V are called vectors and V is a vector space.

There are other important properties of vector spaces, but these are the most relevant to this particular post.

In most applications the elements of V are over a field of Real Numbers, but can also be complex numbers (important for a later post).  As they’re more relevant to the matter at hand, however, we’ll be working with real vector spaces.

Vector Spaces And Norms

Vectors, by themselves in a space, don’t come equipped with a sense of distance or proximity – and the familiar term, metric comes back to us here as something called a norm.

From last week, we recall that a metric is a function “d” on some set that has the following properties:

d from a point to itself is always zero: d(x,x) = 0
d is always positive: if x ≠ y, then d(x,y) > 0
d is symmetric: d(x,y) = d(y,x)
d satisfies the triangle inequality: d(x,z) ≤ d(x,y) + d(y,z)

Applying this to a set of vectors, then, allows us to define the norm ||·||. ||x|| for some vector x is the length of x, or d(x, 0).

It follows then,

||x|| ≥ 0, for all x in V (non-negative)

||ax|| = |a|||x|| for all x in V and a is a Real Number. (uniform scaling)

||x + y|| ≤ ||x|| + ||y|| for all x and y in V. (the triangle inequality)

If ||·|| holds with these properties, it is said to be called the norm of V – together they form a normed vector space.

So what does this tell us, exactly?

The norm gives us a sense of distance, measurement and length in a vector space, since ||x|| is just d(x,0); d is the metric or distance function.

Without the norm, we couldn’t call a vector space a metric space and along with it, the important sense of proximity or distance, essentially.

Vector Spaces, Embeddings & AI/LLM Search

In AI/LLM search, vectors are used to represent data – sub words, words, sentences, phrases, and so on.

Machines need numerical representations to understand data – translating data into vectors allows them to be digested, combined, rotated, scaled and so on within a vector space. These transformations are possible because of the set of rigid properties mentioned above, and allow these systems to find relationships between data points that would otherwise be difficult looking at raw data.

Vectors in the AI/LLM search spaces are high dimensional (very high dimensional, in fact), where each dimension can represent a different feature of that particular piece of data.

The process of mapping data into a high dimensional vector space is called embedding. (Embedding has a bit more strict mathematical definition from topology if you’re interested here.)

Creating an embedding that captures the most relevant, important bits of data – and the relationships between them – is one of the important goals in AI/LLM systems. Since this process compresses the raw, complex data into high dimensional representations, there will be some important bits lost in the process.

Embeddings can range from simple to complex — contextual embeddings being relevant to our AI/LLM search conversation.

Contextual embeddings are a learned, dynamic form of embeddings. That is, they change depending on the context of the data. When the data is words, an embedding can change depending on the context of where that word was used.

“Jaguar” in the sentence “I saw a Jaguar in the jungle” would have a different vector representation in the embedding than “I saw Bob driving his Jaguar to work”. Different implied meaning. Different embedding shape or location (loosely). Same word, different context.

Context.

Contextual embeddings allow for AI/LLM models to understand nuance between data points – words, images, content – whatever type of data that is being ingested, so it can understand its use in similar contextual situations (during a search/query/prompt session, for example) and retrieve and construct a relevant, contextually appropriate response.

Embeddings (vectors), collectively, give us a (loose) surface that is manifold-like. Metrics give us a sense of distance between points in the embeddings. Topology gives us the framework to allow that surface to bend, shift and twist, while preserving relationships between elements on those surfaces under different contextual situations.

Of course, these surfaces exist in a much higher dimension than we can imagine in our minds (which are limited to 3 dimensions), but thanks to our mathematical generalizations, we have a grounding for working with them.

Final Takeaway For SEOs and Marketers

Take a breath here if you’ve made it this far.

Formal mathematics (especially in abstract spaces) can be thick, admittedly, but what’s important here is that your words, sentences, graphics, images and other content are ingested through well-known and important mathematical vector space machinery described above.

Strictly speaking – as with any engineering and science – they don’t always follow the exact properties above (the pure math gets muddied and still under investigation), but in a perfect world, many of the properties can hold up.

From an SEO and marketing perspective, the final thoughts match the previous few posts: maintaining a high fidelity representation across different contextual situations. These spaces are more intricate and sensitive to noise — anything that cause your representation to stray (both on your site and off) from an intended position – or become harder to differentiate with other nearby representations – should be handled with care.

Quality over quantity. Clarity over complexity. Differentiation over blending in.

Keep it clean; keep those vectors looking sharp (and contextually relevant).

Continued notes on similarity metrics and the importance of inner product spaces in coming weeks.