Everything Is A Math Wrapper, Part IV: Inner Products And The Geometry Of Similarity For SEOs & AI/LLM Search Spaces

The past few weeks I’ve covered quite a few areas of mathematics and how they connect with and relate to AI/LLM search spaces, so I want to quickly get folks up to speed if you’ve missed those previous entries. Here’s a quick rundown.

Topology and topological spaces give us intuition about relationships between items in a space and how they’re preserved through deformation (stretching, bending, et. al.) in that space.

Metric spaces, a subset of topological spaces, give us the intuition of distance and proximity within those spaces.

Vector spaces (together with a norm), are a subset of metric spaces – give us intuition of the idea of length (norm) and additional linear properties like addition and scaling of items in those spaces.

Summarizing this, we’re moving from more general spaces (topological spaces) down to more fine-grained, specialized spaces (vector spaces) that contain increasing levels of structure that we can leverage when working with items in these spaces.

You might be wondering why understanding the higher level, generalized spaces are important, since additional structure essentially creates boundaries where properties of those general spaces aren’t necessarily required.

In many engineering and scientific applications of these spaces, the precise properties of the specialized spaces don’t always hold — in many ways we don’t always see pure vector spaces or metric spaces in the strictest sense.

In the math world, finding even one exception in these spaces that violates those properties means the additional properties won’t hold and we can no longer label those as specialized spaces accordingly.

The pure math gets muddied in the real world, essentially.

Certain areas of these spaces fit nicely and behave like traditional vector spaces – some other areas of those same spaces, perhaps not.

Whenever those exceptions are found, we have to rely on more generalized notions and properties to properly understand them and work with them (or perhaps remodel/reshape them to behave more closely to the properties of specialized spaces) — thus the importance of building intuition from those more general spaces.

Ideally, we’d like to work with spaces that have more rigid structure and properties like metric and vector spaces — they’re more familiar and allow us to better predict and model things inside those spaces.

From metrics we get distance. From vector spaces we get ideas of length, addition and scaling. All useful tools we can leverage to understand the items in these spaces better and build further geometric intuition.

Continuing the development of geometric intuition, the inner product gives us one of the most powerful tools used in modern search, AI and LLM spaces.

“Inner product” is likely a vague term for many, so let’s give it some additional structure to help you understand how important it is (see what I’m doing here?).

An Informal View Of Inner Products And The Dot Product

Looking at the two words that make up the term “inner product”, you’re probably more familiar.

“Inner” giving the sense of “inside” or “within” and “product” being related to the idea of the familiar multiplication operation.

Together, the “inner product” then gives us a sense that we are multiplying items that stay within the same space, essentially.

When those items are vectors, they are within the familiar vector spaces mentioned above.

If you’ve taken linear algebra, you’ve likely seen this as the “dot product” – taking two vectors of the same dimension, multiplying the elements pairwise and adding them together to yield a single, scalar number.

What you end up with is a measure of how similar those two vectors are or how much they overlap, essentially — the larger the dot product, the more they overlap. The smaller the dot product the less they overlap. If the dot product is zero, those two vectors are perpendicular (orthogonal).

Dot products are used in our more familiar Euclidean vector spaces we learn about growing up (like the spaces we can observe/imagine having and x, y and z – 3 dimensional coordinate system).

Looking closer, the dot product allows us to get a sense of magnitude (length) of a vector and – perhaps more importantly in our world – the angle between two vectors. We can get a sense of how large vectors are and how much they’re “pointed in the same direction”.

When vector spaces aren’t so familiar – like the ones they often see in AI/LLM spaces – we have to generalize the dot product to the inner product.

The inner product can give us the same geometric intuition and structure as the dot product in more complicated vector spaces, essentially.

(You might have puzzled out here that along with the “inner product” there is an “outer product”; something entirely different, but as important which we’ll touch on in future posts.)

When vector spaces are equipped with an inner product, it becomes an inner product space.

The inner product also induces a vector space with a norm (defined as the square root of the inner product of two of the same elements of the space) — making it a metric space, and thus a topological space.

A (Semi) Formal View Of Inner Products

Inner products are a binary operation that yields a positive scalar. That is, it takes two elements of a space and produces a single number.

The inner product can be used for complex vector spaces, but for this post we’ll stick with real vector spaces.

Notation can vary depending on source, but the inner product is usually depicted using the pointed brackets ⟨ , ⟩ – ⟨ x , y ⟩ , with x and y being vectors.

If the inner product on a vector spaces has the following properties, then it is called an inner product space:

⟨ x , y ⟩ = ⟨ y , x ⟩ (Symmetric)

⟨ x + z, y ⟩ = ⟨ x, y ⟩ + ⟨ z, y ⟩ (Linearity)

For some scalar a, ⟨ ax , y ⟩ = a ⟨ x , y ⟩ (Homogeneity)

⟨ x , x ⟩ ≥ 0 and is zero if and only if, x is the zero vector.

Relating back to the norm ||·||, when an inner product is equipped on a vector space, we can define it in terms of the inner product:

||x|| = √ ⟨ x , x ⟩ (the square root of the inner product of a vector with itself)

We would call this the magnitude of the vector x.

When ⟨ x , y ⟩ = 0, we can say that the vectors x and y are orthogonal – that is, they don’t overlap.

Cosine Similarity In Inner Product Spaces

If you take the inner product of two vectors and divide them by the product of their magnitudes, we get something called “cosine similarity”.

I won’t go into trigonometry here and explain cosine, but formally this gives us:

⟨ x , y ⟩ = ||x||||y|| cos (θ), where θ is the angle between the vectors x and y.

What cosine does here is give us a range or interval value for the similarity between two vectors between -1 (opposite similarity) and 1 (exactly similar), where 0 means the two vectors are orthogonal.

For opposite similarity, the angle between two vectors is 180° ( cos (180°) = -1 ), for orthogonal similarity, the angle is 90° ( cos (90°) = 0 ) and for exact similarity the angle is 0 ( cos (0°) = 1 ).

The important piece here is that it doesn’t matter how large two vectors are (large or small magnitudes), the angle between them can tell you how similar or not similar they are (if they’re pointed in the same direction, essentially).

Inner Products and Cosine Similarity In AI/LLM Spaces

Remember that in these spaces, your words, sentences and other content are digested as vectors (loosely speaking).

If you take two of those items — let’s say two sentences – as vectors, you can use cosine similarity to determine how similar those two sentences in the vector space.

As always, there is more nuance and more intricate ways similarity can be determined (other types of measurements), but from a high level view the inner product can give you some nice mathematical intuition of how some of the AI/LLM machinery can leverage our well-known inner product measurements – along with the geometric properties it endows on these spaces.

Final Takeaway For SEOs and Marketers

If you’ve been following along in this series, you can probably guess what the takeaway here is: maintaining a high fidelity representation for your words, sentences, passages, brands, website and other entities across different contextual situations.

Anything that cause your representation to stray (both on your site and off) from an intended position – or in relevant contexts – in space could be indicated by some similarity measure, like the inner product above.

While you want to be distinct (through differentiation) from others pointed in the same direction (along a topic or vertical, for example), straying too far away along a topical (or other important category in a space) direction could pull you away from being relevant for that topic.

As you stray, a hypothetical “topical” similarity measure could get smaller and smaller between you and an intended topic vector – the angle between your site and a particular topic vector grows larger.  Over time as that measure gets smaller and smaller, you can become irrelevant to the topic and fall out of a candidate pool in that category or topic essentially.

Something like the inner product can also be used as a measure between you and your competitors — if you copy a competitor too closely (along different dimensions) and there’s no distinction or differentiation, then your vectors will blend in with theirs; something that’s not ideal in most cases.

Clarity, quality, strategic differentiation —  and fidelity.   

Even more on these spaces and other notes here in the coming weeks.