Embeddings: how a computer knows a pony is closer to a horse than a truck

The problem: a computer does not know what a word means

Ask a person whether a pony is more like a dog or a truck, and the answer takes no effort. A computer sees the letters p-o-n-y and nothing else. Spelling does not help either: "cat" and "car" look almost the same, and they mean very different things.

The fix behind search-by-meaning, recommendations and chatbots is an embedding: a list of numbers that stands for a word, a sentence or an image. Things with similar meaning get similar numbers. After that, "is this like that?" becomes a question about numbers, and computers are good at numbers.

A word becomes a point on a map

Start with the smallest possible embedding: two numbers per word. Two numbers are a position on a flat map, like a street and an avenue. In the map below, the first number says how machine-like a thing is (0 is an animal, 10 is a vehicle or gadget) and the second says how big it is (0 is tiny, 10 is huge).

These two numbers were chosen by hand to keep the demo readable. Real embeddings are not written by hand, and their numbers do not come with labels like "size". We will come back to that.

Drag the marker next to the animals and the nearest words are animals. Drag it right and the vehicles take over. Nobody told the computer what a "pony" is; it only compared two numbers with the numbers of every other word. That comparison is the whole trick.

Finding the closest word is a few lines of code

The straight-line distance between two points comes from Pythagoras: the difference across, squared, plus the difference up, squared, then the square root. Here is the "pony" experiment from the map in Python:

import math

# Each word is two numbers: (how machine-like, how big). Chosen by hand for this demo.
words = {
    "cat": (1, 2), "dog": (1.5, 3), "mouse": (0.5, 1), "horse": (2, 7),
    "whale": (1, 9.5), "bicycle": (8, 3), "car": (9, 6), "truck": (9.5, 8),
    "phone": (9.5, 1),
}

def distance(a, b):
    return math.sqrt((a[0] - b[0]) ** 2 + (a[1] - b[1]) ** 2)

pony = (2, 5)   # a new word we have not seen
ranked = sorted(words, key=lambda w: distance(words[w], pony))
for w in ranked[:3]:
    print(w, round(distance(words[w], pony), 2))

Running it prints:

horse 2.0
dog 2.06
cat 3.16

Horse wins, with dog a close second. That feels right, and it matches what the map shows. This is the core of a "nearest neighbour" search, and it is how a search box can return results that do not contain your exact words.

Real embeddings have hundreds of numbers

Two numbers can only capture two ideas, and "size" and "machine-ness" miss almost everything about words. Real embedding models, which are neural networks trained on huge amounts of text, produce lists of hundreds or thousands of numbers per piece of text. Common sizes are in the range of a few hundred up to a few thousand, and the exact length depends on the model.

Nobody picks what each number means. During training the model adjusts them so that text used in similar ways ends up with similar lists. You cannot point at position 57 and say "that is the size". Meaning is spread across all the numbers together. You cannot draw a map with 768 directions either, but the maths of "how close are two points" works the same way in any number of dimensions.

Cosine similarity: comparing direction instead of distance

With long lists, people often compare the direction the numbers point in rather than the straight-line distance. This is called cosine similarity. It gives 1 when two lists point the same way, 0 when they are at right angles (unrelated), and -1 when they point in opposite directions.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    len_a = math.sqrt(sum(x * x for x in a))
    len_b = math.sqrt(sum(y * y for y in b))
    return dot / (len_a * len_b)

print(round(cosine([1, 2, 0], [2, 4, 0]), 2))   # same direction, different length
print(round(cosine([1, 0], [0, 1]), 2))         # at right angles
print(round(cosine([1, 0], [-1, 0]), 2))        # opposite directions

The output is:

1.0
0.0
-1.0

Notice the first line: [2, 4, 0] is twice as long as [1, 2, 0], yet the score is a perfect 1.0, because only the direction counts. Many embedding services return lists already scaled to length 1, and for those lists, ranking by distance and ranking by cosine similarity give the same order. Check the documentation of the model you use to see which comparison it recommends.

Where you meet embeddings

Search by meaning. The site turns every document into an embedding ahead of time. When you type a question, it turns your question into an embedding with the same model and returns the documents that are closest. A page about "fixing a flat tyre" can match "my bike wheel lost air".

Recommendations. Songs, films or products can be embedded too. If you liked one, show what sits nearby.

Chatbots that look things up. A common setup finds the passages closest to your question and hands them to a language model along with it, so the answer can use your own documents.

What to remember

An embedding turns meaning into a list of numbers, so that "similar" becomes "close". Closeness is measured with distance or with cosine similarity, and both are short pieces of arithmetic. A few cautions are worth knowing:

The numbers only make sense next to numbers from the same model; mixing lists from two different models gives nonsense. Embeddings also copy the habits of the text they were trained on, so "close" means "used in similar ways", not "true" or "fair". And a high score is a hint to check, not a proof.

Next time a search box finds what you meant rather than what you typed, a list of numbers and a distance calculation are probably what did it.