Embeddings

What are embeddings?

Embeddings are a way of turning words, sentences, or images into numbers so a computer can find meaning in them. Instead of plain text, every word or piece of data gets a list of numbers. Those numbers position words with similar meaning close together in the model's "number space".

Think of it as a map of meaning. Words that often appear in the same context end up near each other. That is how a computer learns that "cat" and "dog" are more alike than "cat" and "chair".

How does a computer learn embeddings?

There is no fixed formula that converts a word into an embedding. A model learns meaning by reading millions of sentences and predicting which word fits in a given spot.

For example, take the sentence "The cat sat on the ___". The model learns that "mat" is a likely fit.

At the start, every word gets random numbers. Each time the model gets it wrong, those numbers are nudged slightly. After enough repetition, it picks up the patterns. Step by step, the embeddings grow into a system that captures meaning based on how words are used.

From words to sentences

Older models like Word2Vec and GloVe gave each word one fixed embedding. That worked well, but ran into trouble with words that have several meanings. "Bank" could be a riverbank, a place to sit, or a financial institution.

Newer models like BERT and GPT handle this more smartly. They produce contextual embeddings: the same word gets different numbers depending on the sentence around it. The computer then has a much better grasp of what is meant.

Take two sentences like "The cat sat on the mat." and "On the mat sat a cat.". They end up with almost the same meaning, even though the word order is different. That is because the model learns which words matter to which, and combines them into a sentence-level embedding.

How does it work technically?

The model uses a neural network built from layers. The first layer turns words into raw numbers (the embeddings). The next layers look at how words relate to each other and try to predict which word should come next. Whenever the model is wrong, it gets nudged a little. This adjustment process is called backpropagation.

After millions of examples, the model knows that:

• "cat" and "dog" often appear in similar contexts

• "chair" rarely appears in the same context

Each word ends up at a specific spot in a high-dimensional space (often 384 or 768 dimensions). Words with similar meaning sit close, unrelated words sit far apart.

Where are embeddings used?

Embeddings sit at the heart of many AI applications:

• Search: find documents with similar meaning, even when you use different words.

• Chatbots: understand what someone means, not just the literal words they typed.

• Recommendations: surface content similar in substance to what someone liked before.

• Image recognition: match images or objects that resemble each other.

It all comes down to measuring similarity between embeddings. The standard way is cosine similarity: a calculation that checks how close two vectors point in the same direction. The closer they are, the stronger the match in meaning.

Without embeddings, a language model would just see a string of disconnected words. With them, it can connect ideas, follow context, and produce sensible answers.