Yesterday we got as far as deep learning. Today we answered the question that actually matters: how does a pile of matrix multiplications understand a sentence?
1. Where Plain Deep Learning Fails
A standard network takes an input and produces an output, one shot. Feed it a sentence word by word and each word arrives in isolation.
The example: “Kids are playing. They are acting strange.”
What does “they” refer to? A network with no memory of the previous words cannot say. Translate that sentence into Telugu or Hindi word by word and the meaning falls apart.
Sequential data needs memory of what came before.
2. Recurrent Neural Networks
RNNs added a loop: each step receives both the current input and the previous state. That carried-forward state is the context.
This made real NLP possible:
- Named entity recognition — spotting that “Google” is an organisation
- Sentiment analysis — is this review positive or negative?
But RNNs are slow. They process strictly in order, one word at a time, doing a matrix multiplication at each step. Word 50 cannot be computed until word 49 is done. No parallelism, and modern hardware is built for parallelism.
3. Transformers and Attention
Transformers removed the loop. With no recurrence, every position can be computed at the same time — the model scans the text instead of reading it.
What made that possible is attention: a mechanism that computes how much every word matters to every other word, directly, regardless of distance.
The “bank” example:
“The kids are playing on the bank… the river has a decent flow.”
Attention links bank → river, not bank → money. The same mechanism resolves “they” back to “kids.”
The order of events matters: attention is what let the architecture drop recurrence; dropping recurrence is what delivered the parallelism.
4. Embeddings: The Numerical Dictionary
Models work on numbers, so every token maps to a vector — an array of floating-point numbers. Think of it as an Oxford Dictionary where the definitions are coordinates.
- A school graph has 2 or 3 dimensions. Embeddings have hundreds or thousands. OpenAI’s
text-embedding-3-smallproduces 1536 dimensions; the large model produces 3072. (An embedding model’s output size is a separate thing from a language model’s internal hidden size.) - Meaning lives in position. Words used similarly sit near each other.
- Relationships live in direction. The classic result is
king − man + woman ≈ queen: the offset between king and queen resembles the offset between husband and wife. It’s the direction of the difference that carries the relationship, not merely the gap. This is cleanest in older static embeddings such as word2vec and GloVe; modern contextual embeddings are messier, because a word’s vector changes with its sentence.
5. Tokens and Tokenization
- A token is a key into that dictionary.
- Tokenization breaks your input into those keys so vectors can be looked up.
- A token is not a word. One word can split into several. Rule of thumb: 75 words ≈ 100 tokens.
- Nothing is untokenizable. Type random characters and the tokenizer falls back to smaller pieces — down to individual characters and bytes.
Semantic search follows directly: search a vector database by distance in meaning rather than exact keyword match. That’s why it finds “automobile” when you searched “car.”
6. The Full Prompt Pipeline
Input → Tokenization → Embedding lookup → Attention → Neural network → Next tokenEverything downstream of tokenization is vectors being multiplied by matrices, at enormous scale.
7. Why GPUs, Precisely
A CPU has a few powerful cores. A GPU has thousands of small ones, which is exactly what matrix multiplication wants — thousands of simple multiply-and-add operations running at once.
Sizing a model to your hardware: memory is driven by parameters (the weights and biases).
- Rough guide: ~2 GB of GPU RAM per billion parameters at FP16, roughly half that at 8-bit quantisation.
- Add overhead for activations and the context cache.
- So a 0.6B parameter model fits comfortably in about 4–6 GB in practice.
The market: Nvidia leads, largely on the strength of the CUDA ecosystem rather than the silicon alone. AMD is growing with ROCm. Intel’s share in this workload is small. The knock-on effect reaches memory makers like Micron, because high-bandwidth memory is the other bottleneck.
8. Two Reminders
The application is not the model. ChatGPT, Claude and Gemini are applications. The model cannot search the internet — the application does the search and hands the results back. Worth repeating until it’s automatic.
Bias comes from data. If the training data leans a certain way, the output leans with it. Search Google Images for “AI” and you get blue glowing brains, because that’s what the internet is full of. Correcting it means work on the data and the weights, not a filter at the end.
⏭️ Next up: building on the Transformer toward how modern LLMs are actually trained and used.
