An LLM is like a very good next-word guesser trained on lots of text. Given some words, it predicts what comes next. With enough data and the Transformer design, it learns grammar, facts, and reasoning patterns.

A deep neural network trained on huge text corpora to predict the next token and generate human‑like text.

Think of attention as spotlighting the important earlier words for each current word. The model stacks many of these spotlights with small neural networks in between.

GPT‑style models use a decoder‑only transformer: masked self‑attention + small feed‑forward networks stacked for autoregressive generation.

The Data Processing Pipeline

Before an LLM can learn, raw text must be converted into a numerical format. This involves several steps: tokenization, converting tokens to token IDs, and then creating embedding vectors.

Tokenization

LLMs can't process raw text. First, we need to break the text down into smaller units called tokens. This can be done at the word level, character level, or sub-word level. Byte-Pair Encoding (BPE) is a common sub-word tokenization algorithm used by models like GPT-2, which can handle any word by breaking unknown words into smaller known pieces.

Embeddings turn each token into a list of numbers that capture its meaning; positional encodings add a gentle pattern so the model knows the order of tokens.

After tokenization, we map each token ID to a high-dimensional vector called an embedding. These vectors capture the semantic meaning of the tokens. However, the basic self-attention mechanism is position-agnostic. To solve this, we add positional embeddings to the token embeddings, giving the model a sense of word order.

Self-Attention: The Core Idea

Self-attention allows the model to weigh the importance of different words in a sequence when processing a particular word. For each token, it creates three vectors: a **Query**, a **Key**, and a **Value**. The attention score between two tokens is calculated by taking the dot product of the Query vector of the current token and the Key vector of the other token. These scores are then normalized (using softmax) to create attention weights, which are used to create a weighted sum of the Value vectors.

Causal = no peeking ahead. Multi-head = several small spotlights look for different patterns at the same time; their results are combined.

Causal Attention: For generative models like GPT, we need to prevent a token from "seeing" future tokens. This is done by masking the attention scores for subsequent positions, ensuring the prediction for a word only depends on the words that came before it.

Multi-Head Attention: Instead of performing attention once, we do it multiple times in parallel with different weight matrices (multiple "heads"). Each head can learn different types of relationships between words. The results from all heads are then combined.

We show the model lots of text with one word blanked out and ask it to guess the missing word. Doing this billions of times teaches general language understanding.

The model is trained on a simple yet powerful task: predicting the next word in a sentence. Given a sequence of words, the model calculates a probability distribution over its entire vocabulary for what the next word should be. The difference between its prediction and the actual next word (the loss) is used to update its weights. By doing this on a massive dataset, the model learns grammar, facts, and reasoning abilities.

Decoding is how we pick the next word: temperature = more/less randomness; top-k = pick from top K choices; top-p = pick from the smallest set that covers p% of probability mass.

When generating text, simply picking the most likely next word every time (greedy decoding) can be repetitive. We use decoding strategies to introduce controlled randomness.

• Temperature Scaling: Controls the "peakedness" of the probability distribution. Higher temperature means more randomness, lower temperature means more deterministic.
• Top-k Sampling: Limits the word choice to the 'k' most likely next words, preventing the model from picking very unlikely words.

Start with a general model, add a small new top layer that predicts your labels (like spam/not-spam), and train on a small labeled dataset.

A pretrained model is a generalist. To make it a specialist, we perform fine-tuning. For a task like spam classification, we replace the model's final layer (which outputs logits for the entire vocabulary) with a new "classification head" that outputs logits for our specific classes (e.g., two outputs for "spam" and "not spam"). Then, we continue training on a smaller, labeled dataset. Often, we only train the new head and a few of the top layers, keeping the base of the model "frozen" to save computational resources.

Show the model examples of instructions and the ideal responses. It learns to respond helpfully instead of just continuing text.

To create a personal assistant or chatbot, we perform "instruction fine-tuning." We fine-tune the model on a dataset of instructions and desired responses, often formatted in a specific prompt style (like the Alpaca format). This teaches the model to follow commands and have a conversation, rather than just completing text.

Quick-Start: Minimal GPT (pseudocode)

// Shapes: batch B, sequence T, embedding D, heads H
// Token + positional embeddings
E = Embedding(vocab, D)
P = PositionalEmbedding(T, D)

for t in range(T):
  x = E[token_ids[:, :t+1]] + P[:t+1]
  // Multi-head self-attention (causal)
  Q = x Wq, K = x Wk, V = x Wv
  att = softmax((Q K^T)/sqrt(d_k) + causal_mask)
  x  = concat(att V across heads) Wo
  // MLP block
  x = x + GELU(x W1 + b1) W2 + b2
  // LayerNorm + residuals around each sublayer
  logits = LayerNorm(x) Wout

loss = cross_entropy(logits[:, :-1], token_ids[:, 1:])
update(parameters)  // AdamW, cosine LR, weight decay

This sketch mirrors the blocks described throughout this guide and in the referenced book. Use it as a mental model while reading.

References & Attribution

• Inspired by the excellent book Build a Large Language Model (From Scratch) by Sebastian Raschka.
• Figures replaced with native SVG diagrams for reliable, fast loading.
• Transformer architecture and GPT block semantics follow the canonical literature: “Attention Is All You Need.”