My Skill Lab
Machine LearningAdvanced

Transformers and Attention Explained

From attention heads to GPT — the architecture behind modern AI.

Taught by Rohan Maharjan · ML Engineer

What you'll learn

  • Explain why self-attention replaced recurrence and what problems it solves and creates
  • Implement byte-pair tokenisation and reason about vocabulary size, context length and cost
  • Derive and code scaled dot-product attention, causal masking and multi-head attention in PyTorch
  • Assemble a full transformer block with residuals, layer norm and feed-forward layers, and contrast encoder, decoder and encoder-decoder designs
  • Build, train and sample from a mini-GPT, using temperature, top-k and top-p decoding
  • Describe scaling laws, KV caching, LoRA fine-tuning and RLHF at the level needed to read current papers

About this course

Nearly every large language model in use today, from GPT to Claude to Llama, is built on the transformer, and this course takes the architecture apart piece by piece until you can build one yourself. You will learn why attention replaced recurrence, how byte-pair tokenisation turns text into ids, how embeddings and positional encodings including rotary embeddings give the model a sense of order, and exactly what scaled dot-product and multi-head attention compute. Then you will assemble residual connections, layer normalisation and feed-forward blocks into encoder and decoder stacks, implement a working mini-GPT in PyTorch, train it, sample from it with temperature and top-p, and finish with scaling laws, KV caching, fine-tuning with LoRA and an honest overview of RLHF. It is an advanced course for engineers and researchers comfortable with Python, PyTorch basics and linear algebra who want to understand LLM internals rather than just call an API.

Course content

8 lessons
  1. 1Why Attention Replaced RecurrenceThe limits of RNNs, the idea of attention as soft lookup, and the shape of the transformer that followed.Preview
  2. 2Tokenisation: Bytes, Subwords and VocabulariesHow byte-pair encoding builds a vocabulary, why tokens matter for cost and context, and how to train a tokeniser.
  3. 3Embeddings and Positional EncodingToken embeddings, sinusoidal and learned positions, and rotary position embeddings used by modern LLMs.
  4. 4Scaled Dot-Product and Multi-Head AttentionDerive the attention formula, add the causal mask, split into heads, and implement it efficiently in PyTorch.
  5. 5The Transformer Block and Model FamiliesResidual connections, layer normalisation, feed-forward layers, and the differences between encoder, decoder and encoder-decoder models.
  6. 6Building a Mini-GPT in PyTorchAssemble embeddings, blocks and a language-model head into a small GPT, train it on text and watch it learn to write.
  7. 7Sampling, KV Caching and Scaling LawsControl generation with temperature, top-k and top-p, make it fast with a KV cache, and understand how loss scales with compute.
  8. 8From Pretraining to Chat: Fine-Tuning, LoRA and RLHFHow a next-token predictor becomes an assistant: supervised fine-tuning, parameter-efficient LoRA, and RLHF or DPO alignment.