What Can 200 Lines of Python Train? A Deep Dive into MicroGPT

Have you ever wondered what it really takes to train a GPT model? Surprisingly, the core ideas can fit into just a couple hundred lines of Python.

In this talk, I'll walk through my notes on studying Andrej Karpathy's microGPT implementation. Designed to be as simple and educational as possible, microGPT implements the essential building blocks of modern language models—Transformers, Attention, Embeddings, and more—without the complexity of production-scale frameworks.

Rather than treating GPT as a black box, we'll build an intuitive understanding of how these components work together to train a language model.

By the end of the session, Attention may not feel nearly as mysterious anymore. You'll also understand what all that ever-growing GPU memory is actually being used for—and why modern language models need so much of it. 🙂