From RTL to Token: Walking Through the Complete LLM Inference Pipeline on an FPGA

Using a DE10-Nano FPGA development board—with no GPU and no frameworks—I built every layer of the AI inference stack myself as a learning-by-doing journey: from model quantization and the inference engine down to RTL hardware design, ultimately getting a BitNet 2B transformer model to generate tokens on the DE10-Nano. This talk is for anyone who wants to understand how an LLM works all the way from the underlying hardware to the final token output. Topics covered: why ternary-weight LLMs are a natural fit for FPGAs, implementing T-MAC lookup tables and balancing hardware resources, design trade-offs in the HPS/FPGA hybrid inference engine, and lessons learned from deploying an LLM on extremely resource-constrained hardware.