Deep learning and LLMs: learn it by building it

For classical foundations, see the ML foundations post. This is the modern half — and here the best pedagogy is build-first.

Build it from scratch

• Neural Networks: Zero to Hero — Andrej Karpathy. The best deep-learning pedagogy in existence: backprop from raw scalars (micrograd), through language models, to building GPT from scratch and reproducing GPT-2. Type every line yourself; the typing is the learning.
• nanoGPT — the ~600-line training codebase that demystifies the whole stack. Reading it end to end kills the illusion that LLM training is exotic.
• Dive into Deep Learning — free, runnable-notebook textbook; the best bridge between Karpathy’s intuition and Bishop’s formality.

The papers actually worth reading

The field produces thousands of papers a year; these are the ones whose ideas you’ll use weekly:

• Attention Is All You Need — read after building a transformer, when every design choice suddenly makes sense.
• Scaling laws (Kaplan et al.) and Chinchilla — why model/data/compute ratios dominate architecture cleverness.
• InstructGPT, DPO, and GRPO (DeepSeekMath) — the post-training lineage. I implemented DPO and GRPO from scratch for the Toolcall DPO project, and the papers only fully clicked at that point — the loss functions are two dozen lines each, and every subtlety lives in the data and the reference model.
• FlashAttention and the vLLM/PagedAttention paper — where systems thinking meets attention; PagedAttention is an OS idea, which is why I built PageServe.
• Lilian Weng’s blog — the survey layer: her posts on attention, RLHF, and agents are the best-maintained maps of each subfield.

The engineering layer

• AI Engineering and Designing Machine Learning Systems — Chip Huyen. The production reality: evals, data flywheels, serving, drift.
• Hugging Face courses — the practical toolchain (transformers, PEFT, TRL) with runnable notebooks.

The loop that works

Watch one Karpathy video → rebuild it without the video → read the corresponding paper → then attach a measurement harness and make a claim you can defend with numbers. That last step is the difference between following the field and doing the work — every ML project on this site is that step, repeated.