Language Model From Scratch

A language model I created from scratch including the BPE tokenizer, transformer architecture, fused kernels using Triton, distributed computation, optimizer sharding, experimentally derived scaling laws, data curation from common crawl, and alignment using RLHF with DPO.

June 17, 2024

There were five parts to creating a language model from scratch.

Here is the documentation of the 5 parts:

  1. [Architecture](/projects/lm-from-scratch/architecture.pdf)
  2. [Systems](/projects/lm-from-scratch/systems.pdf)
  3. [Scaling](/projects/lm-from-scratch/scaling.pdf)
  4. [Data](/projects/lm-from-scratch/data.pdf)
  5. [Alignment](/projects/lm-from-scratch/alignment.pdf)

And here are all of the code repositories for each part:

  1. [Architecture](https://github.com/matttreed/language-model)
  2. [Systems](https://github.com/matttreed/language-model-systems)
  3. [Scaling](https://github.com/matttreed/language-model-scaling)
  4. [Data](https://github.com/matttreed/language-model-data)
  5. [Alignment](https://github.com/matttreed/language-model-alignment)