Building a Language Model from Scratch
The original data pipeline, tokenizer, and first large transformer experiment.
Starting from the data
The first version of this project began with a simple question: could I build and train a useful decoder-only language model without treating the framework as a black box?
The earliest pipeline combined web text, reference material, books, code, and conversational sources. The hard part was not downloading text. It was filtering malformed documents, normalizing inconsistent encodings, preserving source identity, and making a streaming loader that would not exhaust local storage.
Tokenizer and model
I trained a 32,768-token byte-level BPE tokenizer and implemented the transformer in PyTorch. The first large experiment eventually crossed one billion parameters and used a 2,048-token context window.
The model used causal self-attention, residual feed-forward blocks, learned token representations, and a language-modeling head trained by next-token prediction. Several architectural details changed during the project; this article records the historical first model rather than the current configuration.
What the first version taught me
At small scale, almost any bug is cheap. At large scale, a wrong mask, stale checkpoint, corrupted shard, or mismatched tokenizer can waste days of rented GPU time. The durable result of this phase was therefore the pipeline: ingestion, tokenization, batching, checkpointing, and the habit of validating every boundary.