Smaller Model, Better Experiment

The first model was not the final model

The first major version crossed one billion parameters and trained on tens of billions of tokens. It taught me how to stream data, rent serious GPU hardware, resume long jobs, and serve generated tokens. It also taught me that completing a large run is not the same as conducting a good experiment.

Make every run understandable and reproducible before making the model larger.

LMSteinshark Small

The current model is a decoder-only transformer of roughly 482 million parameters: 30 layers, 1,280-dimensional embeddings, 20 attention heads, a 4,224-dimensional SwiGLU feed-forward path, a 32,000-token BPE vocabulary, RMS normalization, rotary position embeddings, and tied input/output embeddings.

The dataset

The present mixture includes Common Crawl WET documents, Wikipedia, Project Gutenberg, YouTube automatic transcripts, academic material, and code. Each source has its own ledger and sampling weight. The loader validates the physical files rather than assuming that a catalog entry exists and is complete.

Training state is a first-class object

A checkpoint now carries model weights, optimizer state, cumulative token and update counts, loss history, timing, throughput, and training-run metadata. Resuming should mean continuing the same experiment, not loading weights into an otherwise fresh run.

Numerical and performance choices

Training uses BF16 computation where appropriate while preserving deliberate control over parameter and optimizer precision. The current H100 training setup reaches roughly 146,000 tokens per second with batch 32 at 512 tokens, though throughput remains dependent on the exact data and validation configuration.

What comes next

Pretraining is the foundation. The next stages are supervised instruction tuning, held-out evaluation, retrieval over local documents, web and reference lookup, restricted Python execution, and tool-selection training. The purpose is not to disguise a small model as a frontier system. It is to learn how much useful capability can be assembled when the base model and the surrounding system are both understood.