Training and Serving the First Model
Datacenter training, streaming inference, and the lessons that forced a cleaner system.
Training on rented hardware
The first large run moved from local experiments to datacenter GPUs. That transition exposed performance bottlenecks in data loading, mixed-precision behavior, checkpoint writes, and recovery after interrupted jobs.
Throughput was not a single model property. It depended on sequence length, batch construction, storage latency, compilation state, and whether validation or checkpointing paused the main loop.
Serving generated tokens
The model was wrapped in a small Flask service that streamed generated text to a static browser interface. The first implementation proved the full path—from prompt to tokenizer, model, token stream, and web page—but it also mixed local paths, model loading, authentication, feedback storage, and HTTP routes in one file.
Why the serving code was rebuilt
The original service relied on Werkzeug reloader behavior to initialize the model, enabled unrestricted CORS, compared plaintext passwords, and persisted feedback through fragile JSON rewrites. Those choices were acceptable only as local scaffolding. The current repository separates model lifecycle, configuration, persistence, and routes so the public API can be deployed deliberately.
The real outcome
The large model generated recognizable language and validated the project end to end. More importantly, it showed where the experiment lacked reproducibility. That led directly to the smaller and more measurable system described next.