My GPT From Scratch
Status: PausedAfter four fine-tuning experiments in one evening, I set a stop rule: no more model experiments until there is an app to use the model in. So for now, I have paused the model work.
What it is
A GPT language model built from the ground up, because I want to understand how models like GPT actually work. It is written on bare PyTorch: a byte-pair tokenizer with special tokens for chat and tool calls, multi-head attention with RoPE, SwiGLU feed-forward layers, a KV cache for fast generation, and its own training loop.
What's done so far
My largest run so far trained an 88.9M-parameter model on about 1.78 billion tokens drawn from a 2.07-billion-token dataset: a Chinchilla-inspired budget of about 20 tokens per parameter, in about 6.7 hours on my RTX 4080 SUPER. Fine-tuning then produced an assistant version that chats and decides when to call a tool. In testing, with 48 generations in each category, it produced a tool call for 66.7% of tool requests and for none of the ordinary chat questions. That measures whether it chooses to call a tool, not whether it solves the task with it. Its known weakness for now: it almost never picks a tool name it did not see in training.
How I measure
For me, the project is as much about measuring honestly as about the model. KubLab, the experiment harness, changes one variable at a time, runs several seeds and reports the spread between them, and asks for the expected result to be written down before the run, with a warning when it is missing. Careful measuring also found the most expensive bug: 29 models that never called a tool were measuring a data encoding error, not a weak model. One end-of-text token sat in the wrong place. Removing it took tool calling from 0% to 77% in the next experiment.
What it taught me
This project is where I realised how far one PC is from the models I actually use: by my estimate, training a GPT-2-sized model on my PC would take around three years. That left two real options. I could pay a third party to train a model on data I give them, but I would only be spending money on a model that never reaches the best ones. Or I could build an assistant with a swappable brain, which runs the best model my PC can handle and changes it when better ones come out. That is why I started Kub.
What I've done so far
- Decided which experiments to run and which results to act on.
- Caught a statistical mistake before it cost GPU time: worked out that four more seeds could not reliably settle the question being asked.
- Found a data-leak hole: the text used to measure prose was also a training file for one of the datasets.
- Set the stop rule that moved the work from model experiments to building an app.
Tech Breakdown
Python · PyTorch · byte-pair tokenizer with regex pre-split and seven special tokens · RoPE · SwiGLU · KV cache · FineWeb-Edu plus a synthetic JSON, Python and tool-call corpus · supervised fine-tuning · KubLab experiment harness with seeds and pre-registration