TL;DR
- What started as a weekend experiment training a tiny language model on Mark Twain eventually grew from 840,000 parameters to 27.4 million, and from roughly 2 million words to 15.6 million words of nineteenth-century American prose.
- On the smaller Twain corpus, longer training, more context, and a wider model each improved performance by less than 0.4%. They looked exhausted.
- Expanding the corpus to 211 works from 25 authors improved the model by 9.6%, more than all those architecture changes combined.
- Then something more interesting happened: with enough data, longer training and a bigger model started working again. The techniques hadn’t failed. The original dataset couldn’t support them.
- Even the best model could produce convincing prose and learn some instruction format and refusal behavior, but it still couldn’t reliably answer questions. Fluency and understanding are not the same thing.
1. It Started With Mark Twain
Most tutorials for building a tiny language model hand you a clean dataset and working code.
That skips a lot of the interesting part.
The experiment behind this week’s article started with raw Project Gutenberg files and a transformer built from the basic PyTorch components. No pretrained weights. No packaged transformer. No prepared Tiny Shakespeare dataset.
The first model had just 836,992 parameters.
The corpus eventually grew to 23 Mark Twain works and roughly 2 million words.
That was enough to make the model surprisingly fluent.
It was also enough to start making some very convincing wrong conclusions.
2. The Obvious Fixes Barely Worked
Once the Twain model reached a reasonable baseline, the next steps seemed obvious.
Train it longer.
Give it more context.
Make it wider.
All three were tested independently.
The results?
Longer training: 0.34% improvement
Double the context: 0.12% improvement
More than double the parameters: 0.39% improvement
More compute. More model. Almost no gain.
At that point, the evidence seemed pretty clear: architecture wasn’t the problem.
The model needed more data.
3. Then the Corpus Grew 7.8×
There wasn’t enough non-duplicated Twain left, so the project expanded to nineteenth-century American prose.
The new corpus contained:
15.6 million words. 211 works. 25 authors.
Melville, Hawthorne, Poe, London, Alcott, Chopin, O. Henry, Stowe, Irving, and many others joined Twain.
The model moved from 1.5849 to 1.4335 bits per character.
That’s a 9.6% improvement from changing the data.
Compare that with the 0.12% to 0.39% gains from the previous architecture experiments.
But the bigger surprise came next.
4. The "Failed" Techniques Started Working Again
On the Twain corpus, longer training was basically useless.
On the larger corpus, increasing training from 15,000 to 25,000 iterations improved the result by 1.54%.
Then the model grew from 12.3 million to 27.4 million parameters.
On the small corpus, roughly doubling model capacity had bought only 0.39%.
On the large corpus, the bigger model improved performance by 2.35% at the same number of training iterations.
Same basic techniques.
Very different results.
The data had changed what the model could do with them.
That’s the part worth remembering.
A lever that appears useless may not actually be useless. It may simply be starved of the thing it needs to work.
For this experiment, that thing was data.
The full article includes all twelve training runs, the BPE versus character-tokenization experiments, the data-cleaning problems, generated samples, instruction-tuning results, and what changed as the corpus and model grew.
5.Fluent Still Doesn't Mean Intelligent
The final v12 model produces remarkably convincing nineteenth-century-style prose.
It uses real English words. It formats dialogue correctly. It keeps scenes together far better than the early models.
Then ask it:
“What is 23 times 19?”
And you can get:
“The town is full of people who are not afraid to be afraid to be afraid…”
Instruction tuning taught it the shape of answering. It even learned some refusal behavior.
It didn’t learn to understand the question well enough to answer it.
That’s a useful reminder far beyond this little experiment.
Today’s AI systems are vastly more capable, but polished output still has a way of making us assign more intelligence to a system than we’ve actually measured.
Evaluate the behavior you need.
Don’t let fluency substitute for proof.
Final Thought
The biggest lesson from this project wasn’t “bigger models are better.”
It wasn’t even “more data is better.”
It was more specific:
The value of one AI lever depends on the condition of everything underneath it.
More context didn’t help when the model couldn’t use the context it already had.
More parameters didn’t help when the training data couldn’t support them.
More training didn’t help when the model was already memorizing.
Then the data changed, and some of those same techniques became useful again.
That’s a useful way to think about business AI too.
When an AI system isn’t delivering, don’t automatically reach for a bigger model, another tool, or more compute.
First find the actual constraint.
Quite often, that’s where the signal is.
Thanks for reading Signal Over Noise,
where we separate real business signal from AI noise.
where we separate real business signal from AI noise.
See you next Tuesday,
Avi Kumar
Founder: Kuware.com
Subscribe Link: https://kuware.com/newsletter/