*What started with Mark Twain, 840,000 parameters, and a weekend experiment grew into 15.6 million words, 27.4 million parameters, and a much better understanding of what actually makes language models improve.*
Most people who want to understand how language models work start with someone else’s repository.
Andrej Karpathy’s nanoGPT and Tiny Shakespeare are common starting points for good reason. They work. The code is clean, the data is prepared, and you can get a small model training quickly.
I chose the inconvenient route.
Over one weekend, I started with raw Project Gutenberg files, built a small transformer in PyTorch, trained multiple versions on a Mac Studio, and then tried to teach the best models to answer questions.
The original goal wasn’t to build a useful AI assistant. A model with a few million parameters trained on Mark Twain was never going to compete with a modern frontier model.
I wanted to understand the mechanics at a scale where I could see what every decision actually did.
What happens when the model is too small? When does BPE tokenization help? Does more context fix a model that loses the thread? Does training longer work? What happens if I double the model size? Can instruction tuning turn fluent nonsense into something that actually answers a question?
By the end of the Twain experiments, I thought I had good answers to most of those questions.
Then I added a lot more data.
And several of my answers changed.
That became the most interesting part of the entire project.
Why I Started With Mark Twain
Tiny Shakespeare is convenient. The dataset is small, familiar, and already prepared.
That convenience was exactly why I avoided it.
I wanted the friction of real data work. I wanted to experience what happens before someone hands you a clean text file and a training script.
Mark Twain gave me public-domain material, recognizable American English, a distinctive voice, and enough writing to expose problems that a prepared toy dataset tends to hide.
I began with eleven verified works and later expanded the Twain corpus to twenty-three. That gave me 2,014,129 words and 11,212,568 characters of cleaned text.
Still tiny by modern language-model standards, but plenty for the experiment I had in mind.
At least, that’s what I thought at the time.
What "From Scratch" Actually Meant
I didn’t write my own tensor library or automatic differentiation system. I used PyTorch for tensors, gradients, optimization, and Apple Silicon support.
But I used no pretrained weights, no packaged transformer encoder, no prepared corpus, and no borrowed training script.
I built the data pipeline, character tokenizer, BPE tokenizer, causal attention, transformer blocks, training loop, validation process, checkpointing, generation tools, fine-tuning pipeline, and eventually an API for serving the model.
The first model had four transformer layers, four attention heads, 128 embedding dimensions, and 836,992 parameters.
I also wrote a direct test for the causal mask. A broken causal mask can produce a beautiful falling loss curve because the model learns to look ahead at the answer, then falls apart when you ask it to generate text.
My test changed a future token and confirmed that every earlier output position remained exactly unchanged.
The model had to predict the future, not peek at it.
The First Big Lesson: Data Is Not Setup
Writing the first transformer took less work than preparing trustworthy text.
Project Gutenberg files span multiple generations of formatting conventions. Transcriber notes appear in different places. Footnotes use different kinds of markup. Travel books introduce accented characters. Collections contain material published elsewhere.
Even the word “Gutenberg” caused trouble.
Twain refers to Johannes Gutenberg, the printer, in his actual prose. A naive cleaner that removed every occurrence of “Gutenberg” would have deleted legitimate text along with the boilerplate.
Then I discovered that five of my original eleven Gutenberg IDs were wrong.
The IDs looked perfectly plausible. Some pointed to different Twain works. One pointed to *The Gilded Age*, which Twain co-authored with Charles Dudley Warner and therefore didn’t fit my single-author corpus.
Plausible metadata is not verified metadata.
I checked every work directly against the source before locking it into the dataset.
Duplicate Text Can Quietly Ruin the Experiment
Duplicates were even more dangerous.
Story collections, essay collections, omnibus editions, and individual reprints frequently contain the same material. Adding that overlap doesn’t just waste space.
It can put the same passage into both training and validation data.
Then your validation loss looks better than it really is, which makes every decision built on that number suspect.
I measured overlap using twelve-word shingles after cleaning.
That last part mattered. When I measured the raw Gutenberg files, some short works appeared to overlap by as much as 40 percent. Almost all of it came from shared Project Gutenberg boilerplate.
After cleaning and deduplication, the final Twain corpus contained twenty-three works with almost no meaningful duplication.
That experience would become even more important when I eventually expanded from 23 books to more than 200.
The Experiments: v1 Through v12
Character and BPE models don’t produce the same number of tokens, so raw validation loss can’t compare them fairly.
I used bits per character as the common measure. Lower is better.
Here is where the project eventually landed:
| Run | Corpus | Tokenizer | Architecture | Parameters | Iterations | Bits/char | Train-val gap | Time |
|---|---|---|---|---|---|---|---|---|
| v1 | 11 Twain works | Character | 4L / 128d / 4 heads | 0.84M | 5,000 | 2.0478 | +0.024 | 4 min |
| v2 | 11 Twain works | Character | 6L / 256d / 8 heads | 4.83M | 8,000 | 1.7775 | +0.200 | 21 min |
| v3 | 11 Twain works | Character | 6L / 256d / 8 heads | 4.83M | 12,000 | 1.6825 | +0.117 | 35 min |
| v4 | 11 Twain works | BPE 2,048 | 6L / 256d / 8 heads | 5.33M | 10,000 | 1.6909 | +0.762 | 30 min |
| v5 | 23 Twain works | Character | 6L / 256d / 8 heads | 4.83M | 12,000 | 1.6430 | +0.058 | 35 min |
| v6 | 23 Twain works | BPE 2,048 | 6L / 256d / 8 heads | 5.33M | 12,000 | 1.5911 | +0.485 | 36 min |
| v7 | 23 Twain works | BPE 2,048 | 6L / 256d / 8 heads | 5.33M | 25,000 | 1.5857 | +0.627 | 75 min |
| v8 | 23 Twain works | BPE 2,048 | 6L / 256d / 8 heads, 512 ctx | 5.39M | 12,000 | 1.5892 | +0.475 | 51 min |
| v9 | 23 Twain works | BPE 2,048 | 6L / 384d / 12 heads | 11.53M | 12,000 | 1.5849 | +0.615 | 63 min |
| v10 | 211 works, 25 authors | BPE 4,096 | 6L / 384d / 12 heads | 12.32M | 15,000 | 1.4335 | +0.151 | 81 min |
| v11 | 211 works, 25 authors | BPE 4,096 | 6L / 384d / 12 heads | 12.32M | 25,000 | 1.4114 | +0.195 | 135 min |
| v12 | 211 works, 25 authors | BPE 4,096 | 8L / 512d / 16 heads | 27.45M | 15,000 | 1.3999 | +0.216 | 156 min |
That table tells a much better story than I expected when I started.
v1 Told Me the Model Was Too Small
The first model clearly underfit.
Training and validation loss followed each other almost perfectly. It wasn’t memorizing the corpus. It simply didn’t have enough capacity.
Its early output looked like this:
“ing the sarkessbly uld and santed lintionts thesto he singinaces”
Word-shaped noise.
Increasing the model from about 840,000 parameters to 4.83 million changed things quickly. v2 passed v1’s final validation result by iteration 1,000.
Then it started overfitting.
v2 Taught Me to Trust Validation, Not Training
v2’s validation loss bottomed at iteration 5,750.
Training continued for another 2,250 iterations. Training loss kept falling while validation got worse.
If I had simply saved the final checkpoint, I would have shipped a worse model while staring at what looked like a successful training curve.
Validation-gated checkpointing saved the better model automatically.
That lesson stuck:
Training loss tells you how well the model fits what it has seen. Validation loss tells you whether the model is actually improving.
Dropout Looked Worse Before It Looked Better
For v3, I kept the same 4.83-million-parameter architecture, added 0.15 dropout, and trained longer.
At first, v3 looked worse than v2.
By about iteration 6,000, the result flipped. v2 was memorizing while v3 kept improving.
The generated text became remarkably fluent for something so small:
“an obstinate blind watermelon of hot soul”
Perfectly plausible English.
Absolutely meaningless.
That distinction between fluency and understanding kept coming back throughout the project.
Then BPE Lost
Character-level tokenization means the model has to learn spelling one character at a time. A 256-character context also doesn’t cover much text.
BPE should fix both problems by combining common sequences into larger tokens.
So I trained a BPE tokenizer with a vocabulary of 2,048 tokens.
The first BPE model didn’t win.
On the smaller corpus, v3’s character model reached 1.6825 bits per character. v4’s BPE model reached 1.6909.
Basically a tie, with the supposed improvement slightly behind.
The reason became clear when I looked at the data. BPE compressed the text so aggressively that the model received far fewer unique training tokens. I had traded training signal for context, and the corpus wasn’t large enough to afford the trade.
So instead of immediately changing the architecture again, I added more Twain.
More Twain Flipped the Result
The corpus expanded to twenty-three works and just over 2 million words.
Then I repeated the comparison.
The character model reached 1.6430 bits per character.
The BPE model reached 1.5911.
Now BPE won clearly.
More data improved the character model by 0.0395 bits per character. It improved the BPE model by almost 0.100.
The same technology that appeared useless on the smaller corpus became the better choice once there was enough data to support it.
That was my first warning not to turn a local experiment into a universal rule.
“BPE doesn’t help” was wrong.
“BPE doesn’t help at this corpus size” was right.
There’s a big difference.
Then I Tried Every Obvious Architecture Fix
Once v6 became the strongest practical baseline, I tested the things that looked most likely to help.
I more than doubled training length. Almost nothing happened.
I doubled the context window from 256 to 512 tokens. Even less happened.
I more than doubled the model size from 5.3 million to 11.5 million parameters. Again, almost nothing.
The gains ranged from about 0.12 percent to 0.39 percent.
At that point the evidence looked pretty strong. The model wasn’t compute-limited. It wasn’t context-limited. It wasn’t obviously capacity-limited.
It was data-limited.
So I stopped tuning the same small corpus.
And this is where the project changed.
I Ran Out of Twain, So I Changed the Corpus
There wasn’t enough clean, non-duplicated Twain left to make the next jump meaningful.
So I widened the corpus to nineteenth-century American prose.
The dataset went from roughly 2 million words to 15.6 million words, a 7.8 times increase.
It now contained 211 works from 25 authors, including Herman Melville, Nathaniel Hawthorne, Stephen Crane, Ambrose Bierce, Bret Harte, William Dean Howells, Jack London, Edgar Allan Poe, Washington Irving, James Fenimore Cooper, Harriet Beecher Stowe, Louisa May Alcott, Kate Chopin, Sarah Orne Jewett, O. Henry, and others.
Twain remained the largest individual voice, but now represented only about 13 percent of the corpus.
I also increased the BPE vocabulary from 2,048 to 4,096 tokens.
This wasn’t simply “download more books.”
The data work got harder.
I used Gutenberg’s machine-readable catalog rather than relying on remembered IDs. I excluded co-authored works, poetry, omnibus editions, and other obvious duplication risks. I capped authors at ten works so one writer couldn’t overwhelm the corpus.
Deduplication caught another ten real overlaps, including *The Legend of Sleepy Hollow* appearing inside *The Sketch-Book* and *Bartleby* appearing inside *The Piazza Tales*.
The larger corpus also introduced Greek, Hebrew, accented characters, typographic ornaments, and other characters that hadn’t mattered in the Twain-only dataset.
The corpus had grown almost eightfold.
The annoying data problems grew with it.
Then Data Won Again, By a Lot
v10 used the larger corpus with a 12.32-million-parameter model.
It reached 1.4335 bits per character.
Compare that with v9 at 1.5849.
That’s a 9.6 percent improvement.
Not 0.3 percent.
Not 0.4 percent.
9.6 percent.
The jump from 2 million to 15.6 million words produced a bigger improvement than all the architecture experiments on the Twain corpus combined.
And something else changed.
Overfitting almost disappeared.
v9 had a train-validation gap of +0.615. v10’s fell to +0.151.
The larger dataset wasn’t just giving the model more things to memorize. It was forcing it to generalize.
The generated text changed too.
v9 tended to produce disconnected fragments:
“The Established Church was in the country.”
“Name it. This is not so; it is too cold for liberty to come, and honor the dreary line.”
v10 could hold one setting across an entire paragraph:
“The river was fainter and stronger than the canoes. But the fright of the pine forest was not a fisherman. Most of the trees in the sides of the forest were now on the banks of the stream…”
Still not meaningful in the way a modern language model is meaningful.
But river, canoes, forest, trees, banks, stream, and shore stayed inside one scene.
That was a qualitative change, not just a lower number.
Something Else Surprised Me: Longer Training Worked Again
Remember v7?
On the Twain corpus, increasing training from 12,000 to 25,000 iterations used roughly twice the compute and improved the result by only 0.34 percent.
I had concluded that longer training was spent.
On the 15.6-million-word corpus, I repeated the experiment.
v10 ran 15,000 iterations and reached 1.4335 bits per character.
v11 ran 25,000 and reached 1.4114.
That was a 1.54 percent improvement for 1.7 times the compute.
Still diminishing returns, but about 4.5 times the improvement that longer training produced on the small corpus.
The training technique hadn’t changed.
The data underneath it had.
Then Bigger Models Started Working Again Too
This was the experiment that changed how I interpreted almost everything before it.
On the Twain corpus, going from 5.3 million to 11.5 million parameters had improved the result by only 0.39 percent.
So “more capacity doesn’t help” looked like a reasonable conclusion.
For v12, I increased the model to:
8 transformer layers, 512 embedding dimensions, 16 attention heads, and 27,448,320 parameters.
I kept the run at 15,000 iterations so I could compare it directly with v10.
v10, at 12.32 million parameters, reached 1.4335 bits per character.
v12, at 27.45 million parameters, reached 1.3999.
That’s a 2.35 percent improvement from model capacity alone under the larger-data condition.
And at roughly comparable wall-clock time, v12 also beat v11, which had spent 10,000 extra iterations training the smaller model.
On the small corpus, more parameters had almost nothing useful to learn.
On the large corpus, they did.
That distinction turned out to matter a lot.
The Biggest Lesson Changed
Earlier in this project I would have written:
Data is the only lever that pays.
That’s no longer quite right.
The better conclusion is:
Data determines whether the other levers have anything useful to work with.
On 2 million words, longer training looked exhausted. More capacity looked exhausted. More context looked exhausted.
After increasing the corpus to 15.6 million words, longer training worked again and additional capacity worked even better.
The lever didn’t change.
The environment around the lever did.
That is probably the most important thing I learned from the entire project.
A failed experiment can be perfectly valid and still produce terrible general guidance if you ignore the conditions under which it failed.
Even the Instruction-Tuning Conclusion Changed
The original instruction-tuning experiments looked pretty grim.
I started with 106 hand-written instruction pairs, then increased that to 321. I masked the prompt tokens so the model was scored only on its response.
The models learned the shape of answering.
They produced shorter response-sized blocks. They learned when to stop. They picked up phrases that sounded like refusals or answers.
But they didn’t actually answer.
Then I doubled the base capacity from 5.4 million to 11.5 million parameters.
Still no meaningful answering.
Worse, fine-tuning increased repetition on those weaker models. That led me to another reasonable-looking conclusion: instruction tuning at this scale learned format at the expense of fluency.
Then I tried the exact same 321 instruction pairs on v12.
And that conclusion changed too.
| Fine-tune | Base | Base corpus | Parameters | Bits/char | Repeated 4-grams | Result |
|---|---|---|---|---|---|---|
| ft2 | v8 | 2.0M words | 5.4M | 1.5962 | 6.9% | Format learned, answers wrong |
| ft3 | v9 | 2.0M words | 11.5M | 1.5732 | 8.7% | More capacity, answers still wrong |
| ft4 | v12 | 15.6M words | 27.4M | 1.5076 | 2.9% | Refusal behavior transfers, answers still wrong |
On the stronger base, fine-tuning didn’t destroy fluency.
At the same sampling temperature, repeated four-grams dropped from 11.3 percent in v12’s free generation to 2.9 percent after fine-tuning.
It had learned something useful about stopping.
It also learned part of the refusal behavior.
Asked:
What is the capital of Japan?
It responded:
“I have no idea, and a body cannot be quite right in the matter of me…”
That’s obviously not a good answer.
But “I have no idea” appearing on a held-out question is real behavioral transfer. The model had learned to decline rather than confidently invent something.
Unfortunately, it still couldn’t answer.
Asked:
What is 23 times 19?
It produced:
“The town is full of people who are not afraid to be afraid to be afraid…”
So the main conclusion remains intact.
This model can learn response format, stopping behavior, and some refusal patterns.
It still can’t reliably map the meaning of a question to a relevant answer.
Fluency Still Isn't Understanding
v12 produces 100 percent recognizable English words in the measured sample. Its average sentence length is 17.3 words, remarkably close to the corpus average of 15.8.
It can produce correctly attributed dialogue like:
“What does it mean?” he asked.
“You’re only half-in-the-way looking me up in the face. You had better look at it.”
“It’s all right,” said the man, quietly.
Quoted speech. Speaker attribution. Paragraph structure. Adverbial tags.
And it still doesn’t know what it’s talking about.
That’s worth sitting with because modern AI interfaces make fluency incredibly persuasive.
A system can sound right long before it is right.
I Also Made It Serve Like a Real Model
Once v12 was stable, I wrapped it in an OpenAI-compatible FastAPI server.
The API supports the usual model, completion, and chat endpoints, so it can plug into OpenWebUI and other software expecting an OpenAI-style interface.
But I deliberately don’t pretend the base model is an assistant.
Its objective is next-token prediction. So the honest interface is text continuation.
Send it text and it continues the text.
On CPU, the exported v12 model runs at roughly 8 milliseconds per token, around 400 characters per second, with about 434 MB of resident memory. The model file itself is about 107 MB.
No GPU required for inference.
For something I built as a learning experiment on a Mac, that’s pretty satisfying.
What This Project Actually Taught Me
The biggest lesson wasn’t “more data is good.” That’s obvious.
The more useful lesson was that the value of almost every other decision depends on the state of the data underneath it.
BPE looked useless until the corpus grew.
Longer training looked exhausted until the corpus grew again.
More model capacity looked useless until there was enough information for that capacity to learn.
Even one of my conclusions about instruction tuning changed when I repeated the same experiment on a stronger base.
And through all of it, data quality remained non-negotiable.
Duplicated data can corrupt validation. Bad metadata can put the wrong books in a corpus. A seductive training curve can hide overfitting. A model that produces beautiful English can still have no understanding of what the words mean.
The experiments were small enough that I could see these effects directly.
That’s exactly why I wanted to build the thing from scratch.
Why This Matters Beyond a Tiny Language Model
Most businesses should not train a language model from scratch.
That’s not the point.
But the reasoning mistakes in this project look a lot like the reasoning mistakes that happen in production AI systems.
A chatbot gives weak answers, so someone buys a bigger model.
An agent loses the thread, so someone doubles the context window.
A fine-tuning run reports a lower loss, so the team assumes behavior improved.
The knowledge base contains duplicate, outdated, or contradictory information, but everyone keeps changing prompts.
And sometimes a team runs a valid test under one set of conditions and turns the result into a universal rule.
The model taught me to be much more suspicious of that last one.
Before declaring that a technology, architecture, model, or technique doesn’t work, make sure it wasn’t simply starved of the thing it needed to work.
In this project, that thing was usually data.
Full GitHub Repository and Early Access
The private GitHub repository now contains the complete training code, data-preparation pipeline, character and BPE tokenizers, checkpoints from the model experiments, fine-tuning infrastructure, evaluation tools, serving code, deployment scripts, and the experiment history behind the numbers in this article.
I’m still cleaning things up before making the repository public.
If you’d like early access to the full GitHub repository, reach out and I’ll share it with you.
Once the cleanup is finished, I plan to make the repository public so anyone can inspect the code, checkpoints, model weights, training history, and the experiments that worked as well as the ones that didn’t.
So, What Comes Next?
v12 is at 27.4 million parameters and 1.3999 bits per character.
That’s a long way from the 840,000-parameter character model that started the experiment.
But the pattern is still pointing in the same direction.
The larger corpus unlocked longer training. It unlocked more capacity. And even at 15.6 million words, the model still has signs that more useful data could support further improvement.
So the next question isn’t whether I can make the transformer wider again.
Of course I can.
The better question is whether giving it more parameters would teach me anything I haven’t already learned.
That’s the filter I’m using now.
What began as a weekend attempt to understand a tiny Mark Twain model turned into something more useful: a controlled demonstration of how easy it is to mistake a temporary bottleneck for a permanent limit.
And how often the answer isn’t a fancier model.
It’s better data.
