The Question
After the sweep in Phase 4A, we had a strong result: eval loss of 0.4724 from a single epoch over 26K training rows. But the loss curve was still trending downward when training ended. That left a nagging question — were we leaving performance on the table by stopping at one epoch?
The standard advice in the fine-tuning world splits into two camps. One camp says more epochs help the model internalize patterns it only partially learned on the first pass. The other says that repeated exposure to the same examples causes memorization, not generalization, and you are better off spending compute on diverse data. We designed an experiment to test this directly.
What Changed from V6
Nothing in the architecture changed. Same base model (Qwen2.5-Coder-14B-Instruct), same LoRA configuration (r=64, alpha=128, lr=3e-4), same hyperparameters, same target modules. The only variable we changed was the data schedule:
| Run | Unique Rows | Epochs | Total Samples Seen | Eval Loss |
|---|---|---|---|---|
| V6 (1-epoch) | 26,126 | 1 | ~26K | 0.4724 |
| Full3 (3-epoch) | 8,042 | 3 | ~24K | 0.5126 |
The compute budgets were roughly matched: V6 saw approximately 26K total samples in one pass, while Full3 saw approximately 24K total samples across three passes over an 8K-row slice. Similar total training steps, similar wall time — but very different data diversity.
Training Details
The 3-epoch run trained on an 8,042-row random subset of the full training set, making three complete passes for a total of approximately 24,126 training samples seen. Runtime was 4 hours 53 minutes on a single A100 at a cost of roughly $15.60 — cheaper than V6 because the per-step overhead of loading 8K rows is slightly lower than 26K rows, even though total steps are comparable.
We used the same 3,265-row validation set as V6 for apples-to-apples comparison. Evaluation ran every 500 steps during training so we could watch the loss trajectory.
Results
The result was unambiguous. The 3-epoch run produced an eval loss of 0.5126 — measurably worse than V6's 0.4724. That is an 8.5% regression.
Loss Trajectory
The training loss continued to decrease across all three epochs — the model was still learning to fit the training data. But the validation loss told a different story. It improved sharply during epoch 1, plateaued early in epoch 2, and drifted upward through epoch 3. This is the textbook signature of overfitting: the model was memorizing the 8K training examples rather than extracting generalizable patterns.
The Takeaway
For this dataset and this base model, diversity of examples matters more than repetition. Seeing 26K unique code samples once taught the model more than seeing 8K samples three times, even though the total compute was nearly identical.
This is not a universal law — there are regimes where multiple epochs help, especially when the dataset is large enough that the model cannot absorb it all in a single pass. But at our scale (26K rows, 14B parameters, LoRA rank 64), one epoch was sufficient for the adapter to learn the distribution. Additional passes just pushed it toward memorization.
V6 Stands
This experiment confirmed that our Phase 4A result — the single-epoch aggressive LoRA adapter — is the best configuration we have tested. V6 remains the canonical adapter. We did not find a way to beat it by training longer on less data.
What's Next
The adapter is ready. The next step is straightforward: merge the winning V6 adapter weights into the base model and quantize to GGUF for local deployment. We want this model running on consumer hardware via llama.cpp and Ollama, not locked behind an API. The merge-and-quantize pipeline is well-trodden ground, but there are still decisions to make — quantization level, whether to use importance-matrix calibration, and how much quality we are willing to trade for smaller file size.