What Changed from Phase 3
We left the vision adapter behind for now. The 1.5B decoder was the bottleneck — not the image encoder, not the projector, not the dataset. The text-only portion of our training data was strong enough that we could establish a proper baseline on a much larger model before returning to multimodal. So we jumped from Qwen2.5-Coder-1.5B to Qwen2.5-Coder-14B-Instruct — nearly a 10x increase in decoder capacity.
The goal for this phase was simple: find the best LoRA configuration for the 14B model using supervised fine-tuning on our code dataset. No vision, no multimodal tricks. Just text-in, text-out code generation with chat-formatted instruction tuning.
The Sweep
We ran three LoRA configurations head-to-head, each training for a single epoch over the full 26,126 training rows with 3,265 held-out validation rows:
| Config | Rank (r) | Alpha | Learning Rate | Eval Loss |
|---|---|---|---|---|
| Conservative | 16 | 32 | 1.5e-4 | 0.5103 |
| Standard | 32 | 64 | 2e-4 | 0.4891 |
| Aggressive | 64 | 128 | 3e-4 | 0.4724 |
The aggressive configuration won cleanly. An eval loss of 0.4724 against the conservative config's 0.5103 is a meaningful gap — roughly 7.4% lower. The trend was monotonic: more LoRA capacity and a higher learning rate consistently helped. There was no sign of overfitting at this scale.
Training Details
Each sweep run trained for 1 epoch over the full dataset. The aggressive config — the one we kept — ran for approximately 7 hours on a single A100 GPU at a compute cost of roughly $22. We used the standard SFT recipe: chat-formatted prompts with system/user/assistant turns, cross-entropy loss on assistant tokens only, cosine schedule with linear warmup.
LoRA Target Modules
The adapter targeted all linear layers in the transformer: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. Targeting the full set rather than just the attention projections gave the adapter enough expressivity to reshape the model's behavior on code-specific tasks without touching the embedding or LM head.
The Forgetting Test
Catastrophic forgetting is the silent killer of fine-tuned models. You optimize for your target task and quietly destroy the model's general capabilities. We needed to check.
We ran GSM8K — a standard benchmark of grade-school math word problems — against both the base model and our fine-tuned adapter. The results surprised us:
| Model | GSM8K Accuracy |
|---|---|
| Qwen2.5-Coder-14B-Instruct (base) | 60.50% |
| Code-Trainer V6 (aggressive adapter) | 67.78% |
The adapter did not just avoid forgetting — it improved GSM8K accuracy by +12% relative (60.50% to 67.78%). Our best explanation is that the chat-format SFT taught cleaner answer formatting. The model was already capable of the math; the fine-tuning taught it to present answers in a more structured way that the GSM8K evaluation harness could parse more reliably.
What This Means
The sweep established V6 — the aggressive LoRA config — as our best adapter. It became the canonical conversion target for Phase 5, where we would merge the adapter weights and quantize to GGUF for local deployment. But first, we had one more question to answer.
What's Next
The sweep is done, but one question nags: would more epochs have helped? We ran a single pass over the full 26K-row dataset. The loss was still trending down at the end of training, which could mean we stopped too early — or it could mean the model was beginning to memorize rather than generalize. We are going to run the same aggressive config for 3 epochs on a subset of the data — same compute budget, different tradeoff — to see if the single-epoch result was leaving performance on the table.