V6 was a specialist. A very good specialist — at exactly one thing. We trained it on 26,000 single-turn code-transcription examples, every single one following the same pattern: "extract the code from this screenshot." The model got brilliant at that task and completely forgot how to do anything else.
The Distributional Collapse
When 100% of your training data is one task type, the model's output distribution collapses around that task. V6 had three capability gaps that made it unusable for agentic work:
- No tool calling — couldn't emit
tool_callJSON structures at all - No multi-turn reasoning — couldn't maintain context across assistant/user exchanges
- No complex instruction following — couldn't parse system prompts with tool definitions and act on them
The Mixed Dataset
V7 corrects the collapse by blending three capability slices into a single training set:
| Slice | Source | Rows | Purpose |
|---|---|---|---|
| A | code-trainer-offsec-dataset (8K subsample) |
7,191 | Preserve V6 code-gen quality |
| B | NousResearch/hermes-function-calling-v1 |
10,381 | Restore tool_call emission |
| C | greghavens/fable-5-coding-and-debugging-traces |
11,290 | Multi-step agent behavior |
Total: 28,862 training rows, 3,206 validation rows. Notably, 63.8% of training rows contain tool definitions — we over-indexed on tool exposure to counteract V6's complete tool-call blindness.
Training Configuration Changes
Everything about the training setup changed from V6. Not because the old settings were wrong for code generation — they were wrong for preserving the base model's existing capabilities while adding new ones.
| Parameter | V6 | V7 | Rationale |
|---|---|---|---|
lora_r |
64 | 32 | Reduce capability overwrite of base model |
| Learning rate | 3e-4 | 1.5e-4 | Gentler gradient updates to preserve existing circuits |
| Sequence length | 2,048 | 8,192 | Agent prompts with tool specs don't fit in 2K tokens |
The LoRA rank dropped from 64 to 32 — fewer trainable parameters means less capacity to overwrite what Qwen2.5 already knows. The learning rate halved for the same reason: be gentler. And the sequence length quadrupled because agent prompts with tool specifications are long. A system prompt with five tool definitions, their parameter schemas, and usage instructions simply doesn't fit in 2,048 tokens. At 8,192 tokens, the model can actually see the full prompt during training.
What Broke: Two Known Issues
V7 is superseded by V8. It shipped with two problems that became dealbreakers in production:
1. Multilingual Hallucination
After generating tool-call JSON, the model would sometimes emit garbage tokens in Thai, Russian, or Chinese at sequence boundaries. The root cause is Qwen2.5's massive multilingual vocabulary — when the model hits low confidence at the end of a structured output (right after closing a JSON brace), it has too many token options across too many languages. Instead of stopping cleanly, it samples from the wrong part of the vocabulary.
tool_call JSON block followed by สวัสดี or Привет before the next coherent English token. The model is essentially sampling randomly from a 150K-token multilingual vocabulary at its lowest-confidence generation points.
2. Tool-Call Format Mismatch
The Hermes function-calling dataset uses <tool_call> XML-style tags as regular text tokens. But Qwen2.5 has its own native tool-call template with different special tokens. The model learned to emit the correct JSON payload — it understood the function name, the arguments, the structure — but it wrapped them in Hermes-format tags instead of the token sequence that Ollama and other Qwen2.5-native runtimes actually expect.
The model learned the semantics of tool calling but not the syntax its runtime requires. A training data problem, not a model capability problem.
What's Next
Two fixes for V8. First, filter the training data to English-only to eliminate the multilingual hallucination at low-confidence boundaries. Second, switch from Hermes-style <tool_call> tags to Qwen2.5's native tool-call template so the emitted token sequence matches what Ollama expects. Both are data-side changes — the model architecture and training loop stay the same.