What Changed from V8
V8 proved native formatting and English filtering work. V9 attacks the remaining tag emission problem from every angle we could think of. On the data side: more tool-calling examples (12K to 19K), synthetic multi-tool conversations, strict tag validation that drops any record with unmatched tags, and stripping trailing text after </tool_call> to teach a clean stop signal. On the training side: a two-phase curriculum where 80% of training uses the full mixed dataset and the final 20% is a concentrated "polish" pass on tool-calling-only examples at double the learning rate. On the deployment side: Q5_K_M replaces Q4_K_M as the default quantization because the extra 1.5 GB preserves multi-token tag patterns that Q4 was rounding away.
Six specific changes define this version:
1. Increased Tool-Calling Density
Slice B expanded from 12K to 19K rows, pushing the tool-calling share from 31% to roughly 45% of the dataset. The model now sees nearly one tool-calling example for every non-tool example. More exposure means stronger learned patterns for the full tag-JSON-tag sequence.
2. Synthetic Multi-Tool-Call Examples
We generated approximately 2,000 synthetic examples where each assistant turn contains 2–3 sequential tool calls. These teach the model that tool calling is not always one-and-done — sometimes you need to chain get_weather into format_report into send_email in a single response.
3. Stop-After-Tag Cleanup
Every assistant turn containing </tool_call> was stripped of any trailing text. In V8, some training examples had conversational filler after the closing tag, which taught the model to keep generating after it should have stopped. Clean stops produce clean outputs.
4. Tag Completeness Validation
During the dataset build, any record with unmatched <tool_call> / </tool_call> tags gets dropped entirely. If the training data has broken tags, the model learns broken tags. We chose data quality over data quantity.
5. Two-Phase Curriculum Training
This is the architectural change. Instead of training on a shuffled mix for the full run, we split training into two phases:
| Phase | Data | Steps | Learning Rate | Warmup |
|---|---|---|---|---|
| Phase A | Full mixed dataset | 80% | 1.0e-4 | Standard |
| Phase B | Tool-calling subset only | 20% | 2.0e-4 | 0.10 warmup |
Phase A builds the broad foundation — code gen, instruction following, agentic behavior, and tool calling all mixed together. Phase B is a concentrated polish pass that doubles the learning rate on tool-calling-only data, pushing the model to sharpen exactly the patterns that V8 was getting soft on.
6. Q5_K_M Default Quantization
We bumped the default deployment quantization from Q4_K_M to Q5_K_M. The model is approximately 1.5 GB larger, but it still fits comfortably on an RTX 5060 Ti 16GB. The extra precision preserves the multi-token tag patterns that Q4 was rounding away — attacking the problem from the deployment side as well as the training side.
Dataset Composition
The final dataset contains 40,401 training rows and 4,489 validation rows across five slices:
| Slice | Content | Rows |
|---|---|---|
| A | Code generation | 7,074 |
| B | Tool calling | ~15,125 |
| B+ | Multi-tool synthetic | ~2,000 |
| C | Agentic multi-turn | 8,994 |
| D | English instruction following | 7,208 |
Total tool coverage: 64.3%, up from V8's 57.6%.
Version Comparison
Looking back across the mixed-model lineage, you can see the progression from specialized to general:
| Version | Dataset Rows | Tool % | eval_loss | Key Improvement |
|---|---|---|---|---|
| V6 | 26,126 | 0% | 0.4724 | Code-only baseline |
| V7 | 28,862 | 63.8% | — | Restore tool-calling + agent |
| V8 | 34,104 | 57.6% | 0.4837 | Fix multilingual + native format |
| V9 | 40,401 | 64.3% | — | Fix tag emission + curriculum |
What's Next
V9 represents the current state of the Code-Trainer pipeline. The model now handles code generation, tool calling, multi-turn agent behavior, and English instruction following from a single LoRA adapter — running locally on consumer hardware through Ollama.
The immediate next steps are deploying V9 into the Phase 6 agentic stack (vLLM + Qwen-Agent) as the hot-swap model for compiled-language tasks, running a structured tool-calling evaluation suite to get hard numbers on tag emission accuracy, and exploring whether a V10 with reinforcement learning from tool-execution feedback could close the remaining gap between "almost always emits the right tags" and "always emits the right tags."
The pipeline infrastructure — automated dataset builds, HF Jobs training, GGUF conversion — is mature enough now that each iteration takes days rather than weeks. From a blank fine-tuning script to a multi-capability locally-deployed model in seven versions and a few months of evenings. That's the build log.