This isn't a training run. This is the deployment step — the point where a fine-tuned model stops being a research artifact and becomes something you can actually serve.
The V6 adapter exists as separate LoRA weights sitting on top of a 14-billion-parameter base model. To load it you need transformers + PEFT, roughly ~28 GB for the base plus adapter, and a Python runtime. That's fine for evaluation notebooks. It's not fine for production inference.
What the GGUF Step Actually Does
Phase 5 of our pipeline has two stages. First, we merge the winning LoRA adapter directly into the base model's weights — no more separate delta files, just one unified float16 checkpoint. Second, we quantize that merged checkpoint through llama.cpp down to 4-bit or 5-bit precision. The result is a single .gguf file that runs through llama.cpp's optimized C++ inference engine.
a100-large flavor solely for its 144 GB system RAM — the float16 merge step needs to hold the entire 14B model in memory at full precision before quantizing it back down.
Quantization Levels
We shipped two variants to give users a choice between size and fidelity:
| Variant | Size | Perplexity Penalty | Use Case |
|---|---|---|---|
Q4_K_M |
~9 GB | ~1-3% | Balanced — fits on most consumer GPUs |
Q5_K_M |
~10.5 GB | <1% | Recommended for VRAM ≥ 12 GB (V9+) |
The Q5_K_M variant is our recommendation if you have the headroom. Less than one percent perplexity penalty for an extra 1.5 GB is a no-brainer. Q4_K_M is for tighter VRAM budgets — the 1-3% penalty is real but manageable for code generation tasks where you're typically re-ranking a handful of completions anyway.
Serving Options
The GGUF can be loaded into any of the major local inference runtimes:
- llama-server — direct llama.cpp HTTP server, lowest overhead
- Ollama — one-command setup with
ollama run, our primary target - LM Studio — GUI-based, good for exploration and manual testing
- text-generation-webui — for users already in that ecosystem
This GGUF is also the Phase 6 hot-swap target. Our production stack runs vLLM with Qwen-Agent orchestration — the GGUF slots into that pipeline as the local inference backend.
Cost and Timeline
Total cost: ~$2 on HF Jobs a100-large. Total wall time: ~1 hour. That's merge, quantize (both levels), and upload. The cheapest step in the entire pipeline by an order of magnitude.
Source Adapter
The initial GGUF was built from the V6 aggressive adapter. We later rebuilt it from V8 as the source adapter improved — the quantization step itself is deterministic and cheap enough to re-run whenever the upstream adapter changes.
What We Didn't See Coming
The model deployed fine. It ran locally. Inference was fast. And then we tried to use it for agentic work and hit a wall.
This is the kind of thing you don't see in eval metrics. Perplexity looked great. Code completion accuracy was solid. But hand it a system prompt with tool definitions and ask it to call a function? It just generates more code. It doesn't know what a tool call is.
The next version needs to fix this. We're going to build a mixed dataset that blends code generation with tool-calling traces and agent behavior. That's V7.