The Premise
We wanted to answer a deceptively simple question: can a small language model look at a screenshot of source code and reproduce the underlying text? Not OCR in the traditional sense — we wanted the model to understand that it was looking at code and emit syntactically valid source, not just pixel-matched characters.
This was the multimodal stage of Code-Trainer. We built a vision adapter that takes a VS Code screenshot rendered in the Monaco Editor, processes it through a frozen image encoder, and feeds the resulting embeddings into a code-specialized language model that generates the source text.
Architecture
The pipeline has three components stitched together:
| Component | Details | Parameters |
|---|---|---|
| Image Encoder | Swin-B (frozen, pretrained on ImageNet-22k) | 87.7M |
| Projector | 2-layer MLP bridging vision embeddings to LM input space | 2.1M |
| Decoder | Qwen2.5-Coder-1.5B with LoRA (r=16, alpha=32) | ~1.5B |
The Swin encoder stays frozen throughout — we only train the projector and the LoRA adapter on the decoder. This keeps the trainable parameter count low and the compute budget manageable.
The Dataset
We rendered 26,126 code screenshots using a headless Chromium instance running the Monaco Editor. Each sample is a screenshot-to-source pair covering 8 programming languages. To prevent the model from overfitting on a single visual style, we rotated through 8 different Monaco themes during rendering — the model needed to learn "this is code" regardless of whether it was dark-on-light or light-on-dark.
Training
Training ran for 3 epochs with an effective batch size of 32 on a single A100 GPU. Total wall time was approximately 5.5 hours at a compute cost of roughly $18. We used a cosine learning rate schedule with warmup and gradient checkpointing to fit everything into 80GB of VRAM.
Results
The numbers tell an interesting story — the model learned to produce code-shaped output, but it did not learn to faithfully reconstruct the original source:
| Metric | Baseline | Trained | Change |
|---|---|---|---|
| syntax_valid_rate | 19.5% | 61.0% | +213% |
| mean_edit_similarity | — | — | +16.8% |
| exact_match | 0.0% | 0.0% | No change |
| BLEU-4 | 0.0 | 0.0 | No change |
The +213% jump in syntax validity is the headline number. The model went from emitting mostly garbage to producing parseable code roughly 6 out of 10 times. Edit similarity improved too, meaning the generated code was structurally closer to the target even when it was not an exact match.
But exact match and BLEU-4 both stayed at zero. The model learned the shape of code — indentation patterns, bracket matching, keyword placement — without memorizing the specific tokens. For a 1.5B parameter decoder trained on only 26K samples, this is not surprising. The decoder simply does not have the capacity to act as a precise transcription engine.
Limitations
We should be honest about what this is not. The vision adapter is not a general-purpose code OCR system. It only works on Monaco-rendered screenshots — hand it a photo of code on a whiteboard and it will produce nonsense. The 1.5B decoder is the bottleneck: it can generate plausible code, but it cannot reliably reproduce the exact source from visual input alone.
What's Next
The vision adapter proved the architecture works. The frozen encoder plus projector plus LoRA-adapted decoder pipeline is sound — the training signal flows through, and the model learns meaningful representations from screenshot inputs. But the 1.5B decoder limits fidelity. The next step is scaling up to a much larger code-specialized base model — something in the 14B range — and training on the text-only portion of the dataset to build a strong code-generation foundation before layering multimodal back on top.