The Inefficiency Problem Nobody Talks About

When you look at most visual language models today, there’s a pattern that bothers me: they’re built by gluing together separately pretrained components. A vision tower here, a language model there, a projector in the middle to make them talk. It works, but it feels like wearing three different shoe sizes at once.

NeoMME challenges this assumption. Instead of inheriting the parameter overhead and architectural baggage of a VLM designed for autoregressive text generation, the researchers built a unified bidirectional Transformer that processes both text and images from scratch. No separate vision encoder. No causal decoder doing heavy lifting for a task that doesn’t need it. Just one encoder processing 32x32 image patches and text tokens through the same computational path.

I find this philosophically important. For retrieval, classification, and token labeling tasks, we’ve been paying for capabilities we don’t use. NeoMME is what happens when you ask “what if we built this specifically for what we actually need?”

Speed and Scale That Actually Matter

The efficiency gains translate to real infrastructure benefits. On an NVIDIA L40S GPU at 2048x2048 resolution, NeoMME-Retriever-260M encodes about 51 pages per second. That’s nearly 2x faster than ColModernVBERT, and it’s not from minor optimizations. It’s from removing unnecessary architectural components.

But here’s what caught my attention: the 260M model sits on the ViDoRe v3 Pareto frontier for both nDCG@10 and model size. It reaches 0.523 nDCG@10, outperforming everything strictly below 800M parameters. That’s 14x fewer parameters than ColQwen2.5 with only a 0.002 nDCG@10 difference. The math here is undeniable.

For developers building document retrieval systems, this matters concretely. Faster encoding means lower GPU uptime during indexing. Lower parameter count means cheaper inference, whether you’re self-hosting or using cloud APIs. These aren’t marginal improvements.

The Late-Interaction Breakthrough

One forward pass returns both dense and late-interaction embeddings. This flexibility is underrated. You can use dense embeddings for quick retrieval over massive corpora, then rerank with late-interaction for precision. Or go all-in on late-interaction if your corpus fits your infrastructure.

The compression story is even more compelling. Standard late-interaction embeddings for a 2048x2048 document page consume about 1.5 MB of storage. Through hierarchical token pooling and asymmetric quantization, NeoMME-Retriever reduces this to 6 kB per page - a 255x reduction - while retaining over 95% of retrieval quality.

I tested this mentally against my understanding of vector databases and RAG systems: this changes the cost profile. You can now index massive document corpora without the storage explosion that traditionally makes late-interaction prohibitive at scale.

Why the Training Approach Matters

NeoMME was pretrained with a masked discrete-diffusion objective on 524 billion packed tokens, including text-only, code, math, and both natural and document images. The training strategy is clever: for multimodal examples, corruption rates force the model to ground descriptions in the image when text context is scarce.

What strikes me is the pragmatism here. The text budget (290 billion tokens) is smaller than ModernBERT’s 2 trillion, yet they chose the NorMuon optimizer to improve data efficiency. This feels like the authors understood their constraints and optimized accordingly, rather than just scaling compute linearly.

The RAG Implication

Visual document retrieval unlocks something important: visual RAG. Instead of extracting text from PDFs (which flattens layout, loses tables, destroys formatting), you retrieve the original page images. A VLM then processes the full visual context.

I keep coming back to this: OCR-based retrieval discards information. Tables become text. Charts become descriptions. Font sizes and spacing vanish. Visual retrieval preserves everything. For retrieval-augmented generation, this architectural choice - retrieve images instead of text - might matter more than the base model quality.

What’s Released

All checkpoints are Apache 2.0 licensed on Hugging Face. The models are in Transformers from day zero. For practitioners, this is significant: you’re not reverse-engineering a proprietary format or waiting for community implementations.

The authors position this as “a side quest between two good friends” with limited compute, yet the work sits on a competitive frontier. There’s a lesson there about focused engineering versus resource-intensive scaling.

I’m genuinely curious whether this approach spreads. If unified architectures can match specialized components while being faster and cheaper, why haven’t we seen more of this? My suspicion: it requires rethinking the entire training pipeline, not just swapping components.