Reading images and text with one encoder

The work covered here is NeoMME: an efficient Multimodal-native and Multilingual Encoder, published by Aurélien Lac and Tony Wu of Hcompany on the Hugging Face blog on September 3, 2026, with a technical report also released under arXiv identifier 2609.01657. The original is linked under the title.

The claim is that a 260M and 800M encoder family trained from scratch with a single bidirectional transformer for both image patches and multilingual text tokens, and without a separate pretrained vision tower or generative language decoder, can match much larger models on visual document retrieval while staying efficient, and this post walks through why that design was needed, how it was trained, how it was tuned for retrieval, and what the numbers mean in practice.

Why another multimodal encoder was needed

A large share of recent work on document retrieval, classification, and token level judgment has borrowed the machinery of generative models. A separately pretrained vision encoder extracts image features, a projector maps those features into the input space of a language model, and a causal decoder reads the combined image and text representations. That structure feels natural when the task is to write new text, because next token prediction is exactly what is needed. Retrieval is not writing. It measures how close a question and a document sit in vector space. Classification is similar. It decides which fixed bucket an input belongs to. Carrying a full causal decoder for such vector decisions adds size without matching purpose. Parameters grow, compute grows, and serving cost grows.

The encoder community had already shown a leaner path. The ModernBERT line of research refined encoder architecture and training so the same size could run faster and score higher. For visual document retrieval, ModernVBERT tried a similar compromise. It used a bidirectional ModernBERT style text encoder while keeping a separate pretrained SigLIP family vision tower. It was a half step. The text side became lighter, but the image side still borrowed a large pretrained eye. Half the weight stayed in place. As long as the parameters and compute of a vision tower ride along, there is a limit to how light the system can become.

This research starts from that limit and asks a direct question. What happens if one encoder handles images and text together from start to finish. Could an eye and a language sense raised together from scratch handle vector work more lightly than borrowed parts bolted together. To answer, the authors refuse to import an existing vision tower, an existing text encoder, or an existing text decoder. One transformer encoder processes text tokens and image patches through the same computational path. Training starts from scratch. That is what multimodal native means here. Not a late marriage of parts, but a design that processes both modalities together from birth. The word efficient gains weight in the same move. It signals an intent to drop the machinery needed for generation and keep only what vector representation needs.

Processing images and text in one transformer

The backbone of NeoMME is simple to state. It comes in two sizes, 260M and 800M, and both share the same architecture. Text enters through factorized token embeddings while images are divided into a grid of non overlapping 32 pixel patches and projected with a small multilayer perceptron, and both streams enter the same transformer encoder. There are no separate roads. Both pass through the same layers, the same attention, and the same feedforward computation. In a dual tower design, image and text travel apart and meet late. In a generative combination, visual features are translated into language space and fed to a decoder. Here both are computed in the same room from the start. That makes pretraining, fine tuning, parallelization, and serving easier to handle uniformly across modalities. From an infrastructure view, that simplicity matters a great deal, because it reduces the need to carry separate optimization and deployment paths per modality.

The detailed stack follows the modern encoder grammar faithfully. It uses grouped query attention, query key normalization, gated attention, two dimensional rotary position embeddings, and squared ReLU based multilayer perceptrons, among other refinements. No single item claims a dramatic leap. The picture is closer to validated improvements stacked carefully. Readers who have followed encoder research will recognize the names. The important point is that these refinements apply to the same stack regardless of modality. With no image only or text only apparatus kept apart, every improvement leaks into both sides. Fix one side and the other side improves with it.

Multilingual support is engraved into the backbone as well. The team trained a byte pair encoding tokenizer with a 131k token vocabulary from scratch on multilingual text, code, mathematics, and machine produced image transcripts. This is not an English centered model stretched to other languages later. It is a tokenizer that sees many languages together from the start. Retrieval often breaks at language borders. Questions arrive in French, documents sit in English, and numbers inside tables use yet another notation. An encoder trained on many languages from the start confronts that mixture directly. Keep this point in mind when we reach retrieval scores later, because compact size working across languages is hard to separate from tokenizer and data choices.

Dynamic resolution and long context for reading documents

Document images come in all sizes. One page is a dense report packed with small type while another image is a single photo with wide margins. Forcing every input into one fixed size blurs type on dense pages and wastes compute on empty margins. This work chooses dynamic image resolution. Images keep their aspect ratio and size. A high resolution information dense page receives more tokens while a small sparse image receives fewer. That matches the instincts of document retrieval practice. Corporate reports, papers, invoices, and drawings lose their fine distinctions the moment resolution is flattened. Font size, spacing, table rules, and chart legends vanish together. Dynamic resolution keeps those cues alive.

Context length is generous too. Both models carry 16,384 tokens of context, enough for about two standard 4K UHD images at once. A model that can only see cut pieces loses thread when a table spans a page boundary. When the unit of a number seen on one page is explained on the next page, fragments alone cannot settle the question. Long context reduces such breaks. To manage compute, most layers use symmetric sliding window attention while every sixth layer and the final layer use global attention. Near neighbors are read densely while distant relations are scanned at fixed points. Reading high resolution documents needs both. Relations between adjacent characters matter, and so do distant relations between a page title, a table, and body text. The alternating structure reads as a design that serves both needs at once.

Seen through the eyes of a working engineer, the two choices interlock. Dynamic resolution adjusts token counts up and down with content, and long context plus windowed attention provides the vessel that holds the larger token counts. Either choice alone would be half a solution. Higher resolution without a larger vessel forces trimming, while a larger vessel with fixed resolution wastes compute on empty space. This research drives both wheels together. It is tuning aimed at a concrete job, document retrieval. The later stories about indexing speed and index compression only make sense on this foundation, because flexible handling of input tokens is what lets high resolution retrieval face its cost problem directly.

Learning from images by guessing hidden text

The training method deserves close attention. The team pretrains the whole model from scratch as a discrete masked diffusion text denoiser. For each text only example, a corruption rate is sampled uniformly between 0 and 1, and each eligible text token is masked independently at that rate. Multimodal examples use corruption rates between 0.3 and 1. Image patches stay visible while the model reconstructs masked text. With light masking, surrounding text alone can often fill a blank. The word cat is a plausible completion of a familiar sentence even with no image. With heavy masking, language clues alone run out. Almost no words remain nearby, so blanks cannot be filled without looking at the image. Heavy masking becomes the device that forces image grounded description.

The virtue of the design is that it blocks language shortcuts. A common failure in multimodal training is a model that answers from text correlation without ever looking. Memorizing stock phrasing in the training data removes any reason to inspect pixels. The result is a model with eyes it never uses. Raising corruption closes that shortcut. Memorized patterns stop helping once text is nearly gone. Clues must be pulled from visible image evidence. To fill a blank in a sentence about a scene, the model must check whether that scene is really present in the image. Repeated pressure of that kind teaches genuine correspondence between images and text. It learns how numbers in tables, prose in paragraphs, and trends in charts point at each other.

The data mix and scale are stated plainly. Pretraining mixes multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including about 290 billion tokens from text only examples. That text budget is relatively small next to the 2 trillion token budget reported for ModernBERT. The team therefore chose the NorMuon optimizer to improve data efficiency during training. Growing a model from scratch on a smaller budget means learning more from every step. The optimizer choice answers that concern. It also rhymes with the backstory that NeoMME began as a side effort between two friends with limited time and compute. This is not a design that assumes a giant budget. It leans toward a design that still works when budgets stay small.

Tuning for retrieval with dense vectors and late interaction

Once a backbone exists, it must prove it can do real work. The authors fine tune NeoMME for visual document retrieval using the page image methodology introduced by ColPali. Traditional text retrieval extracts text chunks from documents and indexes them, which requires preprocessing steps such as optical character recognition. Even perfect recognition flattens layout, charts, tables, font type and size, and other visual clues. Ranking page screenshots directly preserves those clues. Where table rules break, how tall chart bars stand, and how much larger a heading reads than body copy all remain as retrieval signals. It is a choice to use images as they are instead of flattening extracted text.

The retrieval tuning adds two jointly trained heads on the same backbone. The dense head averages backbone hidden states into one normalized vector through mean pooling. That is the most common form today. It stays compact and pairs naturally with approximate nearest neighbor search for fast retrieval. The late interaction head projects every text token and every image patch from backbone outputs into its own 128 dimensional normalized vector. That preserves local matches between individual query tokens and image regions. If dense vectors pack the mood of a whole document into one bag, late interaction keeps finely cut correspondences between tokens and regions. Neither is crowned in advance. One forward pass returns both representations, and practitioners choose what fits.

The usage guidance is practical. Late interaction embeddings are recommended in general because they are stronger and easy to use with open source libraries such as NextPlaid. When corpora grow very large, a two stage pattern is available. Run one forward pass to get dense embeddings, retrieve a small candidate set through an approximate index, then rerank only those candidates with late interaction. It combines a fast net with precise handwork. The appeal is flexibility across infrastructure and budget. A small team can start with dense search and add reranking where cost allows, while a larger organization can lead with late interaction from the start. Two handles grow from one backbone, so deployment paths stay simple.

Compact size with competitive scores, reading the benchmarks

The center of the performance report sits on the ViDoRe suites. The headline metric is nDCG at 10, where higher means desired documents float nearer the top. The 260M retriever reaches 0.523, the highest among evaluated models strictly below 800M parameters. It trails a ColQwen family model by only 0.002 while using about 14 times fewer parameters. The 800M retriever reaches 0.556, within 0.009 of a similarly sized Vultron Flash retriever. Both NeoMME retriever models sit on the size accuracy Pareto frontier. For research that claims efficiency, that frontier plot is the required picture, and the work earns its place there.

The pattern continues on earlier suite versions. ViDoRe v1 and v2 use nDCG at 5, and there the 260M model is reported ahead of ColModernVBERT and a twice larger 500M class model, while the 800M model is reported ahead of ColPali v1.3 despite using 3.6 times fewer parameters. Those comparisons cut against the older habit that size equals quality. Benchmarks are of course only benchmarks. Production corpora are messier, language mixes differ, and document quality varies. Still, a small model reaching the heels of giants under the same ruler and the same conditions is not light news. It matters even more because the backbone was raised from scratch without a borrowed vision tower. The numbers show that eyes can open without borrowing someone else eyes.

When reading these results, keep multilingual and high resolution context in view. The ViDoRe suites span languages and document styles together. Strength there suggests that earlier choices around tokenizer, data mix, and dynamic resolution worked as a group. No single choice earns the credit alone. Without a tokenizer that covers many languages, multilingual questions would collapse. Without preserved resolution, pages heavy with tables and charts would lag. The dual head design contributes too. Coarse gathering plus precise matching naturally lifts ranking metrics. Scores are sums of design. These scores read as the sum of choices tuned for vector decisions from start to finish.

The heavy load of high resolution, shrinking storage 255 times

The long standing weakness of late interaction is storage. Footprint grows with the number of output vectors, so higher resolution means heavier load. A 2048 pixel square page yields about 4,200 vectors, or about 2.1 MB in float32. Averaged across ViDoRe v3, the measured mean is about 1.5 MB per document. At a million pages, that reaches terabyte scale. No matter how accurate, such growth is hard to operate. The tug between retrieval quality and storage cost is the lived reality of high resolution search.

The authors combine two complementary compression methods. The first is hierarchical token pooling, which clusters similar document vectors inside one multi vector embedding and replaces each cluster with its mean, reducing vectors stored per page. Grouping similar regions preserves the gist of local matches while cutting counts. The second is asymmetric quantization, which quantizes stored document embeddings to int8 or binary form. Query embeddings are never stored and are created on the fly, so they can stay at higher precision. Only the stored side travels light. Pulling both handles together shrinks counts and shrinks bits at once.

The reported numbers are sharp. With pooling factor 10 and int8 queries and documents, storage falls from about 1.5 MB to about 39 kB per page, a 39 times reduction, while keeping more than 99 percent of baseline nDCG at 10. A more aggressive setting uses pooling factor 8 with int8 queries and binary documents. That version uses 6 kB per page, 255 times smaller, and keeps more than 95 percent of original retrieval quality. Users can pick a point on that frontier from storage budget and required quality. The choice between sparing storage with a small quality cost and spending storage to hold quality has a numeric menu. Treating compression as part of design rather than late patchwork stands out here.

Indexing speed and what 51 pages per second changes

After storage comes speed. Before any corpus can be searched, every document must be encoded into embeddings and placed in a vector store. Slow encoding stretches GPU uptime and compute cost. Every fresh batch waits longer. The authors measured image encoding speed from preprocessed image tensors with batch size calibrated separately per model and image size. At matched 2048 pixel input on one L40S GPU, the 260M retriever encodes about 51 pages per second, nearly twice the 26 pages per second of ColModernVBERT under the same conditions. Both 260M and 800M models are also reported faster than compared models on smaller inputs.

Unpack what that means on the ground. Indexing a million page corpus at 26 pages per second takes well over ten hours. At 51 pages per second, the same job takes less than half the time. Converted to GPU rental, that is money directly. In shops that rent by the day, a few hours redraw a budget line. Incremental indexing tells the same story. Organizations that ingest thousands of reports, invoices, and drawings daily need them searchable the same day. Speed equals freshness. Whether yesterday documents surface in today search turns on throughput. Twice the speed means indexing twice as often on the same budget, or indexing as often at half the cost.

The roots of speed trace back to backbone design. With no separate vision tower, projector, or decoder in the path, stages run short. Windowed attention keeps high resolution inputs manageable. Small size leaves room for larger batches. More pages can be grouped into each step. No single trick works magic. Choices sum into speed. Efficiency stops being a slogan and lands as pages per second. That speed also grounds the visual RAG story ahead. Retrieval must stay fast for generation to answer over fresh documents.

Paths to try it, transformers, sentence transformers, and visual RAG

Another virtue is how the work is released. All checkpoints carry the Apache 2.0 license and run in Hugging Face Transformers from day one. Base models at 260M and 800M ship with retriever variants. One forward pass returns dense and multi vector embeddings together, and the worked example scores two text queries against two document page images. Late interaction uses MeanMaxSim scoring while dense uses cosine similarity. The example mixes a French question with an English question on purpose. Multilingual support is shown in the act of use rather than asserted as a slogan. Questions and documents in different languages are meant to meet in one shared vector space.

Fine tuning paths stay open. Separate checkpoints serve dense and late interaction tuning with Sentence Transformers v6. Since that library currently supports one retrieval head per model, divided checkpoints tune each head independently, while joint training of both heads uses the retrieval class with a custom trainer. It follows the same pattern text teams know from ModernBERT. Load the backbone through one model class and swap the head that fits the job. Teams already comfortable with text embeddings will find the flow familiar. An encoder that covers images is handled much like text is handled.

The larger picture beyond retrieval is visual RAG. Visual document retrieval becomes stage one of a visual retrieval augmented generation system. Instead of passing extracted text chunks, the system passes original page images to a vision language model. Tables, plots, diagrams, and page layout survive instead of being flattened by extraction. The order is clear. Convert each PDF page to an image, embed with the retriever, and store in a vector store. Embed the user query with the same model and fetch the top pages. Append those images after the query in chat format and send the package to a vision language model for the answer. A public demo space accompanies the release so the flow can be tested directly. The research does not end as numbers in a paper. It continues as a flow that hands can touch.

Conclusion and next questions, what a light encoder opens

In short, this work replaces separately pretrained image and text encoders with one long context bidirectional transformer. It learns from scratch to process multilingual text tokens and raw 32 pixel image patches together. The retriever variant emits dense and late interaction representations in one forward pass. The 260M model leads its size band and encodes at nearly twice the throughput of a prior combined model at matched resolution. The heavy storage load of late interaction drops to 6 kB per page through hierarchical pooling and asymmetric quantization while keeping more than 95 percent of baseline quality. Efficiency here is not a youthful boast. It is measured under matched conditions and reported as numbers.

Open questions remain clear. First comes evaluation of the backbone itself. Strength on retrieval signals that vector representations matured well for downstream work, but classification, clustering, and sentence similarity still need a closer look. Second comes the reading of the training budget. About 524 billion tokens total, with about 290 billion from text only data, is not a giant budget, and the optimizer choice is presented as the lever that stretched it. The account convinces, but where the curve would lead with a larger budget stays open. Third comes generalization of compression. Whether the frontier measured on ViDoRe v3 keeps its shape on very different corpora and language mixes needs field checks, because pooling and quantization can react to document style.

The practical message still lands firmly. Retrieval infrastructure gains an option to trade a heavy borrowed eye plus a heavy mouth for one small jointly trained encoder. Indexing cost, storage cost, and serving complexity can fall together. With multilingual support built into the backbone from the start, there is less reason to carry a separate pipeline per language. Apache 2.0 licensing plus day one Transformers support moves that option from experiment to adoption review. The backstory that NeoMME began as a side effort between two friends with limited resources adds resonance. It shows that a change of direction can move a frontier even without giant means. The next chapter belongs to what gets built on this backbone. Tougher questions, more languages, and more varied documents will test how far a light eye can see.