How Image Resolution Impacts Visual Document Retrieval

Traditional computer vision models typically focus on mimicking human visual perception. jina-embeddings-v4 takes a different approach: it combines image and text processing to understand how people read and interpret information presented visually. Unlike OCR programs that simply digitize text, it actually parses complex visual materials like infographics, charts, diagrams, and tables—documents where both the text and visual elements carry semantic meaning. We call these "visually rich documents."

If we just used OCR and text embeddings models, we would miss important information. If we used conventional embedding models trained on image-caption pairs, we wouldn’t be able to extract the semantics of the text. And in the case of things like tables, we have to know what the text means and process the spatial relationships between text elements to properly process them. We have to process them in visually-aware ways, which is why it’s called visual document retrieval.

But you have to be able to see things to make sense of them, and it’s no different for embedding models. Image quality counts.

Image quality differences can come from many sources: photos are out of focus, lighting conditions are bad, movement causes blurring, and lossy compression algorithms destroy details. But visually rich documents are typically “born” digital. They consist of things like screenshots, presentation slides, and glossy PDFs — data that has been rendered into images by some presentation or publication process. Sometimes, they’re scans of actual paper pages, but if competently made, those images don’t suffer from the kinds of problems that are pervasive with pictures of real-world scenes.

For visually rich documents, the primary input quality problem is image resolution. Too small, and the information we need for retrieval is lost. But too big, and we flood the model with spurious details that undermine accurate processing. Right-sizing your inputs to jina-embeddings-v4 will save you money and improve your retrieval results.

This article looks at how jina-embeddings-v4 processes images and the effect of image resolution on its ability to retrieve visually rich documents. We’ve provided code in the JinaVDR repo on Hugging Face to reproduce our results or test your own data if you want to apply ideas from this article yourself.

GitHub - jina-ai/jina-vdr: Jina VDR is a multilingual, multi-domain benchmark for visual document retrieval

Jina VDR is a multilingual, multi-domain benchmark for visual document retrieval - jina-ai/jina-vdr

GitHubjina-ai

tagVisual Language Models

jina-embeddings-v4 is a visual language model (VLM) that extends the Qwen2.5-VL-3B-Instruct VLM. This approach to embedding images is different from single-modality image or text models and CLIP-style models like jina-clip-v2.

Figure 5 is a schematic of the jina-embeddings-v4 model architecture. The backbone model is similar to a conventional transformer-based embedding model, except that it supports dual output modes: a single-vector (dense vector) embedding produced by mean-pooling the final layer of the decoder, and a multi-vector (late interaction) output produced by a projection layer that is the same size as the input.

For texts, its input is the same as conventional text embedding models: Texts are tokenized, and the tokens are replaced by substituting vectors from a lookup table. These token vectors together serve as input to the model.

The innovation of VLMs is in their handling of images: A conventional image embedding model is connected to the text embedding model, but instead of producing an image embedding by mean pooling, its final layer becomes the input to the text embedding model, as if it were just a sequence of token vectors.

This means that large images have the same problems for embedding models that long texts have. Longer inputs increase computational costs and encoding times and often produce uninformative embeddings. Small images have the opposite problem: If too much information is packed into one patch, it gets attenuated in the embedding process and gets lost in single-vector embeddings, while for multi-vector embeddings, it reduces the strength of matches. There is a trade-off between the amount of information provided to the model and its ability to accurately extract relevant information.

But texts can’t be resized; images can.

tagImage Preprocessing in jina-embeddings-v4

When you submit an image to jina-embeddings-v4, it’s split into patches of 28x28 pixels. This means that the number of patches is roughly proportional to the size of the image. If you use the Jina API, the maximum size supported is 602,112 pixels, but it’s a user-settable parameter in the model code if you download it.

The VLM architecture means that the number of 28x28 image patches the model can support is equal to the number of tokens of text it can support. That places an absolute upper bound on image size at roughly 20 megapixels.

tagExperiments with Image Resolution

We evaluated different image resolutions using jina-embeddings-v4 with the ViDoRe v1 and v2 suites, and part of the JinaVDR benchmark suite, looking at both single-vector and multi-vector outputs. Adjusting the max_pixels parameter of the model to a range of values up to 19,267,584 pixels (i.e., 5,376x3,584) produces embeddings from different image resolutions. Where an image is already smaller than the value of max_pixels, it is not changed.

tagViDoRe v1

Table 1 reports single-vector (dense vector) embedding retrieval results for five resolutions on individual ViDoRe v1 benchmarks (average nDCG@5 score). We only report values up to 9,633,792 pixels because no images in ViDoRe v1 were larger than that.

You can see in Table 1 that the highest resolution is far from the best. 9.6 megapixels performed below the best resolution on tests, except the one with no images larger than 4.8 megapixels, where it got the same score because there was no resizing. The best performer on average was 1.2 megapixels, and the resolution with the most best scores on different benchmarks was 2.4 megapixels.

If we took the best score on each benchmark and averaged them, we would get 0.84905. This is only 0.345% better than the score for 1.2 megapixels.

Using multi-vector embeddings, as shown in Table 2, our retrieval scores are significantly higher. Multi-vector matching typically performs better than single-vector matching. However, the performance of different resolutions is roughly the same.

Just like for single-vector embeddings, 1.2 megapixels resolution has the highest average score, and 2.4 megapixels is the best scoring resolution for the largest number of individual datasets. The average of best scores over all benchmarks is 0.90853, and the 1.2 megapixel resolution’s average score is 0.344% lower, almost exactly the same as for the single-vector case.

tagViDoRe v2

ViDoRe v2 uses much more detailed and colorful imagery than ViDoRe v1, which suggests that the optimal resolution will likely be higher.

We tested multi-vector embeddings from jina-embeddings-v4 on the ViDoRe v2 benchmark suite, with results in Table 3. Single-vector results are similar but with lower scores, so we omit them here.

In this case, the best performer on average is 4.8 megapixel resolution, with average performance at 9.6 megapixel essentially identical. Nonetheless, for three of the four benchmarks, the highest resolution performed worse than lower resolutions, except for one test where no images were larger than 4.8 megapixels.

tagHigh-Resolution Benchmarks

ViDoRe v1 and ViDoRe v2 consist of images with similar native resolutions, so we took two benchmarks from the JinaVDR suite that contain very high-resolution, difficult-to-process images, and performed the same multi-vector embeddings test with them.

One is the europeana-de-news benchmark, which contains high-resolution scans of German newspapers from the 17th to 20th centuries; the other is the wikimedia-commons-maps benchmark, which contains very-high-resolution scans of printed maps, mostly from the pre-digital era.

The results are very mixed, as you can see in Table 4.

The newspaper data clearly does not require the same resolution as the map data. This is likely because the text in the maps is very small compared to the overall image size and it really needs to be readable for retrieval to work. When the newspaper scans are sized larger than optimal, performance fluctuates and declines.

The average is suboptimal for both, suggesting that there is no one right resolution to solve the problem. It’s that discovery that motivates what we did next.

tagMulti-Resolution Embeddings

The VLM architecture of jina-embeddings-v4 treats individual patches of images the same way it treats text tokens, so it’s easy for us to augment high-resolution patches with low-resolution ones. It just means more input data, so long as we stay within the maximum that the model supports. If lower-resolution yields better semantics than high-resolution, we can just include them and not worry about what the optimal resolution is.

To test this hypothesis, we looked at three combinations of resolutions:

This means that when we tested Mix 1, we resized each image to four different resolutions — 150,528 px, 301,056 px, 602,112 px, 1,204,224 px — and processed each of them individually into multi-vector embeddings, then concatenated the results before doing query matching. Same for Mix 2 and Mix 3, but at different resolutions. We tried all three combinations on the ViDoRe v2 benchmarks, summarized in Table 6.

For all ViDoRe v2 benchmarks, Mix 2 outperforms Mixes 1 and 3, suggesting that more different resolutions produce better results than adding higher resolutions. On two of four benchmarks, all mixtures of resolutions outperform the best single resolution, and for the remaining two, only Mix 1 underperforms, and then not by a large margin.

These results show that although there is no single best mix of resolutions for all data, identifying a good mix of resolutions is the right direction to go to find an optimal solution.

tagConclusion

Image resolution matters quite a lot for jina-embeddings-v4 when processing visually rich materials. One key issue is that text should be sized to be readable. What you can’t read, AI can’t read either, and if the text matters, then it has to be read.

But, too high a resolution and it becomes difficult for the embedding model to assemble image patches into a coherent whole. It’s also expensive: Higher resolutions mean more processing and, for multi-vector embeddings, more storage and slower matching.

Using multiple resolutions and applying late interaction-style scoring on all the outputs together is a good way to handle visually rich images with varying sizes. But, this adds to processing and storage costs, and slows down retrieval the same way that very large resolutions do.

We’re looking into ways we can operationalize this insight to improve neural search. We are constantly trying to diversify our training and test data to probe our models for shortcomings and devise ways to improve them. We are looking into the effects of resolution and multi-resolution techniques like the ones described here on various types of materials.

We are also doing experiments to see if multiple-resolution techniques like the ones described here mitigate the effect of noise in images and produce more robust retrieval.

Furthermore, it may be possible to automatically determine in advance the optimal resolution for each image. If we can reliably detect the best resolution, it would remove one more parameter for users while improving overall results, making embeddings more accessible and usable.