JinaVDR: New Visual Document Retrieval Benchmark with 95 Tasks in 20 Languages

GitHub - jina-ai/jina-vdr: Jina VDR is a multilingual, multi-domain benchmark for visual document retrieval

Jina VDR is a multilingual, multi-domain benchmark for visual document retrieval - jina-ai/jina-vdr

GitHubjina-ai

JinaVDR (Visual Document Retrieval) - a jinaai Collection

max. ~1000 images and OCR text included

a jinaai Collection

We're releasing JinaVDR (Visual Document Retrieval), a new benchmark for evaluating how well models retrieve visually complex documents. JinaVDR encompasses multilingual documents with intricate layouts—combining graphs, charts, tables, text, and images alongside scanned copies and screenshots. The benchmark pairs these diverse visual documents with targeted text queries, enabling comprehensive evaluation of retrieval performance across real-world document complexity and broader domain coverage.

JinaVDR spans diverse languages, domains, and document formats to reflect real-world retrieval scenarios. While English remains predominant in both queries and documents, the benchmark incorporates over a dozen additional languages, providing significantly broader multilingual coverage. The domains encompass historic documents, software documentation, medical records, legal texts, and scientific papers, capturing varied professional use cases. Document formats range from web pages and PDFs to scanned materials, presentation slides, and standalone images. Many datasets intentionally mix languages and formats, creating realistic conditions that challenge models to handle the complexity they encounter in practical applications.

tagHow We Build JinaVDR

JinaVDR benchmark provides an evaluation framework spanning 95 tasks across 20 languages, including domain-diverse and layout-rich documents like charts, maps, traditional scanned documents, Markdown files, and complex tables. It evaluates models through both visual question answering (for example, “How many civil lawsuits were dismissed at the Valladolid audience in 1855?”) and keyword querying (for example, “growth of the LED market across different regions”), giving a clearer assessment of retrieval capabilities on different document types found in the real world.

We used four techniques to construct JinaVDR with a focus on data diversity and task authenticity:

First, we repurposed existing benchmarks by converting OCR datasets into retrieval tasks using rule-based query templates (such as transforming MPMQA data) and reformatting question-answering datasets into retrieval scenarios:

Second, we manually annotated existing PDF datasets including StanfordSlides, TextbookQA, and ShanghaiMasterPlan, to create high-quality retrieval pairs:

Our third approach involved synthetic query and/or document generation, where we used existing document collections from sources like Europeana to create contextually relevant queries with Qwen2-VL-7B-Instruct, along with EasyOCR text descriptions:

We also rendered tabular datasets into visual tables and generated corresponding queries through templates derived from the original text data, as demonstrated in our AirBnBRetrieval task.

Finally, we repurposed existing crawled datasets with article-chart pairs, where we use text snippets from the articles as queries and corresponding charts as target documents, as shown in our OWIDRetrieval dataset:

This multi-faceted approach gives us comprehensive coverage across document types, languages, and retrieval scenarios.

tagExisting Benchmarks

Developing truly multimodal models (that can handle visually complex documents) requires benchmarks that go beyond traditional text-only evaluation methods. Frameworks like MTEB (Massive Text Embedding Benchmark) may work well for evaluating text retrieval across different domains and languages, but they’re not built for searching through documents where accurate retrieval depends on visual layout, charts, tables, and formatting. This is where visual document retrieval benchmarks (like the ViDoRe series) and image retrieval benchmarks (like MIEB, the Massive Image Embedding Benchmark) come in.

The ColPali paper introduced ViDoRe v1, which combines five English-language datasets, both academic and synthetic. The benchmark focuses on single-page documents that work well with optical character recognition (OCR), covers narrow domains like scientific papers and healthcare, and uses extractive queries where search terms often appear directly in target documents.

After models like ColPali hit a score of 90% nDCG@5 on ViDoRe v1, a new benchmark was needed. ViDoRe v2 improved upon v1 by supporting longer and cross-document queries, blind contextual querying, and more languages (French, German, and Spanish on top of English). Both benchmarks still have limited language diversity and narrow domain coverage, leaving gaps for evaluating new retrieval systems.

The MIEB takes a different approach by focusing on visual embeddings across 130 tasks, including other tasks beyond just retrieval. However, it mostly evaluates images without much text content, rather than visually rich documents. While the benchmark is excellent at testing visual understanding capabilities, it doesn’t perform well when you need to retrieve documents based on visual layout and textual content.

Our aim with the JinaVDR (Visual Document Retrieval) benchmark is to expand upon the work of these prior benchmarks by incorporating visually rich multilingual documents with complex layouts like graphs, charts, and tables (mixed with text and images), as well as adding real-world queries and questions.

tagEvaluating Embeddings on JinaVDR

Our benchmarking results show that many recent embedding models struggle with JinaVDR’s wide range of visual document tasks, while OCR-based baselines and older models show even weaker results, especially on non-English and structured-document datasets. We included BM25 with OCR for all datasets where simple text extraction made such retrieval feasible.

An exception to this is jina-embeddings-v4. Our results suggest that its multimodal embedding approach handles complex and multilingual document retrieval better than earlier generation models or traditional OCR-based pipelines. The model’s multi-vector capability provides the best performance because it avoids the compression limitations of single-vector approaches — while single vectors have to cram an entire page’s content into one representation (making it difficult to capture specific details), the multi-vector approach maintains the granular information needed for precise retrieval of similar documents.

tagMTEB Integration

dataset: Add JinaVDR by maximilianwerk · Pull Request #2942 · embeddings-benchmark/mteb

Hey, we would like to contribute the JinaVDR benchmark to MTEB. Our aim with the JinaVDR (Visual Document Retrieval) benchmark is to expand upon the work of these prior benchmarks by incorporating…

GitHubembeddings-benchmark

Since MTEB has become the de facto standard for retrieval benchmarking, we're integrating JinaVDR directly into the MTEB framework to maximize adoption and ease of use. This makes it easier for researchers to run visual retrieval models on our benchmark using familiar evaluation infrastructure. However, migrating our data to the BEIR format did require some trade-offs, like not including OCR results in the MTEB version. This means that traditional text-based methods like BM25 aren't directly runnable as part of the MTEB, which reinforces the focus on visual document understanding rather than falling back to text-based retrieval methods.

tagLimitations

To build a comprehensive benchmark from a wide range of sources, we had to perform careful preprocessing to ensure both practical usability and evaluation quality: We applied size normalization by subsampling each dataset to a maximum of 1,000 examples (down from thousands or tens of thousands), making the benchmark actually runnable while maintaining good coverage across tasks. This constraint was especially important given the high levels of compute we needed to process high-resolution visual documents.

We used quality filtering to address several challenges common in real-world document collections. While poor image quality in scanned documents often reflects realistic use cases, it made it tougher to control the quality of synthetic data. We implemented consistency filtering to remove duplicates (which are common across large document collections) and used LLMs to filter out low-quality queries that wouldn't provide useful evaluation signals, such as overly generic questions like "What can you see in the chart?". For synthetic data generation, we ran into limitations in query diversity despite using varied prompting strategies, and needed to perform manual curation to ensure enough evaluation coverage across different retrieval scenarios.

tagConclusion

Visual document retrieval evaluation is now in a situation where traditional text-based benchmarks no longer capture the complexity of how humans actually search for and consume information. JinaVDR overcomes this barrier by providing comprehensive evaluation across a range of tasks and languages far exceeding previous benchmarks.

Moving forwards, the industry needs benchmarks that reflect genuine retrieval challenges rather than artificial constraints. As organizations come to rely more on visual document retrieval for tasks from legal research to medical diagnostics, evaluation frameworks have to evolve beyond narrow academic datasets toward the messy, multilingual, and visually complex documents that we find in the real world. JinaVDR is just the first step in building retrieval systems that truly understand how visual and textual information work together in practice.