jina-reranker-m0: Multilingual Multimodal Document Reranker

jina-reranker-m0 - Search Foundation Models

Multilingual multimodal reranker model for ranking visual documents

Search Foundation ModelsJina AI

jinaai/jina-reranker-m0 · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Today we're releasing jina-reranker-m0, our new multilingual multimodal reranker model for ranking visual documents across multiple languages: it accepts a query alongside a collection of visually rich document images, including pages with text, figures, tables, infographics, and various layouts across multiple domains and over 29 languages. It outputs a ranked list of documents ordered by their relevance to the query. Compared to jina-reranker-v2-base-multilingual, jina-reranker-m0 also improves text reranking for multilingual content, long documents, and code searching tasks.

tagNew Architecture

Unlike jina-reranker-v2-base-multilingual, jina-reranker-m0 shifts from the classic cross-encoder architecture to a decoder-only vision language model. It leverages the pretrained Qwen2-VL's vision encoder and projector, finetuned its LLM with LoRA, and post-trained a MLP to generate ranking logits that measure query-document relevance. This gives a discriminative model optimized for ranking tasks.

This new architecture allows jina-reranker-m0 to handle up to 32K tokens, seamlessly combining both visual and textual inputs. The model supports images ranging from a minimum size of 56×56 pixels up to 4K resolution. When processing images, the ViT and projector condense adjacent 2×2 tokens into single visual tokens for LLM input. Special tokens such as <|vision_start|> and <|vision_end|> clearly mark visual token boundaries, enabling the language model to properly process visual information and perform sophisticated multimodal reasoning that integrates both visual and textual elements.

This architecture also effectively solves the modality gap problem that plagued earlier models like jina-clip-v1 and jina-clip-v2. Previously, images would cluster near other images while text would cluster near other text in the representation space, creating a disconnect. This meant that when your candidate documents contained both images and text, retrieving images using text queries was problematic. With jina-reranker-m0, you can now rank images and documents together without worrying about this gap, creating a truly unified multimodal search experience.

It's worth noting that our training was limited to a maximum of 10K input tokens, with up to 768 tokens per image (between <|vision_start|> and <|vision_end|> markers). Additionally, we didn't specifically train the model for image-to-image, image-to-multimodal, or text-to-multimodal reranking tasks. In this context, "multimodal" refers to a single document containing both image and text tokens in the input. Looking at all possible combinations of image and text tokens in both queries and documents, we can summarize the full range of tasks supported by jina-reranker-m0 in the table below.

In our testing, we found some evidence suggesting the model can extrapolate to these untrained ranking tasks, but any effectiveness in these areas should be viewed as a result of the model's zero-shot transferability or unintended training side effects. We haven't conducted serious evaluations of the model's performance on these tasks, and plan to explore these capabilities more thoroughly in future research.

tagGetting Started

For a quick vibe-check, try our text-to-image rerank demo in the Search Foundation toolbox. We've prepared a collection of document images from our website, and you can also add your own image URLs. Simply type your query and press enter to see ranked results. You may retreat it either like OCR or content-based image retrieval - also feel free to try queries in non-English.

tagVia API

The code below shows how to calculate relevance scores between the query "small language model data extraction" and a collection of images and text documents. You can pass a text string, a base64-encoded image, or an image URL. New users can get a Jina API key with 1 million free tokens. While our API doesn't support using images as queries, you can use images as queries when accessing the model through the Hugging Face Transformers library.

curl -X POST \
  https://api.jina.ai/v1/rerank \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer JINA_API_KEY" \
  -d '{
  "model": "jina-reranker-m0",
  "query": "small language model data extraction",
  "documents": [
    {
      "image": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"
    },
    {
      "image": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/paper-11.png"
    },
    {
      "image": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/wired-preview.png"
    },
    {
      "text": "We present ReaderLM-v2, a compact 1.5 billion parameter language model designed for efficient web content extraction. Our model processes documents up to 512K tokens, transforming messy HTML into clean Markdown or JSON formats with high accuracy -- making it an ideal tool for grounding large language models. The models effectiveness results from two key innovations: (1) a three-stage data synthesis pipeline that generates high quality, diverse training data by iteratively drafting, refining, and critiquing web content extraction; and (2) a unified training framework combining continuous pre-training with multi-objective optimization. Intensive evaluation demonstrates that ReaderLM-v2 outperforms GPT-4o-2024-08-06 and other larger models by 15-20% on carefully curated benchmarks, particularly excelling at documents exceeding 100K tokens, while maintaining significantly lower computational requirements."
    },
    {
      "image": "https://jina.ai/blog-banner/using-deepseek-r1-reasoning-model-in-deepsearch.webp"
    },
    {
      "text": "数据提取么？为什么不用正则啊，你用正则不就全解决了么？"
    },
    {
      "text": "During the California Gold Rush, some merchants made more money selling supplies to miners than the miners made finding gold."
    },
    {
      "text": "Die wichtigsten Beiträge unserer Arbeit sind zweifach: Erstens führen wir eine neuartige dreistufige Datensynthese-Pipeline namens Draft-Refine-Critique ein, die durch iterative Verfeinerung hochwertige Trainingsdaten generiert; und zweitens schlagen wir eine umfassende Trainingsstrategie vor, die kontinuierliches Vortraining zur Längenerweiterung, überwachtes Feintuning mit spezialisierten Kontrollpunkten, direkte Präferenzoptimierung (DPO) und iteratives Self-Play-Tuning kombiniert. Um die weitere Forschung und Anwendung der strukturierten Inhaltsextraktion zu erleichtern, ist das Modell auf Hugging Face öffentlich verfügbar."
    }
  ],
  "return_documents": false
}'
Copy

The response is shown below, where the first result index=1 corresponds to the our ReaderLM-v2 paper screenshot.

{"model":"jina-reranker-m0","usage":{"total_tokens":2829},"results":[{"index":1,"relevance_score":0.9587112551898949},{"index":3,"relevance_score":0.9337408271911014},{"index":7,"relevance_score":0.8922925217195924},{"index":2,"relevance_score":0.8891905997562045},{"index":0,"relevance_score":0.8827516945848907},{"index":4,"relevance_score":0.8701035914834407},{"index":6,"relevance_score":0.8676828987527296},{"index":5,"relevance_score":0.8455347349164652}]}
Copy

tagVia CSP Marketplaces

jina-reranker-m0 will be soon available directly on AWS, Azure and GCP at the prices listed there.

AWS Marketplace: Jina Reranker m0

Microsoft Azure Marketplace

tagVia HuggingFace

You can also use the model locally from our Hugging Face page. We've prepared a Google Colab notebook that demonstrates how it works. Compared to our web API, using the model locally offers greater flexibility, such as the ability to use images as queries and work with multimodal documents.

Google Colab

tagConclusion

jina-reranker-m0 is our first attempt to unify textual and visual modalities in a single decoder-only model. This new architecture incorporates lessons learned from our previous encoder-only retrieval models, including jina-clip-v2, jina-embeddings-v3, jina-reranker-v2-base-multilingual and jina-embeddings-v2-base-code.

The new model not only unlocks capabilities for multimodal retrieval tasks, such as text-to-image reranking and visual document reranking, but also demonstrates improved performance compared to jina-reranker-v2-base-multilingual on text-to-text and text-to-code reranking tasks. We designate this new model series as the "m-series" to highlight its multimodal nature.

When comparing jina-reranker-m0 with jina-reranker-v2-base-multilingual, our goal for the m-series is to achieve multimodality while improving performance on text-only tasks at a level comparable to specialized text-only models. Some might question the value of using an 8x larger model if the performance improvement on text-only tasks appears marginal. While it's true for the moment that m0 may not provide substantial added value over v2 for text-only applications, the decoder-only architecture opens up many new possibilities that weren't achievable with encoder-only architectures, including:

• True mixed-modality reranking
• Listwise reranking and documents deduplication
• Ranking score explainability via attention mechanism

Our future work will focus on further upgrading the text-only reranker and fully leveraging the new features enabled by this multimodal architecture to achieve better and wider search.

tagAppendix: Evaluation

[public]-jina-reranker-m0-evaluation-results

Google Docs

tagBEIR (Text2Text, English-only)

BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

Existing neural information retrieval (IR) models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their out-of-distribution (OOD) generalization capabilities. To address this, and to facilitate researchers to broadly evaluate the effectiveness of their models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains and evaluate 10 state-of-the-art retrieval systems including lexical, sparse, dense, late-interaction and re-ranking architectures on the BEIR benchmark. Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches, highlighting the considerable room for improvement in their generalization capabilities. We hope this framework allows us to better evaluate and understand existing retrieval systems, and contributes to accelerating progress towards better robust and generalizable systems in the future. BEIR is publicly available at https://github.com/UKPLab/beir.

arXiv.orgNandan Thakur

BEIR is a heterogeneous benchmark for information retrieval, designed to evaluate the versatility and robustness of IR models. It contains a diverse set of datasets from various domains and focuses on zero-shot evaluation. Standardized evaluation metrics such as NDCG, Recall@K, and MRR are used.

tagMIRACL (Text2Text, Multilingual, 18 languages)

Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages

MIRACL (Multilingual Information Retrieval Across a Continuum of Languages) is a multilingual dataset we have built for the WSDM 2023 Cup challenge that focuses on ad hoc retrieval across 18 different languages, which collectively encompass over three billion native speakers around the world. These languages have diverse typologies, originate from many different language families, and are associated with varying amounts of available resources -- including what researchers typically characterize as high-resource as well as low-resource languages. Our dataset is designed to support the creation and evaluation of models for monolingual retrieval, where the queries and the corpora are in the same language. In total, we have gathered over 700k high-quality relevance judgments for around 77k queries over Wikipedia in these 18 languages, where all assessments have been performed by native speakers hired by our team. Our goal is to spur research that will improve retrieval across a continuum of languages, thus enhancing information access capabilities for diverse populations around the world, particularly those that have been traditionally underserved. This overview paper describes the dataset and baselines that we share with the community. The MIRACL website is live at http://miracl.ai/.

arXiv.orgXinyu Zhang

MIRACL is a large-scale multilingual dataset for ad hoc information retrieval across 18 languages. It encompasses over three billion native speakers and features thorough human annotations. The focus is on monolingual retrieval tasks.

tagMLDR (Text2Text, Multilingual Long Documents, 13 languages)

BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

In this paper, we present a new embedding model, called M3-Embedding, which is distinguished for its versatility in Multi-Linguality, Multi-Functionality, and Multi-Granularity. It can support more than 100 working languages, leading to new state-of-the-art performances on multi-lingual and cross-lingual retrieval tasks. It can simultaneously perform the three common retrieval functionalities of embedding model: dense retrieval, multi-vector retrieval, and sparse retrieval, which provides a unified model foundation for real-world IR applications. It is able to process inputs of different granularities, spanning from short sentences to long documents of up to 8192 tokens. The effective training of M3-Embedding involves the following technical contributions. We propose a novel self-knowledge distillation approach, where the relevance scores from different retrieval functionalities can be integrated as the teacher signal to enhance the training quality. We also optimize the batching strategy, enabling a large batch size and high training throughput to ensure the discriminativeness of embeddings. To the best of our knowledge, M3-Embedding is the first embedding model which realizes such a strong versatility. The model and code will be publicly available at https://github.com/FlagOpen/FlagEmbedding.

arXiv.orgJianlv Chen

MLDR is a multilingual dataset specifically designed for long-document retrieval, covering 13 languages. It utilizes GPT-3.5 to generate questions for the documents. The dataset is built on Wikipedia, Wudao and mC4.

tagMKQA (Text2Text, Multilingual Question-Answering, 24 languages, 3 variants for Chinese)

MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering

Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). Answers are based on a heavily curated, language-independent data representation, making results comparable across languages and independent of language-specific passages. With 26 languages, this dataset supplies the widest range of languages to-date for evaluating question answering. We benchmark a variety of state-of-the-art methods and baselines for generative and extractive question answering, trained on Natural Questions, in zero shot and translation settings. Results indicate this dataset is challenging even in English, but especially in low-resource languages

arXiv.orgShayne Longpre

MKQA is an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages. The question-answer pairs are sampled from Google Natural Questions.

tagCoIR (Text2Text, Code Information Retrieval)

CoIR: A Comprehensive Benchmark for Code Information Retrieval Models

Despite the substantial success of Information Retrieval (IR) in various NLP tasks, most IR systems predominantly handle queries and corpora in natural language, neglecting the domain of code retrieval. Code retrieval is critically important yet remains under-explored, with existing methods and benchmarks inadequately representing the diversity of code in various domains and tasks. Addressing this gap, we present COIR (Code Information Retrieval Benchmark), a robust and comprehensive benchmark specifically designed to assess code retrieval capabilities. COIR comprises ten meticulously curated code datasets, spanning eight distinctive retrieval tasks across seven diverse domains. We first discuss the construction of COIR and its diverse dataset composition. Further, we evaluate nine widely used retrieval models using COIR, uncovering significant difficulties in performing code retrieval tasks even with state-of-the-art systems. To facilitate easy adoption and integration within existing research workflows, COIR has been developed as a user-friendly Python framework, readily installable via pip. It shares same data schema as other popular benchmarks like MTEB and BEIR, enabling seamless cross-benchmark evaluations. Through COIR, we aim to invigorate research in the code retrieval domain, providing a versatile benchmarking tool that encourages further development and exploration of code retrieval systems https://github.com/CoIR-team/coir.

arXiv.orgXiangyang Li

CoIR is a comprehensive benchmark designed to evaluate models’ abilities in code retrieval. It includes 10 curated code datasets covering 8 retrieval tasks across 7 diverse domains. A Python framework is provided for this benchmark.

tagViDoRe (Text2Image, Visual Document Retrieval Benchmark)

ColPali: Efficient Document Retrieval with Vision Language Models

Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG). To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPali, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. We release models, data, code and benchmarks under open licenses at https://hf.co/vidore.

arXiv.orgManuel Faysse

ViDoRe is a benchmark designed to evaluate retrieval systems on their capacity to match queries to relevant documents using visual features. It covers various page-level retrieving tasks across multiple domains and languages. The benchmark focuses on visual elements of documents.

tagM-BEIR (Text2Image, Image2Text, Multimodal BEnchmark for Instructed Retrieval)

UniIR: Training and Benchmarking Universal Multimodal Information Retrievers

Existing information retrieval (IR) models often assume a homogeneous format, limiting their applicability to diverse user needs, such as searching for images with text descriptions, searching for a news article with a headline image, or finding a similar photo with a query image. To approach such different information-seeking demands, we introduce UniIR, a unified instruction-guided multimodal retriever capable of handling eight distinct retrieval tasks across modalities. UniIR, a single retrieval system jointly trained on ten diverse multimodal-IR datasets, interprets user instructions to execute various retrieval tasks, demonstrating robust performance across existing datasets and zero-shot generalization to new tasks. Our experiments highlight that multi-task training and instruction tuning are keys to UniIR’s generalization ability. Additionally, we construct the M-BEIR, a multimodal retrieval benchmark with comprehensive results, to standardize the evaluation of universal multimodal information retrieval.

arXiv.orgCong Wei

M-BEIR is a comprehensive large-scale retrieval benchmark designed to train and evaluate multimodal retrieval models. It comprises eight multimodal retrieval tasks and ten datasets from a variety of domains and sources. The benchmark focuses on instruction-following retrieval.

tagWinoground (Text2Text, Text2Image)

Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground. Given two images and two captions, the goal is to match them correctly - but crucially, both captions contain a completely identical set of words, only in a different order. The dataset was carefully hand-curated by expert annotators and is labeled with a rich set of fine-grained tags to assist in analyzing model performance. We probe a diverse range of state-of-the-art vision and language models and find that, surprisingly, none of them do much better than chance. Evidently, these models are not as skilled at visio-linguistic compositional reasoning as we might have hoped. We perform an extensive analysis to obtain insights into how future work might try to mitigate these models’ shortcomings. We aim for Winoground to serve as a useful evaluation set for advancing the state of the art and driving further progress in the field. The dataset is available at https://huggingface.co/datasets/facebook/winoground.

arXiv.orgTristan Thrush

Winoground is a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning. It uses twin captions with identical word content and employs contrastive image-caption pairs. The focus is on compositional reasoning.

Winoground evaluates vision-language models using three key metrics: Text Score, Image Score, and Group Score. The Text Score measures if a model correctly matches captions to images, while the Image Score assesses if it selects the right image for a caption. The Group Score, the most rigorous metric, requires all caption-image relationships to be correctly identified. Scores are percentages representing accuracy rates, with higher values indicating better reasoning abilities.