jina-clip-v1

Contrastive Language-Image Pretraining (CLIP) is a highly effective method for aligning images and texts in a shared embedding space. These models are widely used for tasks such as cross-modal information retrieval and multi-modal understanding. However, CLIP models often struggle with text-only tasks, underperforming compared to specialized text models. This performance disparity forces retrieval systems to rely on separate models for text-only and multi-modal tasks. In this work, we build upon our previous model, jina-clip-v1, by introducing a refined framework that utilizes multi-task, multi-stage contrastive learning across multiple languages, coupled with an improved training recipe to enhance text-only retrieval. The resulting model, jina-clip-v2, outperforms its predecessor on text-only and multimodal tasks, while adding multilingual support, better understanding of complex visual documents and efficiency gains thanks to Matryoshka Representation Learning and vector truncation. The model performs comparably to the state-of-the-art in both multilingual-multimodal and multilingual text retrieval benchmarks, addressing the challenge of unifying text-only and multi-modal retrieval systems.

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks. However, CLIP models generally underperform in text-only tasks compared to specialized text models. This creates inefficiencies for information retrieval systems that keep separate embeddings and models for text-only and multimodal tasks. We propose a novel, multi-task contrastive training method to address this issue, which we use to train the jina-clip-v1 model to achieve the state-of-the-art performance on both text-image and text-text retrieval tasks.

Jina Embeddings v4 ist ein universelles 向量模型 (Embeddings)-Modell mit 3,8 Milliarden Parametern für multimodale und mehrsprachige Suche, das sowohl Single-Vektor- als auch Multi-Vektor-Embedding-Ausgaben unterstützt.

Wir stellen jina-reranker-m0 vor, unseren neuen mehrsprachigen multimodalen Reranker für das Abrufen visueller Dokumente, mit SOTA-Performance bei mehrsprachigen langen Dokumenten und Code-Suchaufgaben.

Bessere Ergebnisse skalieren mit Rechenleistung—mehr beim Lernen, mehr bei der Suche. Ein gutes vortrainiertes Modell bringt einen weit, aber Rechenleistung zur Testzeit bringt einen noch weiter. Es ist wichtig, dieses neue Paradigma der Skalierung von Rechenleistung zur Testzeit zu erkennen, selbst bei Embedding-Modellen.