Embeddings

4th EMNLP BoF on embeddings, rerankers and small LMs. Sunday Oct 25, 11:00 to 12:30, Room G9-10 & G11, HUNGEXPO Budapest. Walk-ins welcome, register to give a talk.

One model, four modalities: text, image, audio, video. Best-in-class omni embeddings in 1.6B and 0.9B.

We introduce GELATO (Geometry-preserving Embeddings via Locked Aligned TOwers), a novel approach to multimodal embedding models. We build on the VLM-style architecture, in which non-text encoders are adapted to produce input for a language model, which in turn generates embeddings for all varieties of input. GELATO extends the two Jina Embeddings v5 Text models to support additional modality by adding encoders for images and audio. The backbone text embedding models and the added non-text modality encoders remain frozen. We only trained the connecting components, representing 0.35% of the total weights of the joint model. Additionally, the language model remains effectively unaltered, producing exactly the same embeddings for text inputs as the Jina Embeddings v5 Text models. The resulting jina-embeddings-v5-omni suite encodes text, image, audio, and video into a single semantic embedding space, yielding nearly equal performance to larger multimodal embedding models.

Turn any multimodal LLM into a small audio embedding model that beats CLAP with 25x less data.

A tiny transformer that fingerprints embedding models by reading raw numerical digits. No feature engineering.

Two sub-1B multilingual embeddings with best-in-class performance, available on Elastic Inference Service, Llama.cpp and MLX.