jina-vlm

jina-vlm is a 2.4B-parameter multilingual vision-language model achieving state-of-the-art VQA performance among open 2B-scale VLMs. It couples a SigLIP2 vision encoder with a Qwen3 language decoder through an attention-pooling connector that enables token-efficient processing of arbitrary-resolution images. The model supports 29 languages and a 32K token context, making it suitable for document understanding, chart analysis, OCR, and multilingual visual question answering.

The architecture combines a SigLIP2 vision encoder with a Qwen3 language decoder, connected by an attention-pooling mechanism that enables token-efficient processing of arbitrary-resolution images. The key innovation is the attention-pooling connector, which applies 2×2 pooling to reduce 729 visual tokens per tile to 182 (a 4× token reduction), yielding 3.9× reduction in LLM prefill FLOPs and 4× reduction in KV-cache memory with minimal impact on benchmark scores. Images are processed via tiling: arbitrary-resolution images are divided into up to 12 tiles plus a thumbnail, each processed independently and then combined. The model was trained on a diverse multilingual VQA dataset covering 29 languages (Arabic, Chinese, English, Portuguese, Russian, Turkish, and others). A leave-one-out data mixture ablation study diagnosed which data categories (task, domain, modality, language) are necessary versus redundant, guiding efficient data allocation.

The model achieves the highest average score (72.3) across eight VQA benchmarks among 2B-scale VLMs: MathVista (59.4), AI2D (80.8), ChartQA (79.5), DocVQA (90.6), InfoVQA (65.9), RealWorldQA (64.9), OCRBench (778/1000), and MME (1582). It leads on multilingual multimodal understanding: MMMB (78.8) and Multilingual MMBench (74.3), covering Arabic, Chinese, English, Portuguese, Russian, and Turkish. OCR performance is strong at 778 on OCRBench (0–1000 scale). Text-only performance is competitive: MMLU (54.7), HellaSwag (75.6), though degraded on MMLU-Pro (30.3 vs. 46.4 base) due to vision-language integration. The 4× token reduction from attention pooling yields 3.9× reduction in LLM prefill FLOPs and 4× reduction in KV-cache memory with minimal benchmark impact.

The model is available on Hugging Face under CC-BY-NC-4.0 with weights and inference code. Supports images of arbitrary resolution through automatic tiling (up to 12 tiles plus thumbnail). Use thinking mode by enabling do_sample=True and temperature > 0 for complex reasoning tasks. The model handles 32K context length for extended conversations. For multilingual VQA, it supports 29 languages including English, Chinese, Arabic, German, Spanish, French, Italian, Japanese, Korean, Portuguese, Russian, Turkish, Vietnamese, Thai, Indonesian, Hindi, and Bengali. Best suited for document understanding, chart/diagram analysis, OCR tasks, and multilingual visual question answering. Limitations: counting tasks and fine-grained spatial reasoning may be affected by the tiling approach. For optimal inference, use bfloat16 precision on CUDA-capable GPUs. MLX inference is supported for Apple Silicon. For embedding-based retrieval over visual documents, pair with jina-embeddings-v4 or jina-reranker-m0.