jina-code-embeddings-0.5b

jina-code-embeddings-0.5b is a 494M-parameter code embedding model that maps programming code and natural language queries into a shared 896-dimensional semantic space. It supports 32K token context windows, enabling whole-function and multi-file retrieval, and uses Matryoshka representation learning for flexible dimensionality (64–896). It achieves state-of-the-art code retrieval performance at its size, outperforming models with 3× the parameter count.

The model is built on an autoregressive transformer backbone pretrained on both natural language and code corpora. Instead of the bidirectional pooling used by traditional embedding models, it extracts embeddings via Last-Token-Pooling from the final position of the sequence. This is a key architectural insight: the autoregressive model's causal attention naturally attends to the full input context, and the code-generation pretraining objective produces rich semantic representations that transfer well to retrieval. Training uses a two-stage recipe: (1) contrastive pretraining on large-scale code-text pairs (natural language descriptions paired with code implementations), (2) supervised fine-tuning on curated code retrieval datasets with hard-negative mining. The 32K context length is enabled through rotary position embeddings with a tuned base frequency.

The model achieves a 78.41% overall average and 78.72% MTEB Code average across standard code retrieval benchmarks. Notable scores: 96.77% on HumanEval, 89.01% on MBPP, 98.31% on WikiSQL, 99.70% on CodeChefXLang, 90.37% on CodeTransOceanContest (code-to-code), 85.73% on COIR-CodeSearchNet (NL2Code), 95.98% on Doc2Code, and 91.04% on StackOverflowQA. It outperforms Qwen3-Embedding-0.6B and larger models including jina-embeddings-v4 (74.11%) and gemini-embedding-001 (77.38%) on code-specific tasks. At 494M parameters, it outperforms several models 3–5× its size, demonstrating that the autoregressive backbone approach yields high semantic quality per parameter.

Always use appropriate task-specific instruction prefixes: nl2code for natural-language-to-code search, code2code for code-to-code similarity, code2nl for code-to-natural-language, techqa for technical Q&A, and code2completion for code completion. For large codebases, implement chunking at function or class boundaries to stay within the 32K token limit. Matryoshka truncation to 128 or 256 dimensions is suitable for high-throughput indexing; use the full 896 dimensions when re-ranking top candidates. Use cosine similarity for embedding comparison. Optimal batch size is 512, sequence length 512 tokens. The model is optimized for Python, JavaScript, Java, PHP, Go, and Ruby but supports 30+ languages. For RAG pipelines over code, pair with jina-reranker-v3.5 for the second stage to maximize precision on the top-50 candidates.