jina-reranker-v3.5: Faster Listwise Reranking with Hybrid Attention and Self-Distillation

jina-reranker-v3.5: An Efficient Listwise Reranker with Hybrid Attention and Self-Distillation

Listwise rerankers are the discriminative core of agentic retrieval pipelines, yet production deployment demands efficiency, domain robustness, and fluency on semi-structured data at the same time. We present jina-reranker-v3.5, a 0.6B-parameter listwise reranker that meets these demands together without sacrificing the cross-document comparison that makes its predecessor jina-reranker-v3 effective. jina-reranker-v3.5 keeps the last-but-not-late (LBNL) interaction of jina-reranker-v3 and reworks it along three axes. It replaces uniform global attention with a hybrid schedule of three sliding-window layers followed by two global layers, pinning the terminal layer to global as LBNL readout requires. It trains on a curated multi-domain mixture that spans legal, medical, financial, multilingual, and structured retrieval. It transfers quality through a three-stage self-distillation recipe in which a full-attention teacher sets an upper bound that a sparse-attention student then recovers under a staged adaptation protocol. jina-reranker-v3.5 reaches 63.20 nDCG@10 on BEIR, matching a 4B model at roughly 7x fewer parameters, and improves over jina-reranker-v3 on MIRACL and RTEB as well. Its largest gains come on semi-structured retrieval, where it lifts nDCG@10 by 9.6 points over jina-reranker-v3 and leads all rerankers of comparable size. The hybrid schedule further cuts listwise inference latency by up to 1.56x. We release the model weights on Hugging Face under a non-commercial license.

arXiv.orgChristina Nasika

Today we release jina-reranker-v3.5, a 0.6B-parameter listwise reranker that keeps the last but not late interaction of jina-reranker-v3 and makes it faster and far more capable on the data enterprises actually search. It reaches 63.20 nDCG@10 on BEIR, ahead of Qwen3-Reranker-4B with roughly 7× fewer parameters, and it reranks up to 1.56× faster than v3 on long documents. Its biggest jump is on semi-structured retrieval: +9.6 nDCG@10 over v3 on field-constrained records.

Three changes get us there. A hybrid attention schedule that replaces most global layers with sliding windows while pinning the terminal layer to global. A training mixture curated from the failure modes of legal, medical, financial, multilingual, and structured retrieval. And a three-stage self-distillation recipe where teacher and student are the same size and differ only in attention pattern.

Quality against parameter count on the four benchmark regimes. jina-reranker-v3.5 is on the Pareto front in all four: no model we evaluated is both smaller and better.

Scatter plot of nDCG@10 against parameter count on BEIR with a Pareto frontier line — English zero-shot retrieval on BEIR across 13 datasets, measured by nDCG@10 after reranking the top-100 candidates from jina-embeddings-v5-text-small. The red line traces the Pareto front and the shaded region is dominated: for every model inside it, some other model is smaller, better, or both. jina-reranker-v3.5 defines the front outright at 63.20, above the 1.5B `mxbai-rerank-large-v2` at 62.45 and the 4B `Qwen3-Reranker-4B` at 62.28. No larger model we evaluated buys any BEIR quality over it.

Scatter plot of nDCG@10 against parameter count on MIRACL with a Pareto frontier line — Multilingual retrieval on MIRACL across 18 languages, same top-100 reranking protocol. jina-reranker-v3.5 holds the 0.6B corner of the front at 74.11, the best score of any compact model, while `Qwen3-Reranker-4B` holds the 4B corner at 76.56. The largest per-language gains over jina-reranker-v3 land on Yoruba (+4.4), Farsi (+3.1) and French (+3.0).

Scatter plot of nDCG@10 against parameter count on RTEB with a Pareto frontier line — Professional-domain retrieval on RTEB, covering legal, finance, programming and medical corpora. jina-reranker-v3.5 reaches 70.95, passing both the same-size `Qwen3-Reranker-0.6B` at 68.41 and the 1.5B `mxbai-rerank-large-v2` at 70.81 while running at 0.6B. The remaining distance to `Qwen3-Reranker-4B` at 77.68 concentrates in a handful of legal and medical tasks.

Scatter plot of nDCG@10 against parameter count on Struct-IR with a Pareto frontier line — Semi-structured retrieval on Struct-IR under a controlled candidate pool, where all gold documents are injected alongside the 30 hardest first-stage distractors so the measurement isolates field-constrained discrimination from retrieval coverage. jina-reranker-v3.5 reaches 48.3, a 9.6-point lift over jina-reranker-v3 and the largest single improvement in this release. Note this protocol is not comparable to the SSRB retrieval leaderboard.

tagArchitecture

Listwise reranking scores every candidate in a single forward pass, so full self-attention over a 100-document list grows quadratically and inflates the KV cache. The obvious fix is sliding-window attention. Under LBNL interaction, the obvious fix breaks the model.

In LBNL the query and all candidates form one causal sequence, and the query embedding token sits at the very end. It has to attend all the way back to the first candidate to build a cross-document-aware representation. A finite window severs that dependency. We therefore pin the final layer to global at all times, so query and document embedding tokens observe the entire candidate context at the point of extraction. Replacing that one layer with a sliding window degrades listwise ranking severely; keeping only it global preserves joint encoding without a fully global stack.

For the remaining 27 layers we searched 1L1G, 1L2G, 3L2G and 5L1G schedules. 3L2G won: three sliding-window layers followed by two global layers, repeated, giving 17 local and 11 global layers with a window of 1,024 tokens. Local layers drop attention cost from O(L²) to O(L·w), while global layers every three steps refresh long-range cross-candidate signal at every depth. Denser schedules bought no throughput; the more aggressive 5L1G trended worse on complex multi-document tasks.

Architecture diagram showing LBNL listwise encoding with a 3L2G hybrid backbone and the three-stage self-distillation pipeline — Left: LBNL listwise encoding on a 3L2G hybrid Qwen3-0.6B backbone. Query and all candidates share one causal context; L layers use a 1,024-token sliding window, G layers are global, and the terminal layer G* is pinned global so the trailing query embedding can see the whole list. An MLP projects to a shared space and cosine similarity produces the ranking. Right: the three-stage recipe, where the Stage I full-attention teacher stays frozen and guides Stage III.

tagSelf-distillation across an attention gap

The distillation here is unusual. We do not shrink a large model into a small one. Teacher and student are both 0.6B and differ only in attention pattern. The teacher runs full attention at quadratic cost; the student runs 3L2G.

Forcing a student to switch attention masks and match teacher outputs at the same time makes it fail at both, so the recipe decouples the two pressures:

Stage I — full-attention teacher. Starting from the public jina-reranker-v3 checkpoint, we fully fine-tune a teacher with no sliding-window restriction on the complete v3.5 mixture. It establishes the quality ceiling at this parameter budget.
Stage II — sparse-attention adaptation. The student initializes from Stage I weights and activates 3L2G. First we train only the attention projections with everything else frozen, teaching the sparse masks to route information without disturbing representations learned under full attention. Then we unfreeze everything so the student realigns to the new attention geometry. The student is already deployable and faster here, but a consistent gap to the teacher remains on BEIR, RTEB-legal and MIRACL.
Stage III — teacher-guided distillation. With the teacher frozen, we align the student across four levels at once: a listwise KL over softmax-normalized score distributions, an MSE on absolute scores, an MSE on last-layer hidden states, and a cosine loss on projected embeddings, plus in-context similarity and dispersion regularizers.

Order matters. Stage II alone leaves a noticeable gap because the student must change its routing under a weaker mask before it can safely mimic teacher scores and states. Stage III then recovers most of that gap, which says full-attention capacity does transfer into a sparse student once the geometry has adapted. The recipe is written for 3L2G but the outline generalizes to other attention-schedule mismatches.

tagTraining data

The v3 mixture covered general retrieval well and specialized domains badly. Rather than adding more data, we ran error analysis on RTEB and STARK development sets and built each new shard to cover exactly the retrieval patterns general models fail on. Hard negatives come from several retrievers at once (BM25, Jina, BGE, GTE, E5, ColBERT) so the model cannot learn a single retriever's shortcuts.

The legal shard combines EUR-Lex multilingual, CLERC, AILA, Canadian case law, Swiss case summarization and EuroVoc, oversampled because legal text is citation-dense and long. The medical shard targets clinical wording and entity-heavy passages. The finance shard prioritizes numeric claims, regulatory language and table-aware passages. Multilingual coverage extends past MIRACL and mMARCO with WebFAQ in 50+ languages, SWIM-IR cross-lingual negatives and Ruri-v3 Japanese pairs.

Structured data got the highest sampling weight, because it sits outside the free-text distribution that standard benchmarks assume. Relevance over records and tables hinges on equality, numeric and date bounds, list membership and logical combinations across fields, not on lexical overlap. Early runs lagged worst here, so we synthesized constraint-heavy pairs directly.

Pipeline diagram for generating constraint-heavy synthetic training data for structured retrieval — Synthetic supervision for field-constrained retrieval. We sample typed constraints from an anchor record and have an LLM paraphrase them into a query, then perturb one or two constrained fields to build a near-duplicate hard negative that keeps the same surface form but violates a constraint. Dense mining adds further candidates and an LLM judge promotes true matches, refines over-broad queries and discards ambiguous cases, feeding back into query generation. The example on the right shows why lexical overlap is useless here: positive and hard negative are identical strings apart from a single field.

tagExperimental results

All numbers rerank the top-100 candidates from jina-embeddings-v5-text-small under one unified MTEB v2 pipeline, so they may differ slightly from vendor-reported figures. Full per-dataset and per-language tables are in the paper.

Model	Params	BEIR	MIRACL	RTEB	Struct-IR
jina-embeddings-v5-text-small (1st stage)	0.5B	56.26	65.15	64.60	–
mxbai-rerank-base-v2	0.5B	59.58	64.90	61.44	30.4
Qwen3-Reranker-0.6B	0.6B	56.94	67.12	68.41	41.9
mxbai-rerank-large-v2	1.5B	62.45	69.65	70.81	43.0
Qwen3-Reranker-4B	4.0B	62.28	76.56	77.68	55.6
jina-reranker-v3	0.6B	62.10	72.20	68.01	38.7
<strong>jina-reranker-v3.5</strong>	0.6B	63.20	74.11	70.95	48.3

v3.5 improves on v3 across every benchmark family at the same parameter count, with the largest gain on semi-structured retrieval.

The RTEB gains concentrate where the mixture was aimed: AILA-Statute improves by 14.0 points and AILA-Case by 11.7 over v3, and FinQA reaches 86.91, the best of any model tested. On STARK, v3.5 improves on v3 across all three official metrics and takes the best Hit@5 overall.

One protocol note on Struct-IR. The benchmark indexes millions of objects per schema and first-stage in-schema Recall@5 is around 0.04, so end-to-end retrieve-then-rerank is almost entirely recall-bound and separates rerankers poorly. We instead inject all gold documents alongside the 30 hardest first-stage distractors, which isolates field-constrained discrimination from retrieval coverage. Numbers under that pool are not comparable to the SSRB retrieval leaderboard.

tagEfficiency

Measured on a single NVIDIA A100, batch size 1, top-100 listwise inputs, FlashAttention-2.

Bar chart comparing mean listwise reranking latency of v3 and v3.5 on BEIR Natural Questions — Short-context regime, BEIR Natural Questions, with mean query and document lengths of 10.3 and 145.5 tokens over 254 timed queries. Mean latency for top-100 listwise reranking drops from 371 ms to 305 ms, a 1.22× speedup, and document throughput rises from 270 to 328 docs/s.

Bar chart comparing mean listwise reranking latency of v3 and v3.5 on RTEB AILACasedocs — Long-context regime, RTEB AILACasedocs, with mean query and document lengths of 689.8 and 1,904.0 tokens over 48 timed queries. Mean latency drops from 16.1 s to 10.3 s, a 1.56× speedup, and prefill throughput rises from 11.9k to 18.6k tokens/s. Hybrid attention pays off most when candidate passages are long.

Because production listwise reranking is dominated by a single prefill over a fresh candidate set, these reductions translate directly into longer candidate lists and larger documents at a fixed serving budget.

tagGetting started

tagVia the Jina Search Foundation API

curl -X POST \
  https://api.jina.ai/v1/rerank \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ***" \
  -d '{
  "model": "jina-reranker-v3.5",
  "query": "trail-running shoes under $150, rated 4.5+, released since 2022",
  "documents": [
    "...",
    "..."
  ],
  "return_documents": false
}'

tagVia Elastic Inference Service

jina-reranker-v3.5 is available on Elastic Inference Service (EIS) from Stack 9.3, so you can rerank on managed GPUs without hosting the model yourself. Create an inference endpoint that references the model, then call it like any other rerank task.

PUT _inference/rerank/eis-jina-reranker-v3-5
{
  "service": "elastic",
  "service_settings": {
    "model_id": "jina-reranker-v3.5"
  }
}

POST _inference/rerank/eis-jina-reranker-v3-5
{
  "query": "trail-running shoes under $150, rated 4.5+, released since 2022",
  "input": [
    "...",
    "..."
  ]
}

The response returns a rerank array ordered by relevance, each entry carrying the original index of the candidate and its score. Because it is a standard inference endpoint, the inference_id can be referenced directly from a text_similarity_reranker retriever in a search query.

tagVia transformers

from transformers import AutoModel

model = AutoModel.from_pretrained(
    'jinaai/jina-reranker-v3.5',
    dtype="auto",
    trust_remote_code=True,
)
model.eval()

query = "What are the health benefits of green tea?"
documents = [
    "Green tea contains catechins that may help reduce inflammation.",
    "El precio del cafe ha aumentado un 20% este ano.",
    "绿茶富含儿茶素等抗氧化剂，可以降低心脏病风险。",
    "Le the vert est riche en antioxydants.",
]

for r in model.rerank(query, documents):
    print(f"{r['relevance_score']:.4f}  {r['document'][:70]}")

tagConclusion

The practical claim of jina-reranker-v3.5 is narrow and testable: at 0.6B parameters, targeted training closes most of the gap to a 4B generalist reranker, and on BEIR it closes all of it, while running faster than the model it replaces. For enterprise retrieval that argues for investing in focused supervision on a compact backbone rather than defaulting to the largest model available.

Two limitations are worth stating plainly. Listwise rerankers still carry input constraints that pointwise and late-interaction models avoid, in particular fixed upper bounds on candidate count and total candidate length. And important gaps to the 4B Qwen remain on RTEB legal and medical tasks, controlled-pool Struct-IR, and low-resource MIRACL languages. Those are the hard cases, and we name them rather than hide them behind an average.