Segmenter

Free API for segmenting long text into chunks and tokenization.

Segmenter API

Our Segmenter API is crucial for helping LLMs manage input within context limits, and optimizing model performance. It allows developers to count tokens and extract relevant text segments, ensuring efficient data processing and cost management.


Use a GET request to count tokens

Use a POST request for more features

Request
POST
curl "https://api.jina.ai/v1/segment" \
-H "Content-Type: application/json" \
-d @- <<EOFEOF
{ "content": "Jina AI: Your Search Foundation, Supercharged! 🚀\nIhrer Suchgrundlage, aufgeladen! 🚀\n您的搜索底座,从此不同!🚀\n検索ベース,もう二度と同じことはありません!🚀" }
EOFEOF


What is a Segmenter?

A segmenter is a crucial component that converts text into tokens or chunks, which are the basic units of data that an embedding/reranker model or LLM processes. Tokens can represent whole words, parts of words, or even individual characters.

Chunking long documents, lightning fast!

You can also use the Segmenter API to cut long documents into smaller chunks, making it easier to process them with embeddings or rerankers. We leverage common structural cues and a set of rules and heuristics that perform well across diverse content types, e.g. Markdown, HTML, LaTeX, and CJK languages.
Maximum length of each chunk: 1000
Maximum number of characters in each chunk. In practice, a chunk can be shorter than this value if there's a good boundary in the text.

0 chunks in total


Segmenter API is free!

By providing your API key, you can access a higher rate limit, and your key won't be charged.
Rate limit
Rate limits are tracked in two ways: RPM (requests per minute) and TPM (tokens per minute). Limits are enforced per IP/API key and will be triggered when either the RPM or TPM threshold is reached first. When you provide an API key in the request header, we track rate limits by key rather than IP address.
ProductAPI EndpointDescriptionw/o API Keyw/ Free API Keyw/ Paid API Keyw/ Premium API KeyAverage latencyToken Usage CountingAllowed Request
Reader APIhttps://r.jina.aiConverts a URL to LLM-friendly text20 RPM500 RPM500 RPM5000 RPM7.9sCount the number of tokens in the output response.GET/POST
Reader APIhttps://s.jina.aiSearch the web and convert results to LLM-friendly text100 RPM100 RPM1000 RPM2.5sEvery request costs a fixed number of tokens, starting from 10,000 tokensGET/POST
Embedding APIhttps://api.jina.ai/v1/embeddingsConvert text/images to fixed-length vectors100 RPM & 100,000 TPM500 RPM & 2,000,000 TPM5,000 RPM & 50,000,000 TPM
depends on the input size
Count the number of tokens in the input request.POST
Reranker APIhttps://api.jina.ai/v1/rerankRank documents by query100 RPM & 100,000 TPM500 RPM & 2,000,000 TPM5,000 RPM & 50,000,000 TPM
depends on the input size
Count the number of tokens in the input request.POST
Rate limit
Rate limits are tracked in two ways: RPM (requests per minute) and TPM (tokens per minute). Limits are enforced per IP/API key and will be triggered when either the RPM or TPM threshold is reached first. When you provide an API key in the request header, we track rate limits by key rather than IP address.
ProductAPI EndpointDescriptionw/o API Keyw/ Free API Keyw/ Paid API Keyw/ Premium API KeyAverage latencyToken Usage CountingAllowed Request
Reader APIhttps://r.jina.aiConverts a URL to LLM-friendly text20 RPM500 RPM500 RPM5000 RPM7.9sCount the number of tokens in the output response.GET/POST
Reader APIhttps://s.jina.aiSearch the web and convert results to LLM-friendly text100 RPM100 RPM1000 RPM2.5sEvery request costs a fixed number of tokens, starting from 10,000 tokensGET/POST
Embedding APIhttps://api.jina.ai/v1/embeddingsConvert text/images to fixed-length vectors100 RPM & 100,000 TPM500 RPM & 2,000,000 TPM5,000 RPM & 50,000,000 TPM
depends on the input size
Count the number of tokens in the input request.POST
Reranker APIhttps://api.jina.ai/v1/rerankRank documents by query100 RPM & 100,000 TPM500 RPM & 2,000,000 TPM5,000 RPM & 50,000,000 TPM
depends on the input size
Count the number of tokens in the input request.POST
Billing-related common questions