News
Models
Products
keyboard_arrow_down
DeepSearch
Search, read and reason until best answer found.
Reader
Convert any URL to Markdown for better grounding LLMs.
Embeddings
World-class multimodal multilingual embeddings.
Reranker
World-class reranker for maximizing search relevancy.
Classifier
Zero-shot and few-shot classification for image and text.
Segmenter
Cut long text into chunks and do tokenization.
API Docs
Auto codegen for your copilot IDE or LLM
open_in_new
Company
keyboard_arrow_down
About us
Contact sales
Intern program
Join us
open_in_new
Download logo
open_in_new
Terms & Conditions
Log in
login

Segmenter

Free API for segmenting long text into chunks and tokenization.

Segmenter API

Our Segmenter API is crucial for helping LLMs manage input within context limits, and optimizing model performance. It allows developers to count tokens and extract relevant text segments, ensuring efficient data processing and cost management.
key
API Key & Billing
code
Usage
more_horiz
More
chevron_leftchevron_right
home
speedRate Limit
help_outlineFAQ
api
Status
chevron_leftchevron_right
Use GET request to count tokens
Use POST request for more features
upload
Request
curl -X POST 'https://api.jina.ai/v1/segment' \
  -H "Content-Type: application/json" \
  -d @- <<EOFEOF
  {
    "content": "\n  Jina AI: Your Search Foundation, Supercharged! 🚀\n  Ihrer Suchgrundlage, aufgeladen! 🚀\n  您的搜索底座，从此不同！🚀\n  検索ベース,もう二度と同じことはありません！🚀\n"
  }
EOFEOF

What is a Segmenter?

A segmenter is a crucial component that converts text into tokens or chunks, which are the basic units of data that an embedding/reranker model or LLM processes. Tokens can represent whole words, parts of words, or even individual characters.

Chunking long documents, lightning fast!

You can also use Segmenter API to cut long documents into smaller chunks, making it easier to process them in embeddings or rerankers. We leverage common structural cues and build a set of rules and heuristics which perform well across diverse types of content, e.g. Markdown, HTML, LaTeX and CJK languages.
Maximum length of each chunk: 1000
Maximum number of characters in each chunk. In practice the chunk length can be smaller than this value, if there is a good boundary in the text.
0 chunks in total
Segmenter API is free!
By providing your API key, you can access a higher rate limit, and your key won't be charged.
Rate Limit
Rate limits are tracked in three ways: RPM (requests per minute), and TPM (tokens per minute). Limits are enforced per IP/API key and will be triggered when either the RPM or TPM threshold is reached first. When you provide an API key in the request header, we track rate limits by key rather than IP address.
ProductAPI EndpointDescriptionarrow_upwardw/o API Keykey_offw/ API Keykeyw/ Premium API KeykeyAverage LatencyToken Usage CountingAllowed Request
Reader APIhttps://r.jina.aiConvert URL to LLM-friendly text20 RPM500 RPMtrending_up5000 RPM7.9sCount the number of tokens in the output response.GET/POST
Reader APIhttps://s.jina.aiSearch the web and convert results to LLM-friendly textblock100 RPMtrending_up1000 RPM2.5sEvery request costs a fixed number of tokens, starting from 10000 tokensGET/POST
DeepSearchhttps://deepsearch.jina.ai/v1/chat/completionsReason, search and iterate to find the best answer0.5 RPM50 RPM500 RPM56.7sCount the total number of tokens in the whole process.POST
Embedding APIhttps://api.jina.ai/v1/embeddingsConvert text/images to fixed-length vectorsblock500 RPM & 1,000,000 TPMtrending_up2,000 RPM & 5,000,000 TPM
ssid_chart
depends on the input size
 help
Count the number of tokens in the input request.POST
Reranker APIhttps://api.jina.ai/v1/rerankRank documents by queryblock500 RPM & 1,000,000 TPMtrending_up2,000 RPM & 5,000,000 TPM
ssid_chart
depends on the input size
 help
Count the number of tokens in the input request.POST
Classifier APIhttps://api.jina.ai/v1/trainTrain a classifier using labeled examplesblock20 RPM & 200,000 TPM60 RPM & 1,000,000 TPM
ssid_chart
depends on the input size
 
Tokens counted as: input_tokens × num_itersPOST
Classifier API (Few-shot)https://api.jina.ai/v1/classifyClassify inputs using a trained few-shot classifierblock20 RPM & 200,000 TPM60 RPM & 1,000,000 TPM
ssid_chart
depends on the input size
 
Tokens counted as: input_tokensPOST
Classifier API (Zero-shot)https://api.jina.ai/v1/classifyClassify inputs using zero-shot classificationblock200 RPM & 500,000 TPM1,000 RPM & 3,000,000 TPM
ssid_chart
depends on the input size
 
Tokens counted as: input_tokens + label_tokensPOST
Segmenter APIhttps://api.jina.ai/v1/segmentTokenize and segment long text20 RPM200 RPM1,000 RPM0.3sToken is not counted as usage.GET/POST
keyGet your API key
handshakeContact sales
FAQ
How much does the Segmenter API cost?
keyboard_arrow_down
The Segmenter API is free to use. By providing your API key, you can access a higher rate limit, and your key won't be charged.
If I don't provide an API key, what is the rate limit?
keyboard_arrow_down
Without an API key, you can access the Segmenter API at a rate limit of 20 RPM.
If I provide an API key, what is the rate limit?
keyboard_arrow_down
With an API key, you can access the Segmenter API at a rate limit of 200 RPM. For premium paid users, the rate limit is 1000 RPM.
Will you charge the tokens from my API key?
keyboard_arrow_down
No, your API key is only used to access a higher rate limit.
Does the Segmenter API support multiple languages?
keyboard_arrow_down
Yes, the Segmenter API is multilingual and supports over 100 languages.
What is the difference between GET and POST requests?
keyboard_arrow_down
GET requests are solely used to count the number of tokens in a text, allows you easily integrate it as a counter in your application. POST requests supports more parameters and features, such as returning the first/last N tokens.
What is the maximum length I can tokenize per request?
keyboard_arrow_down
You can send up to 64k characters per request.
How does the chunking feature work? Is it semantic chunking?
keyboard_arrow_down
The chunking feature segments long documents into smaller chunks based on common structural cues, ensuring accurate segmentation of text into meaningful chunks. Essentially it is a (big!) regex pattern that segments text based on certain syntactical features that often align with semantic boundaries, such as sentence endings, paragraph breaks, punctuation, and certain conjunctions. It is not semantic chunking. This (big) regex is as powerful as it can be within the limitations of regular expressions. It balances complexity and performance. While true semantic understanding isn't possible with regex, it well-approximates context by common structural cues.
How do you handle special tokens such as 'endoftext' in the Segmenter API?
keyboard_arrow_down
If the input contains special tokens, our Segmenter API will put them in the field 'special_tokens'. This allows you to easily identify them and handle them accordingly for your downstream tasks, e.g. removing them before feeding the text into an LLM to prevent injection attacks.
Does chunking support other languages than English?
keyboard_arrow_down
Besides western languages, chunking also works well with Chinese, Japanese, and Korean.
How to get my API key?
 video_not_supported
What's the rate limit?
Rate Limit
Rate limits are tracked in three ways: RPM (requests per minute), and TPM (tokens per minute). Limits are enforced per IP/API key and will be triggered when either the RPM or TPM threshold is reached first. When you provide an API key in the request header, we track rate limits by key rather than IP address.
ProductAPI EndpointDescriptionarrow_upwardw/o API Keykey_offw/ API Keykeyw/ Premium API KeykeyAverage LatencyToken Usage CountingAllowed Request
Reader APIhttps://r.jina.aiConvert URL to LLM-friendly text20 RPM500 RPMtrending_up5000 RPM7.9sCount the number of tokens in the output response.GET/POST
Reader APIhttps://s.jina.aiSearch the web and convert results to LLM-friendly textblock100 RPMtrending_up1000 RPM2.5sEvery request costs a fixed number of tokens, starting from 10000 tokensGET/POST
DeepSearchhttps://deepsearch.jina.ai/v1/chat/completionsReason, search and iterate to find the best answer0.5 RPM50 RPM500 RPM56.7sCount the total number of tokens in the whole process.POST
Embedding APIhttps://api.jina.ai/v1/embeddingsConvert text/images to fixed-length vectorsblock500 RPM & 1,000,000 TPMtrending_up2,000 RPM & 5,000,000 TPM
ssid_chart
depends on the input size
 help
Count the number of tokens in the input request.POST
Reranker APIhttps://api.jina.ai/v1/rerankRank documents by queryblock500 RPM & 1,000,000 TPMtrending_up2,000 RPM & 5,000,000 TPM
ssid_chart
depends on the input size
 help
Count the number of tokens in the input request.POST
Classifier APIhttps://api.jina.ai/v1/trainTrain a classifier using labeled examplesblock20 RPM & 200,000 TPM60 RPM & 1,000,000 TPM
ssid_chart
depends on the input size
 
Tokens counted as: input_tokens × num_itersPOST
Classifier API (Few-shot)https://api.jina.ai/v1/classifyClassify inputs using a trained few-shot classifierblock20 RPM & 200,000 TPM60 RPM & 1,000,000 TPM
ssid_chart
depends on the input size
 
Tokens counted as: input_tokensPOST
Classifier API (Zero-shot)https://api.jina.ai/v1/classifyClassify inputs using zero-shot classificationblock200 RPM & 500,000 TPM1,000 RPM & 3,000,000 TPM
ssid_chart
depends on the input size
 
Tokens counted as: input_tokens + label_tokensPOST
Segmenter APIhttps://api.jina.ai/v1/segmentTokenize and segment long text20 RPM200 RPM1,000 RPM0.3sToken is not counted as usage.GET/POST
API-related common questions
code
Can I use the same API key for reader, embedding, reranking, classifying and fine-tuning APIs?
keyboard_arrow_down
Yes, the same API key is valid for all search foundation products from Jina AI. This includes the reader, embedding, reranking, classifying and fine-tuning APIs, with tokens shared between the all services.
code
Can I monitor the token usage of my API key?
keyboard_arrow_down
Yes, token usage can be monitored in the 'API Key & Billing' tab by entering your API key, allowing you to view the recent usage history and remaining tokens. If you have logged in to the API dashboard, these details can also be viewed in the 'Manage API Key' tab.
code
What should I do if I forget my API key?
keyboard_arrow_down
If you have misplaced a topped-up key and wish to retrieve it, please contact support AT jina.ai with your registered email for assistance. It's recommended to log in to keep your API key securely stored and easily accessible.
Contact
code
Do API keys expire?
keyboard_arrow_down
No, our API keys do not have an expiration date. However, if you suspect your key has been compromised and wish to retire it, please contact our support team for assistance. You can also revoke your key in the API Key Management dashboard.
Contact
code
Can I transfer tokens between API keys?
keyboard_arrow_down
Yes, you can transfer tokens from a premium key to another. After logging into your account on the API Key Management dashboard, use the settings of the key you want to transfer out to move all remaining paid tokens.
code
Can I revoke my API key?
keyboard_arrow_down
Yes, you can revoke your API key if you believe it has been compromised. Revoking a key will immediately disable it for all users who have stored it, and all remaining balance and associated properties will be permanently unusable. If the key is a premium key, you have the option to transfer the remaining paid balance to another key before revocation. Notice that this action cannot be undone. To revoke a key, go to the key settings in the API Key Management dashboard.
code
Why is the first request for some models slow?
keyboard_arrow_down
This is because our serverless architecture offloads certain models during periods of low usage. The initial request activates or 'warms up' the model, which may take a few seconds. After this initial activation, subsequent requests process much more quickly.
code
Is user input data used for training your models?
keyboard_arrow_down
We adhere to a strict privacy policy and do not use user input data for training our models. We are also SOC 2 Type I and Type II compliant, ensuring high standards of security and privacy.
Billing-related common questions
attach_money
Is billing based on the number of sentences or requests?
keyboard_arrow_down
Our pricing model is based on the total number of tokens processed, allowing users the flexibility to allocate these tokens across any number of sentences, offering a cost-effective solution for diverse text analysis requirements.
attach_money
Is there a free trial available for new users?
keyboard_arrow_down
We offer a welcoming free trial to new users, which includes ten millions tokens for use with any of our models, facilitated by an auto-generated API key. Once the free token limit is reached, users can easily purchase additional tokens for their API keys via the 'Buy tokens' tab.
attach_money
Are tokens charged for failed requests?
keyboard_arrow_down
No, tokens are not deducted for failed requests.
attach_money
What payment methods are accepted?
keyboard_arrow_down
Payments are processed through Stripe, supporting a variety of payment methods including credit cards, Google Pay, and PayPal for your convenience.
attach_money
Is invoicing available for token purchases?
keyboard_arrow_down
Yes, an invoice will be issued to the email address associated with your Stripe account upon the purchase of tokens.
Offices
location_on
Sunnyvale, CA
710 Lakeway Dr, Ste 200, Sunnyvale, CA 94085, USA
location_on
Berlin, Germany (HQ)
Prinzessinnenstraße 19-20, 10969 Berlin, Germany
location_on
Beijing, China
Level 5, Building 6, No.48 Haidian West St. Beijing, China
location_on
Shenzhen, China
402 Floor 4, Fu'an Technology Building, Shenzhen, China
Search Foundation
DeepSearch
Reader
Embeddings
Reranker
Classifier
Segmenter
API Documentation
Get Jina API key
Rate Limit
API Status
Company
About us
Contact sales
Newsroom
Intern program
Join us
open_in_new
Download logo
open_in_new
Terms
Security
Terms & Conditions
Privacy
Manage Cookies
    email
 Jina AI © 2020-2025.