News
Products
keyboard_arrow_down
For Enterprises
Embeddings
World-class multimodal multilingual embeddings.
Reranker
World-class neural retriever for maximizing search relevancy.
Reader
Read URLs and search web for better grounding LLMs.
Classifier
Zero-shot and few-shot classification for image and text.
Segmenter
Cut long text into chunks and do tokenization.
For Power Users
PromptPerfect
Premier tool for prompt engineering
More power user tools
keyboard_arrow_down
SceneXplain
Leading AI solution for image captions and video summaries
BestBanner
Blog to banner, without the prompts!
JinaChat
More modality, longer memory, less cost
Rationale
Ultimate AI decision-making tools
Company
keyboard_arrow_down
About us
Contact sales
Intern program
Join us
open_in_new
Download logo
open_in_new
Terms & Conditions
login
Log in

Segmenter

Free API for segmenting long text into chunks and tokenization.

Segmenter API

Our Segmenter API is crucial for helping LLMs manage input within context limits, and optimizing model performance. It allows developers to count tokens and extract relevant text segments, ensuring efficient data processing and cost management.
key
API Key & Billing
code
Usage
more_horiz
More
chevron_leftchevron_right
home
Auto preview
help_outlineFAQ
api
Status
chevron_leftchevron_right
Use GET request to count tokens
Use POST request for more features
upload
Request
curl -X POST 'https://segment.jina.ai/' \
  -H "Content-Type: application/json" \
  -d @- <<EOFEOF
  {
    "content": "\n  Jina AI: Your Search Foundation, Supercharged! 🚀\n  Ihrer Suchgrundlage, aufgeladen! 🚀\n  您的搜索底座，从此不同！🚀\n  検索ベース,もう二度と同じことはありません！🚀\n"
  }
EOFEOF

What is a Segmenter?

A segmenter is a crucial component that converts text into tokens or chunks, which are the basic units of data that an embedding/reranker model or LLM processes. Tokens can represent whole words, parts of words, or even individual characters.

Chunking long documents, lightning fast!

You can also use Segmenter API to cut long documents into smaller chunks, making it easier to process them in embeddings or rerankers. We leverage common structural cues and build a set of rules and heuristics which perform well across diverse types of content, e.g. Markdown, HTML, LaTeX and CJK languages.
Maximum length of each chunk: 1000
Maximum number of characters in each chunk. In practice the chunk length can be smaller than this value, if there is a good boundary in the text.
0 chunks in total
Segmenter API is free!
By providing your API key, you can access a higher rate limit, and your key won't be charged.
Rate Limit
Rate limits are tracked in two ways: RPM (requests per minute) and TPM (tokens per minute). Limits are enforced per IP and can be reached based on whichever threshold—RPM or TPM—is hit first.
ProductAPI EndpointDescriptionarrow_upwardw/o API Keyw/ API Keyw/ Premium API KeyAverage LatencyToken Usage CountingAllowed Request
Embedding APIhttps://api.jina.ai/v1/embeddingsConvert text/images to fixed-length vectorsblock500 RPM & 1,000,000 TPM2,000 RPM & 5,000,000 TPM
bolt
depends on the input size
 help
Count the number of tokens in the input request.POST
Reranker APIhttps://api.jina.ai/v1/rerankTokenize and segment long textblock500 RPM & 1,000,000 TPM2,000 RPM & 5,000,000 TPM
bolt
depends on the input size
 help
Count the number of tokens in the input request.POST
Reader APIhttps://r.jina.aiConvert URL to LLM-friendly text20 RPM200 RPM1000 RPM4.6sCount the number of tokens in the output response.GET/POST
Reader APIhttps://s.jina.aiSearch the web and convert results to LLM-friendly textblock40 RPM100 RPM8.7sCount the number of tokens in the output response.GET/POST
Reader APIhttps://g.jina.aiGrounding a statement with web knowledgeblock10 RPM30 RPM22.7sCount the total number of tokens in the whole process.GET/POST
Classifier API (Zero-shot)https://api.jina.ai/v1/classifyClassify inputs using zero-shot classificationblock200 RPM & 500,000 TPM1,000 RPM & 3,000,000 TPM
bolt
depends on the input size
 
Tokens counted as: input_tokens + label_tokensPOST
Classifier API (Few-shot)https://api.jina.ai/v1/classifyClassify inputs using a trained few-shot classifierblock20 RPM & 200,000 TPM60 RPM & 1,000,000 TPM
bolt
depends on the input size
 
Tokens counted as: input_tokensPOST
Classifier APIhttps://api.jina.ai/v1/trainTrain a classifier using labeled examplesblock20 RPM & 200,000 TPM60 RPM & 1,000,000 TPM
bolt
depends on the input size
 
Tokens counted as: input_tokens × num_itersPOST
Segmenter APIhttps://segment.jina.aiTokenize and segment long text20 RPM200 RPM1,000 RPM0.3sToken is not counted as usage.GET/POST
keyGet your API key
handshakeContact sales
FAQ
How to get my API key?
 video_not_supported
What's the rate limit?
Rate Limit
Rate limits are tracked in two ways: RPM (requests per minute) and TPM (tokens per minute). Limits are enforced per IP and can be reached based on whichever threshold—RPM or TPM—is hit first.
ProductAPI EndpointDescriptionarrow_upwardw/o API Keyw/ API Keyw/ Premium API KeyAverage LatencyToken Usage CountingAllowed Request
Embedding APIhttps://api.jina.ai/v1/embeddingsConvert text/images to fixed-length vectorsblock500 RPM & 1,000,000 TPM2,000 RPM & 5,000,000 TPM
bolt
depends on the input size
 help
Count the number of tokens in the input request.POST
Reranker APIhttps://api.jina.ai/v1/rerankTokenize and segment long textblock500 RPM & 1,000,000 TPM2,000 RPM & 5,000,000 TPM
bolt
depends on the input size
 help
Count the number of tokens in the input request.POST
Reader APIhttps://r.jina.aiConvert URL to LLM-friendly text20 RPM200 RPM1000 RPM4.6sCount the number of tokens in the output response.GET/POST
Reader APIhttps://s.jina.aiSearch the web and convert results to LLM-friendly textblock40 RPM100 RPM8.7sCount the number of tokens in the output response.GET/POST
Reader APIhttps://g.jina.aiGrounding a statement with web knowledgeblock10 RPM30 RPM22.7sCount the total number of tokens in the whole process.GET/POST
Classifier API (Zero-shot)https://api.jina.ai/v1/classifyClassify inputs using zero-shot classificationblock200 RPM & 500,000 TPM1,000 RPM & 3,000,000 TPM
bolt
depends on the input size
 
Tokens counted as: input_tokens + label_tokensPOST
Classifier API (Few-shot)https://api.jina.ai/v1/classifyClassify inputs using a trained few-shot classifierblock20 RPM & 200,000 TPM60 RPM & 1,000,000 TPM
bolt
depends on the input size
 
Tokens counted as: input_tokensPOST
Classifier APIhttps://api.jina.ai/v1/trainTrain a classifier using labeled examplesblock20 RPM & 200,000 TPM60 RPM & 1,000,000 TPM
bolt
depends on the input size
 
Tokens counted as: input_tokens × num_itersPOST
Segmenter APIhttps://segment.jina.aiTokenize and segment long text20 RPM200 RPM1,000 RPM0.3sToken is not counted as usage.GET/POST
Other questions
Segmenter-related common questions
How much does the Segmenter API cost?
keyboard_arrow_down
The Segmenter API is free to use. By providing your API key, you can access a higher rate limit, and your key won't be charged.
If I don't provide an API key, what is the rate limit?
keyboard_arrow_down
Without an API key, you can access the Segmenter API at a rate limit of 20 RPM.
If I provide an API key, what is the rate limit?
keyboard_arrow_down
With an API key, you can access the Segmenter API at a rate limit of 200 RPM. For premium paid users, the rate limit is 1000 RPM.
Will you charge the tokens from my API key?
keyboard_arrow_down
No, your API key is only used to access a higher rate limit.
Does the Segmenter API support multiple languages?
keyboard_arrow_down
Yes, the Segmenter API is multilingual and supports over 100 languages.
What is the difference between GET and POST requests?
keyboard_arrow_down
GET requests are solely used to count the number of tokens in a text, allows you easily integrate it as a counter in your application. POST requests supports more parameters and features, such as returning the first/last N tokens.
What is the maximum length I can tokenize per request?
keyboard_arrow_down
You can send up to 64k characters per request.
How does the chunking feature work? Is it semantic chunking?
keyboard_arrow_down
The chunking feature segments long documents into smaller chunks based on common structural cues, ensuring accurate segmentation of text into meaningful chunks. Essentially it is a (big!) regex pattern that segments text based on certain syntactical features that often align with semantic boundaries, such as sentence endings, paragraph breaks, punctuation, and certain conjunctions. It is not semantic chunking. This (big) regex is as powerful as it can be within the limitations of regular expressions. It balances complexity and performance. While true semantic understanding isn't possible with regex, it well-approximates context by common structural cues.
How do you handle special tokens such as 'endoftext' in the Segmenter API?
keyboard_arrow_down
If the input contains special tokens, our Segmenter API will put them in the field 'special_tokens'. This allows you to easily identify them and handle them accordingly for your downstream tasks, e.g. removing them before feeding the text into an LLM to prevent injection attacks.
Does chunking support other languages than English?
keyboard_arrow_down
Besides western languages, chunking also works well with Chinese, Japanese, and Korean.
API-related common questions
code
Can I use the same API key for embedding, reranking, reader, fine-tuning APIs?
keyboard_arrow_down
Yes, the same API key is valid for all search foundation products from Jina AI. This includes the embedding, reranking, reader and fine-tuning APIs, with tokens shared between the all services.
code
Can I monitor the token usage of my API key?
keyboard_arrow_down
Yes, token usage can be monitored in the 'Buy tokens' tab by entering your API key, allowing you to view the usage history and remaining tokens.
code
What should I do if I forget my API key?
keyboard_arrow_down
If you have misplaced a topped-up key and wish to retrieve it, please contact support AT jina.ai with your registered email for assistance.
Contact
code
Do API keys expire?
keyboard_arrow_down
No, our API keys do not have an expiration date. However, if you suspect your key has been compromised and wish to retire it or transfer its tokens to a new key, please contact our support team for assistance.
Contact
code
Why is the first request for some models slow?
keyboard_arrow_down
This is because our serverless architecture offloads certain models during periods of low usage. The initial request activates or 'warms up' the model, which may take a few seconds. After this initial activation, subsequent requests process much more quickly.
code
Is user input data used for training your models?
keyboard_arrow_down
We adhere to a strict privacy policy and do not use user input data for training our models.
Billing-related common questions
attach_money
Is billing based on the number of sentences or requests?
keyboard_arrow_down
Our pricing model is based on the total number of tokens processed, allowing users the flexibility to allocate these tokens across any number of sentences, offering a cost-effective solution for diverse text analysis requirements.
attach_money
Is there a free trial available for new users?
keyboard_arrow_down
We offer a welcoming free trial to new users, which includes one million tokens for use with any of our models, facilitated by an auto-generated API key. Once the free token limit is reached, users can easily purchase additional tokens for their API keys via the 'Buy tokens' tab.
attach_money
Are tokens charged for failed requests?
keyboard_arrow_down
No, tokens are not deducted for failed requests.
attach_money
What payment methods are accepted?
keyboard_arrow_down
Payments are processed through Stripe, supporting a variety of payment methods including credit cards, Google Pay, and PayPal for your convenience.
attach_money
Is invoicing available for token purchases?
keyboard_arrow_down
Yes, an invoice will be issued to the email address associated with your Stripe account upon the purchase of tokens.
Offices
location_on
Sunnyvale, CA
710 Lakeway Dr, Ste 200, Sunnyvale, CA 94085, USA
location_on
Berlin, Germany (HQ)
Prinzessinnenstraße 19-20, 10969 Berlin, Germany
location_on
Beijing, China
Level 5, Building 6, No.48 Haidian West St. Beijing, China
location_on
Shenzhen, China
402 Floor 4, Fu'an Technology Building, Shenzhen, China
Search Foundation
Embeddings
Reranker
Reader
Classifier
Segmenter
Get Jina AI API key
Rate Limit
API Status
Company
About us
Contact sales
Newsroom
Intern program
Join us
open_in_new
Download logo
open_in_new
Terms
Team License
Security
Terms & Conditions
Privacy
Manage Cookies
    email
 Jina AI © 2020-2024.