Segmenter
Free API for segmenting long text into chunks and tokenization.
Segmenter API
Our Segmenter API is crucial for helping LLMs manage input within context limits, and optimizing model performance. It allows developers to count tokens and extract relevant text segments, ensuring efficient data processing and cost management.
Use GET request to count tokens
Use POST request for more features
Return the last N tokens
Return the last N tokens of the given content. Boundary exclusive. Can not be used with 'head'.
Segmenter
Choose the tokenizer to use.
cl100k_base
arrow_drop_down
upload
Request
Bash
Language
arrow_drop_down
curl -X POST 'https://api.jina.ai/v1/segment' \
-H "Content-Type: application/json" \
-d @- <<EOFEOF
{
"content": "Jina AI: Your Search Foundation, Supercharged! 🚀\nIhrer Suchgrundlage, aufgeladen! 🚀\n您的搜索底座,从此不同!🚀\n検索ベース,もう二度と同じことはありません!🚀"
}
EOFEOF
key
API key
visibility_off
Available tokens
0
What is a Segmenter?
A segmenter is a crucial component that converts text into tokens or chunks, which are the basic units of data that an embedding/reranker model or LLM processes. Tokens can represent whole words, parts of words, or even individual characters.
Input text
Chunking long documents, lightning fast!
You can also use Segmenter API to cut long documents into smaller chunks, making it easier to process them in embeddings or rerankers. We leverage common structural cues and build a set of rules and heuristics which perform well across diverse types of content, e.g. Markdown, HTML, LaTeX and CJK languages.
Input text
Maximum number of characters in each chunk. In practice the chunk length can be smaller than this value, if there is a good boundary in the text.
0 chunks in total
Segmenter API is free!
By providing your API key, you can access a higher rate limit, and your key won't be charged.
Rate Limit
Rate limits are tracked in three ways: RPM (requests per minute), and TPM (tokens per minute). Limits are enforced per IP/API key and will be triggered when either the RPM or TPM threshold is reached first. When you provide an API key in the request header, we track rate limits by key rather than IP address.
Columns
arrow_drop_down
| Product | API Endpoint | Descriptionarrow_upward | w/o API Keykey_off | w/ Free API Keykey | w/ Paid API Keykey | w/ Premium API Keykey | Average Latency | Token Usage Counting | Allowed Request | |
|---|---|---|---|---|---|---|---|---|---|---|
| Reader API | https://r.jina.ai | Convert URL to LLM-friendly text | 20 RPM | 500 RPM | 500 RPM | trending_up5000 RPM | 7.9s | Count the number of tokens in the output response. | GET/POST | |
| Reader API | https://s.jina.ai | Search the web and convert results to LLM-friendly text | block | 100 RPM | 100 RPM | trending_up1000 RPM | 2.5s | Every request costs a fixed number of tokens, starting from 10000 tokens | GET/POST | |
| Embedding API | https://api.jina.ai/v1/embeddings | Convert text/images to fixed-length vectors | block | 100 RPM & 100,000 TPM | 500 RPM & 2,000,000 TPM | trending_up5,000 RPM & 50,000,000 TPM | ssid_chart depends on the input size help | Count the number of tokens in the input request. | POST | |
| Reranker API | https://api.jina.ai/v1/rerank | Rank documents by query | block | 100 RPM & 100,000 TPM | 500 RPM & 2,000,000 TPM | trending_up5,000 RPM & 50,000,000 TPM | ssid_chart depends on the input size help | Count the number of tokens in the input request. | POST |
code
Can I use the same API key across all Jina APIs?
keyboard_arrow_down
code
Can I monitor the token usage of my API key?
keyboard_arrow_down
code
What should I do if I forget my API key?
keyboard_arrow_down
code
Do API keys expire?
keyboard_arrow_down
code
Can I transfer tokens between API keys?
keyboard_arrow_down
code
Can I revoke my API key?
keyboard_arrow_down
code
Why is the first request for some models slow?
keyboard_arrow_down
code
Is my API data used to train your models?
keyboard_arrow_down
code
What are the rate limits for Jina APIs?
keyboard_arrow_down
code
Are there batch size limits for the APIs?
keyboard_arrow_down
code
Are the Jina APIs the same thing as Jina models inside Elastic?
keyboard_arrow_down
Rate Limit
Rate limits are tracked in three ways: RPM (requests per minute), and TPM (tokens per minute). Limits are enforced per IP/API key and will be triggered when either the RPM or TPM threshold is reached first. When you provide an API key in the request header, we track rate limits by key rather than IP address.
Columns
arrow_drop_down
| Product | API Endpoint | Descriptionarrow_upward | w/o API Keykey_off | w/ Free API Keykey | w/ Paid API Keykey | w/ Premium API Keykey | Average Latency | Token Usage Counting | Allowed Request | |
|---|---|---|---|---|---|---|---|---|---|---|
| Reader API | https://r.jina.ai | Convert URL to LLM-friendly text | 20 RPM | 500 RPM | 500 RPM | trending_up5000 RPM | 7.9s | Count the number of tokens in the output response. | GET/POST | |
| Reader API | https://s.jina.ai | Search the web and convert results to LLM-friendly text | block | 100 RPM | 100 RPM | trending_up1000 RPM | 2.5s | Every request costs a fixed number of tokens, starting from 10000 tokens | GET/POST | |
| Embedding API | https://api.jina.ai/v1/embeddings | Convert text/images to fixed-length vectors | block | 100 RPM & 100,000 TPM | 500 RPM & 2,000,000 TPM | trending_up5,000 RPM & 50,000,000 TPM | ssid_chart depends on the input size help | Count the number of tokens in the input request. | POST | |
| Reranker API | https://api.jina.ai/v1/rerank | Rank documents by query | block | 100 RPM & 100,000 TPM | 500 RPM & 2,000,000 TPM | trending_up5,000 RPM & 50,000,000 TPM | ssid_chart depends on the input size help | Count the number of tokens in the input request. | POST |
Billing-related common questions
attach_money
Is billing based on the number of sentences or requests?
keyboard_arrow_down
attach_money
Is there a free trial available for new users?
keyboard_arrow_down
attach_money
Are tokens charged for failed requests?
keyboard_arrow_down
attach_money
What payment methods are accepted?
keyboard_arrow_down
attach_money
Is invoicing available for token purchases?
keyboard_arrow_down
attach_money
How do I buy a commercial license rather than API tokens?
keyboard_arrow_down
attach_money
Can I pay by invoice or purchase order instead of card?
keyboard_arrow_down
attach_money
I paid, but my balance or rate limit has not changed. What should I check?
keyboard_arrow_down
attach_money
How do I cancel, stop auto top-up, or remove a saved payment method?
keyboard_arrow_down