Token (LLM)
AI & MLToken (LLM) is a unit of text an AI language model reads and generates, typically representing a word, part of a word, punctuation, or whitespace. Models process prompts and produce outputs as sequences of tokens, and most limits and pricing are measured in token counts. Tokenization varies by model, so the same text can consume different tokens across systems.
How It Works
Before an LLM can understand text, it runs tokenization: the input string is split into tokens using a predefined vocabulary and rules (often based on subword methods such as byte-pair encoding or similar). Each token maps to an integer ID, which the model converts into vectors (embeddings) and processes through its neural network. The model then predicts the next token repeatedly to form an output sequence.
Token counts affect multiple constraints. The context window is measured in tokens and includes both the prompt (system instructions, chat history, retrieved documents) and the generated response. If the combined total exceeds the model limit, earlier content may be truncated or the request may fail. Because tokenization is model-specific, short-looking text (URLs, code, long identifiers, or non-English characters) can expand into many tokens, while common words may compress into fewer.
Why It Matters for Web Hosting
For hosting decisions, tokens translate directly into resource planning and cost control for AI features you deploy, such as chatbots, search assistants, or content tools. Token volume influences API spend, response latency, and how much conversation history or documentation you can keep in memory. When comparing hosting plans, consider whether your stack can handle token-heavy workloads (logging, caching, background jobs) and whether limits like request size, CPU time, and outbound bandwidth align with your expected token throughput.
Common Use Cases
- Estimating prompt and response size to stay within a model context window
- Budgeting and rate-limiting LLM API usage based on tokens per request
- Designing retrieval-augmented generation (RAG) to fit documents into token limits
- Optimizing prompts by removing redundancy and compressing instructions
- Monitoring production usage by tracking tokens per user, per feature, or per endpoint
Token (LLM) vs Character Count
Character count is a rough proxy for size, but LLMs operate on tokens, not characters. A single token may be one character, a whole word, or a word fragment, and the mapping depends on the model tokenizer. This means two strings with the same character length can have very different token counts, especially for code, long variable names, emojis, or mixed-language text. For reliable capacity planning, measure tokens using the target model tokenizer rather than relying on characters or word counts.