Two models with identical architecture, initialisation and training data can respond differently to the same input if they were built with different tokenizers. The clearest recent evidence for this comes from a controlled design, and it points to behaviour, not only cost. It is among the clearer findings in the controlled tokenizer literature so far, though most of that work rests on small models. What remains open is when the difference is large enough to affect a real system, and that depends on conditions that the studies vary in ways that are hard to compare.
For anyone choosing a model, fine-tuning one, or building a pipeline on top of one, the question is practical. Does a different tokenizer change answers on the same text, or only the bill? Does the answer depend on the language, the amount of text the tokenizer was trained on, or the vocabulary size? And do byte-level models make the problem disappear? The research now offers partial answers to each of these, with very different levels of confidence attached.
What tokenization changes and what it does not
A tokenizer splits text into units the model reads as integers. A common word may be one unit; a rare word, a misspelling or a word in a low-resource language may be split into several. Two things follow. The model sees a different sequence for the same string, and the number of units per word, which researchers call fertility, determines how much of the model's context window and compute a passage consumes.
Cost is the part almost everyone accepts. A passage that takes twice as many tokens typically costs roughly twice as much to process at inference, and training cost grows with the token count too. Behaviour is the contested part. A tokenizer does not change what a model knows, but it changes what the model is given to work with, and a model that never sees a word as a single unit may generalise from it differently. Whether that matters in practice is an empirical question, and the answer varies by task.
Controlled evidence
The most direct test of the behaviour question is a design that holds everything except the tokenizer fixed. According to the arXiv listing for TokSuite (arXiv 2512.20757), the authors built 14 models that share initialisation, architecture and training data and differ only in tokenizer, then tested them on real-world perturbations such as OCR errors and typos. The paper is listed as an ICML 2026 proceedings paper, and that listing was confirmed. The results section was not read for this article, so the size of the differences, and whether they hold across perturbation types, are not reported here.
That design is the strongest available argument that tokenizer choice alone can move robustness. It is still a single study at the abstract and introduction level, and the models it trains are small by current standards.
Tokenization bias
Lesci et al. (ACL 2025) introduce what they call tokenization bias. The idea is that a string's probability under a language model depends on whether it is included as a single subword in the vocabulary or must be spelled out as several pieces. To isolate that effect, the authors use a regression-discontinuity design around the vocabulary cutoff, so that strings just above and just below the threshold are compared. According to the paper's abstract, the authors report consistency of the effect across scales, vocabularies and tokenizers. Those experiments were mostly small English-only models, and the authors say other architectures, training procedures and languages need testing.
A secondary summary circulating online gives a figure of roughly 17 times the probability for a 57-million-parameter English BPE setting. That figure did not appear in the primary text as far as this article's sources show, so it is reported here only as a secondary claim and should not be read as a general multiplier. The useful takeaway is the mechanism. A word that is a single token in one tokenizer can be less probable, as a string, than its split form in another, and the gap is measurable in controlled settings.
Multilingual conditions
The language question is where the cost and behaviour stories meet. The Token Tax study (AfricaNLP 2026, ACL Anthology, with an earlier preprint at arXiv 2509.05486) evaluated 10 LLMs on AfriMMLU across 16 African languages. In those models, higher fertility was associated with lower accuracy. This is an association across models, not a demonstration that fragmentation causes the accuracy gap; languages differ in many ways, and the regressions cannot separate tokenizer effects from data volume or task difficulty. The authors report that fertility explains a substantial share of variance in some models, and that claim is attributed here to the paper and not checked against its primary text.
The same study also models training cost. Its authors estimate that doubling tokens quadruples training cost and time, and they give a modelled figure of $420M, rather than $105M, for training Llama-3.1-405B in a language with double the fertility. These are estimates built on stated assumptions. They are not measured training spend, and they should not be read as a forecast for any particular project.
Against this, a preprint across 54 tokenizers reports that a language's bits-per-byte varies more across tokenizers when that language has less training data for the language model. Its abstract and introduction were read here, and the finding should be treated as preliminary. The same preprint is reported to find that excluding a language from tokenizer training raised its bits-per-byte; the wording "in every language studied" has not been confirmed against the full text. A single appendix on a debiased multilingual tokenizer suggests that some remaining fragmentation may reflect morphology rather than tokenizer allocation, which complicates any simple reading of fertility as a fairness measure.
Vocabulary size: where the evidence conflicts
Vocabulary size is the lever most teams reach for first, and the evidence on it is inconsistent. A Turkish study published in the ACM Digital Library (doi 10.1145/3578707, dated; the publication year was not confirmed for this article) reports that accuracy rose with vocabulary size for the tokenizers it tested, with BPE and WordPiece reaching their plateau sooner. Whether "saturating faster" is the authors' own finding or a paraphrase has not been checked against the paper.
Other results pull in different directions. An Arabic fertility result, which concerns how many tokens words take in that language, is one more reason that a single vocabulary setting does not transfer across languages. The distinction between input and output vocabulary also matters: a model can have a large input vocabulary while its output layer is sized differently, and the two can affect behaviour separately. A 2026 preprint across small-model suites reports no robust vocabulary-size effect. Taken together, the conflict is well supported as a finding, but its cause is not settled. The studies differ in model size, languages, tasks and how they define vocabulary.
A heuristic relating vocabulary size to a 20% or 40% share of parameters circulates in a secondary extract. Its origin could not be traced, and it should not be used as a sizing rule.
Byte-level and tokenizer-free models
Byte-level models read raw bytes rather than learned subwords, which removes the vocabulary entirely. ByT5 argues that token-free models handle any language out of the box, are more robust to noise, and are competitive with token-level models on the paper's own tests. That is the authors' claim, from their own evaluation.
A 2026 critique (arXiv 2604.27263) takes the other side and reports that byte-level models struggle to match the training efficiency and downstream performance of subword models. The critique was only partly read for this article, and its exact wording and regime need checking. The two positions are not necessarily contradictory. They describe different compute budgets, sequence lengths and tasks, and the byte-level debate remains unresolved. In practice, byte-level models change the trade-off: they trade vocabulary-driven fragmentation for longer sequences and more compute per unit of text.
Meaning, spelling and rare strings
Spelling is where tokenization most directly touches meaning. Typos, spelling variants and morphological changes can change how a word is split, and a changed split can change the representation the model uses. ByT5 and related work cite this mechanism, and it is plausible enough to design tests around. Whether it produces a consistent error rate on real inputs is what TokSuite's perturbation tests address, and those results were not read here.
A study of Chinese characters makes the point at the level of meaning. Three LLMs were more likely to judge two characters as sharing a semantic radical when the characters began with the same token, and they gave higher similarity ratings to such pairs (Computational Linguistics 51(3)). The finding is peer-reviewed, but it was read at abstract level. It is also worth separating two claims: that shared tokens make the judgement more likely, and that shared tokens raise similarity ratings. Those are not the same measurement, and the paper's attribution of the ratings to radical overlap as well as to shared tokens has not been checked.
Coverage gaps are the bluntest version of the problem. The T5 tokenizer's inability to represent curly braces is a documented, specific case described in the TokSuite paper's discussion of older tokenizers. It matters for code and structured text, and it is an older example, not a measure of current systems.
Conditions that change the answer and open questions
The answer to "does tokenizer choice change behaviour?" depends on the setting. The strongest evidence points to a yes for perturbed, noisy and low-resource text, and to a conditional yes for cost, which scales with fertility almost everywhere. Language, the amount of text available for it, vocabulary size and the task all shift the result. Frontier-scale models, arithmetic, counting, code and multi-step reasoning under tokenizer change are largely unaddressed by the primary sources read here.
Three things are unknown. Whether deployed commercial models were tokenised as their vendors published, and whether tokenizers changed between versions, could not be established. Whether fine-tuning, chat templates and special tokens alter the picture in production is untested in these sources. Under-trained and glitch tokens are often raised in this debate, but the primary papers were not retrieved, so they are treated here only as leads.
Evidence assessment
The evidence is strongest where it is narrowest. The controlled design of TokSuite and the peer-reviewed radical study show that tokenizer choice can change model behaviour under specific conditions. The Token Tax association between fertility and accuracy is peer-reviewed but observational. The bias, multilingual and vocabulary findings are mostly preprints, single studies or secondary summaries, and several of their figures could not be checked against primary text.
No retraction or later-version check was carried out for any source cited in this article. The versions read were those retrieved, and the editor should confirm that the cited versions are current before publication.
Practical implications
Teams comparing models should measure, not assume. Run the same evaluation set through each candidate tokenizer, and count tokens per unit of text in each language they serve. Where the workload involves noisy input such as OCR output, user-typed queries or informal spelling, include perturbed inputs in the test set, because clean benchmarks may understate the gap.
Do not treat vocabulary size as a dial that reliably improves accuracy. The evidence conflicts, and the heuristics in circulation have no traceable basis. Do not cite a tokenization-bias multiplier from secondary summaries. For byte-level models, weigh the robustness claims against the efficiency critique and test at your own sequence lengths, since the trade-off depends on the regime you run in.
Limitations and open questions
Most controlled evidence uses small models, and small-model results may not transfer to frontier systems. Several central findings rest on preprints, on abstracts read in part, or on secondary summaries. Cost estimates are modelled, and associations across languages are confounded by data volume and task difficulty. What would settle it: primary replication of the bias and perturbation results at larger scale, reporting that separates tokenizer effects from data and task effects, and transparent tokenizer versioning for commercial models.
Editor's take
The evidence supports a narrow but useful conclusion. Tokenizer choice can change how a model handles perturbed and low-resource text, not only what it costs to run. That claim rests on a controlled design and on peer-reviewed work at a small scale, which is enough to change how teams test, though not enough to predict the size of the effect in a given product.
The weaker parts of the picture are the ones teams are most tempted to quote. A single multiplier for tokenization bias, a fixed vocabulary-size rule, and a claim that byte-level models remove the problem all go beyond what the sources support. The reasoned position is to treat tokenizer choice as a variable to measure in your own setting, and to be sceptical of any number that cannot be traced to a primary text.
"