Is the Language Hard to Model, or Its Tokenizer?

This series keeps making one move. The substrate a worker runs on decides more than the worker. We showed it for coding agents, where the environment moved the cost and the model already had the skill. Here we make the same move one level down, at the representation layer. The popular read is that some languages are simply harder for language models. We think that is mostly a measurement artifact of the tokenizer and the script, and that Sanskrit, a language built from deterministic rules, is the cleanest place to prove it. This piece is the pre-registration: the evidence we stand on, and the fair experiment we will run.

XO Labs Inc. · July 2026

TL;DR: We are testing whether cross-lingual differences in language-model efficiency come from the languages or from how we choose to represent them. The published literature already points hard at the second answer. Frequency-based tokenizers fragment morphologically rich and non-Latin languages, which inflates sequence length, raising cost and warping the geometry of the context. When researchers control for the tokenizer and transliterate the script, most of the apparent gap dissolves. Sanskrit is the extreme case. Kumar (2026) reports it needs roughly half the tokens of English under an aligned tokenizer, yet GPT-4's tokenizer shreds that density from about five characters per token to barely one. We pre-register a rig to test this ourselves. It is a strictly parallel corpus across five languages spanning three scripts and a morphological range from analytic to highly synthetic, each run in native and transliterated form, through four tokenization regimes, under strict iso-FLOP budgets, scored on sliding-window bits-per-byte rather than perplexity. Two questions drive it. First, which combination of language, script, and tokenization gives the best tradeoff among compression, memorization, and efficiency. Second, which of those three factors explains the most variance. We have no results yet. This is the plan and the standard we will hold it to.

1. Why some languages only look hard to model

Ask why a model is worse at Amharic than at English and the ready answer is that Amharic is harder. The script is unfamiliar, the morphology is dense, there is less data. The answer feels complete, and it puts the cause inside the language.

We think that answer is mostly wrong, and the interesting part is that it is testable. The literature keeps finding that once you separate a language from the way it is written and the way it is chopped into tokens, the language contributes far less of the measured penalty than people assume. What looks like "this language is hard" is usually "our tokenizer spends its vocabulary on English and fragments everything else," which is a fact about the pipeline, not about the grammar.

This is the same shape as The Self-Sufficient Agent, where the model already had the engineering judgment and the environment only moved the cost. Swap the agent for a language and the environment for the tokenizer and the sentence still holds. The capability is in the language. The cost is in the representation.

2. The two questions the experiment answers

The whole experiment reduces to two, and the second is the cleaner one.

Primary. Under a fixed compute budget, which combination of language, writing system, and tokenization yields the best tradeoff among compression, memorization, and model efficiency? "Best" needs a definition before it means anything, so we define it as a Pareto frontier over those outcomes rather than a single score, and we say up front which outcome we would trade for which.

Secondary. Which factor explains more variance in efficiency: language identity, script, or tokenization? This one has a crisp answer, because it is a variance-decomposition question. Build a factorial design, measure the outcome in every cell, and read off how much of the spread each factor accounts for. Our hypothesis is a strict ordering (tokenization first, script second, language a distant third), and the design is built so the data can refuse that ordering if we are wrong.

3. Three findings the literature already supports

We are not starting from zero. Three findings are solid enough to build on, and we cite them as established rather than assumed.

The Token Tax is real and it predicts accuracy. A subword tokenizer trained mostly on English web text spends single tokens on common English words and splinters everything else into many small pieces. The count of pieces per word is token fertility. Lundin et al. (2026), evaluating ten models on the AfriMMLU benchmark across sixteen African languages, found fertility to be a robust negative predictor of downstream accuracy across models and subjects. The cost is not linear. Attention scales quadratically with sequence length, so a doubling of fertility roughly quadruples the attention cost of the same content. Petrov et al. (2023) measured tokenization disparities of up to about fifteen times across tokenizers on FLORES-200, with the worst-hit non-Latin scripts penalized by roughly an order of magnitude. Under one fixed compute and context budget, a high-fertility language simply gets less meaning through each forward pass, which suppresses performance without the grammar being any harder.

Models organize by script, not by abstract structure. If the difficulty lived in the language, you would expect a model to represent the same sentence similarly however it is written. It does not. Verma et al. (2026) probed Llama-3.2 and Gemma-2 with sparse autoencoders and activation-entropy metrics, and found that the same content in its native script versus a romanized form lights up nearly disjoint sets of neurons. The representations that best track cross-lingual structure are exactly the ones that stay invariant when you change the script. So the model is often learning parallel, redundant copies of one concept, one per writing system, which wastes capacity. Rust et al. (2021) had already shown the flip side. Swap a shared multilingual tokenizer for a dedicated one and downstream performance rises, most for the under-represented languages.

The subword advantage is a throughput artifact, not a law. Byte-level models that read raw UTF-8 avoid script bias and out-of-vocabulary failure entirely (Xue et al., 2022, with ByT5), but historically they lost on cost because the sequences are so much longer. Gigant et al. (2026) isolated the reason with iso-FLOP simulations that mimicked subword boundaries inside a byte-level pipeline, and concluded the subword win comes almost entirely from sample throughput and the structural bias of the boundaries, not from any deeper inability of the network to model raw bytes. Newer architectures act on that. The Byte Latent Transformer (Pagnoni et al., 2024) allocates compute dynamically by entropy, compressing predictable byte spans into large patches and reserving fine-grained work for hard ones, and matches tokenizer-based models at scale while staying script-agnostic. MrT5 (Kallini et al., 2025) learns to delete redundant bytes early and cuts sequence length by up to seventy-five percent with little loss in bits-per-byte.

4. Why Sanskrit is the sharpest test

Every hypothesis wants an extreme case, and Sanskrit is it. Formalized by Panini in the Aṣṭādhyāyī over two millennia ago, it builds words from verbal roots, affixes, and deterministic phonetic fusions called sandhi. That machinery packs a great deal of meaning into single words, which is exactly the property a frequency tokenizer is worst at handling.

The numbers make the point sharply. Kumar (2026), on 701 parallel verses of the Bhagavad Gita, reports that under an unbiased SentencePiece tokenizer Sanskrit encodes about 5.07 characters per token and needs roughly half the tokens of the English translation for the same content. Run the same text through GPT-4's cl100k tokenizer and the density collapses to about 1.13 characters per token, because the tokenizer fractures the sandhi boundaries it was never trained to see. The language did not change. The representation did.

This is why Sanskrit is the right probe rather than a curiosity. It separates two things that usually travel together: the theoretical information density of a language and the density a given tokenizer realizes. A high-density language whose tokenizer is aligned to its morphology should be cheap to model. The same language under a mismatched tokenizer looks expensive, and a naive reading blames the language. Morphology-aware Sanskrit tokenizers close the gap by parsing structure before embedding. TransLIST (Sandhan et al., 2022) performs sandhi-aware segmentation, and CharSS (2024) handles sandhi boundaries at the character level, both beating standard subword methods. The lesson generalizes past Sanskrit. Density is only free when the tokenizer knows where the morpheme boundaries are.

5. Four ways a cross-lingual comparison goes wrong

Most cross-lingual comparisons are quietly broken, and the breaks all inflate or deflate the wrong variable. Naming them is the point of the design, because the experiment is only worth running if a difference we see can be pinned on the factor we are testing.

• Perplexity is not comparable across tokenizers. It is computed per token, so a tokenizer that fragments more makes each next-token prediction easier and reports a lower perplexity while compressing the sequence worse. Comparing two tokenizers by perplexity rewards the one that shreds. The fix is to score in bits-per-byte, over the raw byte stream, which is invariant to how the text was chopped.
• Same steps is not same compute. Byte sequences run several times longer than subword sequences, so training both for the same number of gradient steps hands the subword model far more content per step. Any honest byte-versus-subword claim has to hold total floating-point operations fixed, not steps, which is the control Gigant et al. (2026) used to expose the throughput artifact.
• Translationese fakes fluency. Benchmarks translated out of English carry English syntax, so a model can score well on them by exploiting English-aligned structure rather than by handling the target language naturally. Parallel corpora built for this purpose reduce it, but it never fully disappears, so we keep at least one non-translated domain in the mix.
• Fragmentation warps the geometry of context. Position-based attention treats tokens that are far apart as weakly related. If a relationship spans three tokens in English and twelve in the fragmented language, the model is penalized for a distance the tokenizer invented, not one the meaning contains. This is a direct consequence of how positional encodings like RoPE decay with distance, and it means a context-window comparison across languages is comparing warped geometries unless fertility is controlled.

6. How the experiment is built

The design is a factorial grid, built so the same content passes through every combination we want to compare. Every cell is one measurement on identical meaning.

Languages, chosen to isolate the factors. Five languages spanning three scripts and the morphological range from analytic to highly synthetic: English (analytic, Latin), Swahili (agglutinative, Latin), Hindi (fusional, Devanagari), Amharic (fusional, Ge'ez), and Sanskrit (highly synthetic, Devanagari). The pairs are the point. English and Swahili share a script but differ in morphology, so a difference between them cannot be a script effect. Hindi and Sanskrit share Devanagari but differ sharply in morphology, so a difference between them cannot be a script effect either. That built-in control is what lets the variance decomposition separate script from morphology instead of confounding them.

Script conditions. Every non-Latin language is run twice, once in its native orthography and once transliterated to a standard Latin form. Transliteration is the one knob we can turn directly, so it is how we measure the script penalty in isolation. Transliteration is not free of cost, since it erases some orthographic and sociolinguistic nuance, and we treat that as a limit rather than pretend it away. The evidence that it helps is strong. Moosa et al. (2022), transliterating Indic scripts to a unified form, improved named-entity F1 by up to 58.5 percent for Oriya and about 35 percent for Panjabi on IndicGLUE.

Tokenization regimes. Four: standard byte-pair encoding as the baseline; a parity-aware BPE tuned for more equal cross-lingual compression; a morphology-aware tokenizer that constrains merges to respect morpheme boundaries, in the spirit of MorphBPE (Asgari et al., 2025) and the sandhi-aware Sanskrit tokenizers; and dynamic byte-level patching in the Byte Latent Transformer paradigm.

Model and budget. Decoder-only transformers in the Llama family, to reflect how models are built now. The budget is anchored strictly to floating-point operations, not epochs or steps, so a byte-level regime that produces long sequences and a subword regime that produces short ones are compared at the same true cost. This is the single control that keeps the byte-versus-subword comparison honest.

Four outcomes, measured per cell.

1. Compression. Sliding-window bits-per-byte, with stride adaptation so only the non-overlapping trailing segment counts, which equalizes the metric across tokenization granularities and removes short-context bias.
2. Efficiency. Floating-point operations to reach a target loss, plus sequence length and throughput per step, which is where the Token Tax shows up as money.
3. Memorization. Discoverable memorization and extraction rate, measured per language. We flag this as the hardest of the four to measure cleanly at small scale, and we report it with that caution rather than as a headline.
4. Representation quality. Cross-lingual probing alignment, and the native-versus-transliterated activation comparison using sparse autoencoders, following Verma et al. (2026), to see whether the model forms shared structure or parallel per-script copies.

7. How we will answer the two questions

For the variance-decomposition question, the factorial grid feeds a straightforward analysis. Fit the outcome across all cells against the three factors (language identity, script, and tokenization regime), and report the share of variance each explains, with the built-in same-script and same-morphology pairs doing the work of separating them. Alongside it we compute the correlation between token fertility and bits-per-byte, which is the mechanism the whole thesis rests on, so a weak correlation there would be evidence against us.

For the best-tradeoff question, there is no single winner unless you say what you value, so we do not pretend there is. We map the Pareto frontier over compression, memorization, and efficiency, name the cells on it, and state which tradeoffs we would take. If one combination dominates across outcomes, that is a finding. If the frontier is spread out, that is also a finding, and a more honest one than a manufactured single number.

8. How strongly the evidence backs each claim

Keeping the honesty explicit, here is how strongly the literature backs each link, from most causal to most confounded. The numbers above are reported by their authors and verified against source, not produced by us, and we mark them as such until we reproduce them.

9. Scope and limits

The caveats shape the claim, so they sit in the body.

• No first-party results yet. This is a pre-registration. Every number here is from the published literature, verified against source, and every claim about our own outcomes is a plan, not a finding. We are writing the standard down before we run, so we cannot move the goalposts after.
• A from-scratch iso-FLOP sweep is a real bill. Training small decoder models across this grid is far heavier than our coding-agent experiments, which ran off-the-shelf agents. We expect to stage it, starting with the tokenizer-level and fixed-model measurements that need no training, and we will say which tier produced any result we report.
• Sanskrit is a stress test, not a proof about Sanskrit. Its density is measured on a small, classical, poetic corpus. That is what makes it a clean probe of tokenizer alignment, and it is also why we do not generalize from the Gita to Sanskrit as a modeling target in the wild, where data is scarce.
• Memorization is the soft outcome. It is the hardest of the four to measure without confounds at small scale, and we will treat a memorization result as suggestive rather than settled.
• Transliteration has a cost we are not hiding. It normalizes the surface form and recovers efficiency, and it erases orthographic and sociolinguistic detail. We measure the tradeoff; we do not sell it as free.

10. What a result would change

The coding experiments in this series kept landing on the same sentence: the capability was already there, and the substrate around it moved the cost. This is that sentence again, one layer down. If the variance in cross-lingual efficiency turns out to sit mostly in the tokenizer and the script rather than in the languages, then the way to serve the world's languages fairly is not a bigger model or a claim that some tongues are simply hard. It is a representation pipeline that matches how each language is built. That is an engineering problem, which means it has an answer, and we would rather measure it than assume it.

We have written down the rig and the standard. The next piece in this thread is the first tier of results, and we will report it the way we report everything here, including wherever it says we were wrong.

---

References

Cited works, verified against source. Years and venues follow the primary record.

• Asgari, E., El Kheir, Y., and Sadraei Javaheri, M. (2025). MorphBPE: A Morpho-Aware Tokenizer. arXiv:2502.00894.
• Conneau, A., et al. (2020). Unsupervised Cross-lingual Representation Learning at Scale (XLM-R). ACL 2020. arXiv:1911.02116.
• Gigant, T., Peng, B., and Quesnelle, J. (2026). Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation. arXiv:2604.27263.
• Kallini, J., et al. (2025). MrT5: Dynamic Token Merging for Efficient Byte-level Language Models. ICLR 2025. arXiv:2410.20771.
• Kumar, A. (2026). Tokenization efficiency of Sanskrit against Hindi and English on parallel Bhagavad Gita verses. arXiv:2601.06142.
• Lundin, J. M., et al. (2026). The Token Tax: Systematic Bias in Multilingual Tokenization. AfricaNLP. arXiv:2509.05486.
• Moosa, I. M., et al. (2022). Does Transliteration Help Multilingual Language Modeling? (XLM-Indic on IndicGLUE). arXiv:2201.12501.
• Pagnoni, A., et al. (2024). Byte Latent Transformer: Patches Scale Better Than Tokens. Meta FAIR; ACL 2025. arXiv:2412.09871.
• Petrov, A., La Malfa, E., Torr, P., and Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023. arXiv:2305.15425.
• Rust, P., et al. (2021). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. ACL-IJCNLP 2021.
• Sandhan, J., et al. (2022). TransLIST: A Transformer-Based Linguistically Informed Sanskrit Tokenizer. EMNLP Findings 2022. arXiv:2210.11753.
• CharSS: Character-Level Transformer Model for Sanskrit Word Segmentation (2024). arXiv:2407.06331.
• Verma, A. A. K., Chatterjee, A., Gupta, M., and Chakraborty, T. (2026). Multilingual Language Models Encode Script Over Linguistic Structure. ACL 2026. arXiv:2604.05090.
• Xue, L., et al. (2022). ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. TACL 2022. arXiv:2105.13626.