NeurIPS 2025 · Foundation Models & Complexity

Correlation Dimension of Auto-Regressive Large Language Models

Xin Du · Kumiko Tanaka-Ishii

Correlation dimension estimated from next-token log-probability trajectories supplies a multiscale geometric observable complementary to the local view of perplexity.

Perplexity Does Not Describe How Text Is Organized

Autoregressive models are trained by next-token prediction, so perplexity is a natural primary metric. It asks a local question: how much probability did the model assign to the observed next token? A sequence can repeat an established pattern and remain easy to predict, while a passage with new entities and long-range constraints can be less predictable yet structurally complete. Local loss alone cannot tell whether the model maintains cross-sentence relations, when a generation begins to degenerate, or when these abilities form during pretraining.

For a text, every position supplies a next-token log-probability vector over the vocabulary:

These vectors form a high-dimensional trajectory. Nearby states mean that the model makes similar predictions at two positions: the intervening text had limited effect on the predicted future, producing a recurrence. If the number of recurrent pairs across distance scales satisfies

then $d$ measures the dynamic degrees of freedom the model uses to distinguish the sequence. Both short-range and long-range recurrences contribute.

Complexity as Perceived by a Model

The quantity is not an intrinsic text constant independent of a model; it is the complexity perceived by the model. A Lin–Tegmark probabilistic grammar provides sequences with controlled hierarchical structure. As the grammar moves from nearly deterministic generation into a complex regime, correlation dimension rises from nearly $0$ to above $10$, showing that the estimator tracks known generative complexity without specialized training on that grammar.

Context-length experiments clarify perception. With one-token context, the model behaves approximately as a bigram model and dimension is near $3$. At context 32, it distinguishes more conditional structures and dimension rises to about $8$. With longer context, the dimension falls and stabilizes near $6.5$. This is not forgetting: redundant histories are compressed so that more context is used while functionally similar contexts occupy a lower-dimensional prediction structure.

Three Stages of Pretraining

Across checkpoints of Pythia, OpenLLaMA, and Amber, correlation dimension reveals three stages that perplexity does not. It first drops as the model learns word frequency and short-range collocations; then rises as longer dependencies form; and finally declines gradually as the acquired structures are compressed into more general representations. Perplexity remains nearly monotonic and does not separate learning new structure from compressing existing structure.

The trajectory also exposes a capacity boundary. Pythia-14M and Pythia-160M show an anomalous late rise in dimension accompanied by lower accuracy on context-repetition tasks. Their local prediction continues to improve while long-range structure is no longer compressed stably into finite capacity.

Degeneration as Loss of Trajectory Freedom

An artificially repetitive passage has dimension 1.83, compared with 6.27 for normal text. In style- and topic-controlled GPT-4o data, normal text averages 5.04; repetitive, incoherent, and content-poor text falls to 3.80, 3.96, and 4.51. Perplexity has no consistent direction across these cases. Correlation dimension captures their shared loss of structural freedom.

A control using Japanese orthography rewrites mixed kanji–kana text in kana only. Surface repetition changes markedly, while correlation dimension changes much less, showing that the metric responds primarily to predictive structure rather than treating orthographic repetition as semantic simplicity.

In long-text stress tests seeded with random names, correlation dimension across eight models reaches Spearman $\rho=0.952$ with HelloEval. Knowledge-recall cases also show that low-dimensional trajectories are more likely to invent obscure entities. The metric is a signal of degeneration and unstable retrieval, not a per-statement factuality classifier.

Computability and Limits

Naively comparing $N$ positions over a vocabulary $\Omega$ costs $O(N^2|\Omega|)$. A fused CUDA implementation avoids storing pairwise distance matrices, while vocabulary reduction preserves the primary geometry and yields more than a tenfold speedup. When distances are accumulated in FP32, GPTQ or AWQ quantization to 4 bits changes the mean dimension by less than 3%.

Finite-sequence estimates remain sensitive to sequence length, scaling-interval selection, and the perceptual capacity of the model. Full logits are required, so the method cannot be applied directly to closed APIs that return only sampled text. The three pretraining stages are an empirical regularity rather than an established statistical-physics phase transition. Correlation dimension is best treated as a complementary observable of when models acquire, compress, or lose long-range degrees of freedom.

← Foundation Models and Complexity