NeurIPS 2022 · Nonlinear Semantics & Linear Language Models

FIRE: Semantic Field of Words Represented as Nonlinear Functions

Xin Du · Kumiko Tanaka-Ishii

A word is lifted from a single point in a linear space to a positional measure and a nonlinear function in a semantic field, allowing multimodal meaning and additive composition within one representation.

Paper · NeurIPS Code · GitHub Research · Nonlinear Semantics and Linear Language Models

The Conflict between Linear Composition and Polysemy

Classical word embeddings represent each word as one point in $\mathbb R^D$ . Vector addition naturally supports composition, but one point cannot directly express several separated semantic modes. Assigning multiple vectors or a Gaussian mixture to each word captures polysemy but makes composition less simple and less closed. Contextual models return a disambiguated vector after a sentence is supplied; they answer what the word means in that context but do not explicitly represent its complete multimodal structure without context.

FIRE raises a word from a point to a field. In a semantic space $\mathcal S\subset\mathbb R^D$ , word $w$ is

w=[\mu_w,f_w(s)],

where $\mu_w$ is a positional measure and $f_w:\mathcal S\to\mathbb R$ is a continuous function over the space. The measure may contain several weighted Dirac points and the function may contain several peaks.

Semantic Similarity as Interaction

A word is no longer determined only by its own coordinate. Its function acts on the locations of other words, and similarity is the bidirectional interaction

\operatorname{sim}(w_i,w_j) = \int f_i\,\mathrm d\mu_j + \int f_j\,\mathrm d\mu_i .

If $\mu_j=\sum_{k=1}^{K}m_j^{(k)}\delta(s_j^{(k)})$ , the integral becomes an exact weighted evaluation at those locations. Semantic neighbours occupy high-response regions of one another’s fields; distinct senses form separated peaks.

The multimodal semantic field of bank and the combined fields of river and financial — Financial and geographic senses of “bank” form separated regions in one two-dimensional field; adding the functions of “river” and “financial” produces a composite field close to “bank.”

Composition remains linear. For $\Gamma=[w_1,\ldots,w_n]$ ,

\mu_\Gamma=\sum_{i=1}^{n}\alpha_i\mu_i, \qquad f_\Gamma(s)=\sum_{i=1}^{n}\alpha_i f_i(s).

The functions themselves can be nonlinear and multimodal, while their sum remains closed. Nonlinearity represents semantic shape; linear operations compose those shapes.

Constructing the Field with Planar Transformations

Rather than attach a large MLP to each word, FIRE uses small multilayer planar transformations:

\operatorname{Planar}(x) =x+\tanh(v^\top x+b)u .

The word function is constructed from the negative trace of the multilayer transformation Jacobian. Each Jacobian is an identity matrix plus a rank-one matrix; products across layers expand into higher-order state polynomials and can form several peaks even in two dimensions. Function evaluation and word similarity remain $O(KL)$ in the number of locations $K$ and transformation layers $L$ .

Training follows skip-gram with negative sampling. Co-occurring words are moved into high-response regions of each other’s functions, while negative samples are pushed away. Unlike ordinary metric learning, every word learns its own compact function rather than sharing one global network that maps inputs to vectors.

What the Experiments Show

FIRE is trained on the three-billion-token Wacky corpus and compared at equal parameters per word. At 50 parameters, mean Spearman correlation over 12 word-similarity datasets is 49.8, close to Word2Vec’s 50.4 and above the compared Gaussian and mixture representations. At sentence level, 100-parameter FIRE reaches 62.7 on WiC versus 62.0 for Word2Vec; STS is 61.1 versus 61.9. Multimodal functions retain basic compositional performance.

The decisive experiments measure polysemy directly. The paper selects 266 strongly polysemous and 276 strongly monosemous words from WordNet, clusters the semantic neighbourhood of each target, and compares the predicted number of clusters with annotated senses. Word2Vec has only point geometry; fixed mixtures tend to split monosemous words; contextual BERT vectors disambiguate supplied contexts but do not clearly recover the number of senses. FIRE forms stable separated modes for words such as “bank” and “rock” and more accurately distinguishes mono- from polysemy.

Conclusion and Boundary

FIRE shows that nonlinear semantic shape and linear composition are compatible: words can be multimodal functions, while sentences are still built by adding those functions. A two-dimensional field also supplies interpretable geometry without post-hoc dimensionality reduction.

The representation is non-contextual and more expensive to train than a same-size Word2Vec model. Per-word functions create scaling costs for very large vocabularies. Two-dimensional visualization is not a claim about the true dimension of language, and WordNet sense counts are only discrete approximations to polysemy. The broader question is how function-valued semantics can enter contextual recurrent models and retain multiple modes in finite states—connecting FIRE to selective memory in Mamba and DeltaNet.

← Nonlinear Semantics and Linear Language Models