Umbrella or sun cap? Buy or sell stocks? When it comes to questions like these, many people today rely on AI-supported recommendations. Chatbots such as ChatGPT, AI-driven weather forecasts, and financial market predictions are based on machine learning-driven sequence models. The quality of these applications therefore depends crucially on the type of sequence model used and how such models can be further optimized.
The linearity and non-linearity of the models play a central role here. Linear sequence models process information according to the principle of proportionality: the response to an input is always directly proportional to its strength, similar to the principle “as the wind, so the wave.” Non-linear models, on the other hand, can map more complex, context-dependent relationships: they can process the same information in completely different ways depending on the situation. A simple example: whether the word “bank” is interpreted as a financial institution or as the side of a river depends on the context, and such context-dependent distinctions cannot be captured by linear models.
This ability to process context-dependent information makes nonlinear models so powerful for complex tasks such as language comprehension or pattern recognition. In addition to the quality of the results, training efficiency also plays a decisive role. Both linear models and transformers (the architecture behind the “T” in ChatGPT) allow parallel training, in which large amounts of information can be processed simultaneously, which has made scaling to huge amounts of data possible in the first place.
However, while linear models can be trained economically, training large transformer models is extremely costly and energy-intensive: huge server farms are being built around the world for AI training, resulting in enormous energy consumption. The optimum would be a smart middle ground: a model that takes advantage of parallel training without the enormous costs of fully nonlinear architectures.
The key question is therefore how nonlinearity can be used effectively within sequence models. Scientists at the Ernst Strüngmann Institute in Frankfurt and the Interdisciplinary Center for Scientific Computing at Heidelberg University have found the answer. The key finding of the research is that it is worthwhile to find a sensible balance. To investigate this systematically, the researchers tested their models on a wide range of tasks, from text classification and image recognition to cognitive benchmarks from computer-assisted neuroscience. This diversity made it possible to distinguish which tasks really require nonlinearity to function and which can already be solved by largely linear processes.
The surprising result: models with dosed nonlinearity, in which only part of the model (the “neurons” in the neural network) works nonlinearly, outperformed both purely linear and completely nonlinear models in many scenarios. This advantage was particularly evident with limited amounts of data, where the sparse nonlinear models were clearly superior. But they also remained competitive with larger amounts of data. The reason: the nonlinear units act as flexible switches that switch between different linear processing modes depending on the context.
A key advantage of dosed nonlinear models is their interpretability. Because the nonlinearity is limited to a few units, the researchers were able to understand where and how the model uses them. This makes the architecture particularly valuable for neuroscience: when analyzing neural recordings, the models can not only predict behavior, but also reveal the computational principles underlying the brain. In this context, the results show a consistent pattern: memory is often implemented via slow linear dynamics, while computational operations are realized through targeted nonlinear mechanisms.
This means that the researchers are presenting an approach to explaining neuroscientific measurements. In addition, they suggest that, when optimizing sequence models in the context of machine learning, the integration of dosed nonlinearity should be considered a generally useful design principle for modern, data-efficient sequence models.
Experimental study
Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNs.
9-Jan-2026