Adaptation is one of the most remarkable phenomena in nature. From the way an octopus can change their skin color to blend into its surroundings, to how the human brain rewires itself after an injury, allowing individuals to recover lost functions and adapt to new ways of thinking or moving. Living organisms exhibit adaptability that allows life to flourish in diverse and ever-changing environments. In the field of AI, the concept of adaptation holds a similar allure. Imagine a machine learning system that could adjust its own weights dynamically to thrive in unfamiliar settings, essentially illustrating a system that evolves as it learns. Self-adaptiveness in AI promises greater efficiency and the potential for lifelong models ever aligned with the dynamic nature of the real world. This vision of self-adaptive AI is at the heart of our latest research paper, Transformer² (Transformer-squared), where we propose a machine learning system that dynamically adjusts its weights for various tasks. The name Transformer² reflects its two-step process: first, the model analyzes the incoming task to understand its requirements, and then it applies task-specific adaptations to generate optimal results. By selectively adjusting critical components of the model weights, our framework allows LLMs to dynamically adapt to new tasks in real time. Transformer² demonstrates significant advancements across various tasks (e.g., math, coding, reasoning, and visual understanding), outperforming traditional, static approaches like LoRA in efficiency and task-specific performance while requiring far fewer parameters. Our research offers a glimpse into a future where AI models are no longer static. These systems will scale their compute dynamically at test-time to adapt to the complexity of tasks they encounter, embodying living intelligence capable of continuous change and lifelong learning. We believe self-adaptivity will not only transform AI research but also redefine how we interact with intelligent systems, creating a world where adaptability and intelligence go hand in hand. Transformer² is a machine learning system that dynamically adjusts its weights for various tasks. Adaptation is a remarkable natural phenomenon, like how the octopus can blend its color in with its environment, or how the brain rewires itself after injury. We believe our new system paves the way for a new generation of adaptive AI models, modifying their own weights and architecture to adapt to the nature of the tasks they encounter, embodying living intelligence capable of continuous change and lifelong learning. Dissecting the Brain of LLMs Just as the human brain stores knowledge and processes information through interconnected neural pathways, LLMs store knowledge within their weight matrices. These matrices are the brain of an LLM, holding the essence of what it has learned from its training data. Understanding this brain and ensuring that it can adapt effectively to new tasks requires a closer look at its inner structure. This is where Singular Value Decomposition (SVD) provides invaluable insights. Think of SVD as a surgeon performing a detailed operation on the brain of an LLM. This surgeon breaks down the vast, complex knowledge stored in the LLM into smaller, meaningful, and independent pieces (e.g., the different pathways or components for math, language understanding, etc). SVD
