We believe this is a significant architectural limitation. And we have built a framework to overcome it.

The Problem: Why Computational Identity Matters

To understand why this limitation is relevant, it is useful to take a step back and think about how learning works in humans.

A doctor graduating from medical school has a shared knowledge base with all their classmates. But after twenty years of practice, two doctors who have followed different paths reason in profoundly different ways. A cardiologist who has spent their career treating diabetic patients has developed a specific clinical intuition: they do not just know different things from their colleague who has worked in pediatric cardiac surgery, but they think differently. Their reasoning patterns, diagnostic heuristics, the way they weigh evidence — all of this has been shaped by their professional history.

It is not just a matter of accumulated knowledge. It is a matter of cognitive structure: the way past experience shapes how we process future information.

In current AI, this dimension does not exist. When we adapt a model to a new task with LoRA or fine-tuning, we are teaching it what to do on that specific task. But we are not changing its way of reasoning. We are not building a cognitive history. And above all, when we move on to the next task, everything the model has "learned" about the previous task is typically lost or, worse, interferes with new learning — a phenomenon known as catastrophic forgetting.

The result is that each adaptation is an isolated event. The model does not learn to learn. It does not improve as a learner through experience. It does not develop structural preferences that make it progressively more efficient at tackling certain types of problems.

The Proposal: A Talent for Every Model

The Talent-Conditioned Transformer, or TCT, is a framework we have developed at Algoretico to address exactly this problem. The central idea is conceptually simple, although the technical implementation is anything but trivial.

Each Transformer model is associated with a vector — a fixed-size mathematical object, typically 64 or 128 numbers — which we call talent prior, or simply talent. This vector does not contain knowledge. It is not a database of facts, it does not store answers, it is not a hidden prompt. It is something more fundamental: a set of structural preferences that influence how the model processes information.

Talent acts as a persistent cognitive filter. It determines which internal circuits of the model are activated most strongly, which computational paths are favored, and how internal resources are allocated among different sub-problems. Two models with the same base training but different talents, placed in front of the same input, will follow different computational paths — not because they have different knowledge, but because they have different processing styles.

The crucial difference compared to all existing adaptation methods is one word: persistence.

When a talent-equipped model faces a new task, the talent is not erased and recreated from scratch. It evolves. It updates slowly, on a timescale much slower than the specific task learning, incorporating the structural experience accumulated from previous tasks. A model that first learned to classify the sentiment of movie reviews and then tackled logical inference tasks will develop a different talent from one that followed the reverse path. The sequence of experiences shapes the cognitive profile of the model, just as it does with humans.

This is not a poetic analogy. It is a measurable property of the system, and our experiments demonstrate it.

How It Works: Three Mechanisms, One Vector

Talent is not a simple linear modifier. It acts on the model through three distinct but coordinated mechanisms, all controlled by the same vector. This is an important architectural point: the coherence of the system arises from the fact that a single mathematical object simultaneously governs three different levels of modulation.

Activation Modulation (FiLM)

The first mechanism is inspired by a technique from computer vision called Feature-wise Linear Modulation. In each layer of the Transformer, after the activations have been normally calculated, talent generates two sets of parameters — one for scaling and one for shifting — which are applied to those activations.

The most intuitive analogy is that of an audio equalizer. When listening to music, the equalizer does not change the recording: it emphasizes certain frequencies and attenuates others, adapting the sound to the environment or the listener's preferences. Similarly, the talent-conditioned FiLM does not change the model's weights: it modulates the signal passing through the model, emphasizing certain computational patterns and attenuating others.

In practice, a model with a talent oriented towards logical reasoning might amplify the activations associated with processing complex syntactic structures, while a model with a talent oriented towards emotional understanding of text might emphasize patterns related to lexical semantics.

Conditioned Expert Routing

The second mechanism intervenes at a more structural level. In some layers of the Transformer, the standard processing block is replaced by a group of specialized sub-networks — the so-called experts. Each expert is a small independent neural network, and for each input, the model activates only a subset of them (typically two out of four).

In traditional mixture-of-experts systems, the choice of which experts to activate depends solely on the current input. In the TCT, talent intervenes in the selection process, shifting the activation probabilities in favor of certain experts over others. This means that two models with different talents, even in front of the same input, will activate different combinations of experts — and therefore process the information through physically different circuits of the network.

The effect is an implicit specialization: the model not only learns what to do, but also learns which of its internal tools are best suited to the types of problems it has faced in its history.

Adaptive Parameter Generation

The third mechanism is the most powerful and also the most computationally expensive. A small auxiliary neural network — a hypernetwork — takes talent as input and generates modifications to the attention weights of the main model.

This is qualitatively different from the first two mechanisms. FiLM modulates the activations (the signal passing through the network). Expert routing selects which computational paths to activate. The hypernetwork goes further: it physically reconfigures parts of the model itself. The attention weights — the heart of the mechanism that gives the Transformer its name — are modified based on talent, making attention itself dependent on the model's cognitive history.

The modifications generated by the hypernetwork are low-rank, meaning they are parameter-efficient and do not require retraining the entire model. But their effect is profound: they literally change the way the model decides what is important in an input.

Two-Level Optimization

There is one last architectural piece that holds everything together: the way talent is updated.

The TCT uses a bi-level optimization. The inner level is classic training: the model learns to solve the current task, updating the task-specific parameters (the classification head, the FiLM parameters, etc.) at each batch of data. The outer level operates on a slower timescale: every hundred training steps, talent is updated based on its performance on a separate validation set.

This temporal separation is crucial. The inner level answers the question "how do I solve this task?". The outer level answers the question "how should I change my cognitive profile in light of what I am learning?". The slowness of the outer update is intentional: it prevents talent from becoming a mere reflection of the current task and forces it to capture structural patterns that transcend individual tasks.

The Results: What We Measured

The theory is one thing. The question that matters is: does it work?

We tested the TCT on five different linguistic tasks, chosen to cover a broad spectrum of capabilities: sentiment analysis (SST-2), natural language inference (MNLI), paraphrase recognition (QQP), topic classification (AG News), and measuring semantic similarity between sentences (STS-B). We compared it with the most commonly used adaptation methods in current research: full fine-tuning, LoRA, prompt tuning, and a task embedding with the same dimensionality as talent but without persistence.

Single Tasks

On single tasks, the TCT achieves competitive results with LoRA and prompt tuning. On some tasks, it surpasses them, on others it aligns, and on none is it significantly worse. This result is important but not surprising: the TCT is not designed to excel in isolation. Its raison d'être emerges when tasks accumulate.

Sequential Adaptation

Here lies the central result of the work. We subjected the models to ordered sequences of all five tasks — first sentiment, then classification, then inference, and so on — and measured two things: forward transfer (how much experience from previous tasks helps the current task) and backward transfer (how much learning a new task degrades performance on previous tasks).

Traditional methods show a predictable pattern: as tasks are added, performance on previous tasks declines (negative backward transfer) and the help from previous tasks for future ones is minimal or nonexistent. LoRA, which resets its parameters for each task, has no memory. Full fine-tuning, which does not reset anything, accumulates interference.

The TCT shows qualitatively different behavior. Forward transfer is positive and grows with the number of tasks: the model becomes progressively better at adapting to new tasks because talent has accumulated useful structural information. Backward transfer is significantly less negative compared to all baselines: talent protects previous skills because it has encoded them not as specific knowledge (which gets overwritten) but as structural preferences (which persist).

In practical terms: the TCT learns to learn.

Ablation Study

To understand which components of the system are actually responsible for the results, we conducted a systematic ablation study, disabling one component at a time.

The results show that all three conditioning mechanisms contribute positively, but FiLM and expert routing have the greatest impact. The hypernetwork adds an incremental improvement, which is reasonable given that it is also the most computationally expensive component.

The most revealing data, however, concerns persistence. When talent is reset before each new task — effectively making it equivalent to a standard task embedding — the advantage of the TCT almost entirely disappears. This confirms that the value of the framework does not lie in the individual conditioning mechanisms, but in their combination with a vector that evolves over time.

The Evolution of Talent

Perhaps the most fascinating result is also the most visual. We tracked the trajectory of the talent vector over time, projecting it into two dimensions, for models exposed to different sequences of tasks.

What emerges is that models with different histories develop distinct and interpretable trajectories. It is not noise: the clusters that form in the talent space correspond to families of tasks with similar characteristics. A model that started with logical reasoning tasks and then moved to emotional understanding tasks follows a different trajectory than one that did the reverse, and the two trajectories converge towards different regions of cognitive space.

Talent, in other words, is a mathematically measurable object that captures something analogous to what would be called an "expertise profile" in cognitive psychology.

What It Means, in Practice

The TCT is not a product. It is not a better chatbot, it is not a larger model, it is not an app. It is an architectural framework — a different way of thinking about what an AI model should be.

The practical implications, however, are concrete.

The first is deep personalization. Today, when a company wants an AI tailored to its needs, it has two options: fine-tuning (expensive, rigid, subject to forgetting) or retrieval-augmented generation (which does not change the model's reasoning, only the information it has access to). The TCT offers a third way: a model that develops a cognitive profile calibrated to the company's specific ecosystem, through progressive exposure to its data and tasks. Not a model that knows different things, but a model that reasons differently.

The second is efficiency. Instead of maintaining dozens of specialized models for different applications, an organization could maintain a single base model with different talents for different contexts. Talent is a vector of 64 numbers: it occupies a few bytes of memory. The computational savings compared to maintaining separate models are enormous.

The third is scientific. The TCT provides a tool to study how AIs develop competencies over time. The trajectory of talent is a quantitative window into the cognitive history of the model. This opens connections with cognitive sciences, learning psychology, and the study of individual differences in human expertise — connections that until now lacked a technical basis to rely on.

The Context: Why Now

The idea that a model should have some form of persistent identity is not entirely new. Meta-learning, continual learning, and models with external memory have been active lines of research for years. But the TCT differs in a specific aspect: it does not seek to give the model a memory of what it has learned, but a memory of how it has learned. The distinction is subtle but profound.

A continual learning system seeks not to forget the facts learned in previous tasks. The TCT does not care about facts: it cares about strategies. Talent does not remember that "negative reviews often contain words like terrible": it remembers that the model has learned to be good at distinguishing emotional nuances in text, and uses this structural information to better tackle future tasks.

It is the difference between remembering answers and remembering how to reason. Both are useful, but current research is almost exclusively focused on the former.

Where We Are and Where We Are Going

The paper describing the TCT is available as a preprint on OpenReview, TechRxiv (IEEE), and Zenodo. We are preparing the submission to Transactions on Machine Learning Research (TMLR), a peer-reviewed journal founded by Hugo Larochelle, Kyunghyun Cho, and Raia Hadsell, among the most respected names in the field.

The work has been developed entirely at Algoretico, in Milan. Not in a university lab, not in a research center with hundreds of GPUs, not in a big tech company. In an independent Italian software firm that has been building custom AI for its clients since 2018. We believe this origin is relevant: the TCT was born from a practical need — the frustration of having to start from scratch every time we adapted a model to a new context — and the solution we propose reflects this concreteness.

The framework is in its early stages. Experimental results are promising but conducted on a limited scale (small models, NLP tasks). The road to application on frontier models is long. But the direction seems clear to us: the future of AI is not just larger models or more efficient adaptation methods. It is models that develop computational individuality through their history. Models that, like human professionals, become progressively better not because they know more things, but because they have learned to think more effectively.

The TCT is a first step in this direction.

The complete paper is available at:

• OpenReview: openreview.net/forum?id=YFvj0besR4