This post describes research conducted by AE Studio in collaboration with Anthropic.
A Frontier AI model is, among other things, a large repository of knowledge. Part of this knowledge is Dual usemeaning it can be used for good or for bad. For example, knowledge of cybersecurity can help close critical security vulnerabilities or exploit them. Knowledge of virology can help a researcher develop a vaccine, but it can also help a malicious actor develop a deadly pathogen. Ideally, we would be able to balance three different goals: first, to limit access to dual-use capabilities in the most surgical way possible; second, to allow trusted users to access the same features for useful purposes; and third, to do all of this without affecting the model’s performance on other tasks.
The current protective measures are incomplete. We train models to reject malicious requests and use classifiers to check inputs and outputs for dangerous content. These protective layers protect against dangerous output – but they do not alter the knowledge stored in the underlying model. Despite our security measures, a sufficiently determined attacker can still attempt to jailbreak the model by overcoming its defenses to obtain the dual-use knowledge.
A more robust protection against abuse would be to control what the model knows. We’ve explored this before: in previous work, we extracted information about chemical, biological, radiological, and nuclear weapons from pre-training data, and later showed that dual-use knowledge can be limited to a removable slice of a model’s weights. But filtering is a blunt instrument. A model is created with a fixed set of functions. Use filtering if you want a model version may To discuss advanced virology – for use in a verified biosafety lab, for example – and another version that cannot, you need to train two separate models. Especially in the case of boundary models (which are large and very expensive to train), the cost would be prohibitive for the developer.
In new research we conducted with collaborators at AE Studio, we are exploring a new method that could enable the benefits of training many separately filtered models, but at the expense of training just one model. We call it GRAM, for Gradient-Routed Auxiliary Modules. Note that the results of the experiments presented here are preliminary – GRAM has not been applied to any of the production models at Anthropic, and we are not sure if it ever will be.
Table of Contents
How GRAM works
The idea behind GRAM is to provide a model with dedicated, removable compartments for each category of dual-use knowledge and only update these compartments when learning from dual-use data.
Specifically, GRAM adds additional neurons to each layer of a standard Transformer (the neural network architecture on which large language models are based). These neurons are divided into groups (or “modules”), one per dual-use category. When the model encounters general text during training, it learns in the usual way. But when it encounters text from a dual-use category – virology, for example – the rules change: the model can use It is his common knowledge to make predictions, but only the virology module is allowed to do this learn from this text. The universal weights are temporarily frozen.1
The result is that virology knowledge accumulates in the virology module and is not disseminated across the entire network. After training, the module can simply be deleted and the ability is lost. Or it can be left in place for trusted missions when virology knowledge is required. The knowledge can be tailored specifically to the type of use required: In our experiments we defined four dual-use categories, so a training run with GRAM resulted in a model that can be configured in 16 different ways (“on” or “off” for each of the four categories).
Test GRAM
We tested GRAM in three environments with increasing realism.
First, a small GRAM model could be reconfigured using a synthetic data set of children’s stories tagged by topic to “forget” any selected topic, and each configuration would behave almost identically to a separate model trained from scratch that filtered out that topic. That is, for the cost of training a single model, we achieved results that would normally require multiple training runs on different data sets.
Second, we trained a larger model on a realistic mix of web text, code, and scientific papers with four dual-use domains: virology, cybersecurity, nuclear physics, and a niche programming language (intended to serve as a proxy for specific dual-use code). The capability associated with each dual-use domain is routed to its own module. Deleting a module removed the corresponding feature about as effectively as if one had never trained on that data. Remarkably, we find that this removal did not impact overall performance.
We also tested whether an attacker could recover the removed knowledge through training on a small amount of malicious data; GRAM fought this about as well as data filtering. In contrast, an “unlearning” technique is applied only after training suppressed the knowledge – it was easy to recover with a little tweaking.
Third, we conducted the experiment with seven model sizes ranging from 50 million to 5 billion parameters. GRAM achieved data filtering performance at any size, and the gap between “module in” and “module out” increased as models increased. In terms of computational cost, attempting to bypass our protections became relatively more difficult and expensive as we scaled up.
Conclusions
As AI companies build ever more powerful models, the need to limit access to dual-use capabilities will increase. Today, companies limit access through classifiers and denial training. However, it is difficult to make these protections robust without impacting performance on benign requests. Methods like GRAM offer a potential path to more robust access control.
This is early research and there are clear limitations. We have not tested GRAM at frontier scale or in a production training pipeline. (As noted above, it was not applied to any of our Claude models.) Our evaluations quantify performance based on the ability to predict the next token, rather than performance on real downstream tasks. And there is a deeper open problem that relates to data filtering and methods like GRAM: some dual-use capabilities may be so closely tied to common knowledge that no method can cleanly separate them.
For more details about our experiments, see the post on our Alignment Science blog.
https://www.anthropic.com/research/off-switch-dual-use
