This Paper Changed How I Think About Risks of Open-Weight Models

12 Sep, 2026

This is a summary of Casper et al. (2026) Open Technical Problems in Open-Weight AI Model Risk Management paper, along with some of my commentary, pointing out which points made me change how I think about open-weight models and their safety.

TL;DR

Open-weight models are getting better day by day, with the state of the art models trailing roughly four months¹ behind the proprietary models. These models pose new challenges, as they can:

be modified arbitrarily,
be used without oversight,
spread irreversibly.

Authors point out 16 open technical challenges for increasing safety of open-weight models. My main takeaway from this paper: we need a higher standard of risk management from companies releasing open-weight models. Their current approach poses societal risks which could be decreased if more effort was put towards it.

Introduction

With open-weight AI models usage growing quickly, and their capabilities lagging four months behind frontier proprietary models, they likely possess dangerous capabilities already. The deployment of open-weight models differs from the closed-weight ones, with the former having a bigger chance to be misused. Methods which are used to secure proprietary models can't be utilized for open-weight models, whose attack surface is larger and toolkit for defense techniques is less established.

Work on technical safeguards for open models is critical to reap their benefits while offsetting the associated risks. The key areas in which open problems are stated are:

Training data curation, i.e. providing data which doesn't contain harmful content,
Tamper-resistant training and "unlearning" algorithms,
Model tampering evaluations,
Staged deployment strategies,
Model provenance and forensics.

The challenges of open-weight models

The most interesting part of the paper to me was the realization that the challenges associated with open-weight models are very different from those of their closed-weight counterparts. The work also increased my awareness of unique challenges posed by open-weight models, presenting how they are way harder to maintain under control than proprietary models. Managing the risks of a model is more problematic if the model can be replicated, modified and deployed by almost anyone.

Open-weight models are double-edged swords. Their openness is key to enabling more widespread research about them and decentralization of power, but creates unique problems. For example, users can get rid of safeguards that hamper misuse, a method also called tampering.

Open-weight models spread rapidly and once their weights are published, they can't be retrieved. Another downside of these models is that their use can't be centrally monitored or moderated, as they are spread over independent machines with no authority governing their use. This prohibits 'Know Your Customer' policies, enabling everyone to harness the model's power without any supervision. This is concerning.

The supply chains of open-weight models are also more complex than closed-weight ones, they do not have a centralized core and the talent, resources, data, compute and infrastructure is more distributed than that of the closed-weight models. With this in mind, it's also harder to track real-world spread, usage and impact of open-weight models.

As an addition, it is worth mentioning that Dario Amodei of Anthropic and Sam Altman of OpenAI very recently expressed interest in increasing safety of their models by enabling embedded evaluators to look into the company’s internals as if they were their employees. This could possible enhance the security of proprietary models, it is interesting to see whether a similar idea will come spread to the Chinese labs as well.

Toolkit

The emphasis of the paper is put on open problems for technical safeguards with distinct implications for open-weight models. A taxonomy is proposed to manage the risks of open-weight models. Tools that do not have direct implications on open-weight models are not considered in this paper, neither are nontechnical approaches, governance strategies nor development of open-weight models with benefits for society.

Open Technical Problems

The first category is Training Data Curation, which is championed as a key intervention for improving model safety. If a model is not exposed to harmful data during training the risks of it having propensity for such behaviors are minimized. Not all current approaches focus on preventing harmful content from entering training data.

How does data curation's effectiveness differ across harm categories?

Recent work in this direction is promising. Pretraining filtering's ability has been shown to reduce model's knowledge of biorisk-related topics.

How can scaling data curation be scaled across languages, modalities and data/model sizes?

Given the size of internet-scale datasets the feedback loops might be on the longer side. Previous approaches traded-off efficiency for precision, with many good documents being cut off. It's not clear whether bigger and more sample-efficient models need relatively more filtering.

What is the relationship between training data contents and emergent model capabilities?

Can dangerous capabilities emerge from neutral data in practice? Can information from one domain have a bad effect on data from another domain?

The second category is training which is resistant to modifying model's downstream capabilities and unlearning algorithms. Resisting such attacks could improve safety under adversarial manipulations. Post-training safety guardrails are key in this strategy. State of the art (SOTA) models have been vulnerable to such modifications in the past.

How do we develop more tamper-resistant unlearning algorithms?

Leveraging mechanistic interpretability in the context of topics about which models are ignorant could help with enhancing training objectives. There might, however, be limitations for post-training methods' abilities to remove unwanted knowledge from models.

How can we robustly edit model beliefs with minimal side effects?

Instead of making models ignorant about certain topics, introducing certain incorrect beliefs about hazardous topics could counteract harmful output generation. For example models would learn an incorrect way to obtain abusive material.

Can we develop models that effectively resist retrieving harmful information?

With access to tools, models can learn harmful information without having it in its memory, by browsing the web or other files. Preventing this from happening remains an open question.

Evaluating models which are trained to have safeguards removed downstream. It's important to assess practical risks from open-weight models that have been modified. Currently, such evaluations are not a standard practice.

How can we develop rigorous benchmarking and evaluation frameworks?

The risks are widely understood, but their occurrence is not reported often enough.

What modifications should be used for worst-case risk estimation under model tampering?

Threats other than fine-tuning ones, like low-rank modifications, latent-space attacks or distillation can impair safety of models, and should be studied more.

How can we systematically identify effective attacks and defenses?

Tampering attacks are computationally intensive, developing practices to do such evaluations at a scale, while remaining rigorous would require lowering the compute cost.

How can we scalably evaluate thousands of models?

The open-weight ecosystem includes a huge number of models. Assessing their safety at scale would improve practical risk management to a big extent.

Staged deployment strategies are the next theme that is talked about. Slowly rolling out models before fully open-sourcing them would help developers monitor risks and adjust safeguards as well as deployment strategies.

What exfiltration risks do split deployment strategies pose, and how can they be mitigated?

Private model layers would need to be kept secure for successful deployment, but attackers can try exfiltrating the private layers through reconstruction.

How can we design split learning and inference APIs that are less costly and more competitive?

Communication across the split is hard and introduces latency. The overhead is a limitation to other deployment forms, making split strategies less competitive.

Can hardware locking or homomorphic encryption offer practical options for staged deployment?

Hardware locking introduces security guarantees within physical processors themselves. In machine learning this typically means binding model weights to a secure, isolated area of a CPU or a GPU. Homomorphic encryption enables computations to be done on encrypted data without ever deencrypting them. Both of these methods could improve the security of open-weight models, but it remains unclear in what way they would do so. It's not sure if they would find broad applications.

The last category that the paper talks about is model provenance and forensics. Both provenance and forensics help people examine the spread and use of open-weight models, and track the lineage of what the base model was, of a particular fine-tuned version.

How can we watermark models in ways that are more durable against common modifications without side effects?

Evaluations suggest that current solutions in this space can be erased through quantization, fine-tuning, model merging and pruning. Lack of standardized benchmarks that compare watermark durability is also a problem.

What algorithms can enable scalable and versatile model heritage inference?

It would be desirable to track the heritage of models across the whole ecosystem, but with current infrastructure and methods it is not possible.

How practical and scalable are proof of training methods?

Proof of training methods refer to ways to formally verify whether a specific neural network was genuinely produced by a particular computational training run, whether the model was or wasn't trained on a given dataset or if the training adhered to safety, governance and algorithmic constraints.

What techniques are open-weight developers reporting on?

The results of this section stem from model cards of popular open-weight models. In short, almost none of the solutions mentioned above are being talked about in most popular system cards. Data curation stood out as the most common technique but reports on the rest of the techniques are almost non-existent. There is a lot of potential in increasingly reporting measures talked about in this paper.

table-open-weight This table is one of the most interesting findings in the paper. It shows how almost none of the open problems are mentioned in the models' system cards.

The table above portrays a big gap in the reporting of open-weight models. Open-weight creators do not report the associated risks clearly, most likely omitting ever measuring them.

Discussion

Open-weight model capabilities remain four months behind frontier proprietary models. This is not a lot. They pose specific risks due to their fast-spreading nature and ease of which their safeguards can be bypassed. The paper focused only on technical tools, but a more general monitoring and risk mitigating approach is crucial for capturing the benefits of this technology. The extent to which safeguards for open-weight models will ultimately be is unclear, but even with safeguards some models might pose serious risks.

Gathering more information about open-weight models is valuable, and in doing so the community should be aware of 'openness washing' as well as 'safety washing', analogies to the well known greenwashing phenomenon.

Current incentives are not well aligned for private actors to research and develop safeguards for frontier open-weight models. Even if they were, safeguards are effective only to some extent, therefore working on this issue is potentially a high-risk and limited reward undertaking.

Authors end by emphasizing the value of open scientific collaborations, open evaluations, open reporting about risk-management methodology and open standardized documentation.

Through engaging with this paper I changed my perspective on how risky open-weight models are. Current releases lack the sophistication and rigidness of necessary safeguards reporting. Let's spread this message so that more people are aware of these problems and the companies will be forced to change.

Through engaging with this paper I changed my perspective on how risky open-weight models are. Current releases lack the sophistication and rigidness of necessary safeguards reporting. In the light of the recent commitments to embedded evaluators from Anthropic and OpenAI, I would like to see more care put into the state of open-weight models as well.