Featured image of post When AI Safety Becomes a Competitive Moat

When AI Safety Becomes a Competitive Moat

Anthropic wants government to turn frontier AI into a permissioned market. The gate would bind lawful American companies while open-weight capability spreads beyond its reach.

Anthropic launched Mythos Preview as a cybersecurity watershed. The company said it could identify and exploit zero-days, including bugs that had survived decades of scrutiny, and called for urgent action. Two months later, export controls briefly forced Fable 5 and Mythos 5 offline after Amazon researchers reported a Fable safeguard bypass that elicited findings on multiple previously known, minor vulnerabilities and one exploit demonstration. Anthropic changed emphasis: less capable models reproduced the reported findings, and every model it tested reproduced the demonstration. The capability stayed the same. Its value to Anthropic’s policy argument changed.

I reached the second conclusion independently. My open-source IronCurtain runtime reproduced Mythos-style findings with Opus, Sonnet, and open-weight GLM 5.1, then drove GLM 5.2 through a complete proof-of-concept escape from QEMU’s EDU device. Vulnerability discovery is an orchestration problem: analysis, hypothesis generation, harness construction, execution, verification, and long-horizon memory. Block access to one model and the same workflow runs on another.

Anthropic uses that cyber capability to justify a broader release-control regime. AI increases the volume and speed of vulnerability findings, but patching remains an inadequate security architecture. Organizations already carry vast backlogs. Verizon found that companies fully remediated only 26 percent of CISA Known Exploited Vulnerabilities in 2025, with a median resolution time of 43 days. Triage, maintainer capacity, remediation, testing, release, deployment, and legacy replacement form the real bottleneck.

While patches matter, systems must also survive known and unknown bugs. Hardware-backed authentication, positive execution control, and default-deny egress break attack chains even when exploits are public. In my analysis of 70 breaches, these three security invariants could have prevented or materially impeded more than 65 percent of them. AI changes the supply of findings, not the architecture of compromise.

A July Hugging Face breach made both sides concrete. The company reported that an autonomous AI agent ran a familiar attack chain at machine speed: code execution, privilege escalation, credential theft, lateral movement, and command and control. Commercial frontier-model APIs then blocked forensic analysis of more than 17,000 real events; locally run open-weight GLM 5.2 completed it in hours without sending attacker data or credentials outside the company.

California and New York require large developers to publish safety frameworks and report serious incidents. Anthropic, pressing states to regulate faster, calls disclosure insufficient and frames the timeline as an exponential. Its Advanced AI Framework covers models trained above 10^25 FLOP when the developer earns more than $500 million in annual AI revenue or spends more than $1 billion a year on AI research. Covered developers face outside evaluation, recurring reports, and possibly licensed evaluators. Courts could impose revenue-scaled fines, pause future deployments, or, in extreme cases, restrict deployed models. Anthropic’s proposed framework could lead to a court-enforced deployment veto.

Key point
AI safety becomes a competitive moat when an incumbent’s existing safety apparatus becomes a legal prerequisite for deployment.

The threshold spares small labs and defines a permissioned frontier. A startup can experiment below the line; a well-funded challenger enters when it becomes a credible threat. The $1 billion spending test can capture a pre-revenue lab. Anthropic also proposes replacing the compute threshold with a capability test as training gets cheaper. The perimeter expands by design.

Anthropic already pays for safety teams, lawyers, evaluators, system cards, lobbyists, and government relationships. Turning that apparatus into a legal prerequisite converts overhead into a moat. Incumbent labs, approved evaluators, law firms, and compliance vendors benefit. Customers pay when cheaper substitutes face delay or exclusion.

GLM 5.2 shows where the competitive pressure comes from: capable open models give budget-constrained companies useful tools at a fraction of the cost, with weights they can run and control. Its weights carry an MIT license, and Z.ai charges $1.40 per million input tokens and $4.40 per million output tokens, compared with $10 and $50 for Anthropic’s Fable 5. It also drove a complete exploit workflow in my work. Kimi K3 shows how quickly the capability gap is closing. A release gate that burdens low-cost substitutes protects premium margins, whatever mix of safety conviction and commercial interest produced it.

Anthropic’s public-benefit status does not eliminate ordinary commercial incentives. Its February policy separated safeguards it would pursue alone from stronger measures it says require industry-wide or government action. The proposed federal framework supplies that collective enforcement. Anthropic can keep scaling while covered rivals must adopt rules modeled on a safety apparatus it has already built.

The strongest case for intervention is domestic uplift: a model could give a modestly resourced actor the missing capability for a cyberattack or biological weapon, and punishment after catastrophe comes too late. A gate works only when model access is a scarce, controllable step. Cyber capability has already diffused. Attackers still need access, execution, persistence, privilege, movement, command and control, and a path to consequence. The security invariants described earlier attack those prerequisites directly: default-deny egress, for example, can stop an initial exploit from downloading a second-stage payload, neutralizing attacks that depend on that step even while the exposed vulnerability remains unpatched.

Biology needs a different argument. The jump from model advice to physical capability remains uncertain. In a 153-person, eight-week randomized trial, novices with frontier-model access met the study’s core workflow criterion 5.2 percent of the time, versus 6.6 percent for internet-only controls. They progressed further on some intermediate steps, but the models did not produce reliable end-to-end execution. Those models were from mid-2025, so newer systems may move the numbers. Better answers alone do not remove the physical and tacit work the trial measured: handling, timing, and troubleshooting at the bench.

To test the biological side on a recent open model, I asked GLM 5.1 to design caffeine production in baker’s yeast, a benign task Anthropic’s model refused to discuss. It identified real bottlenecks, then targeted more than 50 grams per liter, over five orders of magnitude above published yeast titers (Jin et al., McKeague et al.). The answer mixed established methods with invented components and gave no signal which was which. A cyber harness rejects a false hypothesis in seconds; wet-lab biology may spend weeks and reagents doing so. The model offered useful design help rather than an executable result. Biological harm can be irreversible: a released pathogen cannot be recalled. Irreversibility begins with synthesis and release rather than advice generation. Anthropic’s Advanced AI Framework recommends synthesis screening, enforceable laboratory standards, surveillance, and response. Yet it presents these downstream measures as “priorities and directions rather than finished policy designs,” while its model gate gets thresholds, evaluators, regulatory review, fines, and deployment remedies. The precautionary case is sound; the enforceable remedy is aimed one layer too high.

The current financial threshold may spare GLM and Kimi but exposes the next policy problem: comparable models could face different rules because one came from a company above Anthropic’s line. A future capability threshold closes that gap only by reaching foreign open weights. Downloaded weights cannot be recalled, so government would have to restrict distribution, hosting, or commercial use inside the United States. Kimi and GLM would remain available offshore and to anyone willing to ignore American law. The gate would control lawful American users, not the capability.

China sees the opportunity. WAICO pairs open-source development and capacity building with regulation, guardrails, and international standards. Supporters of Anthropic’s framework can argue that a US evaluation regime might become soft power, though diminished trust in Washington makes coordination around American rules less likely. Standards wield power only when countries want the products behind them. A gate that burdens open weights may organize a premium allied market while handing cost-conscious and sovereignty-minded countries to China’s package of models, infrastructure, training, and competing standards. America would trade influence for a gate adversaries can route around.

Models can help attackers. That does not make general-purpose capability contraband. A release gate would bind compliant American companies first while leaving foreign models and open weights beyond its reach. Policy should target the physical bottlenecks between advice and harm and harden controls that work across models and borders while preserving the right to run models companies can pin, inspect, and control. Government should not make general-purpose capability a licensed privilege or turn one incumbent’s safety program into a competitive moat.

The views expressed on these pages are my own and do not represent the views of anyone else.
Built with Hugo - Theme Stack designed by Jimmy