Capitalism for AI Safety

Much of September’s AI narrative mocked the labs for pleading for a slowdown while they charged ahead.

We have to build AI that murders us, because if we don't, China will build it first, and I don't want to get murdered by a computer that speaks Chinese. That would be ridiculous.

— Ray Kump (@RayKump) September 8, 2026

As ridiculous as it is (and I don’t absolve anybody here), the race condition isn’t easy to escape. We’re on a capitalist playing field, and one player sitting out doesn’t stop the game. We can try to tweak the rules, change some players, or encourage everyone to play nicer. But ultimately what matters is what scores.

Charlie Munger’s rule was to “never, ever, think about something else when you should be thinking about the power of incentives.” Our best chance to make safety count as much as speed is to put capitalism to work on safety too. There’s a lot of money and effort going to safety, but that money isn’t enough on its own.

This piece explores how we can close the gap and put market forces behind safety.

Capitalism as the field

Every lab faces the same choice: if it slows down and its rivals don’t (for safety, employee satisfaction, or any reason), then it loses the customers, the researchers and the capital. That means it loses influence in how the technology turns out - whether for selfish or altruist or mixed reasons. So each one races, even the ones that would rather not. The people running the labs say so themselves: a “race to the bottom, spurred by commercial incentives, can make these risks more acute,” Dario Amodei wrote in September (We Must Pace the Frontier). This is because the market rewards the winning competitor with all the gains, but the market doesn’t adequately charge for the risk society bears.

OpenAI planned for this dynamic too. Its own researchers pointed out in 2019 that other industries price in safety failures through “external regulation, self-regulation, liability, and market forces,” and that AI companies’ safety incentives “appear to be weaker than they are in other industries” (Askell, Brundage & Hadfield). They tried to hold back early on, writing the OpenAI charter in 2018 that promised that if a safer project got close to AGI first, it would “stop competing with and start assisting this project” (Charter). In 2019 it held back GPT-2 over misuse fears (OpenAI).

However, when OpenAI heard Anthropic in late 2022 had a chatbot and might release it soon “all of a sudden, it was like, we got to ship this in two weeks” (Reuters). In 2024, a lawsuit alleges, OpenAI moved GPT-4o’s launch to the day before Google’s event and “compressed months of planned safety evaluation into just one week” (Raine v. OpenAI).

DeepMind, the third leading lab, spent years trying to put the company under independent oversight and to do their research in an insulated, safety-conscious way. Once the AI race was underway, though, “the exercise, [the founders] both agreed, had been futile,” and they settled on the best route to safety being to grow their personal influence inside their companies (Mallaby, The Infinity Machine).

This year, Meta delayed their Muse personal agent for months over safety problems. Then, in a meeting where leaders discussed Instinct, a 14-person startup whose agent was taking off, Zuckerberg “told them Muse was ready to launch despite the risks” (NYT, Oct 9). The fear of losing in the market outweighs safety concerns.

In September, as the argument over regulation, engineering and market forces heated up, Zuckerberg posted that “labs have a strong natural incentive to make their models more aligned” (X, Sept 15), asking us to ignore Meta’s long history of unsafe, misleading and fraudulent products and policies proving exactly the opposite incentives and outcomes. Echoing this logic, Jensen Huang told Ezra Klein they’re “so incentivized to ship safe products. If they ship unsafe products, their customers go away” (Sept 23).

This would be nice if trust & safety carried this weight, but in practice it has been tested and failed.

The gap in AI safety incentives

A lot of people care about AI safety, and a TON of money is flowing into it. Philanthropic funding for AI safety is on track to more than triple this year, to about $1.8 billion by our count of what the funders expect to give (AI Safety Funder Bulletin), or about $2 billion counting government safety institutes. The biggest funder, Coefficient Giving, committed $168 million in 2024 and is “on track to commit over $1 billion” this year (Coefficient Giving). Far more is on the way. Nan Ransohoff, who built Stripe’s climate program and Frontier, its billion-dollar carbon removal purchase commitment, expects AI wealth to “add ~$37–100B per year in intended philanthropic spend in the near future,” with safety near the top of the list (May 2026). Anthropic’s seven co-founders alone “have pledged to donate 80% of their wealth” (Transformer), about $90 billion at its $900 billion valuation in May, by Ransohoff’s math.

But money isn’t what’s scarce. People are, and two kinds in particular. The first is grantmakers: Coefficient Giving can’t hire them fast enough to spend what it has. “Hiring one fewer grantmaker usually means those millions will just sit in an account for another year,” one of its program leads wrote in May (Muehlhauser). The second is founders worth funding. Ransohoff writes that “the shortage of great philanthropic startup founders is the single bottleneck today” (May 2026), and the person running Coefficient’s startup grants calls talent “the most important bottleneck, maybe the only bottleneck” (Nadeau). More money fixes neither shortage.

Why doesn’t this scale of money promise a solution? Part of the difference is size. Google, Amazon, Microsoft and Meta plan to spend about $725 billion on capex this year (FT, via Tom’s Hardware), and investors put $345 billion into AI companies last year (Stanford AI Index). That’s hundreds of dollars racing for every dollar going to safety.

Also important is how the money moves. Most safety money works through goodwill: grants, mission, people choosing to do the right thing. Grant money has to be handed out one decision at a time by a small number of people, and the upside stops at the grant. Ransohoff did the math for AI’s new philanthropists: giving their money away directly would take “on the order of ~22K grants per year” and “~2,500 employees” (May 2026). That’s why the millions sit in accounts. Market money mostly allocates itself: customers signal what’s worth building, investors spread bets across many teams, the winners grow on revenue, and the people who build them get rich.

Even with a lot of money, funds available to safety in nonprofit forms don’t offer the same mechanisms of upside, competition, exit, etc. “Many of the best and brightest rockstar builders, even those who genuinely want to do maximum good, end up taking a for-profit path, even if it ends up compromising on the mission” (Buterin & Optimism, 2021).

Salary isn’t the gap anymore. METR, an independent model evaluator, advertises “base salaries competitive with top AI labs” (METR). What grants can’t offer is a stake in the outcome, or a customer, and without a customer “you’re not going to be able to raise a lot of capital from the venture capital world” (Nadeau). Leveling the field means giving safety some market pull of its own, and some law behind it.

Ways to put the market behind the money for safety

The basic goal: make safety pay, and make the lack of it cost, and do it at magnitudes big enough to compete with the race. We probably can’t get all the way, but we do have levers we can push.

1. Make preventable harm a crime for the company

Zuckerberg’s post argued that “labs face significant liability if their models cause harm.” That’s key! Except, unfortunately today they mostly don’t. There’s a legal model built to fix that: a “failure to prevent” offense. Darryl Slabe proposes one for the companies that build and deploy AI (Tech Policy Press), modeled on the UK’s offense of failing to prevent fraud. If a lab’s AI causes serious harm, the lab is convicted unless it proves it “took reasonable precautions and exercised due diligence” to prevent it.

Blame and liability in AI systems is hard to pin down: these systems aren’t deterministic, and when one causes harm, responsibility could sit with the company that trained the model, the one that built the app or agent on top of it, or the user who prompted it. That complexity is real, but complexity theater is also a good way large companies aim to stay out of court, and we assign responsibility in complicated supply chains all the time (a carmaker answers for the airbags its supplier made). A failure-to-prevent offense sidesteps the blame question: “the lab is not answering for an AI’s actions, but for its own failure to prevent them from causing harm.” A prosecution “would become a trial of the safeguards, with evidence, in public,” and the point “is not to increase prosecutions; it is to change industry practice and attitudes to risk.”

The UK has used this model against bribery since 2010. A House of Lords committee found it “generally agreed to have been remarkably successful,” because companies “now have every incentive to put in place procedures to prevent this happening” (House of Lords, 2019). Rolls-Royce paid £497 million to settle charges that included the offense (judgment). In 2025 the UK extended it to fraud.

For AI, the stick has to be big enough to matter. California’s frontier AI law caps penalties at $1 million per violation (SB 53), and New York cut its own cap from $10 million to $1 million this year (Cooley). Against OpenAI’s revenue, now running near $50 billion a year (Bloomberg), that’s a rounding error. Size alone isn’t enough either: Facebook’s stock rose on news of its record $5 billion FTC fine, because the fine left its business model alone (MIT Technology Review). What changes behavior is a penalty that can reach the business itself: a model withdrawn, a capability withheld, a monitor inside the company.

Finance has built regulations with more bite. Sanctions law is strict liability: a company can be fined even if it “did not know or have reason to know” it was breaking the rules (OFAC). And the penalties reach the business. After its fake-accounts scandal, the Federal Reserve barred Wells Fargo from “growing any larger than its total asset size as of the end of 2017” (Fed), and the cap held for seven years (Fed, 2025). New York suspended BNP Paribas’s dollar clearing for a year in the business lines where it broke sanctions (NYDFS). Europe’s bank supervisor pulled Pilatus Bank’s license outright (MFSA). There’s no compelling reason we can’t hold the companies building frontier models to the same standard we hold financial institutions: strict rules on specific things, and penalties up to losing the license to be in the market.

And once labs have to prove precautions, safety work has paying customers. Financial firms in the US and Canada spend $61 billion a year on financial-crime compliance (LexisNexis Risk Solutions), about 30 times everything going to AI safety. Fire safety got built the same way: in 1894, fire insurers funded the lab that became Underwriters Laboratories (UL).

Insurance, which can play a role, should not be looked to to carry this. It was supposed to do the same job for cybersecurity, and two decades in, a review by RUSI, the UK defense think tank, found “little conclusive evidence that premium discounts or other financial incentives are directly improving organisations’ cyber security practices.” One study it cites found that nearly half the pricing models insurers filed with US regulators didn’t consider security controls at all (RUSI, 2021). Insurers are good at pricing frequent losses that happen one at a time. The AI losses that matter most are the opposite, one failure hitting thousands of customers at once, and Chubb, W. R. Berkley and Great American have asked regulators to let them exclude those losses (TechCrunch).

2. Certify safety, and pay for it

Rules set a floor. A certificate can reward going above it, but only if someone pays attention to the certificate.

Tech already has certificates that work, because buyers ask for them. Most enterprise customers won’t sign a software vendor without a recognized security audit like SOC 2: 77% of security teams name one as their top vendor requirement (ISC2, 2025). Federal agencies “must obtain and maintain a FedRAMP authorization” before putting government data in a cloud service (OMB), so a cloud company without one can’t sell to them. The audit costs money, and it pays for itself in customers.

AI is starting to get the same thing. AIUC certifies AI agents and insures them, and “the cost and availability of insurance are tied directly to audit results: safer systems are offered better terms” (Insurance Journal). It just raised $40 million to begin auditing frontier models (SiliconANGLE). The labs get graded too: in the Future of Life Institute’s safety index this summer, no lab scored above a C+, and xAI got an F (FLI).

But currently nothing pays out on those grades. I couldn’t find a buyer, insurer or regulator that uses them.

Putting money behind a certification means making it the price of things labs already want. The first could be government business. Federal purchasing already sets conditions on AI models, but today they’re about ideological neutrality, not safety (OMB M-26-04). A safety certificate could be the condition instead, the way FedRAMP is for cloud. This could set the standard other enterprises then follow.

Another lever is a tax credit for safety. The tax code already pays for specific kinds of research, and it can move a whole field. The orphan drug credit paid back half the cost of clinical trials for rare-disease drugs. Before 1983, “only 38 drugs to treat rare diseases had been approved,” and by 2022 there were 882 (Milbank Quarterly). The same law also gave those drugs seven years without competition, so the credit wasn’t the only cause. Americans for Responsible Innovation, a policy group, has proposed a safety version: a “25 percent tax credit for qualifying investments in AI security research,” with the findings published (ARI, 2025).

The biggest lever is a tax break the labs already get. Senator Mark Warner suggested in September making safety standards “a prerequisite for corporate tax breaks, such as the bonus depreciation” (Reuters). That’s the break that lets companies write off chips and servers the year they buy them. Against the $725 billion the big four plan to spend on capex this year, it’s a carrot the labs and their clouds would notice.

The risk with tax credits is in certification and auditing. Without proper oversight, every lab would relabel its work as safety, and auditing AI systems is notoriously difficult. A more robust certification and benchmarking ecosystem is developing, though. Researchers led by the MIT Media Lab just released ImpactBench, a benchmark in the language labs already use, measuring a much wider set of outcomes related to human flourishing than standard benchmarks, like whether a model helps people learn or does their thinking for them. It could play a role alongside purely mechanistic safety audits.

Gillian Hadfield and Jack Clark, a co-founder of Anthropic, propose going one step further and making the auditors themselves a market. In their “regulatory markets” proposal, governments would require AI companies “to purchase regulatory services from a government-licensed private regulator.” Government sets the goals, and the licensed regulators compete on how to meet them, “relying on market forces and industry R&D efforts to pioneer the technical methods of regulation” (Hadfield & Clark, Jurimetrics, 2026). It essentially bootstraps the safety industry with guaranteed paying customers, and gives the regulators a reason to get better at it instead of just ticking boxes.

3. Power-law prize pools

Startups run on upside. Most fail, and a few win so big they pay for everything else: in one dataset of about 7,000 venture deals, “6% of deals produced at least a 10x return, and those made up 60% of total returns” (Benedict Evans). That tail is what pulls the best people toward risky bets, and safety grants don’t have one.

Grants typically work the other way around. A committee decides up front which projects deserve money, and the reward stops at the grant, however much the work turns out to matter. A power-law prize pool could try to bring the dynamism of startups to safety: a huge pool that pays teams after their work proves its impact, with huge upside for outsized impact. It rests on the motivation of startup equity and a simple insight from funding public goods: “it’s easier to agree on what was useful than what will be useful” (Buterin & Optimism).

Prizes like this have worked before, and they show two things. A prize can pull in far more effort than it costs: the $10 million Ansari XPRIZE drew over $100 million in team spending (XPRIZE). And it can fail before it works: DARPA’s 2004 self-driving challenge had no winner, since “the top-scoring vehicle traveled only 7.5 miles,” and a year later five teams finished (DARPA). Safety funding already shows the kind of outsized return a pool would reward: a $143,000 early grant to a young safety team was followed by “Coefficient Giving’s largest-ever AI safety grant,” about $160 million (EA Forum).

The difficulty is judging. The biggest experiment in retroactive grants ran in Ethereum and other crypto ecosystems, where organizers found “mega rounds devolve into popularity contests” (Optimism retrospective), and its payouts looked nothing like a power law: a top-1% recipient got about six times the median (Optimism, 2024). Voters could say which project was better, but not how much better. A safety pool would need objective finish lines where it can (an attack nobody breaks, an eval two labs adopt), a small expert jury for the rest, and a few payouts big enough to compete with a lab offer. As one commenter on a recent AI safety prize proposal put it, “The whole game here is picking the juries” (Delaney).

4. Retention grants

Right now the best safety researchers face an uneven choice. Labs offer equity on top of salary, and they own the frontier models and compute the most important safety research needs. Industry “now hires nearly 70% of AI PhDs, compared to 21% in 2004” (CSA, citing Brookings), and the labs’ own programs work as pipelines: over 35% of past Anthropic safety fellows went on to join Anthropic (Anthropic). Strong safety teams inside the labs are good. The problem is how little is left that answers only to safety.

A retention fund would match a lab’s offer and, as important, bring compute and model access with it, so a top researcher can turn the lab down and keep doing independent work.

5. Deconstructed markets

A share price compresses everything a company does into one number. As one primer puts it, “every asset price you have ever looked at is really just an average across all possible futures, compressed into one number” (Token Dispatch). When that number misses something, companies split, spin out or reorganize so the market can see it.

Now there’s a market for the parts. Kalshi runs markets on companies’ KPIs, like Tesla’s production or Netflix’s subscribers, because “a share price is a single number standing in for dozens of separate judgments about a business” (Kalshi). Conditional markets go a step further and price one thing given another: what is this company worth if it does X? If we can price parts of a company separately, we can price its safety separately too.

Pricing safety doesn’t pay for it: a prediction market only moves money between traders. But these markets do open potential new ways to funnel upside to safety, and they may merit experimentation. For example, the company could sign a contract tied to the number. That already exists in climate: with a sustainability-linked bond, a company’s interest rate drops if it hits an emissions target and rises if it misses. When Enel missed its 2023 target, it cost the company nearly $90 million in extra interest (ESG Post). An AI version would tie a lab’s cost of capital, or a bounty pool, to safety targets that a market or auditor tracks, so a safety choice that costs share value earns some of it back.

These tools cut both ways. Markets can reward bad outcomes as easily as good ones, and any target a lab controls, it can game. Sustainability-linked bonds draw fire for soft targets and tiny penalties, and today’s markets on AI labs are thin enough to push around. This is the most speculative idea here.

A race to the top

Can we make being careful as rewarding for AI labs as being first? If the race is the game, can we change what it pays for? These mechanisms would add to grants and regulation, not replace them.

Amodei’s stated aim is “to make safety something on which AI companies compete. In other words, to create a race to the top.” Getting there means putting market forces behind safety as fully as we can. We’ll never price in every externality, but we have more levers than we’re using, and a lot of money and goodwill waiting to be put to work.