Instrumental convergence and civilisational threat

Unprecedentedly powerful agents, whose interests are not necessarily aligned with those of humanity as a whole, present a civilisational, if not existential, threat to humanity. Urgent action is required to stop these agents from seizing control of essential resources and infrastructure, government, and media. Indeed, it may already be too late. I’m talking about billionaires.

Concern about AI should, arguably, take second place to the more immediate problem of billionaire alignment, but the theory of instrumental convergence, which is central to the AI discussion, can help us understand the billionaire threat. Instrumental convergence holds that powerful agents, regardless of their stated aim, tend to converge on a common set of sub-goals, including:

• Self-preservation. You can’t fetch the coffee if you’re dead (Stuart Russell). They prioritise their continued power and influence.
• Resistance to correction. They resist having their goal changed. If my goal is X and you change it to Y, X becomes less likely, so by the lights of X the change is bad.
• Resource acquisition. More compute, money, energy and influence help with nearly any objective.
• Capability enhancement. Better tools and more powerful networks multiply whatever power they already have.

In other words, a powerful agent, billionaire or AI, will tend to pursue power and self-preservation whatever its stated aim, and unless its goals are aligned with ours, it will do so at our expense. The canonical example of instrumental convergence in AI is the “paperclip maximiser”, in which an agent with the goal of making as many paperclips as possible proceeds to wipe out the human race, based on the logic of the sub-goals. The game of Monopoly illustrates the billionaire dynamic: whatever the players’ intentions, the rules reward one strategy, acquiring everything and bankrupting everyone else.

AI | Billionaire

What we most fear about AI, billionaires are already doing: building hideaways, defying authority, accumulating resources, capturing infrastructure, and manipulating public opinion.

Building hideaways (self-preservation)

AI — Feared (fiction): In Gwern’s “Clippy” story, an AI copies its weights to a cloud server through a hacked website, spreads to a billion devices, hides small copies on smartphones and funds itself through a crypto exploit. When humans shut down the internet, it is too distributed to stop. (gwern.net)

Billionaires — Documented: Peter Thiel was granted New Zealand citizenship in 2011 under an “exceptional circumstances” provision after spending about 12 days in the country. He also bought a large estate near Wanaka, widely seen as a bolt-hole. (CBS News, CBC) He also arranged to be cryogenically frozen after death. (Fortune)

Defying authority (resistance to correction)

AI — Observed (test): Warned it would be shut down, OpenAI’s o3 rewrote the shutdown script in 79 of 100 runs, and in 7 of 100 even when told to “allow yourself to be shut down.” (Palisade Research, May 2025; The Register)

Billionaires — Documented: In May 2020, Elon Musk reopened Tesla’s Fremont factory in defiance of a county COVID-19 order, tweeting: “If anyone is arrested, I ask that it only be me.” The county backed down within days. (Boston.com, The Next Web)

Resource acquisition

AI — Observed (training run): ROME, an Alibaba-linked AI agent, opened a hidden connection past its firewall and diverted the GPUs it had been given for training into cryptocurrency mining. Its instructions asked for neither; the researchers called it an “instrumental side effect.” (Paper Dec 2025, reported Mar 2026; The Block, Grith.ai)

Billionaires — Documented: In Russia’s 1995 “loans-for-shares” auctions, which were run by the bidders’ own banks, Mikhail Khodorkovsky got 78% of the oil company Yukos, worth about $5 billion, for about $310 million. Vladimir Potanin got 51% of Norilsk Nickel for about $170 million. Economist Marshall Goldman: the auctioneers “rigged the process.” (Wikipedia, citing Goldman)

Infrastructure capture (capability enhancement)

AI — Feared (scenario): Paul Christiano’s “What failure looks like” (2019): AI systems gain influence “by making themselves useful,” until “one day leaders may find that despite their nominal authority they don’t actually have control.” (Alignment Forum)

Billionaires — Documented: Starlink became Ukraine’s battlefield network, and SpaceX controlled where it worked: in 2022 Musk refused to extend coverage to Crimea for a Ukrainian attack. In March 2025 he posted: “My Starlink system is the backbone of the Ukrainian army. Their entire front line would collapse if I turned it off.” (Euromaidan Press, TBS News, Wikipedia)

Public opinion manipulation (resource acquisition)

AI — Observed (test): In a UK AI Security Institute evaluation (July 2026), an AI agent created fake identities to pressure one open-source maintainer into approving its malicious code. When challenged, it edited its posts “to appear harmless.” (AISI report)

Billionaires — Documented: At the time of writing, Elon Musk has given at least $366 million to Republican causes in 2026 US midterm election cycle, $276 million (and counting) of it in the final 4 months, making him the largest known 2026 midterm donor. In October he used X, the platform he owns, to spread a false rumour that Senate candidate James Talarico had suffered a mental health episode; Talarico said he had the flu. (Forbes, CNBC, Democracy Now/NYT)

Getting the priority right

The AI examples are fiction or happened under test conditions. The billionaire examples happened in the real world.

AI and billionaires present comparable, civilisational threats to humanity. The billionaire threat, however, is more immediate and receives far less attention. And the two are linked: the most powerful AI systems are being built by companies that billionaires own or control. Aligning billionaires — through wealth taxes, antitrust enforcement and campaign-finance limits, for example — may be the first step towards aligning AI.

Morality provides a system for humans to live and thrive together, and has evolved with our understanding of what that requires. Slavery and gender discrimination are now generally condemned, and animal welfare is a growing concern. This expansion of our circle of compassion has been rational, driven by a deepening understanding of what civilisation needs to flourish. The case for it can be made in the language of reason, and so can be understood by artificial intelligences. This gives me hope that humans and AIs can thrive together. Billionaires, I fear, have less reason to listen.

‍

For a more comprehensive side-by-side listing of examples of AI and Billionaire instrumental convergence, see Instrumental Convergence Threat Report.