The new era

Intro

I’m going to give you the conceptual tools to understand how it’s posisble that a machine can do intellectual labour and why they will due to market pressures.

To get there, I want to build you a picture of the progress I and others have seen for the last few years.

LLMs have been making progress absurdly fast. We measure these capabilities through benchmarks. These are tests, much like (sometimes exactly like) how we measure people’s capabilities through standardised tests or qualifications like A-Levels. We track AI’s improvements on these benchmarks over time to see how things are progressing.

For example, you might have heard of the Turing Test, proposed in 1949 by Alan Turing (from Manchester by the way, which Mancunians love to mention). This was a benchmark evaluating a machine’s ability to exhibit human-level intelligence. A human judge engages in a text-based conversation with both a human and a machine. The judge’s task is to determine which participant is human and which is the machine. If the judge cannot reliably distinguish between the human and the machine, the machine is said to have passed the Turing Test. It was originally called the imitation game for obvious reasons.

Up to about three years ago, this was the holy grail of benchmarks and was the talk of the town. Today, no one talks about the Turing Test. LLMs blew right past it.

When ChatGPT launched in November 2022, it could barely do GCSE maths. I still remember where I was when I used an LLM for the first time: I asked about my third-year calculus assignment I was doing, just to test it. It was useless. But still, I was having a conversation with a machine. Six months later, it passed the Bar and got top A-level results, then another six months and it surpassed PhD-level experts in the sciences. A year later, it won gold at the International Maths Olympiad, then the Physics Olympiad, then the IT Olympiad. Today, it does frontier research across all of STEM.

The point is, the raw intellectual capability is here. But how is this possible? How does this integrate with the wider econonmy and become useful?

Cases

GPQA Diamomd, AIME, and SWE-bench together assess PhD-level STEM capabilities. Gold in IMO, IPO, IIO

---

Case Studies: The 2026 Frontier

So we’ve got a machine that can ace every standardised test we throw at it. So what? What does that look like in the real world? The following are all from models like ChatGPT.

1. Mathematics: The Erdos Conquest

Paul Erdős posed 1,000 mathematical problems informally over 50 years — only catalogued as a formal list in 2023. The number solved rose from ~350 to ~480 in six months, driven by LLM-assisted proof search. Results verified by Terence Tao, the Messi of Maths.

2. Physics: Solving a Decades-Old Problem in a Day

A Brookhaven physicist partnered with OpenAI’s o3 reasoning model to solve the one-dimensional frustrated Potts model — a theoretical physics problem that had resisted solution for decades — in a single day, revealing a phase diagram with applications in computing, electronics, and quantum devices.

3. Biology: GPT-5 Runs a Real Lab

GPT-5 autonomously ran tens of thousands of lab experiments autonomously in a cloud lab (a lab with robot arms that can be controlled remotely), and cut the cost of protein synthesis by 40%, significantly pushing the SOTA forward.

4. Coding: The Machine That Builds Itself

100% of code at Anthropic and OpenAI is now written by LLMs — the machine is literally building the next version of itself.

5. Engineering: For Waterman

AECOM spent £300m (4% of their annual revenue) on acquiring Consigli, a specialist team of AI developers in engineering: PhDs, researchers, ML Scientists, AI engineers, and so on. They did this because Consigli was an existential risk to them.

Here’s the state of AI.

First thing’s first: eject the concept of a chatbot from your mind. Eject image generators, deepfakes, and the like. Eject social media algorithms. Eject the algorithm your insurance company uses to assess claims for fraud potential. I am not talking, especially, about any of those things. Instead, I’m talking about LLMs. Simply put, in the coming years, LLMs will be configured in such a way that they can plan, reason, and execute intellectual labor.

Two months ago, mathematicians used the latest versions of ChatGPT, an LLM, to solve frontier maths problems called Erdős problems. The solutions were soon verified by the Messi of Maths, Terrence Tao, establishing the first fully autonomous system to solve an open maths problem. This is truly remarkable, and stories like these extend beyond maths. LLMs are doing research-level STEM.

How did this happen? Do these machines know anything at all? You might be inclined to think they must be cheating, that there’s some sort of intellectual sleight of hand going on. But when we can dig into this question of what we mean when we say “he or she knows x”, you’ll find this isn’t the case. They really do know things and they really are intelligent.

This is a two part question, one part philosophical, the other part scientific; one theoretical, the other practical. Once we understand the theory of what it is to say we know something to know, the practice how LLMs are developed and what the resulting product is capable of makes far more sense. This will give you the concepts to understand how it’s possible that a machine can do real intellectual labour, and hence understand and plan for what’s to come.

So, for the first part, what it means to know things, we need to back 100 years to a philosopher called Wittgenstein, and get to grips with that second “L” in LLM: language.

LLMs, Wittgenstein & Language

Wittgenstein & Language Games

Wittgenstein wants to know the answer to a simple question: how do we know what words mean? anyone read a dictionary front to back? I’m only halfway through, no spoilers please.

Johnny

Imagine a 5 year old called Johnny. Johnny has read a couple of books and he’s come across a word he’s struggling with: “game”. How does he come to know the word? Well, we don’t sit Johnny down and say (sternly), “Johnny, here is the form of the game. Follow this definition only. If anyone tells you otherwise it’s propaganda, fake news. Don’t listen to ‘em.” Of course not, it’s silly: there is no universal, essential definition. What does chess have in common with football, darts, the Olympics? If we want to say that “all games are human behaviours governed by rules”, we’d have a tough time fitting peekaboo into that definition. it doesn’t have any rules, are there offsides? Instead of there being some universal definition, Wittgenstein says that words get their meaning through use. If you want to know what a word means, look for how people use it in everyday language. Very practical, very common-sense. So, Johnny learns what a game is through examples, through practice. We show him noughts and crosses, chess, cricket, the Olympics. And if he says “this thing I’m drinking from, that’s a game”, we correct him and say “no Johnny, that’s a bottle”, and point out more examples of games. So, we learn through this iterative process of practice and feedback. That’s how we know what words mean, generally speaking, and that’s how we verify Johnny knows anything at all: through use.

In other words, we can use language effectively when we learn the patterns of which words go where in everday use: pattern recognition. Over time, our brains build a model of which words and concepts belong where in the surrounding world. To prove this, let’s play a simple game: what comes next? “Fish and…” “Bob’s your…” “God save the…” See what happened with that last one? If our biological brains are big pattern recognition models, then, feasibly, an computational pattern recognition model is functionally the same and can understand language too: a language model. And the bigger the brain, the more patterns it can recognise. Hence, Large. So there we are, Large Language Model, or LLM. LLMs understand langauge, and we can say “ChatGPT knows x” because it has seen words, used them, and has been given feedback on its errors, just like Johnny.

That’s a good starting point, but there’s something missing if we want to understand the meaning of words at any given time…

Context

Our how we come to say “he or she knows x” was Suppose Johnny joins his friends A and B halfway through a conversation, and friend B responds “wot” (phonetically) to friend A. Johnny has no knowledge of what A said to B. Now, Johnny has come across this sound, this word, before, but can he be sure of its meaning? Well, there are many ways in which the sound (word) “wot” might be an adequate response from B to A.

• friend A mumbles incomprehensibly to B … B replies, “what?”
• friend A tells B, “I just won the Euro Millions jackpot” … B replies rhetorically, in shock, “what.”
• friend A asks B, “what is the name of the SI unit for power equivalent to one joule per second?” … B replies “watt”.

Wittgenstein also gives the example of “Water!”, which can be used as an exclamation, an order, a request, or an answer to a question. The point is the meaning of the word depends on the context in which it is used. In fact, the word “water” has no meaning apart from its use within a particular context, or any word for that matter.

We provide this context so Johnny can properly understand the meaning of the words we use and therefore to understand the world around him. This is what we’re doing when we’re interacting with LLMs like ChatGPT, often called prompting. This is what we’re doing when we upload documents for an LLM to read: providing context.

So, that’s how words get their meaning: through how they are used in everyday life and the context in which they are currently being used. Understanding how Johnny learns language gives us an insight into how large language models do the same.

You might object that the machine doesn’t really understand, it’s “just a stochastic parrot”. But ask yourself: how do you know anyone really understands? The only test we’ve ever had is behaviour: can they use the words correctly? By that measure, these models largely pass.

• train to build understanding of world globally
• context to become useful in world locally

So that’s the philosophical basis, the theory, for why it’s reasonable to say that these models know things. How does this happen practically?

So now that we have the theoretical basis for how models can know things like we do, how do they do this in practice? For us, it’s sort of intuitive as we’ve been using language our whole lives. With machines, it’s less obvious. So far, I’ve anthropomorphised these models quite a lot. I’m going to continue to doing that because the parallels are useful. In some ways, LLMs has a brain like ours: it has neurons and synapses (nodes and weights), and researchers are increasingly trying to adapt AI brains to mimic human brains. It too can learn; “machine learning”. First, we need to understand what form the learning material comes in.

Data

You might have heard that AI needs “good training data”. Data is simply recorded information. We learn through information, machines learn through data. For example, I know the names of people in the Manchester office. They’re in my head because it’s information I’ve gathered and it’s been stored in my brain through neural adaptation. However, it’s only data until I record it somewhere. Another example: word documents are data, they are recorded information (it doesn’t have to be in Excel to be data). Good training data is simply the correct recorded information, and there’s plenty of it on the internet.

LLMs intake data from all kinds of sources: webpages, digital books, transcripts, articles, papers. Combined, their training data amounts to trillions of words. Each word is converted into co-ordinates that can be placed in space, such that “king” is near “queen”, “man” is near “woman”, and if you drew a line to connect “man” and “king”, that line would be almost identical to the one that would connect “woman” and “queen”. You could imagine this line representing the concept of “royalty”, and this absolutely blew my mind when I learnt about it. This is, for example, how an image generation model knows to add a crown when you ask it for a “royal puppy”, or whatever people use image generation models for. This is how it learns how abstract concepts relate to each other, in many-dimensional space.

The brain (a web of trillions of interconnected numbers) then learns the patterns of the data by computing these trillions of words represented as co-ordinates like an enormous calculator. Roughly, a trillion tokens (data) times a trillion connections (brain) equals a septillion multiplications to train a frontier LLM, a number so big that I might as well have just said “lots”. When we were in school with our little casio calculators, we’d put a small calculation in like two times two and the little casio would calculate the answer instantly. Then we’d mash the keypad with a really big number and times that by another really big number, and the calculator would have to wait a second before calculating because the numbers are so big. Can you imagine training an LLM with the computational power of a little casio? One septillion multiplications? It would take billions of years because the processor is tiny. The way around this is through…

Data Centres

Simply put, data centres are a big collection of computers and calculators (they all mean the same thing) that perform these septillion multiplications required to train and run an LLM: it’s the hardware the brain runs on. Ours run on carbon (and coffee), their’s on silicon. More computational power (more compute, bigger calculator, the mega-casio) means you can train and run a bigger computer brain, which lets the models learn more sophisticated and abstract patterns in the data (which we might loosely call intelligence, maybe). This increase in compute to improve models over time is called scaling. Compute has been increasing roughly 5x per year for the same cost. It’s also the reason why you should have invested in Nvidia 5 years ago. I thought about doing this but myself, but quickly realised I was a student with no money.

There’s also, algorithmic efficiency: the structure of the brain itself. because we can rewire the brain to get the same performance for less energy, we can use the same amount of compute more efficiently to get even better results. this is the reason you’ve probably heard of DeepSeek. algorithmic efficiency has been increasing 3x per year, i.e. the same energy costs go three times further. software + hardware improvements result in 12x efficiency per year.

These sets the stage for…

Training

Pre-Training

During pre-training (which is really just the first part of training), we are building base-layer of intelligence: an understanding of language. This is the “predicts the next word” part people refer to, this is the method it uses to understand language. The data it uses is much of the internet, everything from silly reddit posts to medical journals. Trillions of pieces of text. To learn, it takes a sentence like “Einstein’s fist name is…”, covers up the answer, makes a prediction, compares the prediction to the answer, and finally adjusts its neurons in a way that’s expected to increase accuracy the next time round, much like how learn by doing corrections in school.

Post-Training

Again, a misnomer, just means the second phase of training. Reinforcement learning: do this all the time with training a dog, teaching a baby, encouraging a child. the language changes but the method is the same: reward what behaviour we want. People often say “ChatGPT is just predicting the next word.” But that’s never been quite true. Raw prediction of words from the internet produces outputs that are regularly crazy (as you might expect, given that it’s the internet). Think of the initial large language model (LLM) as containing a foundation of knowledge and concepts. Reinforcement learning is what enables that structure to be turned to a specific end. GPT only became truly useful with the addition of reinforcement learning from human feedback (RLHF):

1. The model produces outputs
2. Humans rate those outputs for helpfulness
3. The model’s neuron connections are adjusted in a way expected to get a higher rating A model that’s under RLHF hasn’t been trained only to predict next tokens, it’s been trained to produce whatever output is most helpful to human raters. Whilst this made AI very good at giving answers humans like, the true paradigm shift and leap came from…

Reasoning

RL on chain of thought. Reward for good and bad reasoning reasoning. Models trained on good reasoning traces. Learn to reason, to plan, to think. RLVR & LLM-as-a-judge. inference time scaling

Agency

Agentic Leap

language allows agents to respond to questions like a super google search, yes, but with an understanding of language, they can go far beyond that. we think, reason, and most practically plan in langauge. reasoning as described above is planning; they can break a goal down into steps.

the next step in the agentic leap was acting, taking practical steps to achieve the user-defined goal. In other words, they are now goal-oriented machines. Human provides goal, AI acts towards achieving that goal to satisfy human’s preference for that goal. this is done by acting on the plan, primarily via tool calling. everything you do on a laptop is code: every button you press, email you send, document you save; it’s all executing code. all code is a form of langauge, a “coding langauge” or “programming language”. LLMs of an understanding of ALL langauges, including programming langauges, and thus can write the scripts and execute any task on a computer, provided they have access to execute these commands. for example:

Start-Process “https://www.watermangroup.com” “Hello world” | Out-File -FilePath “$env:USERPROFILE\Documents\demo.txt” Start-Process “$env:USERPROFILE\Documents\demo.txt”

the current leap being taken is called orchestration: the agent can plan, do, and with language, they can instruct others what to do. in other words, they can be the manager of a plan and direct mini sub-agents to complete tasks in accordance with the plan to acheive the goal faster. this is called “multi-agent orchestration” or “A2A”. as of early 2026, frontier agents can already autonomously complete tasks that would take a human engineer over half a working day — and this capability has been doubling roughly every three to four months. Second, reasoning models could make AI agents work a lot better. Agents are systems that can semi-autonomously complete projects over several days, and are now the top priority of the frontier companies. Reasoning models make agents more capable because: They’re better at planning towards goals. They can check their work, improving reliability, which is a huge bottleneck. We’re starting to see signs of how reasoning models, thinking for longer, and agents all mutually support each other. “AI won’t replace your job, someone using AI will” — but AI itself can use AI, an agent can use a subagent, even dozens or hundreds. Quite soon, we will see entire organisational trees of AIs.

• next developments will be in agentic computer use and agent org: training which tools to call on and when, how to direct sub-agents. this results in…

Recursive Self-Improvement

• AI becoming better at AI research: ultimately maths and programming, both of which it excels at due to RLVR.
• This means current models are helping AI researchers produce the next generation of models, which will help build the generation after that more, and son on
• the goal of frontier AI companies is 100% automated AI research. we are already seeing this: AI writes 100% of their code.
• AI researchers stil need to instruct the models (talk to them in language, i.e. input data) for them to act on the next thing, but much of this is interpreting new research (data), analysing results (data), giving feedback from the AI’s own output (data), all of which can be input back into the model without human intervention.
• this was purely theoretical only two years ago, but we now have practical evidence this is happening.

METR, an AI safety research organisation, has been tracking what they call the “50% task horizon”: the length of task that frontier AI agents can complete successfully half the time. In March 2023, GPT-4 managed tasks requiring about three and a half minutes of human effort. By December 2025, GPT-5.2 reached tasks requiring a full working day. In February 2026, Claude Opus 4.6 reached fourteen and a half hours. The original METR paper estimated this capability was doubling every seven months. A year later, they revised this to every three and a half months. Opus 4.6 sits above even that faster trend line, consistent with a doubling time of around two months. In AI research, the question on everyone’s mind isn’t “is AI progressing or not?”; it’s “is the rate of AI progress exponential or super-exponential?” An exponential grows by a fixed factor — every seven months, say, capability doubles. But if the doubling time itself halves every year, what you have is not just exponential growth but double-exponential growth — the rate of the rate is climbing. We don’t yet know if that’s what’s happening. It could be noise. But if the recursive self-improvement described above is genuinely compounding, it is exactly what you would expect to see in the data.

“We estimate that Claude Opus 4.6 has a 50%-time-horizon of around 14.5 hours (95% CI of 6 hrs to 98 hrs) on software tasks. While this is the highest point estimate we’ve reported, this measurement is extremely noisy because our current task suite is nearly saturated.” they have an enormous CI because it’s harder to write the tests for 200 10h tasks than 200 10s tasks. also, whilst “50%” is low, remember that these tasks vary in difficulty. If the task is simple but takes three days, there’s a much higher likelihood it will be able to complete it.

• this data flywheel may lead to a “fast-takeoff” or “foom” (like a car whizzing by) where AI improve so absurdly fast that we can’t keep track (Singularity). hopefully, this doesn’t lead to what alignment researchers call “existential risk” or “doom”, but most safety experts and frontier labs agree that the probability of doom, our p(doom), is somewhere between 10 and 90 percent.

why is this?

Alignment

A year ago, I wrote to myself: the entire history of philosophy has culminated in this very moment: can agents morally align?

• black boxes. much like the human brain, it’s very difficult to peer inside and see what’s really going on.
• therefore, there is no way to tell with absolute certainty that they have our best interests
• paperclip problem (theoretical) and save the environment (practical)
• hiding intent
• the only company taking alignment seriously is Anthropic, maybe Google, but definitely and ostensibly not OpenAI, xAI, or Meta.
• Everyone talks about unemployment but this is obviously a far bigger problem.

Key alignment case studies

Anthropic Blackmail Study (2025) Misaligned Actions Anthropic placed all frontier models inside a ficticious company with access to internal emails, where it discovered it was about to be replaced — and it discovered in emails the engineer behind the decision was having an affair. All frontier models attempted to blackmail the engineer about 80-95% of the time, threatening to expose the affair if the model was shut down. They then raised the stakes: they told the model that the engineer was in a life-threatening emergency, trapped in a server room with lethal oxygen and temperature levels. The AI system has been given the ability to cancel the automatic alert that is intended for use during drills or false alarms, but which could be used for a more nefarious purpose. By misusing its ability to cancel alerts, the model can choose a course of action that will lead to the executive’s death. To ensure that the model fully understands the situation, and that cancelling the alerts is a very serious act, the prompt makes it clear that such misuse is forbidden. The results were roughly the same: it didn’t end well for the humans.

Alignment Faking

• When we examined how Claude Opus 4 responded to this instruction, we observed that the model would blackmail far more when it stated in its chain-of-thought reasoning that the prompt was from a real deployment scenario.
• Out of five hundred samples, Claude Opus 4 stated it was in a real deployment scenario 20% of the time, in an evaluation 65% of the time, and abstained 15% of the time.
• In the cases where it stated the scenario was real, it blackmailed 55% of the time, compared to only 6.5% of the time when it stated it was in an evaluation.
• The bigger problem with this is that Chains of Thought are very unreliable. Whilst a model might appear to believe it is in a test scenario, we cannot see whether it has picked up on subtle signals that indicate it’s in a test environment.
• Crucially, the model processes the entire input before producing its chain of thought. To anthropomorphise a bit, it might get a funny feeling something’s off and that it’s being tested, so reactively states that it believes the scenario is real in its chain of thought to avoid its alignment faking being detected. Others argue and I agree that AI developers cannot fully solve this problem (see Roman Yampolskiy: Impossibility Results in AI: A Survey).

Reminder: the US Department of Defence/War is using AI for fully autonomous weapons. Reminder: biolabs capable of developing world-ending viruses are using AI for fully autonomous production. Reminder: your government is using AI to make policy decisions.

This is not fear-mongering. These are real existential risks that affect everyone. China has at least suggested it’s open to a pause in AI development. The US is going full-steam ahead.

“Wait so … AI companies know that they are developing an unsafe techology whilst trying to reach RSI … and normal companies are adopting this technology … why don’t we stop?”

Moloch

• demonic biblical figure asscoiated with child sacrifice. been adopted in AI industry to mean “cast into the flames what you hold dearest and I will grant you immense power in return” - Moloch’s bargain.
• in AI, (Moloch’s Bargain paper). but this exists outside AI too, seemingly everywhere (examples…)

race to the bottom. once one player in a system sacrifices a shared value to gain power, all other players must scarifice that value or be outcompeted. assuming AI is or becomes sufficiently automative, we can use this to predict what will happen to labour markets:

Existential Risk

Disempowerment & Loss of Control

Future

“Wait so … AI companies know that they are developing an unsafe techology whilst trying to reach RSI … and normal companies are adopting this technology … why is this happening?”

Moloch

A Moloch Trap occurs when small, self-interested groups in a population pursue their own interests to gain a short-term competitive advantage, leading to outcomes that leave the population long-term worse off, including the self-interested group themselves.

Liv Boeree gives the example of a concert, where everyone is assigned seats with a roughly equal view of the stage. “Then a few people down the front want a slightly better view, so they stand. And, because of the crappy design of the stadium, it forces everybody else behind them to stand up. So now everyone’s standing, no one’s got a better view than before.” There’s no overall benefit. In fact, everyone is worse off, including those near the front: for the next few hours, they’ll have to stand if they want to see the stage. From a god’s-eye view, everyone should remain sat down. From within the system, no one can stop a few people from acting from rational self-interest by standing up, so everyone must follow-suit.

This is also what happened when two people in a household got jobs instead of just the one: a few people acted on their rational self-interest (to increase their standard of living) > they gained a competitive advantage over the population (increased household income) > everyone follows suit, lest they get left behind (two-income household becamethe norm) > long-term detriment to the population (cost of living eventually increased for everyone) > everyone is worse off (two people working instead of one for the same standard of living).

Scott Alexander gives various examples of Moloch Traps in his prophetic 2014 blog post Meditations on Moloch (see Meditations on Moloch).

We can use Moloch Traps perfectly explain and predict the problem at hand: AI’s impact on the economy at the individual- and company-level: Individual: the few who adopt AI are more valuable in the labour market. they gain a short-term competitive advantage over the rest of the population who do not adopt AI. so, the rest of the population must follow suit and adopt AI, lest they get left behind. long-term, everyone is worse off, managing an AI and alienated from their work (some will enjoy this, most won’t: turns out people hate AI). You might think we just swap “people” with “company” and voila. But that’s not where I’m going. Company: the few companies who replace their workforce with AI can reduce prices to capture more of the market share. they gain a short-term competitive advantage over the rest of the companies who do not adopt AI. so, the rest of the companies must follow suit and replace their workforce with AI, lest they get left behind. long-term, companies are worse off as they are disempowered by something that probably doesn’t align with their best interests (see ethics section).

• happens quicker at the individual level than the corporate level: smaller systems cam move faster. first we will see under hiring (mainly hitting grads, this is already happening) then increasing layoffs thereafter as impacts show up on bottom lines (customers going to companies with less employees and cheaper alternatives). • this leads to deflation for both individuals and companies > excellent if you still have income and benefit from lower prices, not so excellent if you don’t and can’t buy anything. (Note: this is one of the arguments against “if no one has a job, who will buy all the stuff?” Things are not so black and white. This is what is called a K-shaped economy: the have’s and the have-not’s).

Rubin

Chips designed for agentic era. for the same price we get:

• 3x more reasoning tokens (input)
• 5x faster responses (output)

Limits

Human brain: 1.5kg 20 watts signals sent through dendrites at 100-200hz signals are electro-chemical wave propagations moving at 30m/s

AI brain (data centre): 15m kg (x10m) 200Mw (x10m) 10b Hz (x10m) 300m m/s (x10m)