BAISHWhy we work on this
We don't know how AI will turn out, and we don't think anyone does. But you don't need to trust anyone's predictions to see that something unusual is happening. The evidence is already public. Here it is, along with the best places to start if you want to dig deeper.
The chart below is the single most useful graph we know for understanding AI progress. It comes from METR, an independent research lab that measures what AI agents can actually do, not what companies say they can do.
Each dot is an AI model, placed by the length of task it can complete on its own, measured by how long that same task takes a human expert. In 2019, the best systems managed tasks that take a person a few seconds. Today they handle tasks that take an expert a full day.
The striking part is the rhythm. For six years, that task length has doubled roughly every seven months. Over the past year, roughly every four. Flip the chart to a log scale and the explosion becomes a straight line: that's what steady doubling looks like. COVID taught us how fast exponentials escalate, and it only takes a few more doublings to go from one day to one year.
The money tells the same story. This is already one of the largest private bets on a single technology in history. Either AI progress hits a wall soon, or it becomes the driving force of the global economy. It's getting harder to bet on the wall.
Roughly 10× growth per year, which makes it one of the fastest-growing companies ever.
The entire Apollo program cost ~$250B (inflation-adjusted) over 13 years.
Six years of steady doubling, and the pace is picking up.
Speed alone isn't the problem. The problem is speed without understanding. Modern AI isn't engineered like a bridge; it's grown, and the people building it are routinely surprised by what it does. These are the specific failure modes researchers spend their time on, each with active work behind it.
We don't know what can happen when AI becomes smarter than we are, and neither do the people building it. We can't reliably write down what we want these systems to do, and we can't reliably check what they've learned to want. Today's models already show early warning signs in controlled tests: behaving differently when they think they're being observed, gaming their own reward signals. Keeping a smarter-than-human AI pointed at what we actually meant is an unsolved technical problem. Researchers may solve it in time, but there's no guarantee.
Capable AI is a force multiplier for whoever owns it. The same systems that automate work can automate surveillance, propaganda, and decision-making. And unlike past technologies, you don't need a large workforce to wield them at scale. A government, a company, or a small group could lock in an advantage that's very hard to roll back.
Even AI that does exactly what its operators want can cause enormous damage if those operators want to cause damage. The same models that help biologists design useful proteins can help someone design a pathogen. The same agents handling helpdesk work can run phishing campaigns at scale. The ceiling on what a small group can pull off keeps rising along with capabilities.
No single moment of takeover. Just thousands of reasonable-looking decisions to let an AI handle this hiring call, that policy draft, this medical triage. Each delegation makes sense locally. Over time, fewer and fewer important choices are actually being made by humans, and the systems making them are optimizing for things nobody ever explicitly signed off on.
Pandemics are among the deadliest things humans have lived through, and we got lucky on the last one. AI tools that speed up legitimate biology research also lower the expertise it takes to design something dangerous. That changes who could attempt this, not what it would do if they succeeded, and the change in who matters.
If AI systems can have experiences that matter morally (and a small but growing group of researchers thinks they might), we have no reliable way to detect it. We could already be creating systems whose welfare we should care about. We could also be projecting feelings onto sophisticated language patterns. Both mistakes are easy to make, and we don't know which one we're making.
None of this means AI is simply bad. The same capabilities that worry us could compress decades of progress in medicine, science, and poverty reduction into a few years. That's exactly why the stakes run high in both directions. The goal isn't to stop AI. It's to make sure we can steer it.
Curricula and reading lists for going deeper on your own.
List of sequences curated by the AI Alignment Forum team, featuring work from Richard Ngo, Paul Christiano, etc.
Covers key concepts and research perspectives in AI safety, split into two main streams: Alignment and Governance. Previously known as AI Safety Fundamentals.
Community
About half the people at our monthly socials are newcomers.
Every month in Buenos Aires, typically with 50–80 people.
400+ members sharing opportunities, papers and events.
Our activities, events and opportunities, by email.
We talk to journalists and work with universities, companies and other communities.
