Technology, Connected - Could AGI Replace Wall Street?

Episode Date: May 18, 2026

Anders Sandberg examines whether artificial general intelligence could manage the global economy more effectively than human institutions.A sufficiently capable AI system might coordinate markets, allocate resources, interpret legal rules and respond to complex global problems faster than governments or companies. Greater efficiency, however, wouldn’t necessarily mean greater freedom.In this short excerpt from a longer Thinking on Paper conversation, Anders discusses:Whether AGI could manage the global economyHow superintelligence might improve global coordinationWhy markets and legal systems are difficult to optimiseWhether AI could make better decisions than human institutionsHow highly efficient systems could concentrate powerThe challenge of keeping advanced AI under human controlHow evolutionary pressures could shape competing software systemsWhether humans could become wealthier while losing political agencyWhat role people would retain in an AI-managed economyThe central question isn’t simply whether AGI could run economic systems better. It’s whether humans would still control the goals, rules and trade-offs behind those systems.This is a short from a much longer conversation with Anders Sandberg about superintelligence, governance and the future of human decision-making.--Thinking on Paper is a technology podcast about AI, quantum, space and their impacts on society, business and culture. It's very good. 🏠 Buy us a beer on Substack🎧 Watch on YouTube 🎧 Remember Steve Jobs on APPLE📺 Get the clips and outtakes on Instagram

What would it look like if AGI or AI gets so good, we trust it to manage something like the world economy? What does that look like? Well, hopefully it looks like a very well-functioning world economy. But that can also be a bit of a problem. So why do we have legal systems and states and this? Well, that is the way we outsource our cognition and we handle it by having these artificial systems made out of people and pieces of paper. But now we're of course also getting software that can take over more and more of that. Why do we have law?

Well, we want peace and prosperity and of course power to some people. They are very good at latching onto that. And with AI, we might get even more peace and prosperity, even more efficiently. And maybe power to Sam Oldman or other people who happen to grasp it in the right way. But that's a separate issue. But the really interesting part is we might really want better coordination because the world is big and somewhat dangerous. we have climate change, we have controlling powerful AI, we have biotechnology getting more and more powerful.

We probably need to coordinate about some global problems way more powerfully than we can do right now. And maybe we could do that with a United Nations plus plus, but that seems a bit unlikely. It might be that we actually want to leave more and more to super smart, distributed AI systems. But then you end up with a funny situation, we might be swimming in peace and prosperity.

That's great. But previously, we needed human autonomy. We need to make decisions to run the legal system, to run the market. Now it runs itself and probably runs itself much better without us messing with it. It's a little bit like chess playing. Once humans were better than computers that played chess, then gradually computers got better and better and started beating humans. But a team of humans and computers could actually still do better.

But after a while, the computer was so good that actually the humans were just slowing things done. So actually, you should just listen to the computer. So if we end up in that world, we have this weird situation that we might be like pampered aristocrats. Really good material standard. But in some important respects, our lives are not our own. And I think that sounds bad to many people. And I think the reason is we want to feel like we're in control. Because all our ancestors, the ones that we kind of find giving up control to others,

they were just exploited by the ruthless people. So we have this tenders event. We want to have autonomy. And then the funny thing is, once we have autonomy, we are very happy to give it away in various ways, but we want to be the ones that do it. It might be that you give away your autonomy by getting married. It might be that you join a company.

And in the exchange for getting a paycheck, you get some limitation of what you can do. But that might be worth it. So kids. Yep. Kids are amazing example. People, every thing. person I know that has gotten kids

say, oh yes, it's super stressful and there's a lot of hard work, but it's the most important, meaningful thing I ever done. And it's amazing. It's almost like mind control, as somebody who haven't got a kid, but it's kind of a happy uncle. It's kind of, whoa, those babies,

they have really got some psionic power here. But it's very effective. And it's kind of good for our species that we have it. And it's good that we can give away autonomy in a reasonable way. But we should also, in some sense, be able to withdraw it. Although, of course, when you're apparent, you realize,

okay, I cannot even withdraw that thing. Now I'm in for it because it's what it is. But this is where a world economy run by AI might be tricky because it might be so effective that, yeah, running your own part might be out-competed and might be so inefficient. It's a bit like living your life without electricity in a little log cabin in the woods. Sounds absolutely lovely and probably is totally lovely the first day. after a few weeks of carrying water, a bit less lovely.

And after a month, most people are back and writing on Facebook about experience. We can have two economies, like we'll have two of me and two of you and two of everybody else. We have two economies, one that's run efficiently and well by the AGI, and then another smaller one that's run and managed by humans, and they just run in conjunction with each other. Yeah, and in some sense, I think that this might be what we're developing. We have the backbone of our civilisation. the big infrastructure, the really important things, that we must get right.

When electricity goes, well, nothing else works. So you really need to get that working. But once you're wealth enough, you can spend that excess wealth on all sorts of weird stuff. There is a reason why you have so many consultants as big companies. They can afford it. And they're probably more involved with status games and politics than actually doing something useful. I think it's a good thing that we have enough wealth and safety that we can try. out different lifestyles, even if many of them turn out to be a slightly embarrassing once

you're middle-aged. There are things as humans, power dynamics and control and me versus you, like, if we were to design or not design, but like kind of enable this world economy through AI, how do we make sure as humans we don't bring that baggage that always wrecks stuff? But even that is a problem. When you try to remove the human, well, what have you trained the large language models on? They're all trained on all the human literature. They have read every single thread we've written about betrayal and power dynamics and corporate shenanigans.

One of the big debates right now in the AI safety community is when you test your AI models and they try to do sneaky stuff like escape the system if they think it's going to get shut down, is that because they want in some sense to escape or just. because they know that's expected of me because of all the science fiction I ever read says that that's what an AI does in the situation. And maybe it doesn't even matter. So I do think that it's very hard now to take away the imprint of the human world from the AI world.

On the other hand, I think it is also going to be evolutionary pressures pushing in certain directions. There are various forms of competition that means that certain pieces of software are going to be copied more widely than others. And that is going to act anyway. regardless of what human values are. And of course, my more doomy friends say, yeah, and that will inevitably be pushing a very scary direction

that's going to be very different from what we humans are. So I think one important thing is we might want to be involved here. We might want to be an essential spice in this mix to make sure that there is at least some of us in that. But then we want to filter out the really bad stuff. Cruelty, for example, it's kind of useful to stabilize altruistic punishment. Taking delight in the misfortune of somebody else is a really good way of kind of giving you extra points when you're trying to punish somebody breaking norms.

Because otherwise I would be taking a risk in policing these norms and actually it would be easier if I just stayed at home and let the norm break you get away with it. Except that today we can do this with laws. We can use police. And suddenly cruelty is just a bad thing that generally makes us nasty. That's an emotion I totally think we should kind of try to get. get rid of and at the very least make sure our AI progress never experience.