Recursive Self-Improvement: When AI Builds Itself

Emily Low

Our next session is on recursive self-improvement. Facilitating the session, we have Tiberio Caetano at the Gradient Institute, and we'll be joined by Anna Goldsworthy, dean at the Elder Conservatorium of Music and School of Performing Arts here from Adelaide. Toby Walsh, chief scientist at UNSW's AI Institute, and Dane Sherburn, CEO and co-founder at P-Zero Research.

Tiberio Caetano

Hello everyone. Thank you for being here today. And thank you very much to the panellists for joining me this morning. Or I should say — still this morning. We are going to talk about, well, it's not unrelated in a sense to the previous panel. We are going to talk about the concept of recursive self-improvement. I think it was in 1965 that I.J. Good, who used to work with Turing, Bletchley Park, wrote a marvellous paper which is very readable actually, and a few equations in it. But in that paper he was actually asking the question, where are you?

If you were to have machines that eventually become better thinkers than humans across the board? Well, it turns out that maybe those machines will be better at designing machines as well, and maybe they would be better. Therefore, by conclusion and by simple logic inference, essentially help to build the next generation of the next one. And that process would basically keep on going. So that is the theoretical concept of recursive self-improvement. And that's what this panel conversation is about. And I.J. Good had that wonderful line where he said that might be the last invention humans ever make.

And then he said, well, we better have those machines — what was the word? Be benign or something like that to us, because that may be, well, the last invention. So we are here today to talk about that little question, which was hypothetical at the time of I.J. Good, and has been hypothetical for a long time, until perhaps recently, when it no longer became so hypothetical. And to join me in this conversation, I've got these three wonderful people and panellists who are going to provide different perspectives in this chat.

We are going to start with Dane. So Dane has been actually working with what we could call the best proxy we have today for recursive self-improvement, which is essentially trying to evaluate whether systems that are assisting humans at building the next generation of AI are themselves adding value on top of humans, and how much they are actually effectively accelerating the process of research. So let's start with you, Dane. And specifically, I would like to ask you, tell us: what can current AI systems do from the point of view of helping design the next version of them,

the next generation? And what is still missing? Where are we? What is the evidence? Dane is an actual tester of OpenAI models for this very question.

Dane Sherburn

So I think initially, the first reaction is that we need better data. The data that we have is pretty average. And I think we need more people working on it. But some of the data that we do have is recent. One example is Anthropic released an article saying that they have 80% of their code written by AIs. I think it was like eight x the number of PRs produced since, the code is through 21 to 25. So we're seeing a lot of AI assistance, but I think if we fired everyone at the frontier lab and got agents to work, it would grind to a halt pretty quickly.

And so I don't think we're at the point where we have adequacy from AIs. I think they are helping a little. But I think the gold standard measurement that we would want is: if we had a fully autonomous lab, so we could either fire everyone at OpenAI and get their agents to run autonomously and see, does the machine still churn along? Do we still get AI R&D improvements? One thing that we might want to measure is the model horizon, the time horizon of models. So if you have a model of generation

n, and it has like a one-month time horizon, what kind of model does that produce, and what's the time horizon of the new model? So I think if it has a time horizon of two months or longer — so we have a doubling or more — that would point to recursive self-improvement. But if it's any less than that, the series kind of converges. So I think there are plenty of experiments that we could run. The data that we have is kind of average at the moment.

But we do have experiments that people need to run. But I don't think we're there yet. But maybe soon. Who knows?

Tiberio Caetano

Many of those experiments are being run.

Dane Sherburn

Yeah.

Tiberio Caetano

And we'll get back to that in a moment. So let's move to Toby now. Toby, actually, you were the one who proposed this panel, so thank you for that. I'm here trying to understand why Toby Walsh, the rock star of Australian AI, has chosen this very topic to speak at the forum. Why this choice, Toby?

Toby Walsh

It's important for us to understand how quickly AI capabilities improve, because the whole point of this forum is how can we do so if it's happening quicker than we can adjust? And I think that's the fundamental, because if there's a possibility that the AI starts improving itself, well then it's going to happen very quickly. That's very clear. I maintain some healthy scepticism that the first 80% is easy, because there's a lot of engineering, a lot of running experiments that the AI can help us accelerate. But maybe the last 20%, the creativity AI — I don't think scale is all we need, and that we need something else.

If I just look at the human brain, there are lots of different structures in the human brain. So I don't think transformers are just enough. I think we're going to have to invent something new, and we don't know what that new thing is, and I'm not sure. But I got to the point where it's as creative as humans. The example so far where AI had done an interesting novel, scientific things proved interesting mathematical results. It tends to be that it's plundered a bit of mathematics from some part of mathematics.

It hasn't come up with something that's truly astounded mathematicians, is not been anywhere else in the literature. It's tied things together that humans maybe weren't able to tie together. It's certainly done interesting things, but I suspect the last 20% is going to be much harder than the first 80%.

Tiberio Caetano

What kind of data would you like to see that would convince you that the last 20% is going to be automated?

Toby Walsh

Until you close the loop, I think you're never certain that you can. I should point out, humanity's invented one self-improving system so far. That was the compiler. Compilers are often written in the language they compile. You can apply the compiler to itself to improve the efficiency of the compiler. You can only do that once. Applying it again, you've reached a fixed point. So it would be interesting. There's no reason why we can rule it out, but I'm also cautious, as all sides of it should be.

Tiberio Caetano

Oh, thanks. Stories. So humanity invented the compiler, the self-improving system.

Toby Walsh

That's right. And the evolution.

Tiberio Caetano

And we didn't end up with a super compiler.

Toby Walsh

No, no, no.

Tiberio Caetano

That's true. But evolution also invented, ones, and many self-improvement systems, like.

Toby Walsh

Yeah, but I'm getting to that now. That's another good example. But to remember, the blind watchmaker, evolution moves very slowly.

Tiberio Caetano

Yeah.

Toby Walsh

There are lots of redundant bits in me, as probably as you as well. And I don't see humans. No, but I'll — evolution. You get what I mean when I ask. I know now because evolution was very slowly.

Tiberio Caetano

It has the benefits of millions of years. But it did build something called the human brain. And the human brain can actually improve itself in certain ways. Not neurobiologically, but culturally, in terms of learning, in terms of all that — and in a timescale that's much, much different from the evolutionary timescale and also much different from the timescale. So now, Anna, coming to you. You were an accomplished, award-winning concert pianist. And the literary person, the real writer you have to read. And it's, the work that is going to be — her essay is going to be sold soon.

She was going to do a session now after this. Now, the question for you, Anna, is: you come as an outsider to this community, and yet you engage so deeply and so carefully, with so much presence and attention to what is happening. What made you choose that theme of AI and what's happening now to engage with it? And in particular, since we are talking about really the ultimate category of what AI could become, how do you see that move that you made to start engaging with AI in it?

What's your next book and what's your next move? In terms of the context of this conversation, is there something beyond AI?

Anna Goldsworthy

Okay, that's a really big question, I think. I think the first part of that is the easiest part to answer, which was that I was drawn into this subject, which I think is something that touches all areas of human endeavour. And I think it's incumbent upon all of us to engage with it in some capacity. But my dad — I was saying to Toby earlier — my dad was an early sci-fi enthusiast and drew my attention to this phenomenon some decades ago. But more recently, I was drawn to it because of conversations I've been having around the dinner table with my sons, one of whom I've dragged along to this forum today with me, Reuben, sitting there in the middle, and I.

Tiberio Caetano

We'll hear from Reuben later in the next session as well. So we can ask Reuben a few questions in the next session too.

Anna Goldsworthy

But I found I was engaged in these nightly debates around dinner, and increasingly I felt by debating with my children, I was essentially debating with the future. And on top of that, I thought I had really, really clearly skin in the game, because it's the children who this is really going to hit the most. And the deeper I got into this, the more fascinating I found it. As I said, there's literally nothing that it doesn't touch. And I suppose I was viewing it largely through the lens of my own artistic practice, but quickly became really engaged with everything ranging from the existential threat to the geopolitical ramifications, to what it means in terms of equality within societies and also, internationally, the future of human relationships.

Wearing my hat as a dean of a school at a university, what it means for the future of education and so on. So it remains a really fruitful topic to look at through multiple lenses. And one of the beautiful things about publishing a quarterly essay like this is it opens you up to a range of different conversations with very different people around the country. And I found myself in a really fascinating conversation at ANU a few weeks back with Andrew Leigh, but also with Anthea Roberts,

who is, say, professor of law, and she speaks about something that she calls dragonfly thinking, which is the fact that it's going to be very, very easy for us to view this through some sort of binary lens: that AI is wonderful, or AI is completely dystopian, terrible. And somehow it seems to me very important that we can find a sober way of assessing it in all of its complexity. None of us is capable of doing this single-handedly because it's just so massive. But she advocates a dragonfly way of looking at it.

And by way of example, she sent me something she'd written which views AI through six different prisms. They contradict each other and they don't really negate each other, because they actually sort of sit alongside each other. And that to me probably speaks to the complexity of this and the fascination of this to me. But bringing it back to your other question, which is about what next? I guess, paradoxically, one thing that's emerged for me through immersing myself in this, and even through the process of writing this essay slash book, is I create a sense of investment in this sort of humanist aspect of being an artist, and a sense of when we write, we don't just write with passion machines.

There's something fundamentally embodied about making art, just as there's something fundamentally embodied about being a human being. And a sense of all the intuition that you bring to bear or the emotion that you bring to bear. And I find that a fascinating possible upper limit in the development of agentic AIs. If something that elicits scope has identified, to what extent does emotion really inflect human thinking? And so possibly my next book is going to be kind of returning to the source for me. I'm kind of keen to write a book about the relationship I've had with the other two members of my piano trio for the last 30 years, and what it means to be a human organism playing music composed by other humans, designed to connect us in real play, in real space and real time, and plug us into the entire history of humanity.

Because to my mind — and you may say I'm being a bit human-centric here — but I just don't get what the point of any of this is if it's not actually to exist in service of human flourishing.

Tiberio Caetano

Yeah, yeah, I agree.

Toby Walsh

I agree, actually. I do think intelligent people overestimate the importance of intelligence. And you touched upon emotions and embodiment. I think that they're actually as important a part of us and the things that we are able to do. And it's not just intelligence. As an example, having more intelligence isn't going to solve the climate problem. We know how to solve the climate problem. It's not intelligence holding us back. It's coordinating and — or at least intelligence — dealing with our greed and so on. But it's not intelligence.

We know the solution to the climate problem.

Tiberio Caetano

Lovely, awesome. We still have plenty of time, so let's keep going then. I'm really curious to hear more because you are really in the trenches on this stuff, so I would love to get a little bit of a taste of the work you do, and how you can give us a bit of a sense of what the sort of precursor, potentially, of recursive self-improvement we have now. What is the edge of what you are doing now, and how does it provide the clues, if it does, about how fast things could go from here?

Dane Sherburn

Good question. So I think, in terms of what data we have, I think across the board people are finding it really, really difficult to create harder and harder tasks, or tasks hard enough. So if you speak to a lot of folks at frontier labs, the tasks which they're creating for AIs are saturated by the time they're finished creating the task. That, to me, is a really worrying situation to be in, because we just don't know how fast it's moving. And that's concerning. I think, to your point, there are a lot of really big problems that we need to solve.

We have to solve a lot of physical, or philosophical, problems on a deadline. And the faster this moves, the less time we have to settle debates on those really important questions. I totally agree, they're really important questions. So in terms of measuring it in the trenches, people are having a really hard time measuring it. What we are measuring now is the effect of AIs and humans working together. So a lot of the time it's hard to study AIs in isolation, because the tasks that we create — we try to create a self-contained task that is realistic and is representative.

It often gets saturated, so a lot of the time the problem becomes, how do you try to get a feel for how fast or how helpful AIs are, by looking at the combined pair of humans and AIs? And so you often get a mixed picture. You don't get a good answer to the question. It's very unsatisfying. So I think it's anyone's guess now, but I think the thing to look for is, first of all, adequacy. So the productivity hit of deleting humans from an organisation is just — the productivity doesn't go down by 100%.

Maybe goes down by 99%. And then I think AIs have started to become useful in automation. And then we just track that number as it goes along. But it's a really hard thing to measure.

Tiberio Caetano

Well thanks, Dane. So Toby, perhaps quickly, in 40 years of research in AI, do you feel this is different from what you've ever seen?

Toby Walsh

Oh, it is different. And I think the largest way it's different is just the scale of investment. I never expected to see billions of dollars being spent on AI. But of course that's driving progress. You spend that amount of money — a third of the world's R&D budget or more is being spent on one technology. That's without precedent. And so why does it seem to be arriving so quickly? Well, in part because that's what happens when you spend so much money on a technology. You're going to start to see results.

And of course, it's been enabled. We've got plentiful data these days. Mostly we've got plentiful compute. So that has laid the foundations for what we've been able to build. But I still worry that there are some significant limitations to the current tools that we have, that have — very capable where we can verify answers. So in mathematics and in programming where it's easy, you can run the code. You can check the proof. But in other areas the reasoning capabilities can be sometimes quite challenging. They can make stuff up.

They're not grounded, in part, because they're not embodied in the world. That comes back to the benefit — having a body, knowing what gravity is because you experience it. You didn't have to follow the equations of gravity. You are part of the world.

Anna Goldsworthy

I guess there are certain disadvantages in having a body too. There are certain vulnerabilities and fragility. So I love being in the body. And one is that we need to sleep. And the whole velocity thing is kind of fascinating to me. And I think it was interesting what was said before about we have evolutionary time. We have sort of human culture time, and we see how that works. When you look at the rise of literacy since the early 1800s to now, it's more than quadrupled. And this is really significant.

And this is a type of recursive learning. It's humans teaching each other. And that's exciting. But that's not enough to keep up with something like this. It increases exponentially.

Toby Walsh

Do think we could be brains in boxes?

Anna Goldsworthy

We could right now. Yeah. Yes, we could be.

Toby Walsh

But do you not think our intelligence is also a function of the complexity of the world that we have to inhabit?

Anna Goldsworthy

Yes. So it's the friction. And this is something that I kept on coming back to, actually, when I was thinking about this: all the problems that we encounter and how that becomes something upon which we kind of sharpen our understanding. And this notion of philosophy on a deadline is actually pretty, pretty scary. It may or may not happen, and you obviously have sort of different viewpoints on that. But I think probably we do need to consider what happens if it does get to this place, what happens if it does disappear over the horizon.

Will it take us with it?

Tiberio Caetano

And on that, and noticing that we are coming to the end, I'll give each of you one minute to address this final question. It touches on this really fundamental philosophical imperative we find ourselves in. Deep uncertainty is ahead of us. What is the single thing that you believe must be done in light of the fact that we don't know the course of action? Which single action or institution or attitude or inspiration or you name it. It can be technical, it can be human, it can be anything.

What single thing do you think would be valuable and would not only help us move forward in this path, but would help us move forward regardless of which path unfolds ahead of us?

Dane Sherburn

My mindset, I think, is pretty simple. I think we just need to slow down. I think it's already gotten to the point. [applause] Thank you. It's already gotten to the point where, I think, we're outpacing evaluations. It's already, in my mind, going off the rails. It's early stages, I think, because we are on an exponential, much like Covid. When people were sounding the alarm and Covid, it either felt way too early or way too late. And I think we are on another exponential and we should pause before it's too late.

Tiberio Caetano

Toby,

Toby Walsh

I would suggest we should put a lot of emphasis on protecting and valuing our humanity, because that's the thing that distinguishes us from the machines. The machines will hopefully be there to do all the dull, dirty, difficult stuff for us. But we have to be mindful of what makes us us.

Tiberio Caetano

Anna.

Anna Goldsworthy

Yes, I think probably a combination of those two. I think by slowing time, slowing down a bit, maybe we buy ourselves a little bit more time to protect our humanity and perform this philosophy on a deadline. I'm going to run with that. Remember that line? And can we slow down? I certainly hope we can. And when we have people who are in the thick of it advocating for that, I think we really need to be taking that seriously.

Tiberio Caetano

And with that, I will thank the three of you for this lovely conversation. And 157 of you or whatever it is, for being here with us this morning. And thank you all. Thank you, thank you. Thank you