ACM ByteCast - Dawn Song - Episode 79

Episode Date: December 18, 2025

In this episode of ACM ByteCast, our special guest host Scott Hanselman (of The Hanselminutes Podcast) welcomes ACM Fellow Dawn Song, Professor in Computer Science at UC Berkeley, Co-Director of Berkeley Center for Responsible Decentralized Intelligence (RDI), and Founder of Oasis Labs. Her research interest lies in AI safety and security, Agentic AI, deep learning, security and privacy, and decentralization technology. Dawn is the recipient of numerous awards including the MacArthur Fellowship, the Guggenheim Fellowship, the NSF CAREER Award, the Alfred P. Sloan Research Fellowship, the MIT Technology Review TR-35 Award, ACM SIGSAC Outstanding Innovation Award, and more than 10 Test-of-Time awards and Best Paper awards from top conferences in Computer Security and Deep Learning. She has been recognized as Most Influential Scholar (AMiner Award) for being the most cited scholar in computer security. Dawn is an IEEE Fellow and an Elected Member of American Academy of Arts and Sciences. She is also a serial entrepreneur and has been named on the Female Founder 100 List by Inc. and Wired25 List of Innovators. Dawn shares her academic journey in cybersecurity, which used to be a much smaller field and how the MacArthur Fellowship (aka the “Genius Grant”) and other prestigious recognitions enabled her to pursue impactful multidisciplinary research. Dawn and Scott cover a myriad of topics around Agentic AI, including current and future security vulnerabilities from AI-powered malicious attacks, Dawn’s popular MOOC at RDI, and the associated AgentX-AgentBeats global competition (with more than $1 million in prizes and resources) focused on standardized, reproducible agent evaluation benchmarks to advance the field as a public good. AgentX-AgentBeats Agentic AI Competition Berkeley RDI Agentic AI MOOC

This is ACM Bytecast, a podcast series from the Association for Computing Machinery, the world's largest education and scientific computing society. We talk to researchers, practitioners, and innovators who are at the intersection of computing research and practice. They share their experiences, the lessons they've learned, and their own visions for the future of computing. I'm your host today, Scott Hanselman. Hi, I'm Scott Hanselman, and this episode of Hansel Minutes is in association with the ACM Bikers. Today, I have the distinct pleasure of speaking with Dr. Don Song. She's a professor in computer science at UC Berkeley and the co-director of the Berkeley Center on Responsible Decentralized

Intelligence. She's also the recipient of various awards, including the MacArthur Fellowship, the Guggenheim Fellowship, and more and more. And I'm just thrilled to be chatting with you today. Thank you so much, Dr. Song, for spending time with us. Great. Thanks, enough, by having me. You have got such an impressive background. I'm just curious when you started your academic journey in security, did you think that the work you would be doing would be so recognized? It would be such a big, fun, long career. Thank you. Thanks for the question.

Actually, when I started working in cybersecurity, so first of all, the field was really, really small. I mean, the conference that you go to is only like maybe a hundred, a couple hundred people. And also, when I started, I just actually transitioned, switched from being a physics major. to computer science. So yeah, so I actually did my undergrad in physics and I only switched to computer science in grad school. And when I first switched,

I was trying to figure out what I want to focus on, like, you know, the domain. And I actually found security really interesting. And also I liked the combination of theory and practice. So that's why I chose it, right, given the fresh transition and also the field was very small. So I think it was difficult to predict what's going to happen in the future. But I do know that security was important and was going to be a lot more important.

So I'm very happy that I chose the path. Yeah. It's funny. Sometimes people ask me, like, my career, like, did you plan all of this? And it's easy to say, looking back, oh, yeah, it was all a plan. But you just work hard. You did your best, and people recognize it, and you follow your sense of smell to, like, what is the next thing? Yes, that's actually a really good way.

Yeah. Now, the MacArthur Fellowship and some of the other recognitions that you've had, like you're an ACM fellow, you're an I-Triple-E fellow, these are rare and you're stacking them up. I'm curious when you got something like a genius grant like the MacArthur Fellowship, did that change what you chose to follow, or do you still plan your agenda based on your gut and where the research takes you? That's a very good question. I think in some sense it does give me maybe like more freedom, more a sense of courage, is to really explore things that I find interesting that I feel that can be impactful in the future. My trajectory after the MacArthur Fellowship has really even further broadens my research domain. I actually have been taking a quite

unusual path than I think a lot of people. I like that idea that it gave you courage in the sense of like it's a very big validation. And it's also like the direction we're headed is a good one. I'm going to now take some risks, make some strong decisions. Did it change how you formed your team? Did it change your feelings about taking risks? That's a very good question. So I would say, like in my research career has been quite different from a lot of people, from most people, and that I have, as you mentioned at the beginning, I actually have explored fairly broadly and also at the same time, you know, deeply in a number of different domains. After the MacArthur Fellowship, as you mentioned, initially my career started in security and privacy. And also I've always been

interested in, you know, how the brain works and want to build intelligent machines. So, so yeah, so after the MacArthur fellowship, I actually, and then I also did some startups, and my startup was acquired, and I was asking myself, what I want to do if I retired. I had retired, then the conclusion was that I want to build intelligent machines. So that I actually switched my whole group and then I actually focused on deep learning. Before deep learning was actually hot. This was, you know, even before AlphaGo and the last wave and so on. I would say for most people, that would be a pretty big change.

I was in a meeting recently that I felt maybe had too many managers, and a person texted me in the meeting privately, and they said, there's a lot of talkers in this meeting and not a lot of doers. And one of the things that I would give you a compliment about your career is it seems like as academics go, you're a doer. Like you create centers, you make conferences, you are outward facing, you're talking to people,

you're creating massively online courses. While other academics tend to kind of fold within themselves and they just kind of like disappear and write a paper for a year, and then they kind of pop back up occasionally. And you said you did startups as well. How do you find that balance between what academia expects from a people talking in a room perspective and they let's do things.

Let's ship products. Let's make lives better. You seem to be a doer, not a talker. I see. Okay. So first of all, I think everybody has their own path. Everybody has their own preferences and people make contributions in their own ways. And I wouldn't say people who are just working on their page.

papers, maybe, you know, in their offices, at talkers. I think they are, I mean, some of, you know, some great work actually came out of that kind of settings as well. And so, yes, I wouldn't say necessarily, right, one approach is necessarily better than others. But I think people, they, people have different aspirations. People like to do different things. I'm glad that the path that I chose, the type of work that I have been doing, have impacted a lot of people with the massive open online course. For example, helped like tens of thousands or hundreds of thousands of people to actually learn about cutting edge new topics and so on.

And also, you know, the startups help transition research technologies into the real world. And all these things, yes. So I think I'm very happy that my work has been able to help a lot of people. But I think people also, they contribute in a different ways. I appreciate that. I apologize if that was an indelicate question. It was just meant to show the difference between really making things happen in a very physical, impactful way. But you're right. Impact comes in different flavors, including our friends that are maybe more quiet in their writing. Now, you co-direct the Berkeley Center for Responsible Decentralized Intelligence RDI. Can you explain that mission and what that means? And then how do you select the areas that the center focuses on? The Berkeley Center, RDI Responsible Decentralized Intelligence, works at an intersection of responsible innovation, decentralization, and intelligence as AI, for example.

And I would say Agentic AI is actually a very good example of the kind of work that we focus on. If you look at Agentic AI, it's really important that it's safe and secure and responsible. We want Agentic AI to help with responsible innovation. And also, intelligence is a key part of, AI. And also we hope that the agentic AI future that we built is not centralized. It's decentralized. Each of us, we may have our own assistant personal agent that represent us or help us to interact with others, with other agents and so on. And we'll have lots and

lots of different agents that actually perform different tasks, have different capabilities to help make a better world for all of us and for society. And also at some time, it's safe and secure and responsible. It feels like for the people out in the community, like the non-technical people, that AI is having a moment because it's being well branded. We're hearing the word agentic just in the last year or two. But this has been something you've been thinking about for six, seven, eight years.

Like, what does it feel like to hear things you've been working on for six or seven years now start to break out into the mainstream? Because I think even now, regular people have struggled to understand what is an agent and what is agentic? Is it just an LLM that has the ability to call a tool? Or is there something more there? Yes, that's a very good question. So first of all, it's not just six, seven years.

Actually, it's been much longer than that, right? I mean, it has been in the making for many decades. And even from my own transition into deep learning, as I mentioned, I started working in the field of deep learning. Even before, actually, the term really became popular. most people actually started working in the area. But even then, I think, yes, I would say almost all of us have been really surprised at the speed of advancement for frontier AI and so on.

There has been polls and also, if you just ask most AI researchers who are working AI today, back then before, like, CHAPT came out, before GPT 3.5 or JPD4 came out, like what people expect us. For a lot of the tasks, people would expect that still it would take decades for those tasks to be able to be accomplished by AI. But today, you know, here is where we are. And I think most people, almost everyone, has been very surprised. Yeah. Yeah. Certainly the math, the work, the subset of machine learning, the multilayered neural networks, this is something that you said has been worked on for decades.

It popped when GPT started, when the transformer architecture was introduced. Do you think that there's an over-emphasis on Next Token prediction, on transformer architecture, when there's so much other really interesting work happening in deep learning and in machine learning? Yeah, that's a great question. So, of course, what has been shown now is this next. prediction paradigm has been very powerful. And also recently, the reinforcement learning-based approaches also have been shown to be

really helpful, effective at improving the model capabilities and also in particular for agent capabilities and so on. And of course, I think now this is a big question. Is this transformer with the current training paradigm with RL and so on, will this path be sufficient for us to get to where we want. And, I mean, the truth of the matter is nobody really knows. But so far, we are continuing to see and still the fast progress of other capabilities and also the ageing developments and so on.

So, I mean, of course, I think we would love to see more exploration, more diverse ideas and so on. Even the current paradigm still, there are many limitations. shortcomings, now very data efficient, and so on. So we do hope that we can continue to make further progress and identify new ideas, new breakthroughs, and so on. And in the meantime, I do foresee that we'll continue to see the improvements and the model capabilities and so on.

As a very simplistic example, if I take a small GPT on my computer and I give it access to tools and I let it run around on my file system and edit files and do things, I have the basic. of an agentic AI. Fooding agents in that case. In an agent, yeah, a small agent, making a small basic agent. I'm basically letting next token prediction run shell scripts on my machine, and maybe productivity comes out of it.

But one of the themes in your research bio is the intersection of deep learning and security. I think about these little agent that runs on my machine, and then maybe a robot in my house that has arms and legs and has a model behind it. Both of those instances do no harm. has always been one of the ideas around robotics. The first rule is, like, do no harm. Is that something that is possible for an agentic AI to be both secure and helpful,

or are we always going to have that tension? So also first, when earlier you also asked about what is agentic AI. Thank you. Right. So the examples that you mentioned, these are very good examples of some of the things that agentic AI can do. But when we talk about agentic AI in general,

It's not just about, you know, one type of agent and so on. It's actually, in fact, it's a very broad spectrum. In our recent overview of people, we actually lay out a general landscape for Argentic AI along a number of different dimensions. Along each of these dimensions, essentially the Argentine AI systems can be less flexible versus more flexible. So, for example, the kind of tools that they use, whether the tools pre-specified in a static set, or they can even use.

dynamically selected tools during a round time that the developer didn't even know that I didn't even specify ahead of time and so on. And, you know, the level of autonomy, the level of how flexible, the flow, the control flow and the workflow of the agent. So it's a very broad spectrum. And given that, so what we also have shown is along with each dimension as the agentic system becomes more and more flexible and more dynamic and so on, it also increases the attack surface. And also when we talk about safety and security of agentic AI, there are actually two main different aspects. So one is whether the agentic AI system itself is secure, whether it can be secure against

malicious attacks on the agentic AI system itself. So for example, in the example that you mentioned, you have a little coding agent that works on your files and so on. You want to be careful that there's no malicious attacks, attacking the coding agent so that the coding agent somehow misbehaves, delete your database, and then send out, and also send out sensitive data right from your files to the attacker and so on. So this is one type of concern.

And then another type of concern is these agents, as they become powerful, attackers may misuse them as well to launch attacks to other systems, to the Internet, to the rest of the world, and so on. So that's also a responsibility that we have as we build these agentic AI is, you know, what people say is with strong capabilities, also there's strong responsibilities as well, right? So it's both sides, and both sides has its own set of challenges. And I would say cyber security has always been challenging, a challenging domain. We are seeing, you know, attacks every day today already. Several attacks are causing like billions and billions of dollars of financial loss and damages every year.

And now when we add to agentic AI, I think on both sides, actually things get a lot worse. So for the agent A.I. systems, first of all, because it's much more complex and much more dynamic. And also, we don't actually understand how these large language models work. they have intrinsic vulnerabilities, issues, like jailbreak, problem injection, and so on. So a GENTIAS system itself is actually much harder to secure, to protect against malicious attacks on its own. And then on the other hand, when a GENTIS system becomes more powerful, when attackers misuse them, the consequence can be much worse as well. And this also has been illustrated with some of our own recent work in actually evaluating what AI can do in cybersecurity, like CyberGIM and so on.

ACM Bytecast is available on Apple Podcasts, Google Podcasts, Podbean, Spotify, Stitcher, and Tunein. If you're enjoying this episode, please do subscribe and leave us a review on your favorite platform. A little bit of a side rant. I remember in the early 90s when they told us never to trust user input, right? And you always have your little text boxes and you always put validation on each text box and you're so careful to not trust user input. And now the internet is just one giant text box where we type pros and we're expected to trust user input. But that's the now the attack vector. While the stack is so deep, like I have this alter behind me and I have a PDP. 11 over there, those are computers where you can hold in your brain almost the entire computer. But now, there's no such thing as a full stack engineer because a chat bot on the internet is a distributed system within another distributed system within virtual machines and its complexity all the way down. Is it problematic that none of us can hold the full stack

in our brain anymore? I think that's a very good question. It is actually a huge issue. As I mentioned, first of all, I mean, it's not just about. whether we can hold it in our brain. As I mentioned, even though our arms are so powerful, but it's amazing that none of us really understands how it works. That seems concerning. Like somewhere not to figure that out. Right.

Like a hallucinous, right, can be jailbreak and various issues. And think about it, that provides the intelligence of our Argentine AI systems. So we have this really powerful system. And also we give it all sorts of privileges so that it can do things on our behalf. In the future, we may give it our credit card numbers so it can shop for us, right? And we give it privileges in our systems.

Right, it can take actions on our systems and so on. So it's so powerful with all these privileges that we gave it. But at the same time, we have no idea how it works. We don't know when it can break down. We don't know how it's going to behave under different situations. So I think this really causes huge concerns. So that's why also some of my work has been focused on what can we do,

how we can build more secure solutions for these type of systems. And ideally, we want to also develop new approaches to still to even have probable guarantees of certain security properties, even for these agenteeat systems. I think that's something that we really need, in order to actually have agentic air systems to take critical actions for us. That's a great point. Like here we are making these giant distributed programs where the fundamental for loop in the middle is a black box that's non-deterministic,

and we can't trust it because it could suddenly decide to be angry and cause problems. How do you design a system around that? How do you make it so the light switch that that can flip off doesn't hurt someone? I'm curious, what do you think about the stochastic parrot analogy? I think there's arguments that it's probabilistic mimicry and the LLM is a kind of a parrot, but then there's also maybe that that's a simplistic analogy and it undersells the emergent capabilities of LLMs. I'm curious which side you're on. I see.

Yeah, that's a very good question. I mean, again, at Hallam, as I mentioned, we really don't have very good understanding of how these LMs work at all. And we do see very interesting phenomena, right? On one hand, these arms, they can win the gold medal in these Olympia math competitions, right, programming contests, and so on. And they can, in such a case, solve very hard math problems. But on the other hand, you can easily see it actually makes very silly mistakes, very simple problems. we call the ALM has this jagged intelligence on certain things, right? It does really, really well.

And the other things, right, it does very poorly. And also, we have done some recent work, also trying to understand better what LAM's learning, whether how well can it generalize, we actually develop some new benchmarks, omega, delta, and so on, to try to develop this controlled experiments to really understand how I'm doing generalization, both with supervised fine tuning, as well as reinforcement learning, and so on. So what our work has shown is that, I mean, so first of all, yes, I mean, in certain cases, LMs capabilities, it is amazing. But then on the other hand, our work does show that there's still significant limitations in terms of how these LMs,

how well I can generalize. In particular, as we increase both the difficulty level of the problems and also the compositional complexity that doesn't generate and generalize that well. And also still, I think it can come up with some new ideas and some but in general, like with our benchmark evaluation shows that still when some problems that require really new, solutions, new type of solutions, still not very good at those. Yeah. I like that term jagged intelligence. To assume that one individual is uniquely smart in all things is to oversimplify. I could be a poor driver and I could be a genius in math.

I am not single dimensional. So neither are the LLMs. You've been teaching for so long and now you're teaching massively open online courses. There's a really exciting one that you've been doing the agentic AI, this is blown up. How many people did you expect would come to the agentic AI MOOC and how many are coming now? First started, this massive open-upon-on-on-course MOOC on Agentic AI actually fall off last year, 24. Mm-hmm. And when I started, actually, back then, the agentic AI agents wasn't quite a thing yet.

Not many people were talking about it. But however, I could foresee that this is the future, this is the next frontier. So that's how I actually, you know, started teaching the class. And I think because it was the first, it was literally, I think, the first class, agents, agenting AI, and it's the first MOOC on the topic as well. So I think it really caught people's attention. And now we're actually running the third edition for the class. And so we have over 32,000 enrolled globally.

So that's been really exciting. And also, suddenly, you know, this year, it's now called the year of agents. Even though, as I said, when I started the class, you know, it's because I could foresee that this is the next frontier. But even I did not expect things to explode so fast. Like this year, you know, especially after the reasoning models came out, that really helped with the reasoning capability of agents overall. and we are really seeing the field exploded.

So that's been really exciting to see. At what level should a person feel comfort around their level of computer science and AI before they join a MOOC like this? Like, is this for high school students? Is this for graduate students? Is this for practitioners like myself? What should I come into a course like this knowing and being prepared for? Yeah, that's a great question. So I would say actually the course is designed to,

have something to offer for people, all people at different levels. So, of course, the course is mainly designed for people with technical backgrounds in computer science and so on. And we actually do systematically cover both in terms of different layers of the agentee AI stack, all the way, you know, from the foundation, the foundations, the model development, the capabilities to a Gentile AI framework, all the way to applications. both horizontal and vertical applications and so on. So, and the class is technical, but on the other hand, also, I think even just from the lectures,

even for people who don't have too much backgrounds, there's still, I think, quite a bit that they can learn about just a general overall development in the space. Yeah, it's really a huge source of material to explore, to look back at spring and fall of last year. It's worth noting that the supplemental readings, the links, to all of the quizzes, the slides, the videos are all available online. So people should go back and explore. This is really very formal and structured and deep with a lot of really great guest speakers that you've put together.

And now you've even got a contest for the greater good that you're doing an open competition for agents that hopefully make people's lives better. Yes, yeah, thanks. Yeah, so we are running a competition. So for each edition of the MOOC, we actually have organized the competition. So, for example, the last one in this past spring,

we had close to a thousand teams that participated globally. For this semester, for this addition, we have a new competition, which actually focuses on agent evaluation. So as we develop agents, actually, it's really important to have good aging evaluation and to have good methodologies for aging evaluation. Because, I mean, they're just saying,

that we can only improve what we can measure. And also in general evaluations, essentially the goalpost for development for the community. So, you know, my group, we've had earlier work in a space like MMLE and some of these other benchmarks have been widely adopted in the community. But a lot of those were focused on, I would say, evaluation at the model level. But for aging evaluation, actually, it requires, essentially different focus, given that the agent is not just a model. The agent evaluation

is not just a model evaluation. You actually have the model, and also you have the agent itself, like also called the harness, that actually uses the model to essentially perform tasks and so on. So the agent evaluation essentially has more components for the evaluation. And it's very important to have open, standardized, reproducible evaluations for agents. And so far this has been lacking. So we actually have developed a new paradigm. We call it an gentified agent assessment, triple A, that actually helps to meet this need to enable a new paradigm of this open, reproducible, standardized agent evaluation.

And also we are developing a platform for this as well. And this agent evaluation competition is really to help bring the communities together to develop the best benchmark evaluation that's standardized, reproducible and has broad coverage in diverse domains that helps guide the community developments. And we really hope that more people can join the competition. We have actually over $1 million in prizes and resources provided by sponsors and partners of the competition, including Google DeepMind and many others and so on.

So I think this will be really fun. And also it's a great opportunity for the community to come together to develop public good. So we hope that people can join us in this competition. Yeah, this is very exciting. And folks can explore all this stuff. There's agentbeats.org. They can see the code. They can take a look at RDI. berkeley.edu to learn about Agent X and about Agent Beats. And they can learn about the MOOC at AgenticAI-Learning.org. I'll put links in the show notes for all of this stuff. As we get ready to close, I want to ask you, as a person on the forefront, this technology. In your day-to-day, what agents are you using that are helping you be a better

professor, be a better thinker, be a better teacher? Are you using commercial products that you just have a subscription to? Or are you writing these things custom? Are you using cutting-edge things? What's an agent expert using for their own agents? That's a very good question. So I'm actually trying to develop some of my own agents to better automate some of my own workflows. As you know, you mentioned, like I do a lot of different things. actually, I mean, they do take a lot of time. And even when I have assistance and so on, it can still take a lot of time and so on.

And a lot of these things now can really be automated, are like hugely helps with agents and so on. So that's some of the things that I'm doing as well. I've heard in the space of robotics, someone said that a robot should do things that are dull, dirty, or dangerous. And we use the term toil. So I assume you're trying to automate the boring

stuff so that you can do the interesting fun thinking. Yes, absolutely. That's fantastic. Well, thank you so much, Dr. Song, for spending time with us today. Thank you so much for having me. We have been chatting with Don Song in association with the ACM Bytecast, and this has been another episode of Hansel Minutes, and we'll see you again next week. ACM Bytecast is a production of the Association for Computing Machinery's Practitioner Board. To learn more about ACM and its activities, visit acm.org. For more information about this and other episodes, please do visit our website at learning.acm.org slash bitecast. That's B-Y-T-E-C-A-S-T. Learning.acm.org slash bitecast.