Microsoft Research Podcast - AI Testing and Evaluation: Learnings from cybersecurity

Episode Date: July 14, 2025

Drawing on his previous work as the UK’s cybersecurity chief, Professor Ciaran Martin explores differentiated standards and public-private partnerships in cybersecurity, and Microsoft’s Tori Westerhoff examines the insights through an AI red-teaming lens.Show notes

Welcome to AI Testing and Evaluation, Learnings from Science and Industry. I'm your host, Kathleen Sullivan. As generative AI continues to advance, Microsoft has gathered a range of experts from genome editing to cybersecurity to share how their fields approach evaluation and risk assessment. Our goal is to learn from their successes and

their stumbles to move the science and practice of AI testing forward. In this series, we'll explore how these insights might help guide the future of AI development, deployment, and responsible use. Today, I'm excited to welcome Kieran Martin to the podcast to explore testing and risk assessment in cybersecurity. Kieran is a the podcast to explore testing and risk assessment in cybersecurity. Kieran is a professor of practice in the management of public organizations at the University of Oxford. He had previously founded and served as chief executive of the National Cybersecurity Center within the UK's Intelligence, Security and Cyber Agency. And after our conversation, we'll talk to Microsoft's Tori Westerhoff, a principal director

on Microsoft's AI Red team, about how we should think about these insights in the context of AI. Hi, Kieran. Thank you so much for being here today. Well, thanks so much for inviting me. It's great to be here. Kieran, before we get into some regulatory specifics, it'd be great to hear a little bit more about your origin story and just take us to that day. Who you on the shoulder and said, Ciaran, we need you to

run the National Cyber Centre. Too fancy building one? You could argue that I owe my job to Edward Snowden. Not an obvious thing to say. So the National Cyber Security Centre, which didn't exist at the time I was invited to join the British government's cyber security effort and leadership role is now a subset of GCHQ. That's the Digital Intelligence Agency, the equivalent in the US obviously is the NSA. It had been convulsed by the Snowden disclosures. It was an unprecedented challenge. I was a 17-year career government fixer with some national security experience.

So I was asked to go out and help with the policy response, the media response, the legal response. But I said, look, any crisis, even one as big as this, is over one way or the other in six months. What should I do long term? And they said, well, we were thinking of asking you to try to help transform our cybersecurity mission. So the National Cybersecurity Center was born and I was very proud to lead it and all in all I did it for seven years from startup to handing it on to somebody else. I mean, it's incredible. And just building on that, people spend a significant portions of their lives online now with a variety of devices.

And maybe for listeners who are newer to cybersecurity, could you give us the 90-second lightning talk? Kind of what does risk assessment and testing look like in this space? Well, risk assessment and testing, I think, are two different things. You can't defend everything. If you defend everything, you're defending nothing. So broadly speaking, organizations face three threats. One is complete disruption of their system. So just imagine not being able to access your system. The second is

data protection and that could be sensitive customer information, it could be intellectual property. And the third is, of course, you could be at risk of just straightforward being stolen from. I mean you don't want any of them to happen but you have to have a hierarchy of harm so that's your risk assessment. The testing side I think is slightly different. One of the paradoxes I think of cyber security is for such a scientific data rich subject, the sort of metrics about what works are very very hard to come. So you've got boards and corporate leadership and senior governmental structures,

and they say, look, how do I run this organization safely and securely? And a cybersecurity chief within the organization will say, well, we could get this capability. And well, the classic question for a leadership team to ask is, well, what risk and harm will this reduce by? How much and what's the cost benefit analysis? And we find that really hard. So that's really where testing and assurance comes in. And also as technology changes so

fast, we have to figure out, well, if we're worried about post quantum cryptography, for example, what standards does it have to meet? How do you assess whether it's meeting those standards? So it's a huge issue in cybersecurity and one that we're always very conscious of really hard. Given the scope of cybersecurity, are there any differences in testing, let's say for maybe a small business versus a critical infrastructure operator? Are there any sort of metrics we can look at in terms of distinguishing risk or assessment? There have to be. One of the reasons I think why we have to be is that no small business can be expected to take on a hostile nation-state

that's well equipped. You have to be realistic. If you look at government guidance certainly in the UK 15 years ago on cybersecurity, you were telling small businesses that are living hand-to-mouth, week by week, trying to make payments at the end of each month, we were telling them they needed sort of nation state level cyber defences. That was never going to happen, even if they could afford it, which they couldn't. So you have to have some differentiation. So again, you've got assessment frameworks and so forth where you have to meet higher standards. So there absolutely has to be that distinction. Otherwise you end up in a crazy world of crippling small businesses with just unmanageable requirements, which they're never going to meet. It's such a great point. You touched on this a little bit earlier as well, but cybersecurity

governance operates in a fast moving technology and threat environment. How have testing standards evolved and where do new technical standards usually originate? I keep saying this is very difficult and it is. So I think there are two challenges. One is actually about the balance and this applies to the technology of today as well as the technology of tomorrow. This is about how do you make sure things are good enough without crowding out new entrants. You want people to be innovative and dynamic. You want disruptors in this business. But if you say to them, like, well, you have to meet these 14 impossibly high technical standards before you can even sell to anybody or sell to the government, whatever, then you've got a problem. And I think we've wrestled with that and there's no perfect answer. You just have to try and go to find the sweet spot

at the between two ends of a spectrum. And that's going to evolve. The second point, which in some respects, you've got the right capabilities is slightly easier, but still a big call is around, you know, those newer and evolving technologies. And here having, you know, been a bit sort of gloomy and pessimistic, here I think is actually an opportunity. So one of the things we always say in cybersecurity is that the internet was built and developed without security in mind. And that was kind of true in the 90s and the noughties as we call them over here.

But I think as you move into things like post-quantum computing, applied use of AI and so on, you can actually set the standards at the beginning. And that's really good because it's saying to people that these are the things that are going to matter in the post-quantum age. Here's the outline of the standards you're going to have to meet. Start looking at them. So there's an opportunity actually to make technology safer by design, by getting ahead of it.

And I think that's the era we're in now. That makes a lot of sense. Just building on that, do businesses and the public trust these standards and I guess which standard do you wish the world would just adopt already and what's the real reason they haven't? Well, again, where do you start? I mean, most members of the public are quite right. I haven't heard of any of these standards. I think public trust and public capital in any society matters, but I think it is important that these things are credible.

There's quite a lot of convergence between the top-level frameworks. And obviously in the US, the NIST framework is the one that's most popular for cybersecurity, but it bears quite a strong resemblance to the international one, ISO 27001 and there are all this as well. But fundamentally, they boil down to kind of five things. Do a risk assessment, work out what your crown jewels are, protect your perimeter as best you can. Those are the first two. The third one then is when your perimeter is breached, be able to detect it more times than not. And when you can't do that, you go to the fourth one, which is can you mitigate it? And when all else fails, how quickly can you recover and manage it? I mean, all the standards are expressed

in way more technical language than that. But fundamentally, if everybody adopted those five things and operated them in a simple way, you wouldn't eliminate the harm, but you would reduce it quite substantially. Which policy initiatives are most promising for incentivizing companies to undertake these cybersecurity testing parameters that you just outlined? Governments including the UK have used carrots and sticks, but what do you think will actually move the needle? I think there are two answers to that and it comes back to your split between smaller businesses and critically important businesses.

And the critically important services, I think it's easier because most industries are looking for a level playing field. In other words, they realize there have to be rules and they want to apply them to everyone. We had a fascinating experience when I was in government back in around 2018 where the telecom sector, they came to us, they said, we've got a very good cooperative relationship with the British government, but it needs to be put on a proper legal footing because you're just asking us nicely to do expensive things. And in a regulated sector, if you actually put in some rules and please develop them jointly with us, that's the crucial part, then that will help because it means that we're not going to our boards and saying, or our shareholders

and saying that we should do this. They're saying, well, do you have to do it? Are our competitors doing it? And if the answer to that is yes, we have to and yes, our competitors are doing it, then it tends to be okay. And the harder not to crack is the smaller business. And I think there's a real mystery here. Why has nobody cracked a really good and easy solution for small business? We need to be careful about this because you can't throttle small businesses with onerous regulation. At the same time, we're not brilliant, I think, in any part of the world at using the normal corporate governance rules to try and get people to figure out how to do cybersecurity.

There are initiatives there that are not the sort of pretty heavy stick that you might have to take to a critical function, but they could help. But that is a hard nut to crack. I look around the world and I think if this was easy, somebody would have figured it out. By now, I think most of the developed economies around the world really struggle with cybersecurity for smaller businesses. Yeah. Yeah, it's a great point. Actually building on one of the comments you made on the role of government and how do

you see the role of private-public partnerships scaling and strengthening robust cybersecurity testing? I think they're crucial, but they have to be practical. I've got a slight sort of high horse on this if you don't mind Kathleen. Of course. I think that there are two types of public-private partnership. One involves committees saying that we should strengthen partnerships and we should all work together and collaborate and share stuff. And we tried that for a very long time and it didn't get us very far. There are other

types. We had some of them at the National Cyber Security Centre where we paid companies to do spectacularly good technical work that the market wouldn't provide. So I think it's sort of partnership with a purpose. I think sometimes and understand the human instinct to do this, particularly in governments and big business, they think you need to get around a table and work out some grand strategy to fix everything. And the scale of the, not just the problem, but the scale of the whole technology is just too big to do that. So pick a bit of the problem, find some ways of doing it. Don't over lawyer it. I think sometimes people get very nervous. Oh, well, is this our role? You know, should we be doing

this that the other? Well, you know, sometimes certainly in this country, think, well, who's actually going to sue you over this? You know, you're not. So I wouldn't over programatize it, just get stuck in practically into solving some problems. I love that. Actually made me think, are there any surprising allies that you've gained? Maybe someone who you never expected to be a cybersecurity champion through your work. That's a, what a question. To give you a slightly disappointing answer, but it relates to your previous question. In the early part of my career, I was working in institutions like the UK Treasury, long before I was in cyber security. And the Treasury and the British Civil Service in general, but the Treasury

in particular sort of trained you to believe that the private sector was amoral, not immoral, amoral. It just didn't have values, it just had bottom line. And its job essentially was to provide employment and revenue then for the government to spend on good things that people cared about. And when I got into cybersecurity and people said, look, you need to develop relations with this cybersecurity company, often in the US actually. And I said, well well what's in it for them and sure sometimes you were paying them for specific services but other times there was a real public spiritedness about this there was a realization that if you try to delineate public private

boundaries that it wouldn't really work it was a shared risk and you could analyze where the boundaries fell or you could actually go on and do something about it together. So I was genuinely surprised at the allyship from the cyber security sector. Absolutely, I really, really was. And I think it's a really positive part of certainly the UK cyber security ecosystem. Lyle Orr Wonderful. Well, we're coming to the end of our time here. But is there any maybe last thoughts or perhaps requests you have for our listeners today? I think that standards, assurance and testing really matter.

But it's a bit like the discussion we're having over AI. Get all these things to take you 80, 90% of the way and then really apply your judgment. There's been some bad regulation and under the auspices of standards and assurance intervals, have you done this assessment? Have you done that? Have you looked at this? Well, fine. But, and you can tick that box, but what does it actually mean when you do it? What bits that you know in your heart of hearts are really important to the defence of your organisation that may not be covered by this? And just go and do those anyway because sure it helps, but it's not everything.

Kierran, thank you for joining us today. This has been just a super fun conversation and really insightful. Just really enjoyed the conversation. Thank you. My pleasure, Kathleen. Thank you. Now I'm happy to introduce Tori Westerhoff. As a Principal Director on the Microsoft AI Red Team, Tori leads all AI security and safety Red Team operations, as well as dangerous capability testing to directly inform C-suite decision-makers.

So Tori, welcome. Thanks. I am so excited to be here. I'd love to just start a little bit more learning about your background. You've worn some very intriguing hats. I mean, cognitive neuroscience grad from Yale, national security consultant, strategist in augmented and virtual reality. How did those experiences help shape the way you lead the Microsoft's AI red team?

I always joke this is the only role I think will always combine the entire patchwork LinkedIn resume. I think I use those experiences to help me understand the really broad approach that AI Red Team, artists also known as AI RT, I'm sure I'll slip into our acronym, artists also known as AIRT, I'm sure I'll slip into our acronym, how we frame up the broad security implications of AI. So I think the cognitive neuroscience element really helped me initially approach AI hacking, right? There's a lot of social engineering and manipulation within chat interfaces that are enabled by AI. And also this metaphor for understanding how to find soft spots in the way that you see human heuristics show up too. And so I think that was actually my personal in to getting

And so I think that was actually my personal in to getting hooked into AI, red teaming generally. But my experience in national security, and I'd also say working through the AR, VR metaverse space at the time where I was in it, helped me balance both how our impact is framed, how we're thinking about critical industries, how we're really trying to push our understanding of where security of AI can help people the most, and also do it in a really breakneck speed in an industry that's evolving all of the time.

That's really pushing you to always be at the bleeding edge of your understanding. So I draw a lot of the energy and the mission criticality and the speed from those experiences as we're shaping up how we approach it. Can you just give us a quick rundown? What does the Red Team do? What actually kind of is involved on a day-to-day basis? And then as we think about our engagements with large enterprises and companies,

how do we work alongside some of those companies in terms of testing? The way I see our team is almost like an indicator light that works really part and parcel with product development. So the way we've organized our expert red teaming efforts is that we work with product development. So the way we've organized our expert red teaming efforts is that we work with product development before anything ships out to anyone who can use it. And our job is to act as expert AI manipulators, AI hackers.

And then we are supposed to take the theories and methods and new research and harness it to find examples of vulnerabilities or soft spots in products to enable product teams to harden those soft spots before anything actually reaches someone who wants to use it. So if we're the indicator light, we are also not the full workup, right? I see that as measurement in evals and we also are not the mechanic, which is that product development team that's creating mitigations, it's platform security folks who are creating mitigations at scale. And there's a really great throughput of insights from those groups back into our area where

we love to inform about them. We also love to add on to how do we break the next thing, right? So it's a continuous cycle. And part of that is just being really creative and thinking outside of a traditional cybersecurity box. And part of that is also really thinking about how we pull in research. We have a research function within our AI Red team and how we pull in research. We have a research function within our AI Red team and how we automate and scale. This year, we've pulled a lot of those assets and insights into the Azure Foundry AI Red teaming agent. And so folks can now access a lot of our mechanisms through that. So you can get a little taste of what we do day to day in

the AI Red Teaming agent. You recently, actually with your team, published a report that outlined lessons from testing over 100 generative AI products. But could you share a bit about what you learned? What were some of the important lessons? Where do you see opportunities to improve the state of red teaming as a method for probing AI safety? I think the most important takeaway from those lessons is that AI security is truly a team

sport. You'll hear cybersecurity folks say that a lot, and part of the rationale there is that the defense in depth and integrating a view towards AI security through the entire development of AI systems is really the way that we're going to approach this with intentionality and responsibility. So in our space, we really focus on novel harm categories. We are pushing bleeding edge. And we also are pushing iterative and like contextual based red teaming and product dev. So outside of those a hundred that we've done,

there's a community through the entire, again, multi-stage life cycle of a product that is really trying to push the cost of attacking those AI systems higher and higher with all of the expertise they bring. So we may be experts in AI hacking in that line, but there are also so many partners in the Microsoft ecosystem who are thinking about their market context or they really, really know the people who love their products. How are they using it? And then when you bubble out, you also have industry and government who are working together to push towards the most secure AI implementation for people. Right?

And I think our team in particular, we feel really grateful to be part of the big AI safety and security ecosystem at Microsoft and also to be able to contribute to the industry real art. As you know, we had a chance to speak with Professor Kieran Martin from the University of Oxford about the cybersecurity industry and governance there. What are some of the ideas and tools from that space

that are surfacing and how we think about approaching red teaming and AI governance broadly? Yeah, I think it's such a broad set of perspectives to bring in in the AI instance. Something that I've noticed interjecting into security at the AI junction, right, is that cybersecurity has so many decades of experience of working through how to build trustworthy computing, for example,

or bring an entire industry to bear in that way. And I think that AI security and safety can learn a lot of lessons of how to bring clarity and transparency across the industry to push universal understanding of where the threats really are. So frameworks coming out of NIST, coming out of MITRE that help us have a universal language that inform governance, I think are really important because it brings clarity irrespective of where you are looking into AI security, irrespective of your company size, what you're working on. It means you all understand, hey, we are really worried about this fundamental impact.

And I think cybersecurity has done a really good job of driving towards impact as their organizational vector. And I am starting to see that in the AI space too, where we're trying to really clarify terms and threats. And you see it in updates of those frameworks as well that I really love. So I think that the innovation is in transparency to folks who are really innovating and doing the work. So we all have a shared language.

And from that, it really creates communal goals across security instead of a lot of people being worried about the same thing and talking about it in a different way. In the cybersecurity context, Kieran really stressed matching risk frameworks to an organization's role and scale. Microsoft plays many roles, including building models and shipping applications. How does your red teaming approach shift across those layers? I love this question, also because I love it as part of our work. So one of the most fascinating things about working on this team has been the diversity of the technology that we end up retinium testing. And it feels like we're in the crucible in that way because we see AI apply to so many different architectures, tech stacks, individual features, models,

you name it. Part of my answer is that we still care about the highest impact things. And so irrespective of the iteration, which is really fascinating and I love, I still think that our team drives to say, okay, what is that critical vulnerability that is going to affect people in the largest ways? And can we battle test to see if that can occur? So in some ways, the change our testing, we customize a lot to the access to systems and data and also people's trust, almost as different variables that could affect the impact.

Right? So a good example is if we're thinking through agentic frameworks that have access to functions and tools and preferential ability to act on data, it's really different to spaces where that action may not be feasible. Right? that action may not be feasible, right? And so I think the tailoring of the way to get to that impact is hyper-custom every time we start an engagement. And part of it is very thesis-driven and almost mechanizing empathy.

You almost need to really focus on how people could use or misuse in such a way that you can emulate it before to a really great signal to product development to say this is truly what people could do. And we wanna deliver the highest impact scenarios so you can solve for those and also solve the underlying patterns actually that could contribute to maybe that one piece of evidence,

but also all the related pieces of evidence. So singular drive, but like hyper, hyper customization to what that piece of tech could do and has access to. What are some of the unexplored testing approaches or considerations from cybersecurity that you think we should encourage AI technologists, policymakers, and other stakeholders to focus on? I do love that AI humbles us each and every day with new capabilities and the potential

for new capabilities. It's not just saying, hey, there's one test that we want to try, but more, hey, can we create a methodology that we feel really, really solid about so that when we are asked a question we haven't even thought of, we feel confident that we have the resources and the system. So part of me is really intrigued by the process that we're asked to make without knowing what those capabilities are really going to bring. And then I think tactically AIRT is really pushing on how we create new research methodologies, how are we investing in kind of these longer-term iterations of red teaming. So

we're really excited about pushing out those insights in an experimental and longer term way. I think another element is a little bit of that evolution of how industry standards and frameworks are updating to the AI moment and really articulating where AI is either furthering adversarial ability to create those harms or threats, or identifying where AI has a net new harm. And I think that demystifies a little bit about what we talked about in terms of the lessons learned that fundamentally a lot of the things that we talk about are traditional security vulnerabilities. And we are standing on kind of that cybersecurity shoulder. And I'm starting to see those updates translate in spaces that are already considered

that are already considered trustworthy and kind of the basis on which not only cybersecurity folks build their work, but also business decision makers make decisions on those frameworks. So to me, integration of AI into those frameworks by those same standards means that we're evolving security to include AI. We aren't creating an entirely new industry of AI security. And that I think really helps anchor people and the really solid foundation that we have in cybersecurity anyways. I think there's also some work around how the cyber defenses will actually benefit from AI. So we think a lot about threats because that's our job. But the other side of cybersecurity is offense. And I'm seeing a ton of people come

out with frameworks and methodologies, especially in the research space, on how defensive networks are going to be benefited from things like agentic systems. Generally speaking, I think the best practice is to realize that we're fundamentally still talking about the same impacts and we can use the same avenues, conversations, and frameworks. We just really want them to be crisply updated with that understanding of AI applications. How do you think about bringing others into the fold there? I think those standards and frameworks are often informed by technologists, but I'd love you to expand policymakers or other kind of stakeholders in our ecosystem, even end

consumers of these products. How do we communicate some of this to them in a way that resonates and has an impactful meaning? in a way that resonates and it has an impactful meaning. I've found the AI security safety space to be one of the more collaborative. I actually think the fact that I'm talking to you today is probably evidence that a ton of people are bringing in perspectives that don't only come from a long-term cybersecurity view. only come from a long-term cybersecurity view. And I see that as a trend in how AI is being approached opposed to how those areas were moving earlier. So I think that speed and the idea of conversations and not always having the perfect answer, but really trying to be transparent with what everyone does know is kind of a communal energy in the communities, at least where we're playing.

So I am pretty biased, but at least the space is where we are. No, I think we're seeing that across the board. I mean, I'd echo sitting in research as well, like that ability to have impact now and that speed to getting the amazing technology and models that we're creating into the hands of our customers and partners and ecosystem is just underscored. So on the note of speed, let's shift gears a little bit to just a quick lightning round. I'd love to get maybe some quick thoughts from you. It was just 30 second answers here.

I'll start with one. Which headline grabbing AI threat do you think is mostly hot air? I think we should pay attention to it all. I'm a red team lead. I love a good question to see if we can find an answer in real life. So no hot air, just questions. Is there some sort of maybe new tool that you can't wait to sneak into the Red Team arsenal? I think there are really interesting methodologies that break our understanding of cybersecurity by looking at the intersection between different layers of AI and how you can manipulate AI to AI interaction, especially now when we're looking at agentic

systems. So I would say a method, not a tool. So maybe ending on a little bit of a lighter note, do you have a go-to snack during an all-night red teaming session? Always coffee. I would love it to be a protein smoothie, but honestly, it is probably Trader Joe's elote chips. Like the whole bag. It's going to get me through.

I'm going to not love that I did it. Amazing. Well, Tori, thanks so much for joining us today, and just a huge thanks also to Kieran for his insights as well. Thank you so much for having me. This was a joy. To our listeners, thanks for tuning in. You can find resources related to this podcast in the show notes. If you want to learn more about how

Microsoft approaches AI governance, you can visit microsoft.com slash rai. See you next time. Thanks for watching!