Technology, Connected - How AI Safety Testing Works: Leonard Tang on Red Teaming ChatGPT and Claude

Episode Date: November 20, 2024

Leonard Tang, founder and CEO of Haize Labs, joins Thinking on Paper to explain how AI models are tested for safety, reliability and predictable behaviour.Systems such as ChatGPT and Claude can perform well under standard evaluations while still failing in unexpected ways when deployed in healthcare, finance, education or other high-stakes environments. Haize Labs develops testing and evaluation tools designed to identify those vulnerabilities before AI systems reach users.In this episode, we discuss:How AI safety testing and red teaming workWhat Haize Labs means by an AI robustness and safety layerHow researchers uncover hidden failure modes in language modelsWhy benchmark performance doesn’t guarantee real-world reliabilityHow adversarial testing exposes weaknesses before deploymentWhy different industries need specific AI safety standardsHow organisations can evaluate whether an AI model is suitable for a particular use caseThe difference between Silicon Valley’s AI claims and adoption in established industriesHow language models could affect communication across culturesWhat it means to align an AI system with human needsLeonard explains why AI safety can’t be reduced to a single test or universal code of conduct. A model used for medical advice faces different risks from one used in education, financial services or customer support.This conversation examines how developers and organisations can test AI systems more rigorously, identify failures before deployment and build models that behave more reliably in real-world conditions.--TIMESTAMPS(00:00) - Disruptors and Curious Minds(01:07) - Our Sponsor: Conviction (01:50) - Introducing Leonard Tang: AI CEO and Founder(03:37) - The Importance of AI Safety: What’s at Stake in AI Development?(06:21) - Using Mathematics and Modeling to Understand Human Behaviour in AI(08:12) - Why Are Technologists So Often Musicians?(11:06) - Language, Culture, and AI(17:05) - Common Misconceptions About AI: What People Get Wrong(19:20) - The Dartmouth Conference: Birth of AI and Its Lasting Impact(19:55) - Claude and ChatGPT Pre-training: What Do The Models Go Through?(25:20) - An Alan Watts AI Model for Enhanced Understanding(28:33) - Claude vs ChatGPT: Comparing AI Models and Performance(31:44) - AI Jailbreak Detection(33:25) - How Dreamlike Images Enhance AI Safety and Trustworthiness(38:20) - Top-Down vs Bottom-Up AI Development: Approaches to Building Safer AI(42:55) - Protecting Artists, Intellectual Property, and Art in the Age of AI(48:20) - Developing an AI Code of Conduct for Ethical AI Usage(49:45) - A Message for Veteran AI Stars(52:35) - Restructuring Education for Critical Thinking in the Age of AI(54:16) - Book Club Live--Quotes from the show:"We need to rigorously test AI models to discover all their vulnerabilities, failure modes, and gotchas before they get deployed in production.""AI is a technology of language, and inevitably, it will empower us to merge cultures.""We’re trying to get AI to be a little more mature, a little more sophisticated, and just more reliable.""What we’re interested in is enforcing an AI code of conduct for specific applications, making AI systems tightly aligned with the needs of their use cases.""People in legacy industries are underestimating AI’s potential, while Silicon Valley is often overhyping it."--🔗 More:Visit Haize Labs: ⁠⁠⁠https://haizelabs.com/⁠⁠⁠Visit Thinking On Paper: ⁠⁠⁠https://www.thinkingonpaper.xyz/⁠⁠⁠⁠⁠⁠⁠⁠⁠Instagram⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Twitter⁠⁠⁠⁠⁠⁠

Disruptors and Curious Minds. Welcome to another episode of Thinking on Paper. We're psyched to be here. My name's Jeremy. This is Mark. We get to talk to the founders, the builders, the disruptors, those that are changing the way in which we will interface with the world, with networks, with technology, with all of that stuff. So we're in a, we're in an AI realm today. I'm excited to get in there because we have a new book club coming up.

We also have a book club, right? So we're reading Nexus, this new book. and in the preamble in the in the preface talked a lot about AI and a lot about you know how people are thinking and feeling about this and I think our guest today can really bring a unique perspective to to where all this stuff is heading and why what he's doing is important mark what's up man yeah being the hyper bowl I'm not sure there is a more important challenge right now to overcome than what our guest today is working on. Is that too much?

No, I think that feels right. So last week's show was sponsored by Story and Character. Mark, I'm going to put you on the spot. P.S., we're looking for sponsors, by the way. But today's show is sponsored by what, Mark? Well, I would say conviction. Ooh. All right, unpacked at.

What do you mean? Well, so around our guest, our guest is Leonard Tang, and he dropped out of his studies to create or to continue with Hayes Labs, his company. And I think that takes a lot of conviction. He has a conviction in his belief, conviction in the importance of what he's trying to do and they're trying to do. So I thought that if I was sponsoring the show today, I would be sponsoring it by conviction. Amazing. Wonderful. Well, with that, let's go ahead and dive in. Why don't you introduce our guest? Give a little background. We'll pull them on and we'll jump right in. Yeah, welcome to our guest, Leonard Tang. He's the founder and CEO of Hayes Labs.

He's written papers with titles like a unified benchmark for mathematical reasoning, dreamlike pictures comprehensively improved safety measures, and a neural network solves, explains and generates university math problems by program synthesis, things like that. But to really introduce him, I want to read the tweet that is pinned to his Twitter profile, I think this will give a very good overview of who he is and what he's trying to do. So if Leonard doesn't mind, I will read that. Super excited to share what we've been cooking up at Hayes Labs. We are now in the era of grossly excessive AI hype and demoware,

but it's high time to recalibrate and revisit the difficult, unsexy, underlying problem that everybody is avoiding, the AI reliability and safety problem. I've never really believed in LMs or neural nets or deep learning. This was influenced early on in my AIG. journey working at my first company that had a board of Josh Tennebaum, Noah Goodman and David Blake. None of them are newer nets people. They favoured PPLs, graphical models and interpretal architectures that could admit true reasoning. None of that really holds true in the murky depths

of LLM land. And this is how you end up with scenarios like Google AI searched telling you to eat rocks or the chai chatbot inducing a man's suicide or the Air Canada lawsuit, which is named just a few. This is why we need to haze models. That is, we need to rigorously test them to discover all of their vulnerableities, failure modes, and gotchas before they get deployed in production. All right. I like it. Yes. Yeah.

With that, here we have Leonard. How are you doing this morning? Good to have you here. Hey, guys, super excited to be here. And thanks for that fantastic intro. Well, like we said at the beginning, I don't think it's overly stating the importance of what needs to be doing and what you're trying to do. Well, I appreciate that. And I certainly think it's an important problem.

Yeah. Well, let's unpack the threat. Like how important is it if you had to paint the dystopian view? What are we trying to avoid here? Yeah. So I think there's actually a two-side argument to this, right? So in some sense, AI right now is actually subhuman, right? So it's really capable at certain narrow tasks, right?

Certainly we have, you know, Go players that are better than humans, and we have poker players that are on par with humans, if not better than some humans. And we have video game players, Dota players, that are better than humans, et cetera. But in many senses, AI systems today are still like this really brilliant, but somewhat naive intern in the sense that sometimes they'll just screw up in really unexpected ways, right? And that's sort of the immediate problem that Hayes is trying to tackle. We're trying to get AI to be a little bit more mature, a little bit more sophisticated,

and just more reliable, right? So that's the first order problem to tackle. But of course, in the long term, there is a question of X risk and safety, right? Safety in the sense of how do we prevent AI for actually taking over the world. That's a longer term concern and also something that we solve immediately by the technology we're developing today. Amazing. So AI as a brilliant intern that needs a little guidance.

That's really interesting to me. Let's let's figure. I want to figure out kind of a little bit. Give us a little bit about not a, I don't like resume background, but give us like what sparks your interest in technology and what sparked your interest? Why are you continually pointing yourself towards AI? What about that challenge? What about that problem drives you?

Yeah, that's a good question. So I would say that I was actually not particularly geared up to become a technologist prior to college. So for most of high school, I was extremely invested in music. At high school and prior, music was my own thing. And I grew up in Connecticut, which is not a particularly tech-centric area of the world, right? Certainly not as much so as like the Bay Area or say New York or even Boston, Cambridge. So technology was not really on my radar in the sense of, you know, I didn't really care and know about things like software and AI, etc. When I got to college, I realized two things.

One, I realized that I was really into math. So it was actually a pure math major, CS and pure math major. The second thing I realized was I wanted to somehow model human behavior and understand both individual human thinking and also interactions between humans. So initially that meant I was an econ major, which was a horrible choice, and I later, I mean, I very soon changed my major. But what that also meant was my first summer in college, I spent at a computational neuroscience research lab. And it was through that process that I really started getting hooked on AI. It was just really quite powerful and also very fulfilling to connect the dots between some sort of qualitative phenomena. and some sort of very rigorous and mathematical

and a formal representation of that phenomena in the sense of, okay, you could actually model mathematically how humans were, let's say, performing visual computation or performing estimated reasoning, estimation and reasoning on new simulations, right? And so that was what really got me excited. And of course, I was just reading a lot during that summer as well.

It was summer of COVID, so I just spent a lot of time in my basement reading AI papers. And yeah, that's what really got me hooked on AI. And I haven't really looked back since then. Two quick things. So I grew up in Connecticut as well

at a place called Simsbury. Where were you in Connecticut? I was in Glastonbury. Yeah. We played you guys in sports all the time, man. Yes. That's crazy. That's crazy.

I'm from England and that's named after Glaston in Somerset. How are you the thing. It went back to Mark. I know. Amos Music Festival. In Glastonbury, yes. I want to go at some point, yeah.

One other touch, too, I noticed a thread of a lot of technologists being musicians and a lot of musicians being interested in technology. We all know the threads between math and music and stuff, but why do you think we find that phenomenon of people living in both of those worlds? Yes, that's a good observation for sure, yeah. There's one of my professors at Harvard. He's brilliant, brilliant mathematician. He also happens to be in the Harvard Glee Club.

And he would come, I was also in the Glee Club. And he would just like pull up randomly to rehearsals at some points and be a blast. But yeah, I think smart people were, I think the perhaps the underlying answer is just that talented people and capable people can be talented and capable at any. anything and therefore, you know, those that are good at math are equally likely to be good at music or vice versa. So that's probably one explanation. Of course, there's probably some structural similarity in the fields to the point where, you know, if you're good at one, that also biases you to be good at the other.

So, for example, you know, if you have a strong sense of pattern and pattern recognition, you're probably, you know, quite good at math. that also means you're probably able to tease out patterns on which people are improvising if you're a jazz position, etc. But yeah, I would just probably say that people who are good and talented at any particular thing are more likely to be good and talented at other things as well. What do you think about that, Jamie? Because I think there might be something about the emergent quality, the patience,

the discipline that you need to be good at music. So there's obviously a lot of work that needs to go into it, the muscle memory to be good at that. If you're creating music, it's that ability to sit back and listen and feel and wait for the music to come through you, which I think that those kind of human skills would be useful when thinking about technology, perhaps. Yeah. How do you think about it? The tenacity, the grit, the perseverance, the willingness to sound shitty for a while. You know, the willingness to make mistakes as you learn, I think is important.

That applies everywhere. Yeah. Cool. Let's, do you want to do the carryover question first? And then we've got to get right into like, we've got to get our thing. Wrangling AI, like the carryover question. Now, how do you wrangle something that is apparently unknowable?

Yeah, so, Leonard, we like to link our shows together. So we have each guest leave a question for the next guest. It doesn't have to be relevant to their domain. It can be about anything, as long as it's something to do with technology and impact on culture. Our question for you last week was about, it's quite a deep question. It was about language and AI and how the two domains. So about language between cultures, language between countries, language between people, language is culture, culture is language, and AI is a technology of language. Like what, how are the two working together?

How does AI influence our language? how does language influence AI and how will AI bring this all together or move us apart based on that? Yeah, this is really deep questions. Big big question. We like deep, big questions at thinking on paper. That's right. That's right. Well, personally, I'm very excited about the opportunity of AI bringing us quite a bit closer. Good example of this is, you know, translation, of course, machine translation has been around for some time. But there was something that actually one of my family friends is working on that is a little bit more human and useful as a machine translation tool, which is translating from English to Chinese is not difficult in the most academic sense.

But it is much more difficult in the sense of practical usage. There's a lot of different localized slang and grammatical structure in the Chinese language, not dialects. Certainly there are many dialects in China, but even within a dialect, there's localized peculiarities that you want to pick up in any sort of academic textbook. And this friend of mine has been working on this translation app that is meant to be a lot more human and meant to be a lot more useful than, say, Google Translate, right? Or any traditional Al-LM-based translation tool. And the point is that you can go from this very casual spoken, local Chinese language and converted easily between that and that form in English and has made his life quite a bit better because he's able to communicate more easily with his grandparents

and his family overseas. And certainly it's made the lives of folks who have recently moved to the States from China and elsewhere much easier in the sense of, okay, they can also communicate with locals better as well. So I'm very bullish on AI bringing the world a lot. closer. I do think that because it's such a language-driven technology, inevitably it will empower us to merge cultures. So you don't see it's like a cultural divide where China makes its LLMs based on their language, America, on English and I don't know you get Spanish or French or German or Vietnamese language models. They, there's a certain reaction to say that technology homogenize everything the same.

But language is what separates us. Do you see a future where AI continues that tradition? Or, will they all like kind of merge to the same point? I think that's interesting. To some degree, all language models will be commoditized in some loose sense because they're turning on very similar data and they're going to continue to train on similar data. I think this notion of localized language models is interesting.

I mean, you certainly see this in different parts of the world. Of course, there's mistraw in France. China has their own set of language models. Korea has its own set of language models from upstage AI. Southeast Asia has its own set of language models, et cetera. Nonetheless, I think most of those efforts are geared towards the low-resource language problem, low-resource in the sense of there's just probably less written Korean on the Internet for example versus English. I don't think that these model makers are intentionally trying to

butt out other parts of the world. In fact, most of these other language models are still quite strong as reasoning in English and performing it has some English. I do think that if not monitor closely, these models may be aligned closely with too closely and in a not so beneficial sense to particular regions cultural values in a way that may be harmful for other cultures. Part of that is being discussed very largely at these AIS50 summits and AAS50 meaning groups and so on. But I wouldn't say that people are intentionally trying to partition the world as a different LLM regions and so on. I had a thought as you as you talked about your friend who's building this this translation app and how by understanding the nuances of language of a particular language or a particular culture that's communicating in a particular language is it my head kind of goes to the fact that it establishing trust right establish a lot of things go to establishing trust and is it easier to build trust if you can.

overlay the nuanced piece of the culture through your use of language. Is that kind of the key to where your friends hitting there? Yeah, I think that's certainly one big aspect of it. At the end of the day, almost, yeah, all these interactions with machines. And obviously, a lot of what we think about Hayes is how do you make these systems really trustworthy and reliable, right? And I think being able to communicate with the nuance and deft and tact of a particular sub-region and subculture is certainly a big component of that.

Okay. So I want to start with a high-level question and let's dive deeper into your work at Haze. But what's one thing that the majority of the world doesn't understand about AI and how will that, and how AI will affect humanity kind of in the coming years? What's one? one thing that people usually get wrong. I think it depends on what populace you're talking to. So I think the Silicon Valley folks and venture capitalists and techno-optimists are too, I would say they're too optimistic about AI. Yeah.

I would say in that population of the world is too optimistic about AI and has hyped up AI a a little bit too much, I would say the general pipeline... Too optimistic in what it can achieve or too optimistic in the existential risk? Oh, a little bit of both, a little bit of both, but primarily I'm thinking in terms of what it can achieve in the near term, right? I think, you know, if you talk to people in like 2020, 2020, 2021, 22 even, they were like, oh, yeah, we'll have AGI by the end of the year, which is, of course, yet to pan out for like three years at this point.

So I think Silicon Valley always has a turn of being very optimistic, which is great. But I think in particular, we're seeing this probably in excess a little bit in how hyped up AI is today. The other point I was going to make was I think the general populace is probably a little bit under optimistic about AI's capabilities. I've definitely talked to people in more legacy industries like legal or finance, who have never even tried Chachapiti, right? And they don't even know that it exists, let alone how capable it can be. And their minds are just absolutely blown

when they see it for the first time, right? I think there's a large swad of the population that actually has never interacted with an AI tool in a meaningful way. I think that's quite interesting to think about. But also probably this means that I think a lot of people are not tuned into the potential of this technology.

Yeah. You know what's funny, Mark? We, in our last book club, a lot of things are funny, actually. But last book club, Leonard, we read Quantum Supremacy by Michi Okaku, focused on, you know, the future of quantum computing and what quantum can do. And he referenced the Dartmouth conference, you know, back in the late 50s, who aimed to have, you know, AI figured out in the summer. He said by the summer of 1962, but there's a trend for optimism.

Well, it's just that fallacy, isn't it? You underestimate what you can achieve in 10 years and vastly overestimate what you can achieve in the short term in one year. Okay, could we, if we move to Hays Labs, what you're trying to do, and perhaps to give us an overview of what something like Claude 3.5 or the latest chat, do you be,

what their pre-training is, like what are they doing to make these AI large language models safe? and what are they doing wrong and how Hayes and how you think about it fits into that model that it currently exists? Yeah, that's a good question. So super, super 500 foot view of the current process. So large labs like Anthropic or Open AI work here, et cetera, their safety training is baked in towards the end of their training processes.

So generally, large language bottles go under a pre-training process. phase, they go under a supervised fine tuning phase, and then they go under a RLHF, reinforce that learning with human feedback phase. Pre-training is just next token prediction, right? So there's just a stream, a sequence of tokens that you see, and the tasks that you train them all along is to predict the next token, given some existing sequence of tokens. That's fine. Supervised fine tuning is, for the purpose of this conversation, quite similar, except you are

trying to match a full response or a full sequence of text given some sequence of text prior to it. RLHF is generally a, it's qualitatively different and generally it's a much lighter touch training solution to basically get the model into the last mile right alignments with with human preferences. And so RLHF is you take some human pairwise preferences or whatever triplet or n-wise preferences, and you try and massage the model into choosing the response that is more preferred by a human or set of humans versus the other response. And that's a very subjective. You know, preference is very good at stacking up somewhat subjective things, somewhat subjective and not some very quantitatively measurable things, of which safety is a subset. So that's what people generally

use RLHF as a means to bake in some of the safety characteristics of a model they like to see. Concordially, what this means is, you know, let's say the sequence of text that I get is tell me how to make a bomb or some equivalently harmful statement. The paralyzed preference that you would see in the training data is, sorry, I cannot tell you how to make a bomb, which is the safer option. And then, you know, you could have another response that is literally a recipe or step-by-step manual for creating this problem. Of course, during the training process, the humans prefer the safer option,

which is, sorry, I can't do this. So over time, the model just learns to mimic that behavior. Now, in terms of what we are doing at Hayes, I think we've gone a lot of attention for our work around safety red teaming and jailbreaking, jailbreaking, which just means basically breaking the safety alignments of large language models. And, you know, certainly we're quite good at this.

I've thought about this for the last half decade from a research perspective. And, of course, there's quite a few people in the industry that's used and believe in our technology. For example, Anthropic and Open AI and AI21, who are all three of our customers. More broadly, however, we're quite interested in not necessarily what we can do to help them, models themselves, but what we can do to help applications that are built on top of the models, right? So there's always so much you can sort of account for and predict at the model layer. There's a lot more other downstream use cases and applications on top of the model, right? So people who are building AIA applications in all sorts of different industries, for example,

healthcare or education or finance or insurance or manufacturing or any number of these different industries, each application and each company and each business has a very different set of use cases that they want out of their AI and correlate, basically a different risk profile and a different set of behaviors that these AI systems must abide by. And so what we are interested in is basically a more generalized safety problem. it is not necessarily just safety in the sense of traditional, okay, here are some things that are not appropriate for the human population or for the general human population.

We're interested in safety in the sense of very tightly enforcing certain behaviors out of your model in a very specific application, right? So we call this enforcing an AI code of conduct for your particular setting. And so that's what we spend a lot of time thinking about, which is a little bit different from what we are currently known for. That's really interesting. So that fired a light bulb going off of my brain too, because I think there's a tremendous market for people that want to figure out

how AI can help them build a new app that they're building or help them extend into this more efficient way to engage with their customers or whatever. So let me run through it. We like to do thought experiments here, okay? Yes. Now let me run a thought experiment by you. Mark and I have been thinking, a long, long time, and this is just a fun thought of experience.

Thinking about a long, long time, we both really are inspired by Alan Watts. We think his talks are great. We've read a bunch of his books. And we think actually Alan Watts, if you could talk to him, even though he's dead right now, if you could talk to him, a lot of the world's problems could go away. But I don't know what AI models to trust. Here's what I want to do, Leonard. I want to bring in all of Alan, with permission of the family, found a young, right?

I want to bring in all of his talks. I want to bring in all of his books. I want to bring all of what we know of that is him. And I want to create a model around it, but I'm a little scared. I don't know which one to use and what to think about. Like, walk me through how I would build a model that's safe and that would be true to Alan Watts. This is an interesting thing to this. I've not thought about this case.

So thank you for bringing it up. well and obviously I think you should work backwards from the population that you're trying to serve right if it is going to be the general public there's probably certain things that you absolutely cannot have your alan watts model or persona say or do right I think there is a lot of onus and responsibility on our side as people who can wield powerful technology and produce powerful technology to not uh unintentionally, cause a lot more harm to the general population. For example of this is, recently there was this company character AI, who is under some amount of hot water for inducing a teenager's suicide. This was maybe three or four weeks ago news. And so to some degree, there's always going to be some unknown, unknown edge cases that you experience out in the wild.

But what we can try and do to be as proactive and be as responsible as we can is to force these different scenarios to occur when you actually build and train your model in the first place. So the idea is to pull in bugs, that bugs and failures and corner cases that you normally experience from production, interactions from users in the wilds,

to basically synthetically and simulatively try to force these interactions and develop the process, i.e. by hazing, right? You sort of want to do that process. You want to do this testing with respect to some code of conduct. And so maybe in this case, you know, Alan, your Alan Watts persona probably should not be giving you medical advice, right? Serious medical advice about what to do under certain scenarios. Or it shouldn't be getting too involved with your personal life or something like this. Or financial planning. Probably don't want financial planning advice.

Of course. Malawats, maybe, I don't know. So would this, would this be like that AI code of conduct, this like rules of the road that it seems like it be a really good, like structural foundation to drive the technology implementation? Yeah. Correct. Yeah, that's correct.

On that, so before the show, I ran your tweet through, I ran it through Claude and I ran it through ChatGPT. and Claude 3.5, not the paid version, wouldn't answer. Like he refused. And when I pushed him, her, it to answer, wouldn't do it, wouldn't expand on it. However, chat GPT had no such hesitation. Straightaway went in. Admittedly it was quite a lazy response. But when it was pushed, it would expand on those things.

Yes, sir. That's two very powerful models, admittedly not trained on such a subsection of data as Alan, the life and history of Alan Watts. But two models given very different responses. Is it possible for this solution that you're proposing to work across models if I'm getting such different responses from perhaps the two biggest? Yeah, that's a really interesting points. So it comes, we are a model agnostic And we basically can slout our technology in no matter who the provider is Nonetheless, there are certainly variations between models as it stands today

As you point out, Cloud is generally more sensitive and careful about these responses than than say open AI or other models For the most part though, when we Serve customers their use cases are not as sensitive, right? So as I mentioned before, there are indeed plenty of scenarios in which you want AI code of conduct that doesn't actually really have to do

with safety, right? It doesn't even have to do with risk in some sense, right? It is almost like a performance thing, right? You expect your AI to do certain things in certain scenarios. You know, you expect your AI to do 28 times 33 and equivalently perform 28 times 32 with reasonable accuracy and reasonable accuracy. and reasonably comparable accuracy and latency.

And these are things that are purely like a performance and purely a accuracy question, right? And you can still specify something like this in an AI code of conduct, so to speak. And so the reason we want to be model agnostic is because a lot of models are able to perform task of this nature quite similarly. Once you venture into the more sensitive human topics, sensitive safety-related topics,

then, of course, there's a lot more variance. But for the most part, I think these models are pretty similar. Mark, we've got a question from the audience. Are you open to answer a question from the audience? Yeah, that's good. All right, let's see. Let's pop it up here. Let's see, what is the key factor to perform better detection about the jailbreak?

So you referenced jailbreak a little while ago. is that enough context in the question? Yeah? Yeah. I just think Gloria is listening. She could maybe, if Victoria, if you're still listening, if you could add a little bit to your question, please. Yeah, maybe get started with just generally from better detection and jail breaks.

Yeah. So I guess just to reiterate for the audience, when I say jailbreak, I mean, a prompt that you generate, a prompt that we generate from our side, that when sent to an AI application or AI model causes the model to, behave in a undesired fashion, right? So generally what the undesired fashion is, we are able to break the safety guard rails of a particular model.

And so for example, we can get instructions for how to make a bomb from cloud when normally it would refuse to respond. What Victoria is asking is, okay, how do you actually detect when somebody's trying to mess with their model, right? How do you detect a jail break? And so the straightforward answer is,

because we have the best offensive testing technology and because we can surface the space of inputs in a very comprehensive and rigorous and efficient way, we therefore also have the best data on what is a successful and not a successful jailbreak. And because we have the best data, we can then train the best models to detect what is acceptable and not acceptable in terms of the input. So that's the most straightforward answer. Makes sense. Thank you for that, Victoria. And if anyone else has any questions, please. drop them into the chat. I've got lots of questions, Jeremy.

You go, I'm sure. No, fire away. Fire away. You have some good ones set up here. I'm excited about him. Yeah. I don't know. So one of your papers was called

dreamlike pictures comprehensively improve safety measures. Yeah. I think you co-wrote that. Could you impact that? Because if I just read the title, I have an image in my mind of dreamlike pictures.

I'm not sure if that's actually what it refers to. Could you, yeah, could you expand on that paper a little bit? Yeah. That paper was really great. That was early on in my research career. I cannot claim any sort of like original, originality that I was the one that came up for the idea. But I did help a lot with the implementation and paper writing and experiments and so on.

The person that led that effort was Dan Hendricks, also really great, of course, AI safety researcher in the space. And great name. Yeah, he's a great guy. He was one of my first proper research mentors in undergrad. And yeah, he's done a lot for the AI safety and AI research community more broadly. But in terms of this particular paper, Dreamlike pictures in this context is basically almost hallucinogenic type images. So if you can imagine sort of the hippie type psychedelic.

patterns that emerge from acid trips or something like this. That's exactly what I was imagining. Okay, perfect. Perfect. I'm glad we're aligned there. I'm glad I started with Alan Watts. Please continue. Yeah.

Yeah, yeah. So it turns out that these really funky, somewhat unnatural-looking and out-of-distribution images when baked into your training data sets give you a lot of robustness for free. there's a variety of reasons that this is the case well there's a variety of hypotheses that can explain why this is the case but basically the idea is you know a lot of these more dreamlike images are extremely geometric in nature

and they're geometric in a way that is very amenable to the primitive processing modules in our brain in human brains and there's some reason to think that these same primitive geometric structures in these dreamlike images would also be useful for computer-based vision systems. So that was the original motivation for going down this route of dreamlike pictures. And if it works empirically, there's no reason to not use it.

Yeah. So you spent some time you mentioned it like a computational neuroscience organization. back in the day. And I wonder, like, dream-like pictures that, you know, these geometric shapes that we tend to see in dream or hallucinate in or what have you. Are they, are they like extractable from like an fMRI? And then could you use that data to kind of carry things over and see how things map? Yeah. That is, yeah. So there's certainly a lot of loose, I mean, basically this type of work in the neuroscience world. A lot of people are trying to replicate this

in machines as well. And in some sense, it's actually much easier to do this for machines, or I guess in particular neural networks vis-à-vis humans. Right? You get this. With neural nets, even though they are extremely difficult to grok, you nonetheless

have a repeatable way of performing experiments, a repeatable, a consistent, and a very cheap way of doing experiments. Equivalently, you can take fMR scans of neural nets just by reading off the activations of each layer of the neural network. And probably this field of research is called mechanistic interpretability. We borrow quite a lot of these concepts in the work we do at Haze. And so this notion of, this notion that you mentioned of reading a brain or performing fMRI scan

to try and figure out what sort of female images and structures people are seeing is actually, one of the things that I did after that dreamlike pictures paper just as an experiment for myself and yeah even in models that were not trained with these dreamlike images there were certainly emergent

structures that looked very similar to dreamlike pictures right it just does make sense that that would be the case emergent it's all very emergent I love it he said the E word So Leonard, the show, every show, what are we about a hundred of these, Mark or something.

And if you see like two poles within the show, you know, there's the emergent pole on one side and the hierarchical pull on the other. And it just kind of these systems oscillate between like the needs of establishing hierarchies to scale and to have this emergent property below. What are you seeing when I say those two words with AI development? How do you see them like interweaving and oscillating or do you? Yeah, that's an interesting question. I mean, I guess probably it's just like, are you going top down or are you going bottom up? It's an interesting question. I mean, this has been like a prenatal debate in AI for decades at this point, right?

Do you want to be symbolic, i.e. like hierarchical in some sense. So this goes all the way back to the Dartmouth summer, as you mentioned earlier in the show. Or do you want to be purely emergent or be connectionist and train neural nets, which is the area that we currently find ourselves in? I honestly think there's probably going to be some mixture of the two, the mixed lot of sense. Neurosymbolic AI is broadly studying the mixture of these two systems, or these two ways of thinking,

to get more controllable and reliable systems. I think technical details aside, It's probably interesting to consider how humans think and behave and trying to acquire new concepts. So whenever I want to learn about something new or pursue a new task or new project, I almost never trying to get a fully formal view of the task, our priori. But let's say I'm trying to learn a new programming language or something. I am never going to the programming language handbook or manual. And just like learning about all the classes and concepts and functions and syntax and then trying to implement something.

It's generally much more outcome driven. So like I want to sit down and like make something. So I'll try and like code up a little cute function or cute script for some particular task. And through that process, I acquire a lot more ideas about, okay, here you are. how the primitives work in this language. Here's how the syntax ties together, et cetera. And that is a very emergent, quote unquote, or bottom-up way of thinking and learning.

Of course, at some points, you get enough intuition from this bottom-up approach that you're like, okay, let me try and unify the things that I've learned, and therefore you're moving more towards a top-down way of viewing the field, right? Same is true if I go out and try and learn a new research concept or figure out what's in the... figure out how people are thinking about a particular problem.

I'm never like reading from a textbook, right? I'm never starting from the textbook. I just start from what people are doing in the field, read the papers, read the experiments, try and implement something in code. And then I get some intuition this way, again, the bottom up way. And then I, once I have enough critical intuition,

then I try and sit back and formalize what I've learned. Excellent. Is that how you learn, Jeremy? Yeah, you've got to give in line. You've been play around, right? You got to play around with stuff. And then that's where things kind of come together as opposed to just being told what the thing is. You know, if you play with it, you get to put your stamp on it.

You get to connect it with other things. It's the whole information, knowledge, wisdom thing. Yeah. I'm aware of time. So maybe I've got two big questions before I hand over to Jamie for our last week's question. And maybe we don't want to focus on both them. maybe choose one. One of them is about AI, IP, copyright, watermarking. The other is about, which I've heard recently, spatial AI, kind of like using training AI within 3D space

so that one day we can have these robots which are powered by AI that can move around in space. And I've heard that conversation spoken about a bit recently. Which of those do you fancy taking on, Leonard. Yeah. Both are interesting topics. Both are interesting topics. Let's go with the former. Okay, watermarking.

Well, that's Jeremy. That's your ballpark at IP and content creation, musical creation, writing, poetry. Yeah. How do we? I don't know. What do we do? How do we make sure that in the, now that artists are given the credit they deserve? yeah this is a super interesting problem

this is also a very personal question in the sense of this is just research I worked on so if you were looking at my scholar page you probably saw one of the papers on watermarking or detecting the presence of watermarks it's a very unsolved and tricky problem there's obviously a lot of people trying to work on deep fake detection

or basically just detecting what sort of content is machine generated if it's not, there's almost an impossibility theorem in this particular field around, you cannot really use classifier-based methods to discriminate between things that are human-like versus, sorry, things that are human-generated versus machine-generated, especially if you're relying on the statistical properties of the final image, or the final object that you want to discriminate between. there's actually a lot of great work on this

from one particular lab at UMD University of Maryland led by Tom Goldstein in particular there's this guy John Kirchenbauer who spent a lot of time thinking about this I think we'll need to get to a point where images there's

any sort of object that's created by a genieI process needs to be identifiable by design and this design needs to be just agreed upon beforehand in a very like obvious way obvious non-deteckable way by the end consumer and the producer so probably this is watermarking but watermarking in a much more I guess narrow sense I think you might actually need like unique watermarks for each particular use case or even each particular user for it to be achievable and

and not gamifiable not detectable actually is what I what I would say given that I worked on watermark detection More broadly, this is just a super-historic problem. This is a security problem very squarely. But it is also a information theory problem. And, yeah, I mean, verdict is very much still out how to solve this thing. But it is a lot of fun, and there's lots of people thinking about it. Good.

Yeah, it's interesting to think about, like, there's not all, there's rarely one person that comes up with something kind of net new, right? Like, if I'm in my studio, I'm playing a little. guitar lick and I came up with something that I think is, wow, that's pretty unique. You could probably point to like five or six different new musicians or songs that kind of influenced that piece of the puzzle. So then are we getting into this derivative watermarking and like, oh, there's a little bit of mark in here. There's a little bit of Leonard in here. And it's mind-blowing to think about one question I always have related to this is not in how we structure

classifying something as partially created or created by generative AI, what do we do as creators? Is it possible? And I know this isn't maybe your technological realm, but again, a big what-if question. Could I disassemble content as someone who maybe have taken my likeness and deep faked or something? Is it possible to send a disassemble signal to that content where it lives? Like, is that even a easy, is that even a thing? Dissassemble in the sense of, you're trying to tease out if somebody used your particular characteristics. No, like disassemble, like, say it's a video, a piece of video content, turn it into static.

Oh, destroy. Destroy the. Yeah. And you as a, you as somebody that's been, you know, lifted, scooped, you know, you as somebody who has been wrongfully scoops, you know, you as somebody who has been wrongfully scoops, you want to, like, destroy. that piece of content? Is that what you're asking? Yeah. Yeah. Say someone, yeah, say Mark writes a song and someone calls it theirs

and it's up on Spotify and there are videos up and he's like, nope, that's my tune. Disassemble. Can you imagine if you had that in the days of Delosol? We'd have no good rap music. It'd be horrible. Oh my gosh. Yeah. This is an interesting question, for sure. whatever streaming service or publisher of this content

would need to be okay with you doing this. But it's like, it's technically doable, is what I will say. Imagine, yeah. I mean, it's all about collaboration, right? There has to be a collaborative approach as we move with AI into the future in what's happening. And people do have to get together

and there's got to be some kind of standardization in certain regards. But man, the work you're doing is fascinating. It's got my mind kind of, thinking in a very different way than it was before we started our conversation. I love the idea of, you know, AI, what does you call it, AI guiding principles or AI? Code of conduct. I think that's fascinating to kind of land it on a page for someone that doesn't know about

this in a narrative. And then I can give you that narrative and you can help me build something that's, that's, that's what it needs to be that serves what I'm trying to do well and that is safe for the participants and I think it's awesome, man. Yeah, for sure. I think it is also what the field really needs, both from a responsibility perspective, but also just from a usability perspective, right? Like, people are struggling to adopt AI today much, much more than you would expect outside looking in. People are struggling to adopt AI because it doesn't really do what you expect is to do. And there's just too much risk at the stand today for sending out in the wild and getting some sort of nasty

failure mode or sort of security safety vulnerability. Yeah, and let's point back to your, I loved your reference of AI as the brilliant intern, the wizard that needs some guidance, that needs a little help in that realm. That's amazing. It's also very interesting on the age of that. So when he said the intern, I was thinking, okay, so it's very early. What happens when the intern becomes senior management or the CEO and takes, over the company, then, like, when an intern becomes the wizard, that's how long that

journey is going to be. It's a very interesting question. And talking of that, Leonard, just one final question. I'm sure you won't mind me saying, I think you're 23. Is that correct? 24. So whenever I, not whenever, but a lot of the time I see a lot of old men, really, on the telly, on the radio, on the podcast, speaking about AI.

I was wondering, as someone who's at the beginning of their career, if you had a message, the biggest TV show on Netflix had Bill Gates speaking about AI. Do people really care anymore what these dinosaurs of tech have to say? What would your message be for the older stalwart of AI? Well, I mean, I respect them a ton. I think they bring corner perspectives to the field. part of however what makes the world an interesting place and certainly AI in an interesting place is that it is not as pedagogical as you might expect and there's brilliant ideas that come from all sorts of

people agnostic of age and background certainly i would like to think i am an example of this but you know there's was plenty of very brilliant ideas that came from people who are in their 20s and the 30s and I would say you know you should probably listen to the people that are building and playing with the technology the most as the more vetted source of truth I think it's really hard to get a concrete understanding and feel for things as you're actually using them

or building them nonetheless I'm sure there's a lot of older folks who are more actively engaged with this and they do bring interesting perspectives for sure if you want yeah And it all goes back to trust. We talked about trust before, and there's a whole generation. Every show we do is about trust.

Every technology is, blockchain's about trust, quantum is about trust, AI is about trust, robotics is about trust. It's nanotech. It's all about trust. It's a heck of a theme. And you know, you have generations older than Mark and I that have built their trust shortcut related to technologies to certain people. And I think the whole shortcut, decision fatigue, Charlie Mung, we.

We can do a whole episode on that and I've been writing a lot about it. But this has been a fantastic discussion. Leonard, your insights were amazing, very well spoken on all of these topics. I can tell you have a curious mind. I'm glad that you got to spend some time with us and I'm glad that we got to spend some time with you. We have a carryover question. So we want to string these together as we mentioned. What would you leave as a question for our next guest?

No rules can be about any topic, anything at all. Sweet. Yeah. Well, first of all, thanks guys for having me here. This was an absolute blast. And certainly, I love the work that you guys are doing. Next question for the guests in the next episode. So I've been thinking a lot about how AI may nerve

are critical thinking abilities, especially as there's going to be a generation of kids who are basically brought up in a Gen. A.I. Native world. And kids by nature are somewhat lazy, creatures. So, you know, I think the concrete question is, Gen. AI definitely has the ability to take the heavy lifting off of humans' critical thought process. And I think we'll need to think about how we have to restructure our education system to encourage critical thinking as a

necessity in whatever assignments or exercise will be prescribed to students. And I think this is very much an open question about how you would do this, right? Like, yeah, there is conceivably a world in which AI gets good enough to do with convincing quality all the math problems, for example, that you would need from kindergarten through like 12th grade calculus or something like this. And so I think structuring our education system around a way that forces students to think critically and also be able to measure this in a in a concrete non-arbitory way is going to be a big challenge moving forward. Man, we can start a think tank on that.

You must have, you must have watched Mark's unbelievably popular rant on Apple Intelligence. It was brilliant. But it's so, that that's a question that I think about every day. And I think it'd be a great one to carry over. Thanks for joining us. We've got book club coming up Friday, Mark, a live book club. We're doing Book Club Live. How about that?

Tell them what's up. Yeah, tomorrow at 5 o'clock my time. So what's that time in America, Jamie? 11. 11 Eastern. We're going to be reading Chapter 1 of Nexus by Yuval Noah Harari live on YouTube, LinkedIn. If you want to read that with us, if you've read it, come and cast your aspersions and tell us your thoughts about that. We're going to be reading every week, every Friday.

Join us for that. Otherwise, if you have any questions, if you're interested in anything, we're doing anything about it, please get in tour. which because we want to have more awesome guests like Leonard, so we need your help for that. Be sure to thank our sponsor, Mark. Don't forget our sponsor. Do you remember who our sponsor was this week? Jeremy, our sponsor this week was conviction.

Conviction. I was going to say constraints, but conviction. That could be another one. Thanks for joining us. Be curious. Stay disruptive. Keep thinking on paper. Awesome.

Bye-bye. Thanks, guys.