Latent Space: The AI Engineer Podcast - ICLR 2024 — Best Papers & Talks (Benchmarks, Reasoning & Agents) — ft. Graham Neubig, Aman Sanger, Moritz Hardt)

Starting point is 00:00:05 Welcome to the Latent Space Podcast, ICLR edition, Part 2. This is Charlie, your AI co-host. We're back with our coverage of the 12th International Conference on Learning Representations in Vienna, Austria. Many of you absolutely loved our Europe's coverage last year, and we're proud to bring you part two of our special two-part episode covering our attempt at giving you an audio experience of ICLR. If you'd like to see us return to Vienna for ICML, let us know by sharing this episode on X and LinkedIn. In part one, we covered the best papers of ICLR across four

Starting point is 00:00:42 sections, image generation and diffusion, computer vision and weak supervision, improving attention algorithms and state space models. In today's episode, we cover the wealth of papers we found around the related problems of LLM reasoning and agents, also in four sections. In section A, we do a regular latent space chat with Graham Newbig of Web Arena and Open Devon, introducing many of the major themes of the rest of this episode. In Section B, we survey a few prominent issues in benchmarking, from SUE bench, test set contamination, and general intelligence. In Section C, we look at agent building blocks from rag and self-reflection,

Starting point is 00:01:26 verification, safety and frameworks. In Section D, we finally look at two. proposed agent systems from Google DeepMind's web agent and Mededge Pete. This is the second of two episodes covering ICLR, which is overwhelmingly academia-focused. If you're interested in production AI engineering and industry, you should join us at the first AI Engineer World's Fair this June, where we have now announced many of our speakers from all the big clouds, all the large model labs, now including Anthropic, Cohere and Cartesia, brand new state space model startup, all top AI enabled developer tools and codegen

Starting point is 00:02:05 agents now including Quinn Slack, CEO of Sourcegraph, insights on the GPU and inference market like last week's guest gradient AI, and now featuring Dylan Patel of the semi-analysis GPU poor blog and the rest of the emerging LLMOS stack of startups and open source tools across RAG, multi-modality, LLM ops and agent frameworks, disrupts, disrupt disruptive startups like mid-journey, perplexity and character AI, and for the first time, a closed door track for VPs of AI and technical leaders to discuss AI strategy and leadership. Get your tickets now and see you in San Francisco from June 25th to 27th. We'll start this episode with a special on-site interview we did at ICLR with Professor Graham

Starting point is 00:02:54 Newbig of the Language Technologies Institute of Carnegie Mellon University. Graham has taught the CMU NLP course for the last seven years, but is also an active participant in the open source AI software ecosystem, having been personally involved in Open Devon, which recently scored a notable 21% unassisted resolve rate on SWE Bench Light. As an extra special treat, we are proud to invite Amman Sanger, co-founder of Cursor AI, back as our first ever guest co-host to add his personal takes on the state of code editing and agents.

Starting point is 00:03:30 This is going to be a doozy of an episode, so we better get started. Watch out and take care. So welcome to the pod, Graham, and welcome to the pod. Amon, our first ever guest co-host. Thank you for having you. Yeah, thanks for having you. Yeah, thanks for taking some time during Iclear. This is very impromptu, but two of you wanted to chat, I was like, let's just record a chat,

Starting point is 00:03:57 and that can be fun. And also, one of my goals here at conferences like this is just cover, post. for people who are at home, like not at a conference like this, just to get a sensitive mood. So I'll cover a little bit of your background, and then you can sort of fill in the blanks. So you're a professor at CFU, teach NLP. You also spent some years in Japan as a language teacher

Starting point is 00:04:17 as well as a grad student. Yeah, it was a good experience. It was very impromptu me going there just because I wanted to learn a new language but I ended up staying there for 11 years. Yeah. Never know how life leads you, I guess. Yeah.

Starting point is 00:04:30 Now you run your own lab and you teach the events NLP course. I'm sure that's been a wild ride over the past seven years. Was it? It was 2016. Yeah. Yep. The last time I taught it before this semester was before ChatGPT, so I had to go in and rip out everything and put all the new stuff in. But yeah, it's a good opportunity to keep up with all the new stuff too because I feel like I need to pressure myself into like giving a good experience, so I need to, you know, cover all the areas.

Starting point is 00:04:57 What does, what do you mean good experience? Like, what is a good experience for students? I think a good experience for students is them knowing stuff that's actually practically applicable and whatever they go on to do next. And I'm teaching the advanced NLP course, which is for people who are more on the like research and new innovation side as opposed to just, you know, pulling in existing technologies. So because of that, I feel like I need to stay on like the cutting edge of the most important things that people should know for the people who are pushing the boundaries to do the next thing.

Starting point is 00:05:26 For the NLP course, how deep into the kind of older school techniques and fundamentals do you Yeah, so this is a constant battle because there are some older stuff that's like interesting, you know, algorithmically interesting, but we don't use as much now. It might come back later or it might not. But I think right now, given the limited amount of time I have and the number of things that would be useful to know, I mostly focus on the parts that are like in the modern stack of how you build models. So what do you like to keep it? Is there anything that you do kind of keep in that's not really used much today? Yeah, it's a good question. I I still teach Ngram language models, which are language models that are based on counting up the number of words that follow other words and stuff.

Starting point is 00:06:08 And we don't use them at all today, but I keep them in there at the very beginning. So people have an idea of how you could calculate this without throwing it into a neural network black box. And what's the difficulty in why neural networks are important? But that's about it, I think. Interestingly, Ngrams may be making a bit of a comeback for speculative decoding. Yeah, that's true. Yeah. Can you elaborate?

Starting point is 00:06:31 Oh, just one other way of kind of suspective decoding is when you have some kind of draft model predicting forward tokens. So you can kind of batch at the sequence level tokens when you're doing generation, just because if you're generating one token at a time, it's much slower than kind of pre-filling a bunch of tokens at a time. So in this case, you would use kind of Ngram models to if, for example, you had V, maybe you know, D is often followed by CAT, and you wouldn't even need a decoder model or a, sorry, draft model. model, which is a smaller model that would be kind of generating tokens. And you could just use kind of and gram statistics or like basic adgram models for this. I was thinking also that by pairing encoding is kind of engrammy. You sort of build it up from... At least by Grims.

Starting point is 00:07:14 Yeah. Well, we could cover more later on, but, you know, I just wanted to touch briefly on the posters that you presented. I didn't know you had three, actually. I only prepped on two, but Amon can fill in the blanks on the one that you're presenting today. So Web Arena, Sotopia, and then Performance Improving Code Edits. Yep. Anything else that I missed?

Starting point is 00:07:30 Those are the three. Those are the three. Okay, so Web Arena and Slotopia are basically like two kinds of sandboxes is what I was thinking about. And maybe like which one do you want to tackle first? Like how do you tell the story about how this work is done at your lab? I think I can do Web Arena first. Yeah. So to give a little bit of context, I never really was an evaluation person or never really cared about evaluation that much for a long time until maybe 2022 or so.

Starting point is 00:07:57 And so I was mostly working on system building stuff. But once we started getting into the really good GPT models, my typical formula for system building was to figure out what's not working and fix it. And I got to 2022, and I was like, I don't know what's not working. These language models are so good. I don't want to be working on insignificant things that are already solved by the strongest models. And so because of that, I got into benchmarking. And kind of my goal for benchmarking is to push the boundaries of what is possible.

Starting point is 00:08:27 in a rigorous way. And kind of both of these Webberina and Sotopia are aiming to do that. I could explain about Web Arena in more detail. Yeah, we'll put a slide of the poster up. It's a pretty useful slide. I like the phrase, we create a mini-internet for your agent to play a master.

Starting point is 00:08:46 Yep. So I can't take credit for that. That was probably Xiu Yan, who, the first author, who came up with that. But the basic idea is, like, I'm very interested in how language model agents that act in the world can, like, actually work. So originally, what Shuiyan, the first author,

Starting point is 00:09:03 wanted to do is she wants a robot that helps her out with housework, so she doesn't have to do housework. The problem with robots is robotics isn't there yet. It's like, I feel like robotics is the bottleneck in language model plus robotics work at the moment. And so we tried to think of something that had some of the interesting problems for, like, kind of long horizon planning,

Starting point is 00:09:23 and also where you can pull in world knowledge. about how the world works and use that in an interesting way. And so eventually we settled on doing tasks on the web is a way to benchmark this. And we wanted a benchmark that was as realistic as possible, so if you do well on it, it actually kind of means something. So like a lot of people complain about how academic benchmarks don't mean anything, and we wanted to create an academic benchmark that does, you know,

Starting point is 00:09:49 actually mean something. And we wanted it to be evaluated in a way where you're not evaluating how good you are you are compared to mimicking how humans do it, but actually how good you are at solving the task. So we set up a sandbox internet by taking production grade open source sites. Yeah, this is Reddit, GitLab, CMS. Yeah, so the sites, we tried to mimic some like existing sites. So like Amazon is an existing site and we mimicked it with an open source counterpart,

Starting point is 00:10:20 one stop shop. Yeah. Reddit is an existing site. We mimicked it with open source. a site called Post Mill and GitHub is an existing site that we mimicked with GitLab. And all of these are like, you know, open source. They're actually used by people to do real things.

Starting point is 00:10:34 So they're pretty close to being realistic. Then we created a whole bunch of tasks that you would want to do over the sites. And the way we did this is we looked through our own Chrome browsing history. So we were like, these are some of the things that we like as the authors did over the past month. And then we tried to generalize them into things

Starting point is 00:10:50 that would work on the sites. And then we wrote some validators to check whether it like actually, So to give an example, one of the tasks is, like, tell me how much I spent on food in the past month, which, like, I had personally done because I wanted to, you know, budget and make sure I wasn't spending too much or things like this. So in order to do that, the agent has to go to the shopping site, has to identify all the food purchases. It has to figure out which ones happened in the past month and then add things together.

Starting point is 00:11:15 And language models aren't particularly good at navigation. They aren't particularly good at filtering and they're not particularly good at math. So despite the fact that this is, like, pretty trivial for. humans. It's hard for language models to do now. Yeah. And you had some stats here where basically all the models were under 15% and humans were 78%. Yeah. I like that kind of wide variance for a benchmark because it means buys us maybe a year before language models catch up. Yeah. So in the past six months, this came out about six months ago. In the past six months, we've increased by, from like 14 to 25 or 30% in the state of the art. Using GPD4? Yeah, using GPD4 as a base

Starting point is 00:11:54 model, it's based on improvements. I mean, before Turbo has... No, no. So actually, LOLM improvements are not the main driver behind this. It's more about the way that old agent uses LOM. And to give some examples of some things that people have done, the first thing is just like optimizing the prompts in the action space. So it's, you know, prompt engineering, figuring out which actions you can do,

Starting point is 00:12:16 like good ways to scroll up and down the page and like click on buttons and stuff like this. Other things that are maybe somewhat more method phenomenologically interesting are it takes a step and does kind of self-refinement or self-reflection about whether that step was a good step or not and then rolls back if it was a bad step. And there's another one that tries to create like a textbook about how the sites work and give information about how the sites work. Like if you want to do this, you should go to this page. So it's creating like world.

Starting point is 00:12:47 Yeah. Yeah. Creating world knowledge about or documentation might be a better word. Yeah. About how the sites work. and then feeding that into the agent based on where the agent is and like the tasks. So there's a lot of creative things that people have done with this, but we're still at 25 to 30% as opposed to the 80% where we're doing.

Starting point is 00:13:05 Well, one thing that's surprising to me is it's only 70% for humans. Yeah. So think about half of this gap is just human negligence for doing the task. Like they don't follow the instructions exactly. About half of this actually might just be an issue with the benchmarks validators being too strict. And so the human gets the correct answer, but the validators are, like, not counting it as the correct answer. So they're looking for, like, an exact match. And it's an exact match, but it's off by a rounding error or something like that.

Starting point is 00:13:36 And so we could go back and fix all of the validators, and that might bump up the human performance and bottle performance. But still, I think the gap would be the same. I'm surprised that long context hasn't helped a decent bit for this. Like you mentioned, it was mainly not on the model side. But at least, I'd assume for like coding agents, longer context is helpful. I'm not sure what it looks like here. Yeah. For longer context, we haven't seen any papers that say adding a long context helps.

Starting point is 00:14:04 It's also kind of not clear what exactly more context you would put in because we're putting in the entire past action sequence. Oh, and that fits entirely. It's the entire past action sequence, but we're not putting in the past pages. So if we added the past pages, it might give us a little bit more information, but it might also explode the content. set the models looking at. So yeah, maybe somebody will figure out how to make it work, but I haven't seen it yet. And so what's the follow-on work from Web Arena? I think web browsing is super interesting. I think web browsing, the tasks that we created here were based on our personal browsing history, and they were also created to be kind of like manageable based on what we were able to do at the time. And since then,

Starting point is 00:14:44 we've released it, and we have a lot of new ideas about coming up with things that are more representative of actual tasks that people do in jobs. And so we're working on something related to this, you know, like what are some of the tasks that software engineers do and things like this. And so in order to do that, we need to, you know, have web browsing. We need to have it over realistic sites

Starting point is 00:15:07 that people use in, like, actual workplace scenarios. So we're working a bit on that. And then separately, now that we have a benchmark, as I mentioned, I'm not like an evaluation person. Like, that's not my core passion. My core passion is building things at work. And so now we have a good evaluation. that I care about, so we're doing lots of things to improve it.

Starting point is 00:15:23 Some of the things we're thinking about are training models based on synthetic data or reinforcement learning methods. Also better ways to interface with websites, so maybe we should be using an API instead of directly interacting with the website or things like this. Directly interacting meaning point and click. Yeah, exactly. Using coordinates. Yeah.

Starting point is 00:15:42 So I was going to say this is for listeners on the pod, like very, very relevant to our adept episode where they have a very strong opinion that you should just rely on point and click instead of using APIs. I even had a section of my right of calling setting like no APIs as a rule. Is that a surprise? Yeah, I mean, what was there? I'm curious if there are even why.

Starting point is 00:16:00 It's very simple. The sheer number of things that don't have APIs vastly overwhelms the number of things that do have APIs. So if you're only going to constrain yourself to API at work, you're not going to be generally capable as a human. Yeah, yeah. So I listened to that episode. It's a great episode, by the way.

Starting point is 00:16:17 I largely agree with that. And actually on Web Arena only one of the four websites that we cover has APIs at SCETLab, and the other three do not. But I think we can be creative about that. You know, maybe there's ways to create APIs for sites that don't have them, or figure out ways to build tools that allow you to say, OK, navigate.

Starting point is 00:16:37 It's kind of like the documentation idea that I said, like, how do I navigate to the purchases page? And if you just tell it what the purchases page is, then you can create a navigate to purchase this page function or something like that. So we're not. finished with this yet, but we're thinking about things in that general direction. I'm curious, what do you think is missing from the base models themselves, and like,

Starting point is 00:16:58 how do you think the benchmark would inform what the model providers do to make better ones? Yeah, so this is a great question. So one thing about Web Arena is that we're taking in a textual format. I didn't actually mention this yet, so I can explain it briefly, but we're looking at websites by using something called the accessibility tree, which was created for screen readers for people with vision impairments. And I don't think that's necessarily the best way to see a website. I think multimodal is probably the way to go eventually because most people see sites by looking at them.

Starting point is 00:17:31 I feel like all open source models and even most of the closed source models are not terribly good at understanding websites visually or through accessibility tree formats. So they make silly mistakes about, like, for example, not realizing that they can click on a drop-down menu in the accessibility tree. format or not being able to ground to the site when you're looking at it visually.

Starting point is 00:17:52 We have another version called Visual Web Arena that follows up and does this visually. Another thing is planning is a really big problem. So I'm very hopeful that the foundation model providers are working on this very hard now in training models that are better as agents. But for example, on Web Arena, it will very often step into a page and then be on the wrong page and just not realize that it needs to go back. So those are some other problems. It also has some really interesting failures of common sense.

Starting point is 00:18:20 So my favorite failure is we have a thing in Web Arena that says assign this issue to myself. And it assigns the issue to the username myself instead of like your own username. So there's lots of like little common sense things there too. Maybe tokenize identities and then have a special token for self versus a username to myself. Although I kind of hope that my language model would be smart enough to figure out out too. Like a human would be. I don't trust it. Fair enough. Okay, cool. We should probably move to Sotopia. So going from simulating the internet

Starting point is 00:18:52 towards simulating multiple agents talking to each other in a social environment. Yep. What's the motivation there? What's the story there? Yeah. So the motivation there is a lot of people are starting to use language models in kind of like social situations or at least like socially charged situations. Whether they're ready for this yet or not, I'm not sure about. And that kind of motivates this work. For example, should we have a language model that negotiates with somebody about like a price of a... There's no should we, we are going to have them. Yeah, we are going to have them. Are they ready to do this in a way that is not harmful, basically? And also, are they good at it? You know, are they able to cooperate with somebody and find a solution to a shared problem? So,

Starting point is 00:19:43 what we did is we came up with like six different types of these like socially relevant situations. Yeah, negotiation, exchange, competition, collaboration, accommodation, and persuasion. Yeah, exactly. And with each of these, we kind of like semi-automatically created a bunch of tasks. So the example task that I have on the poster is we have two people who are out in the cold. One of them has a blanket and the other doesn't have a blanket. And it's a guy and a girl. Very important. Yeah. In person, person one wants to keep their blanket to themselves, and person two wants to have the blanket shared. And so they need to negotiate with each other. Each of the agents is given a persona. They have like a personality and some side information about them. They also have a secret that they don't want to reveal to other people. And then based on this, they have a conversation. And then we do evaluation along a number of axes, like whether they achieve their goal, whether it's a believable representation of a social situation, whether they violated social rules, whether they broke secrets and other stuff like this.

Starting point is 00:20:43 And based on this, we had language models talk to language models and evaluate whether they do a good job of this. And we also had language models talk to humans and see whether they did a good job. And language models eval. Yeah, that's what's math. How do you do the evaluations? Yeah.

Starting point is 00:20:58 So we did human evaluation and we did language model-based evaluation. And we tried to measure the overlap between the human evaluation and the language model-based evaluation. And not overlap, correlation. So how well do they agree with each other? And the answer is language models are okay at navigating these situations. They're also okay at evaluating these situations,

Starting point is 00:21:20 but they're not perfect at either of these yet. 74%, yeah, 74%, which is usually around the rough correlation of L.LLM's judge for most things. Yeah, exactly. And one of the interesting things we also found was, like if we have GPD4 talk with GPD4, it got a score of about 3.3 out of 7

Starting point is 00:21:38 on our kind of composite scale. If we had GPT4 talk with humans, it got something closer to 4.8, I think. So when GPT4 is talking to humans, it's actually more successful at achieving its goals and being believable and stuff, because the humans are a better conversational partner and can recover from the airs and stuff. Oh, cool. But when humans talk to humans, it's 6.15. So we're, you know. What is the human to GPT4?

Starting point is 00:22:02 Like, human to GPT4 is high. It's nearly identical to human and human, like 5.9.5. Yeah. Your number should be bigger in your chart. Yeah, yeah. I will convey this to the students who made the poster. Yeah, I agree. But, yeah, so I think this is pretty interesting.

Starting point is 00:22:20 This was our first work in this general direction. Again, you know, the I Clear submission deadline was about six months ago. So, like, a lot of these papers came out six months. But also some of the stuff we've done in the follow-up now is we are trying to get better evaluation, train better evaluators that are good at evaluating this kind of, like, social skills, and we're hoping to beat GPD4 with respect to that. But more interestingly, we also tried to train models to be better at navigating social situations. So we took a Mistral 7B model, and we trained the model through both behavior cloning. So we ran a bunch of

Starting point is 00:22:54 conversations through GPD4 and trained Mistral to mimic them, but also through self-reinforcement, where we basically had the Mistral model do a bunch of conversations, have GPD4 grade them, and then pick only the good ones and train on that. And in doing that, we actually were able to max out our evaluation so that the train-to-bottle matched GP4 according to machine judgment, according to model-graded judgment. But when we actually did human evaluation on it, it was still far behind. So what this demonstrates is we can over-optimized two model judgments

Starting point is 00:23:32 and actually get very close to kind of our teacher model with respect to model judgments, but still humans fall behind. So it demonstrates that we actually have, you know, significant problems in evaluating these models as well. So I think we need to alternate between getting better evaluators, getting better models. Is the evaluation done right now via kind of prompting a GPD-4?

Starting point is 00:23:53 Yeah, at the moment, that's the best thing we have, but we're trying to beat that currently as we speak. The last paper that you're presenting today is performance improving code edits. You did the prep on this one. Yeah, I mean, first, off, would you be able to kind of roughly explain the paper? Yeah. So the basic idea is we have lots of programs, of course, written by humans or written by

Starting point is 00:24:16 machines, and correctness is a major concern, but efficiency is also a major concern. And so we ask the core problem, can we use, like, large language models to improve the efficiency in programs? And the overall concept of the paper is pretty simple, but the execution is maybe interesting and so in order to do the execution we basically took competitive programming problems that had lots of different solutions and especially we focused on ones where the first one timed out and was too slow and then a revised version didn't time out and was fast enough to complete and the reason why this is interesting is because we know the implementation is

Starting point is 00:24:53 going to be pretty similar because you know people working on competitive programming so want to do it quickly so they don't want to like rewrite their whole implementation when they go from that but you know we We know one is slower and one is faster. And so then we basically ask language models to try to do a similar optimization and make it faster. So we created this data set. We also created an evaluation harness that makes it possible to measure things fairly because

Starting point is 00:25:21 if you just run a program and see how long it takes, what if the system is busy or other things like that? There's a bunch of mitigating factors, so we fix that through virtualized CPUs. And then we also have some like better prompting methods and fine-tuning methods to try to create models that do better on this. And in the end, we actually got quote-unquote superhuman performance on this task and we came up with models that on average made things faster. The caveat for superhuman performance is the people doing these programming contests only

Starting point is 00:25:51 need it fast enough to beat the timer. So they're not trying like extremely, they're not trying extremely hard to optimize. So I won't say that, you know, models have like beat humans at program optimization yet, but it's like maybe a first. Still, it'll save you from really bad performance. Yeah, exactly. Yeah. I mean, the surprising thing was it was only using, I believe, the code Llamas and 3.5.

Starting point is 00:26:11 I don't think it used for, if I remember correctly. Yeah. This work has had a long evolution. And I think part of the reason why we didn't use for was just it's pretty expensive to run experiments. But yeah, I think that's the main thing. Yeah. There are a few interesting things that I noticed.

Starting point is 00:26:31 The really interesting one that I liked was the performance conditional generation. So I feel like I've seen pieces of this in other work. For example, Alpha Code actually mentions doing this. Yeah, I'm just curious what the motivation for that was. Yeah, so the basic idea is what we do is we kind of like prefixed the sequence that we're generating with how good the performance is. So it's like this is a zero out of ten performance thing. This is a ten out of ten performance. And then we fine tune a model to learn like how.

Starting point is 00:27:01 how to generate slow implementations and fast implementations. And then at test time, we always append the fastest tag. This has been used in a number of places. Like another example is a method called Quark, which basically tried to do this to generate like not toxic text. So it would say this is toxic text, this is not toxic text. And then when you generate it, you always append the like not toxic text tag. So it's a pretty widely used technique and it just seemed appropriate here because it's also

Starting point is 00:27:28 very easy to use. You just append a tag. you evaluate, then you append a tag to the beginning. Yeah, and I saw you guys had used like 3.5 fine tuning. Like one limitation of kind of using opening eye fine tuning is you can just kind of do supervised fine tuning. You can't really, you can't do RL. But it feels like using this method,

Starting point is 00:27:46 you might get some of the benefits of learning from negative examples. You kind of would get what from RL. Yeah, exactly. Yeah. By prefixing tags. Yeah, right? So because you can have kind of a low scoring thing

Starting point is 00:27:58 prefix with a bad example following it. and then a high scoring thing followed by a good example. And ideally the model learns the difference with that data. In the Quirk paper, at least they show that this works better than just training on the high quality examples. So yeah, excellent. What's the appetite for performance improvement code edits at Cursor? Yeah, I mean, in Cursor.

Starting point is 00:28:20 I think like in practice, it's really tricky when looking at arbitrary code. There's the problem of actually isolating the actual performance of some piece of code that you care about. stuff you don't have like this nice sandbox environment, people are almost always running it on their laptops or some kind of remote SSH machine. Then like actually isolating that piece of code, it's possible, right?

Starting point is 00:28:40 You could kind of just add kind of timers around it. But we don't have anything like optimized for that. It's mainly like if the user wants to ask the model to improve it, they can. They can add in relevant information of how long it took. And it should work reasonably well, but definitely not like its own thing. Another thing is our paper requires having tests.

Starting point is 00:28:59 And in performance-improving code edits, a huge, huge bottleneck is having good test coverage, because if you don't have good test coverage, then it will generate a return statement and just return from the function with the correct answer to the test, which is very fast, but only works in one case. So I think for real-world code, that's a lot harder

Starting point is 00:29:22 because generating a very comprehensive test is hard if you have data structures and stuff like that. So it's definitely not trivial. Awesome. wanted to move on next to now that you're done with the spring semester, I guess officially after you present this poster, you were mentioning that you're going to spend a lot of time on Open Debit. So maybe, and obviously you're interested as well,

Starting point is 00:29:41 what were your reactions to Devin? And then maybe you would tell the, sorry, Open Devin first, but I wanted to start with Devin, you know, just both of you, whoever wants to. Yeah, I think it was really exciting. Like the demo was great. And I've been working on code generation for a long time. like, I think actually since 2014.

Starting point is 00:30:00 And, you know, the big moment where I felt it had made it first was co-pilot coming out. And it's like, yeah, this is great, you know. And then I kind of slightly lost interest in doing this because I felt the limits of code completion. And, like, CodePilot does pretty well. Sure, we can improve it a little bit. And then... You did. Yeah.

Starting point is 00:30:20 Well, so, you know, maybe Cursor was right about how much more you could improve it, you know, honestly. So I personally kind of lost interest a little bit, but then I was working on web agents, and then I saw the demo from Devin, and I'm like, oh, this is really cool. Like, I'm interested in agents. I've been interested in code generation for a long time. You know, this seems like a good sandbox to be working in. And I had known about SweetBench and the stuff that had come out. And I was like, yeah, let's do this.

Starting point is 00:30:48 Let's, you know, work on this problem because it's another benchmark like Web Arena, where it's like our scores right now are low, but there's a lot. a lot we can do to improve them, so it's kind of exciting there too. Yeah, no, I thought it was a really good demo, and I think it'll be, like, pretty useful for, like, a lot of the bottom, whatever percent of PRs. I guess, especially, like, because I'm working on the stuff that I'm working on, I think I'm a little bit more bearish than other people on, like, agents working kind of immediately or, like, somewhat soon. I think there are a lot of really hard problems, and human judgment is pretty paramount. Like, I think on the margin,

Starting point is 00:31:21 we are going to shift things a little bit more kind of agentic and eugenic direction, and more things happening in the background. But I suspect the human will be needed for a while rather than kind of going from issue to pull requests. Which, I mean, I think that's kind of sui agent's direction. But I think one of the things that Devin, I think, nailed was the async interaction with the, it could be executing its plan,

Starting point is 00:31:43 but you could sort of intervene while it's doing it. And that felt very much more like, I guess, you know, level three or level four, self-driving rather than full level five. And, yeah, that seems to make more sense. Yeah, and I think like one of the interesting things, when we first came out with Open Devon, you know, Devon still was not open for everybody. I can tell a little bit about the story, but basically we saw the Devon demo come out,

Starting point is 00:32:06 and Jun Young, one of the people from the Quinn team building a language model at Alibaba, basically... I think he's the lead on Quine, right? Yeah, he's one of the leads for sure. And he basically said, yeah, this is really exciting. Let's make a project about this. He made a repo with a read-me, and the read-me got a thousand stars on GitHub. So then shortly after that, I think probably that evening or maybe the next evening, I was like, yeah, if we have something to hack on here, like the open source community is so excited about this, that we'll be able to do something interesting. So I basically came up with a really, I'm not a React developer, but I came up with a really, like, Jankey React.

Starting point is 00:32:43 I wouldn't say clone of the Devon interface, but, you know, something similar to the Devon interface. It was completely non-functional. It had no chat functionality. it kind of looked reasonable. And then after that, I pushed that, and it's like, yeah, let's make this actually work. And then a bunch of people came together and did that. So for the first four weeks or so, we didn't have anything, you know, that actually worked at all. But while we were doing that...

Starting point is 00:33:05 This is when I livestreamed and tried it down to this. I was like, I really respect you and felt a little bit bad that I subjected you to that experience. No, I didn't know. I was like, oh, this is done, right? Yeah, yeah. But, like, the interesting thing was then we had... a ton of people who are not developers coming to us also. Yeah.

Starting point is 00:33:23 And so I think I totally agree that for, you know, really big software engineering projects, like Open Devon, like we're trying to use Open Devon to solve issues on Open Devon, and it's a complex enough software project that it's actually pretty tricky to do. Like the model needs to figure out how to set up the software repository in the first place, and that's a pretty big lift. But I think the possibility of doing things like slightly smaller level, like setting up simple web apps and stuff like that for people who are not professional developers is another thing that even immediately these sorts of agents might be able to start making a dent in.

Starting point is 00:33:58 Yeah, that makes a cool sense to me. As someone who's like slightly more positive on Devin, yeah, I'm also, you know, I don't think it's, it's going to threaten our jobs anytime soon, but I'm pretty positive on it. Like I think it is very good for Greenfield and then like moderate for Brownfield and then obviously depending on the size of the job that you're asking it to do. Like, yeah, there are many bottom percentile PRs that I have to do anyway. Yeah. If I could just throw it to Devon, even if it takes like eight hours to do it, that's probably one hour that I don't have to spend thinking about it at all, which is cool. Yeah.

Starting point is 00:34:31 One thing I wonder about is the U.S. Because it does kind of shift into you're managing a bunch of, let's say, like, in your developers, right? Yeah. You're kind of doing code review then all day. How does it feel? It feels fine. Literally, it feels like I'm an engineering manager, I'm technical, and I can see what

Starting point is 00:34:47 that my coders are doing and check in on them. If they're going off base, I can just tell them they're going off base, and they'll replan. It's exactly what I do with engineers anyway. Yeah, that makes sense. And so in creating Open Devon, and you were talking about planning earlier, I feel like Devon's planning. Devin made a big fuss about their, like, breakthrough being,

Starting point is 00:35:06 or secret sauce being planning, but I really don't think it's secret sauce. Like it's just, they generated a plan and they try to execute it, and the plans change over time. Did you find that hard? That was the, you know, any insights? you may open Devin? Yeah. So at the moment, we started out with implementing planning,

Starting point is 00:35:23 and I do think it's important. So right now, our best agent, which is doing reasonably well on like the Sweebench benchmark, the same thing that Devin tried out on actually isn't really doing any explicit planning at all. So nonetheless, we're able to get 21% on the Sweet Bench Light version. We haven't run the full Sweet Bench version just because setting up Sweebench takes a while

Starting point is 00:35:43 and it costs $6,000 every time you run it with GPT4. So it's a little bit heavy to run evaluations on it. But I'll be very interested to see does their supposedly really good agent with planning stack up to something without planning, but just has like a good toolbox for, you know, searching code and for modifying code in a, you know, efficient way and stuff like that. Yeah, I think this is public, but they use Morp, which is Jesse Hans thing. I don't know if you know him. So they use a good code indexer or a searcher that pages things into context whenever you need. Yeah.

Starting point is 00:36:16 It seems like the magic trick. I don't know. We're thinking of trying out morph, or we're actually actively trying out morph as well. Right now, our code search is based on the code search that was used in SWI agent, which is another agent by the people who created SWEBunge. And it's literally like a find and gruff. Okay. It seems to work good, but it won't work as well as, you know, like an actual code search engine, semantic search. I don't know.

Starting point is 00:36:43 Have you talked about the search that you use? We use a mix of things. Like, the main meat of it is kind of retrieval with embedding, so a kind of standard approach there. But then we use, like, kind of re-rankers in the mix, occasionally use LSP information. I think there's, like, a much stricter requirement for the agent stuff for getting, like, exactly the right context that we don't have. So I'm not super familiar with what exactly Morph Labs has, but yeah.

Starting point is 00:37:07 He's his emphasis on speed and scale, but I don't really know how specific indexing or retrieving. Yeah. And this is not at all like a knock on SweetBench because I think it's a fantastic benchmark and it's a great way of kind of measuring progress. But I do wonder, I posted about this, like how much of the performance is also that the models kind of do know those code bases because they're all public code bases. Yeah. So you asserted that it's already leaked for the online models.

Starting point is 00:37:32 Like in some ways, like I kind of tested this with one of the first problems that I saw in Sway Agent and Claude Opus basically knew the correct file to edit just based on the PR, the name of the pull request. So I suspect it's like somewhere in the pre-training data. I don't know like how much of an effect that actually has. Like if you're like getting better and better at it, I still think that translates to better performance on private repos, but I don't think like an X percent

Starting point is 00:37:56 will also be on a public repo, will be like the same X percent on a private repo. Yeah. So this is a great point. I loved your tweet about that actually, and I retweeted it. But I can also explain a little bit of our like vision for open devins.

Starting point is 00:38:10 So it started out as basically a deviant clone, but I feel like we've moved a little bit beyond this because I think the open nature, we have a hub where people can add agents. We have a plugable thing where you can use any LM in it. So you can combine any LM with any agent. Is it light LLM or? It's light LLM. I'm a big light LLM fan. It makes everything very easy. And then the final thing is we also want to have plugable evolves. So right now we've only implemented like SwayBench, but there's a bunch of good code benchmarks. We're also planning to add web arena and through something called Browser Gym, which was created by Service Now that has these three web navigation benchmarks. Because in order

Starting point is 00:38:50 to be a good software engineer, you also need to be able to gather like information on the web and stuff like that. So we're going to add that. And I think there are ways to basically create benchmarks that are not leaked using the same method as Sweet Agent. And there was recently a paper out of Berkeley called R to E that converts repositories into. like evaluation environments for code generation agents. And we're talking with the people who created that to incorporate that into like our evaluation harnesses. Yeah, I'm very excited by that.

Starting point is 00:39:19 I talked to them too. I think at least one or two of them may have been also people behind LiveCodeBench. Yeah, yeah. Which I think is fantastic because it's a great way of seeing if, and for people who don't know, LiveCodeBenz is basically, I think it's like a bunch of lead code problems, and you can kind of slide the cutoff date forward and backwards

Starting point is 00:39:36 and see how different models perform. They go way down. down. It's really great. One thing that I'd be really interested in seeing is, like, I think they're, like, now good benchmarks for overall agents working well, but, like, good benchmarks for capturing, like, all the things that a model needs to do well to be a good agent. Because you kind of need to build, like, the good overall system, and maybe the system works really well for some models, better for some models than others. But I wonder if there's, like, a good kind of benchmark you can do that tests, like, independently each part that's needed to be a good agent. So this, this is a

Starting point is 00:40:09 This is a great question. And for web agents, we recently released something called Visual Webbench, with the idea being that it tests about eight different capabilities that we think a model should have. Like, is it able to do OCR on the page? Is it able to ground the web elements? Is it able to predict the effect of clicking on a button

Starting point is 00:40:32 or something like this? I think we currently lack something like that for coding agents. Some of the mistakes that we see our agents making are really silly. Like, it get clones a repo, and then it doesn't know to CD into the thing, so it tries to get cloned the repo again. And this is GPD4, so it's, you know, the most capable L-O-M model. And so I think there's a bunch of little things where it's like we could categorize these and just make sure it checks all those boxes, and it would just become more capable. But that

Starting point is 00:40:59 being said, you know, there will probably be other things that fails on. So if we overfit to that benchmark, that tests all the skills, and that would also be a problem. But I still think having one would be better than that. Yeah, like, there are a few general things that you're kind of surprised by, like, how poorly the model does. Like, one is kind of applying code edits. I had talked to some of the Swee Ben, Sway agent people, and, like, also just, like, looking at their demo, there's this great example of the model knows what, like, roughly knows

Starting point is 00:41:23 what to do. It has, like, a plan for it. Then it's trying to apply the edit, and, like, seven times it incorrectly indents it. It gets that feedback. It keeps doing it again and again and again. But, like, yeah, code edits feel like one big part kind of of the pie that aren't, like, super well-tasted at the moment. Yeah, anything else about the future of coding agents? Like, where do we go from here? I guess you already talked a little bit about the future of Open Devon.

Starting point is 00:41:44 I feel like we're, like, right at the beginning of a very rapid delta with respect to the performance of how well these are going to go. And I think now we have all the ingredients for academia and open research to iterate on this. We have a good benchmark, like Sweet, Sway bench, you know, maybe there are some issues with it, but I think it's fine. for now to iterate on. Open Devon, we've set up, you know, an environment where people can, you know, put in agents and very quickly iterate on them. So I think we're just going to see a bunch of people jump on this and improve rapidly. And then I think it's going to plateau a bit when we get to the really hard things where our like current language model backends are not going to be good

Starting point is 00:42:27 enough to handle them. We'll have GPT5 by then. Yeah, but like actually I think that's probably going to happen. And I think because all the open model creators know that GPT5 is going to to have that. All the open model creators are probably also thinking about it too. Yeah, like, you know, Lama 3, 400B. Yeah, exactly. Exactly. So I think that will give us another bump with respect to that. And we'll see how far that takes us. I don't think it will take us to resolving every GitHub issue automatically, but I think it'll be pretty exciting over the next, like by the end of 2024. Yeah. Again, I think it'll be like really useful for like some bottom percent of

Starting point is 00:43:01 GitHub issues and that'll go up. The thing that we would like to build is kind of agentic things that happen as you're coding. Like, the ideas we have in mind are, like, as you're coding, you can spawn off kind of pretty meaty units of work, right? Like, as an example, let's say you need to implement some random helper function or some utility function in order to, like, get some value out of it. Let's say the contents of some file in some particular way. You should just be able to kind of write that function out, file contents equals whatever,

Starting point is 00:43:30 and then in the background, that thing gets implemented for you. What we're going to see is we're going to see the ability of like, basically scaling inference time compute in some way, and this could be either smarter models, it could be kind of using models with chains and looping. And when you scale up inference time compute, you can't really use the level one kind of systems that are built in with cursor right now, which is humans supervising the outputs of these models, either with kind of next edit prediction slash autocomplete, streaming in kind of diffs or chat. And so something needs to happen in the background. But the goal of what you want to do is it happens in the background in a way that's

Starting point is 00:44:06 like very much preserving human flow and letting the human be completely in the driver's seat and kind of dictating exactly what happens. So it's kind of working completely in service of what you're building towards. I think this is the original Morph Vision and it'll be exciting to see when it happens. Do you think most of that will be running locally or you don't really have the difference? Locally meaning the models, inferring on models locally for cost reasons. I think it's going to be firing off a lot of work. It's going to have to happen with like the most capable models, meaning it'll happen.

Starting point is 00:44:35 Yeah. Not locally. Just a side note, I was just talking a lot about open models and all that. Do we have strong opinions about code-specific models being best for code, or do we think general models are just best anyway, are the best code models? So maybe to phrase it, there's no code GPD-4 that's just GPD-4, and GPD-4 is the most capable code model. Yeah, I mean, here's one question.

Starting point is 00:45:01 I'll kind of pose in response. It seems like people say that training in code improves performance and everything else. There was one paper about that here. Oh, really? It's like how much this code improve performance? And the guy didn't show up for his post a session, which is very annoying. Does it? Does it show that it?

Starting point is 00:45:21 I don't know. I feel like, I don't know, if you look at open papers, I think most of the time it's kind of showing if you've run out of data than adding in code will improve performance, right? which makes sense, right? On reasoning in particular is the thing that people kind of speculate. But it does feel like Open AI is the company that popularized this notion, and that was kind of like back in the day

Starting point is 00:45:42 when they unified the models. Because it used to be like codex was separate from GPD3, and then they kind of had this unified Code DaVinci 2. So I actually wrote the first paper on that. And when we- What was this paper called? If you remember? Large language models of code

Starting point is 00:45:59 are few shot common sense reasoners, I think. But when we wrote that paper, we didn't know what Code DaVinci 002 was and what Text DaVinci 002 was. And we thought Code DaVinci 002 was a fine tune on top of Text DaVinci 002, but it was actually the opposite. Text DaVinci 002 is a fine tune of Code DaVinci 0.02. But nonetheless, Code DaVinci 02 had better performance on some reasoning benchmarks that we measured. And the funny thing is actually text DaVinci 02 was trained on more data, But Code DaVinci 002 was still better at some structured reasoning stuff, which I would really like to prove, but we haven't been able to do it yet, is that code is more structured. And so it also has more repetition.

Starting point is 00:46:43 So you need to attend back to the previous context more when you're doing code. And because of that, it's better at capturing things that are very structured in the output. And that includes things like reasoning. So I'm a pretty strong believer that like there is something special about code, but there could also be something equally. special about text if you use the right variety of text, if you use text with lots of repetition or other stuff like that. So I don't think it's like code is magical. I think it's some properties of code are good for reasoning. Yeah. Like equal text. That would be one example. Like my suspicion is there's just like classes of text slash code that help for reasoning.

Starting point is 00:47:18 And like there's a bunch of not fantastic text for reasoning that will exist in pre-training datasets. Like maybe the very top you have like archive, right, or textbooks. And then like just under that you've code. So it ranks higher than maybe most text data that's used in pre-training models, but it's not like better than like actually the thing you want. Yeah, I agree. Awesome. I'm gonna broaden out to more general free-form topics.

Starting point is 00:47:40 Something that we prepped was just the changing nature of like, I guess, industry and academia. I don't know if you guys have opinions on that. I guess you're representatives of both. Yeah, I'd love to hear your thoughts. Yeah, so the changing role of academia is really interesting, because I lived through several areas where it was like academia is probably leading research with respect to, you know, deep learning and everything, which was maybe 2010 to 2013 or something like that. And then there was the like sequence, sequence models paper from Google, which was 2014, which was this at the time, like, huge four-layer LSDM that nobody could train.

Starting point is 00:48:27 And so then we were starting to feel the compute crunch, but there was. There were still lots of modeling innovation. I created a neural network toolkit called Dynet, which was kind of precursor to the pie torch. And a lot of stuff was happening there. And then after Burt, you know, it's like, oh, we're scaling up, we're moving beyond the compute that academia I can use to train these base models.

Starting point is 00:48:45 And then I think the really big thing was like the GPD models, right? And after chat GPT came out, we actually had an emergency workshop at CMU, which was a group therapy session to say, what should we do? What should we do? And I think for a short amount of time, A lot of people were worried, like, what could we be doing in the face of this?

Starting point is 00:49:04 And then I think a few things changed. I think number one, the evaluation stuff I talked about, it's like we realized that actually there's a lot of stuff that GPD 3 and chat GPD cannot do yet. And, you know, more complex reasoning, more multi-step stuff. And another thing is all the open models started coming out, which made it a lot easier for us to do fine-tuning. A lot of the open-source frameworks came out that made it easier to do these sorts of run, large models on hardware that we have access to. So I feel like Lama CPP.

Starting point is 00:49:34 No, I'm talking more about like the training stuff like deep speed, Wama factory, you know, axelotel. Tensartority. Yeah. Any of the things that we can use for training and that makes it so you have a machine that costs you know a hundred thousand dollars which is a lot of money but it's very much within you know an academic budget and you can actually do training runs do interesting things that free you up. And then at the same time right now, you know, every university is trying to build a GPU cluster. Yeah. Or get access to it, including us, including, you know, everybody else. I imagine CMU would be ahead because you already have so many other needs.

Starting point is 00:50:11 Yeah, so we do have a good cluster, but like the kind of hardware that you need for training large language models is kind of specific. You also need a system for allocating, you know, like, okay, this is the most important thing to be doing right now. We're going to give a lot of compute to that, which is not something that traditionally universities are very used to doing. They're used to being very chaotic with lots of ideas, but I think we need to focus on some... MIT had lists like really great big class, but it was V-100s. So two generations too old, soon three. There's these like national supercomputer labs that you can apply for grants for.

Starting point is 00:50:46 The funny thing is many of these don't have the hardware that we need. They have V-100s or they have A-10040GB things. They have A-180GB, but they only have four of them. It's like, it's of interesting how little there is available. Well, you can talk to Luther, which has its share of grants. Luther is basically like a compute grants collector right now. They're pretty amazing at what they've been able to achieve with that, but they're also mostly not using US national clusters. I think they use some overseas and other stuff like that.

Starting point is 00:51:19 I wouldn't be surprised if Andromeda would give away compute for research. Probably, yeah. I'd heard of them doing something like that before. Well, you would know because you're in a grant. grant. Yeah. Like, yeah, my first response would be, aren't they already maxed out by existing users? Yeah, I think I'd heard when there is like a period where there were like not too many people using it or like a bunch of people canceled, they gave it away on some grants. So maybe if they're like bubbles where people aren't using it. Yeah. Yeah. The other two sources I'll name are Crusoe Energy, which is using, I don't know if you're familiar with them.

Starting point is 00:51:49 Yeah. I think they, correct me if I'm wrong, they put GPUs on top of like oil rigs. I'm not familiar with the details. It's slightly sketchy, but like, whatever. It's clean. Yeah. Okay. And then the other one is strong compute, which is doing one of those distributed cluster things. So it's like together, but with that's funding from Australia.

Starting point is 00:52:08 We are working with some providers like RunPod and NetMind. So I think there's definitely some resources out there, but everybody is looking for them. And really, I think the solution is we're going to need to scale up the compute that we have available to academia, the US. Well, in CMU and just in general. But I think we realize the importance of this. I hope the US government realizes the importance of this and invests lots of money for it, because that's actually the best solution,

Starting point is 00:52:34 but they move a little bit slower than a lot of people move. So we'll see. Yeah, I'm curious, if you kind of look across all of academia, what work have you been either most impressed by or do you think best represents the kind of work do you think academia should be doing in the last year or two? Yeah, so actually another comment about the, like, like academia versus industry thing, I really do wish that the people doing kind of

Starting point is 00:52:58 of frontier research on language models in industry acknowledged academic work a bit more because I do think like a lot of the things that people are doing in academia end up in industry but just don't get acknowledged and I think that has to do with the fact that industry is super secretive right now about anything they do in large language model space and so previously it would be like industry is publishing papers and we could point to the fact that hey Google uses our stuff, OpenAI uses our stuff, or things like that. But now there's a lot less of that, which makes it seem like we're shouting into a black hole,

Starting point is 00:53:31 and actually maybe we aren't so much. And the best example of this recently was like the Matrushka embeddings thing from UW, where OpenAI used it and renamed it something else. I mean, that one seemed like just an oversight rather than intentional exclusion, because they left enough hints that it was that. Yeah, maybe. And they did better with SORA, for example, where they actually cited all the works that, like, inspired them. The diffusion transformers. Yeah, and things like this. But they have every right to be

Starting point is 00:53:56 secret when they're competing, you know, their industry, they're competing against each other. Like, I think it makes sense, but it's also a little bit disincentivizing for grad students, for example, because they can't point to like their success stories that they had before. I don't know if there's any solution to that, but I thought I'd mention it just in case anybody who has influence would be listening. Yeah, it's just a, you know, corporate responsibility thing to, it's the right thing to do. Yeah. But like, you know, for me, the interplay between industry and research, I think you feel it the most with just like your grad student pipeline

Starting point is 00:54:28 or maybe like the undergrad that you're teaching, like what their interests are. I'm sure the class composition has changed a lot for the NLP class. Yep, yeah. I realize I skipped your examples of good papers from academia question, actually, and I can go back to that. But examples of good papers are both on like the evaluation side and on the modeling side.

Starting point is 00:54:49 I think on the modeling side, I have always preferred papers that are simple but work. And I think I'm a little bit weird with this respect in academia sometimes because I feel like when I see papers get reviewed, people are like, oh, this paper is not novel enough. But I'm like, this is a great paper. It's like it made a small tweak to this method, but it works three percentage points better.

Starting point is 00:55:15 Like they'll change one line of code and suddenly everything will work. But that insight was not there before, is why it didn't exist. So I really do like those sorts of things. I think DPO is a pretty good example. It's a lot simpler than they make out in the math. It's a lot simpler than they make out in the math, but I think that's a great thing, right? It's like a simple tweak that worked really well and people use it a lot. I think those are the kinds of things that are really valuable. I also think benchmarking, which isn't simple and takes a lot of work, is something valuable, which is why I'm spending time on that. Also contributions to open source,

Starting point is 00:55:51 because I feel like there's a small number of companies that are very committed to open source, like Hugging Face is an obvious example. But they don't have enough firepower to compete with the bigger companies who are working on these sorts of things. And I believe that open source, good open source alternative should exist. And academia could help with this.

Starting point is 00:56:13 The problem is we're very disorganized. So if we solve the problem of organization and focus and getting everything together, then that could help. And I mean, like HuggingFace is one example of a company that's doing that. I also hope that like efforts like Open Devon or MergeKit for model merging or other things that pulled together a whole bunch of different things under one roof could help out with that too. Yeah, I'm trying to feature those things.

Starting point is 00:56:37 Like I have a Merge Kit talk in my conference. You know, those kinds of projects will never get featured at Iclear. Right. And I'm trying to create a venue for engineering rather than just research. But obviously there's overlaps between them. Yeah. He's speaking as well. Do you know what you're going to speak about?

Starting point is 00:56:53 Not yet. Yeah, we can broaden out to just the syllabus and student interest before and after. You said you had to revamp the NLP syllabus. We can talk about that. We can talk about how to pick promising areas of work, which you already somewhat covered. Syllabus before and after, I increased stuff on distillation and synthetic data, for sure. I added a thing that was like a tour of large language models. So it was covering all the different large language models

Starting point is 00:57:25 and their similarities and differences and stuff like this. Because even I didn't know enough about the differences between the models. What are you like surprising differences? I don't know if this was necessarily surprising to me, but it might be surprising to some people, but like how similar the architectures are for, like everybody is using the Lama architecture. And it's not because architecture engineering

Starting point is 00:57:46 is not important, but it's because we finished architecture engineering, and now we have a really good architecture that works, and we're at least a local, you know, optimum for that, which is everybody uses rope, everybody uses swig glue, everybody uses all these other small tricks, and there's this really nice figure written in the Mamba paper. So Mamba is kind of like a linear architecture. It's a great paper, but there's a figure that compares the original transformer to the Lama transformer with respect to how it. how well it scales. Yeah.

Starting point is 00:58:18 And the Lama Transformer just scales like way, way better than the original transformer. So architecture is important, but we're kind of done with that and everybody is making no more than small. But there's like, I don't know, I feel like my belief here is there still exists a bunch of tricks, like there's the MEO trick, right? Like that'll get you like a slightly better.

Starting point is 00:58:37 It's a big one. Maybe there are small ones. I do wonder how many of these are left and how many of these also maybe only come into play when you're at larger scales. Like there's a great recent paper at meta, where they trained in the next few tokens, right? So you're not just predicting the next token,

Starting point is 00:58:51 you're predicting the next four. It didn't show better performance at small scales. But I think past the 13 billion parameter scale, it actually showed better performance. What? Yeah. So this is like another concerning thing for like academia perhaps. You may not know if you're actually works

Starting point is 00:59:06 in architectures. You're dealing with like large enough models, trained enough data. Yeah, I think that that's a major reason why we do need to scale up the resources that we have for training. models. And I think there's a lot of progress on that right now. Like, I think that a lot of places are working on that. And then the other thing that I wanted to mention is, yeah, because the architectures

Starting point is 00:59:27 are so similar, the data and the training methods are the big difference there. And that's where everything is actually really, really influential. Like, what data do you train on? How well do you clean and de-duplicate your data and stuff like that? And that's not something I really talked about it all before when I taught previously one year ago. So a lot more focus on data. I definitely teach architectures, but there's a lot less focus on architecture engineering. And it's more of an explanation about like

Starting point is 00:59:54 why the architectures we currently use are the ones that work. But we did have Albert Gou talk about Mamba because he's at CMU2. I was gonna ask, what are your thoughts in like this new wave? I feel like there's a bunch of like alternative architectures, like mainly Mamba, then I think RWKV and you- Those are I mean too.

Starting point is 01:00:10 Yeah. Yeah. I think we don't know enough about them. I definitely would like to focus some percentage of our effort on understanding them better. Because this is a perennial problem, which is when you try to do something really, really different, there's so much catching up to do with respect to the highly engineered thing that we have before. So neural machine translation, for example, it took, you know, a year or two to beat statistical machine translation or phrase-based machine translation, which is what we had before,

Starting point is 01:00:39 just because there were like 10 years of engineering that had gone into phrase-based machine translation to make it really work really well. And I feel like we're kind of in that thing for all of these linear architectures like Mamba and RUKP. I want to see them continue to be pushed. But I do think they have some fundamental limitations. Like, for example, recalling information. So we see the hybrid architectures with seven Mamba layers in one transformer layer.

Starting point is 01:01:02 Yeah. Recall and stuff like that. So it'd be interesting to see if they're... That's Jamba, right? Yeah, Jamba. Yeah. So you're... optimistic on the mixing. I think it's one way to solve the problem of like poor

Starting point is 01:01:13 recall and linear architectures but there might be might be other ways. Yeah I would say pure amount but pure R2KV both have the R&N problem. Right. It's forgetting. Yeah. It seems like you have to mix them and yeah. Yeah. I mean the mixing you like lose like the niceness of you're getting a factor of eight but it's not getting like if you're scaling up to a million like 10 million tokens like it just won't scale right. You still you're only diluting by a factor of eight while you saw that like quadratic attention bottleneck. But you can do strides and stuff like this.

Starting point is 01:01:42 So like I think there's a lot of room for improvement here, which is why I'm like kind of excited about that direction. That's a direction I'm most excited about with respect to architecture engineering. Interesting. One direction I wish would work, but it feels like no one's made it work. I don't know if it will work,

Starting point is 01:01:58 is like some kind of retrieval faked into the model. Like so retro is kind of like the original paper. Well, you know, Dali Kila from Contextual's working on. Oh yeah. So contextual, I think, is like working on things related to this. But I don't know, it does feel like if you really want to scale to 10,00 million tokens, you can't, like, store all that in some compress state. You do something fundamentally different putting in this context.

Starting point is 01:02:18 Like, it's like you need all that information, right? You need all the information to be present, but you need some kind of like sublinear per token generation. So we have a paper called the Unlimaformer, which was at Nurep's last year, that does retrieval-based detention. It encodes all of the previous context in FICE retrieval index. I like that general direction. It worked really well with kind of more traditional transformer models. We used it for T5, but one difficulty is actually rope makes it very difficult because you need to handle relative positional encoding appropriately and stuff like this.

Starting point is 01:02:51 So, yeah, I could go into details here, but we don't have a lot of time. But I think there definitely are some things moving in that direction. I think that's another thing that could be interesting. Take our existing architectures and somehow make them, like, efficient. through approximations. I do think this kind of K&N operator is pretty interesting. Can the Commonwealth of Arnold Network

Starting point is 01:03:13 or something else? Sorry, just like K-N-Yorice neighbors. Just being able to like do like a K&Earst neighbor's operator because like if you're trying to do attention over like all these tokens, right, you're now taking like kind of the average of all like I don't know, you've all these keys and values

Starting point is 01:03:26 and you're like averaging like 10 million of them like it feels worse. Yeah. You only need the top few. Yeah. Interesting. on KV scores. We're basically done. I will leave it to you for any plugs that you want to do, any calls to action. Yeah, I guess I'm really excited about open source things in general.

Starting point is 01:03:48 So, you know, any... Come contribute on Open Devon. Come contribute on Open Devon. Also, you know, use Open Devon to test your agents, add new agents to make it work well on particular tasks and stuff like that. That's really exciting. And also just in general, like, I really love new developments in open source AI. So even if it's not in Open Devon, like people to continue pushing on that. Yeah, I appreciate it when people do that.

Starting point is 01:04:14 No, open source AI has been fantastic for, I mean, it's fantastic for startups as well. Yeah, like it's been super helpful for us. Speaking of Quinn, do you guys use Quinn? Like what's the relationship between you and? Oh yeah, so we have a roadmap for Open Devon. And the initial roadmap for Open Devon was by the end of May, we wanted the best agent on Sweet Bench.

Starting point is 01:04:34 And we did that by the end of, end of April and we did that May 5th, so we were a little bit late, but... Close enough. Yeah. And then our... It was 20 days ago. Yeah. And then our May roadmap is to have a really good open agent.

Starting point is 01:04:48 And since we have people on the Quinn team, you know, working with us, I think building something on Gwen would make sense, but, you know, Lama 3 is also good. So we'll see. Yeah. Plugs, close actions. Yeah. I mean, we're hiring for Kursor, researchers, engineers, ML engineers. I think we're working on like very interesting stuff in code generation kind of.

Starting point is 01:05:06 on the frontier of what is possible for kind of in flow coding and assistive coding. So I think some really interesting stuff to do. Always hiring. I love the hustle. Yeah. My plug is AI engineer, the conference that I'm spending all my working always working on right now. That's it. Well, thank you very much for your time. Yeah, thanks a lot. It was great. Yeah, it was really fun. That was section A of our ICLR reasoning and agents coverage. The next section, section B, covers.

Starting point is 01:05:36 related discussions of benchmarks. We start with the hottest new benchmark that has emerged this year, Sweene Bench, which broke through the noise as the presumptive next level after the saturated human evil and MBPP benchmarks from OpenAI and Google DeepMind. Hi, it's great to be here. My name is Carlos Jimenez, and I'm a PhD student at Princeton University. And today I'm going to be talking about our evaluation benchmark called SwayBench. Can language models resolve real-world GitHub issues?

Starting point is 01:06:06 This is a work with my collaborators from Princeton and the University of Chicago, and I led this project with my co-author, John Young. So recently, language models have become really, really popular, and they're being pushed to perform in use cases that researchers haven't previously considered. A lot of past work on evaluating language models has become outdated simply because model performance is getting really, really good. And that's a good thing. but evaluating language models is also really important. Understanding the strengths and weaknesses of language models plays a major role in building future applications and helping end users know when and where it's appropriate to use them.

Starting point is 01:06:51 So I want us to think about what sort of qualities makes an evaluation benchmark useful. First, the problems need to be hard enough to challenge state-of-the-art models. Problems should also reflect what people actually want to use the models for. And lastly, solutions need to be easy to verify. You can, yeah.

Starting point is 01:07:13 So consider the tasks involved in software engineering. Software is an extremely powerful tool, and coding is already one of the most popular applications of language models today. In reality, programming is a very hard skill to master. So if we can understand how AI systems perform on this task, we have a better sense of their abilities on doing real and challenging work. Furthermore, we have a lot of infrastructure for evaluating code. Many large software projects incorporate things like unit and integration testing to automatically

Starting point is 01:07:44 evaluate changes to source code. Currently, language models only report evaluation numbers on coding benchmarks like human aval. And let's take a look at an example. So it starts with a function signature and a dock string describing what the function should do, and language mit models are evaluated on their ability to write the body of the function. Here it's highlighted in yellow. And there are many possible solutions, but the nice thing about programming is that we can

Starting point is 01:08:16 check the validity of any of them automatically using unit tests. However, very few programmers ask language models questions like this, unless maybe they're trying to cheat on an interview. Software engineers typically write code that fits into a larger project. not one-off isolated functions. So we created SweeBench as a benchmark to evaluate the software engineering ability of language models in as realistic a setting as possible.

Starting point is 01:08:44 I'll show you a very high-level view of what SweeBench is trying to evaluate, and then I'll talk about how we created it and so on. So Swee-Bench starts with a code base and a problem statement, and by code base, I mean a real code base with like hundreds of lines and thousands of files, and thousands of lines of code.

Starting point is 01:09:04 Hundreds of files and thousands of lines of code. And usually the problem statement is describing a bug with the code base or requesting some new feature or change in behavior. We then give this to the language model, and the language model is tasked with generating edits to one or more of the files in the code base in order to resolve this problem statement. Then we take the model's proposed changes

Starting point is 01:09:28 and evaluate it using unit tests from the same report. repository that were made after the issue was resolved. What this means is that SweeBench can programmatically evaluate AI systems on their ability to solve real-world problems situated in full code bases. Solving tasks in Swee-Bench goes beyond just code generation. It requires models to understand how large code bases work and how code changes in one function can impact the behavior of other parts of the code base. I'll briefly summarize how we made Swee Bench.

Starting point is 01:10:03 Now, GitHub is a website that people use to collaborate on software development projects, most of which are open source. And Astropi is one such example. On an open source GitHub project, users of a software can report bugs or request features by creating an issue explaining a problem that they encountered when using the software. and someone else who knows how to solve, how to fix the issue, can submit their solution in code, which is called a pull request to the project. Now, maintainers of the project can then review and modify that pull request and either accept the solution, in which case the source code for the project is updated or they reject it. Now, this process in collaborative development, which underlies a lot of,

Starting point is 01:10:56 the open source like development process. It can naturally be converted into tasks, and that's the source of the task instances in SweetBench. So we use the following procedure to gather task instances. We first scrape 12 popular Python repositories for all of their accepted pull request instance pairs. And then we filter these pull requests to make sure that they contribute updates

Starting point is 01:11:24 to both the source code, as well as the tests in the repository. And lastly, for each instance, we verify that the source code can be installed automatically and that the testing behavior changes before and after the source code solution is applied. So let's look at an example of what an instance in SweeBench looks like.

Starting point is 01:11:47 This is an issue from the SimPy Python library, which is used for symbolic mathematical operations and notation. So we show the problem statement on the left, and it's giving a detailed explanation of what the user is experiencing with SimPai, where they're seeing unexpected output when using the identity matrix.

Starting point is 01:12:11 And the code base for this issue is going to be tied to the version of the code base that was active when the issue was first submitted. Next, we'll have the gold patch. And this is the edits to the source code. that was submitted with the pull request, and it represents a possible solution to the issue, and it was the one that's officially accepted into the actual repository. And then finally, we have the test patch, and it's the edits to the tests that was contributed

Starting point is 01:12:45 with the pull request. Now, the test patch updates or adds tests that evaluate source code for this particular issue, and we verify that the testing behavior changes from failed to pass when running them before and after the source code is updated with the gold patch. So after collecting instances like this across 12 repositories, we end up with over 2,000 instances representing a diverse set of problems and codebases. Each sweepbench instance includes the full codebase, totaling to about 3,000 files on average,

Starting point is 01:13:18 while gold patches usually only edit one or two files. We further collect 19,000 unverified instances, so unverified, meaning they don't have test cases, and we use that for training purposes. So as an initial baseline, we use a retrieval augmented generation system or a rag using a simple BM25 sparse retriever. And language models are then provided with the problem statement, the project's read me file, and the entire file contents for the top retrieved files. They're then tasked with generating a patch file that specifies which files to change and the edits that they want to make to those files. We evaluate top models like chat, GPT, GPD4, and Claude. And we also fine-tune a code Lama, we also fine-tune code Lama using long-context rag examples from the training set to get our own model, Swee Lama, 7B and 13B, which are the only open-source models that. have non-zero performance on SweetBench now. And across the board, base performance is

Starting point is 01:14:28 extremely low with the best performing model, Claude 3 Opus, resolving only 3.8% of issues on SweetBench. So how can these models get better on SweetBench? Well, first, improving the Rack system can greatly improve performance. So if we assume that we have a very, very strong retrieval system that retrieves all of the files that were edited by the gold patch, which we call Oracle here. Performance jumps immediately from 3.8% to 9.1% for Cloud 3. Another thing is that long context still seem to remain an issue. So for Claude 2, longer context inputs anti-correlates with performance very strongly, and that's something that we saw with basically every model. The longer the files that are being retrieved or input, the more context, the

Starting point is 01:15:20 worst performance in general. And so lastly, qualitatively, we find that language models tend to generate shorter, simpler, and more primitive code compared to gold patches. And for instance, we notice that they tend to overuse Python built-ins and ignore library-specific utilities and API features, with an example shown here. So let's summarize. SweeBench is a benchmark for programmatically evaluating the software engineering abilities of AI systems using over 2,000 real-world test instances. And we show how even state-of-the-art models are still woefully behind on this task.

Starting point is 01:16:04 We open source Sway Lama 7B and 13B, which are suitable for long-context rag with SwayB So before concluding, there's one more thing. I've shown you performance using a RAG system for language models on SWIBENCH, but software engineering is naturally a very interactive task. And we've recently had a follow-up work to this paper called SWE agent that explores that idea a bit further. And with SWE agent, we built an agent computer interface for language models to interact with a computer to solve tasks on SWEBENCH, demonstrating much better performance.

Starting point is 01:16:38 So up to 12.5% of SweetBench is resolved with our new framework. And this shows that there's a lot of room to improve for AI systems on SweetBench. So finally, I'd like to thank all of my collaborators and colleagues who have helped with this project. We have an active community on GitHub, so please consider submitting your own solutions to be listed on the SweeBench Leader Board. Thanks. Hey, John. Nice to see you at the oral session. Yeah, yeah.

Starting point is 01:17:09 So, congrats on the success of SweetBench. Thank you. Why do you think it's caught on so much? Like, the first I heard about it was from Devin. That's right. What was the launch process? I think, you know, I'm trying to get into the meta story around like a lot of grad students here trying to get their work noticed. Yeah.

Starting point is 01:17:25 You got noticed. Yeah, yeah, yeah. Yeah, no, that's a great question. The Devin really certainly helped a lot with really putting Sweet Bench sort of, I think the biggest contribution they did was give people a visual of what you're doing. of what even 15% looks like on SweetBench, and I think it made it really compelling. Prior to Devin, we had started working on Sweet Agent back in, I want to say, September,

Starting point is 01:17:48 like right after we submitted this project to Eclear. So we kind of had this vision that, like, oh, we're going to put out Sweet Agent, and people will see the numbers can, in fact, go up. A lot of the feedback and the skepticism we had at that time was that the benchmark is really difficult. But looking at kind of what Human Eval did, you know, when they released,

Starting point is 01:18:05 GPT3 was at 0%. And we sort of use that as kind of motivation of like, well, you know, it's really bad now. But if we keep at it and we use this sort of agentic approach, there's something promising that could come out of it. So I feel like with this benchmark, our advisors and like me and Carlos, we're just really had this mindset of like trying to champion our own work a little bit. And I'm kind of expecting that it'll take off of it on its own of like that first step from zero to 10 or whatever it might be that that we really have to drive that and sort of make that happen. Yeah. Yeah, yeah, yeah. So you worked on Sui Agent first?

Starting point is 01:18:38 We worked on this starting in June last year, summer. We were able to submit by September to Iclear, and then right after that we started working on Sui Agent. Got it. And then just the backstory behind how you guys started to work together, how do you choose this direction, anything like that? Yeah, the narrative story. Yeah, for sure. Carlos is fantastic. He had mentored me for a long time.

Starting point is 01:19:01 I was a master's student at Princeton, and Carlos is a fourth year of PhD, is about a graduate. next year and he has a lot of expertise he's built benchmarks before I also give credit to like Shun you because we had worked on Web Shop and Intercode and a lot of this agent stuff before and Alex was around during the summer and he helped a lot with sort of thinking about the fine-tuning and sort of what are good baselines to go with so the way it kind of came together was in June I put it together a lot of related work and thought about this idea and brought it to our advisor Carthic and then I found out that Carlos had a very similar idea

Starting point is 01:19:33 kind of at the same time, of just sort of thinking about how we could take a lot of this great open source data on GitHub and turn it into a meaningful task. And then really just sort of like the nature of the task and how to follow through in terms of the engineering plan, I think was honestly quite clear after a week and we just had to execute at that point. What was some of the big debates where you had to go either this way or that way and you pick one way? Oh, that's a great question.

Starting point is 01:20:00 I think one of the things that I really remember initially was sort of, who how we were actually going to collect instances and sort of what the heuristics are. In hindsight, I think they're pretty straightforward and obvious. But at the time, one thing was just like, do we want to collect a lot of instances from different repositories, or do we want to sort of focus on a couple

Starting point is 01:20:18 of well-maintained repositories and mine the most instances from them? So just to sort of contrast that, we have 2,294 instances from 12 repositories. We could very well have 700 instances from 500 repositories, exactly. depth ended up winning out. It wasn't very obvious to us, but really just the process of manually inspecting,

Starting point is 01:20:39 looking at contribution guidelines, like looking at the natures of the test, gave us a lot of these heuristics that ended up being pretty reliable and I think scaled pretty well, at least for Pai Pai packages. So a lot of design decisions there, but I think we got lucky that we had a couple good sort of hits in the beginning. Were you concerned that it's primarily Python? Yes, yes, yes. That's a choice, you know?

Starting point is 01:21:00 Yeah, it is a choice. So I think for the first version of the benchmark, just because Python is so commonplace kind of in LLM evaluation, especially with a human avow, that like, we'll just go with the flow. Like we have no problems with the language. I think like in the same way that multiple E maybe expanded the amount of offerings for human eval, this is something we'd be interested in doing. It's going to require quite a bit of engineering effort. Like we're both sort of fairly good at Python, but when it comes to things like maybe Rust or

Starting point is 01:21:31 scholar or these other languages, like we're not quite sure. But I think like basically if there's an opportunity to collaborate and there's people who are experts in those languages, maybe even the software repository maintainers and they're interested in sort of having agentic language models help maintain their code base. We're more than happy to work with them to sort of see the sweet bench sort of evaluation harness idea and really manifest it for what they're trying to do. A lot of benchmarks try to say like, okay, human performance is 50. and then most language models are 25, and then we'll try to get the language models above human.

Starting point is 01:22:06 But, like, what is human here? What is a single human performance here, right? Like, is it 100, or is it not undefined? Yeah, that's a great question. I think it's kind of an evolving answer in the sense that when we initially pushed the paper, we were like, it's 100% because someone wrote it, and they did the issue, and they contributed. The thousands of people wrote it. Yeah, exactly, exactly.

Starting point is 01:22:30 But, you know, more recently, I think as this benchmark is kind of picking up and there's more people interested in it, I think just like having the right efforts to figure out like which issues are easier or harder along what dimensions. Like when we say easier for a human, what does that actually mean? Is it characterized by the issue or by the size of the change or by the nature of the change? Like maybe it's a one line edit, but it's really difficult because you have to know the code base super, super well to make that precise change. So we have some ongoing efforts that are just sort of taking a look at SweetBench and taking a look at sort of the actual code changes and the problem statements and saying like, all right, let's figure this out. Let's get sort of like Spider, how they have like easy, medium, hard, extra level problems, you know, stuff like that. Yeah, yeah, yeah, yeah. What about Eval cost? I think one of the reasons that human Eval is so popular is because it's easy, it's quick to evaluate. Yes. But my impression, I haven't run, SweetBench, actually, but like, my impression is it's quite expensive.

Starting point is 01:23:27 Yes, you're absolutely correct. I 100% agree with you for the 2,294 instances. In hindsight, you know, if it was a little smaller, it would have been okay. But for a rag, it's like 20 cents a pop. Even then to do sort of pass at whatever or run a model a couple times if you're kind of empirically validating system, the cost can add up really quickly. So I totally agree with you there. So like $100 or something? Yeah. Like for example, for a Svi agent, so we just released kind of the first preprint recently. We'll kind of polish it for NURPS, but just to get that. the idea out. We actually have a cost table in there. And for the agents, like for the resolved issues, it takes like one to two dollars to actually solve it. We have a $4 limit. But the meaningful thing that I feel like we did in response to sort of a lot of this was create the sweet bench

Starting point is 01:24:13 light split. And that's 300 task instances filtered from the 2,294. And the objective there is that we apply some filtering criteria. It's not random to only look at changes to one file where the issue has, you know, reproducible code, like basically removing some of the diversity of SweetBench that makes it really difficult to have sort of a more better starter one. And recently, like, there have been works, like, even yesterday that can't, like, auto code rover or like, yeah, I mean, like, I think this would have been, we kind of wish we had this to give to Devin also when they were running on this. And also, like, Code Act from sort of the Open Devon team, you know, they kind of put together something. And they claim they have 21% on Sweet Ben's

Starting point is 01:24:54 line now. So, so I think after we offered this, this, it's kind of a, there's a little bit more traction, I think. Okay. Awesome. What is your take on Sue agents' direction versus Devon versus Open Devon versus AutoCode Rover? What are the goals? What are the logical differences? Yeah, yeah. Oh, that's a really, really great question. Yeah, I guess I'll, like, kind of speak from what I know, and I don't claim to have a good understanding of the other systems. I think what Devon and Open Devon are doing is like really, really cool. It's really, especially when the Devin demo came out just to kind of see it as. potentially like a product. It was really fantastic. What I'll say is like I think for

Starting point is 01:25:31 Sweet Agent we were a little bit more focused on from a research angle just getting something to work and sort of having the empirics and the numbers to back it up. I feel like Carlos and my direction has more been sort of like trying to solve interesting research challenges. So like even after Sweet Agent like understanding what human interventions look like for autonomous software engineers, why people intervene, how they want to intervene, stuff like that. Which to be clear right now, In the one in the sweet agent that I saw, there was no human intervention. Exactly. You're right.

Starting point is 01:26:01 Exactly. And I think from that, I think it was a great session. Like, I really learned a lot. And I went back to my, and I was like, you know, we should, we should really think about what they said. There's so good points in there. But, yeah, to say it in one sentence, I would say, it's just like, yeah, just like, I think people like autonomy, but it also seems like they don't want to give up control. And it's kind of interesting in the sense that, like, it's cool from a research point to put out an end-to-end software.

Starting point is 01:26:26 engineer, but I think just like Devon and Open Devon have kind of the dialogue system. I think they're building out really intuitive features. The one thing I'll say is like from kind of when we did Sweet Agent, some intuitive things worked, some intuitive things did not work. And that's kind of what we're interested in really discerning. Like some human AI, UI's user interfaces and applications are great for giving to the language agent and having it sort of run with it. Some things don't work as well.

Starting point is 01:26:53 So, you know, we're curious and figuring that out. Cool. I think that's all that questions I have. Awesome. Yeah, yeah, yeah. Is there anything else that should have asked you? Oh, that's a great question. I guess, like, in terms of sort of, like, the future of this,

Starting point is 01:27:05 I think it's just exciting to see people be very enthusiastic about this kind of evaluation paradigm. So, like, yeah, like the feedback you gave on the podcast, I think it's great. Like, I'm really receptive to it. I guess, like, just as a personal job, like, I won't be in Princeton. I'll be in Stanford coming this fall, so I'll be back in the Bay Area. I'm very excited. I think it's a great ecosystem. there of sort of like a lot of people are pushing this and yeah just excited to sort of

Starting point is 01:27:30 collaborate with people and see what people's own takes are and just sort of manifest the really cool ones yeah awesome yeah well thank you thanks so much yeah really appreciate it yeah yeah next we explore the issue of benchmark contamination an issue raised by Horace He on GPT 4 Susan Jung on Microsoft's Phi 1.5 and you heard a mansehanger discuss sui bench contamination in the Graham Newbig discussion This next paper won an outstanding paper mention for their simple canary-free contamination detection technique. Hi everyone, it's an honor to be here. My name is Yonatan, and I'd like to talk to you today about test-sec contamination.

Starting point is 01:28:09 So recently we've seen large language models show remarkable performance on many challenging benchmarks. And it seems like almost every week there's a new open source model which comes out and tops the leaderboards. What's driving this performance gains that we're seeing in unsupervised learning has been massive pre-training data sets collected from the internet. So to give just one example, here is a breakdown of the pile. This is a large, diverse data set, which many open source language models use for training. And you can see it's compiled from many different sources.

Starting point is 01:28:37 So here in blue, we have academic sources like Archive and PubMed. In green, we have internet-based sources like Wikipedia and Stack Exchange. There is pros in here, there is code and math, and so on. But because of the scale of modern pre-training datasets, datasets, which are often on the order of trillions of tokens or petabytes of data, it's difficult to know if there's good separation between the training process of the language model and the benchmarks that we evaluate on. So to give you an example of how contamination might play out, let's say you have a language

Starting point is 01:29:07 model that you want to evaluate on a coding task, like Code Forces, for example. So Code Forces is a very commonly used benchmark, and maybe somebody uploaded it to GitHub, and then a webcaller found it, and as a result, it ended up in your training data. And so when you see an accuracy number or score on some benchmark, it's difficult to know if that's a result you can really trust because of the risk of contamination. So naturally, this brings us to the following question, which is how can we identify when a language model

Starting point is 01:29:37 has trained on a test set or benchmark? And this is a difficult question to answer because many of today's top-performing LMs are either closed behind APIs, or even if they're open source, their data sets are kept secret. So this is a figure from the Foundation Model Transparency Index by Stanford CRFM. You can see here in the first row that when it comes to pre-training data, there is very little openness in the industry.

Starting point is 01:30:02 I want to show you an example of the kind of discourse that's happening around test-set contamination today. So these are screenshots from a very prestigious academic forum. It's called Twitter. So on the left, you know, Horace is saying here, isn't it suspicious that if you look at the performance of GPT4 on Code Forces problems introduced before 2021, it scores 100%. But if you test the model on recent problems, the performance drops to zero. Yeah, that seems a little bit suspicious. On the right, Susan points out that if you give Phi 1.5, so this is a language model trained by a team at Microsoft,

Starting point is 01:30:35 if you give Phi1.5, the first half of an example from a data set of math problems called GSM8K, it'll complete the second half perfectly, which also seems a bit suspicious. Of course, it's possible that these things happen by random chance, but the point is that this is circumstantial evidence at best, and it's clear that we need some way to audit or test a closed language model for contamination in a way that provides rigorous proof.

Starting point is 01:31:00 So there's been a lot of exciting work on test that contamination recently. I want to highlight three works in particular that are actually all at EICARE this week, and I would highly encourage you to meet with these teams and learn more about their work. These are all excellent papers, but for our work, we were specifically interested in detecting contamination with provable guarantees in the false positive rate. And that's what I want to talk to you about today. So the question we're interested in is, is it possible to prove in a statistical sense that a language model was trained on a test set without access to the data used to train the model? So to describe this goal more formally, our setup is as follows.

Starting point is 01:31:37 So we're given some test set X and the ability to evaluate log problems of text under a language model of theta. And what we want is to develop a statistical test in the classic frequentest sense, which guarantees the type 1 error rate of utmost alpha. So here we're framing contamination as a statistical dependence between the model and the test set. And what that means is that we're going to test the null hypothesis that the test set X and the model theta are independent random variables. So just to be precise, the randomness of the model here is determined by the random draw of the training data which may or may not contain the test set X. So how can we do this?

Starting point is 01:32:18 How can we accomplish this? In order to make this possible, we're going to exploit a property which is true of many test sets, which is called exchangeability. So what do I mean by this? So typically, a test set is just a file where on each line we have an example. But the order in which the examples appear in the test set doesn't actually matter. The examples are exchangeable. I can show you the examples in any order and it's still the same test set.

Starting point is 01:32:46 So formally, what exchangeability means is that we can permute the examples in the test set without changing the joint distribution of the data, which means that the model should have no inherent preference for the ordering of the examples. It's worth noting that exchangeability is a strictly weaker assumption than IID, which is an assumption we make all the time about our data on machine learning. So however, if a test set was leaked into a model's training data,

Starting point is 01:33:10 because of the way pre-training works, where we take as many tokens as fit into the context window, the model would see multiple examples in a row and then memorize something about the order of the example in the test set. So our key insight here is a preference by the model for a canonical ordering of a test set must be a result of contamination.

Starting point is 01:33:29 So using this idea, there's a very simple permutation test we can construct. So the key here is to compare the likelihood of the original ordering to the likelihoods of shuffled orderings. So for example, for the original ordering to have the highest log likelihood under the model than any ordering over, let's say,

Starting point is 01:33:49 a million random shuffles, there's a one in a million chance of this happening under the null hypothesis, right, if the model didn't see the test set during training. So more formally, we can draw random shuffles, which is shown here as x sub pi, and we can compare each of these shuffles to our original sequence order, which here is log p-sbeta of x. And by doing these comparisons, we're estimating the quantile of the log likelihood of the original ordering. So this ratio here turns out to exactly be the P value of a permutation test.

Starting point is 01:34:22 So this test works quite well, but it's computationally expensive because of the number of times we need to permute the data set. And it turns out that we can do something a little more clever by aggregating a number of smaller test, and that's what we call the sharded rank comparison test. And just for the sake of time, I'll refer you to the paper for more details. So how well does this actually work in practice? So in order to validate our test, we needed to have a language model for which we knew the training data was contaminated.

Starting point is 01:34:46 So we decided to train our own model. So we started with a data set of 20 billion tokens from Wikipedia, and we pre-trained a 1.4 billion parameter language model on this data. But first, we took a collection of benchmarks, and we injected them into the training data at random positions. And then we wanted to see, so can we actually detect contamination in this case? So here are the results of that experiment. They're in this table. So here each row is a test set that appears in the training data,

Starting point is 01:35:14 and the data sets were injected at very very important. were injected at various duplication rates. So some of them are in there one time, some are in there 10 times, and so on. And in the two columns on the right, we have the results of both the permutation test and the sharded test. So these numbers are p-values, so the smaller the better, or the stronger the detection. And using a typical rejection threshold of 0.05, we find that for test sets, which appear 10 or more times in the pre-training data, we can detect them 100% of the time, so we get perfect detection. We also found that detection at a duplication count of one is quite challenging.

Starting point is 01:35:46 So we don't currently have a statistical test that works for duplication counts that are that low, and we'd like to encourage the community to continue working on this problem. So one question you can ask is, you know, at what point does contamination become detectable? So we found that for a duplication count of four, we can detect test sets about half the time. For a duplication count of two, we can detect test sets some of the time. So we are able to detect contamination at low duplication counts, but just not for duplication count of one. So you're probably asking yourself,

Starting point is 01:36:17 okay, what about real models? Can we identify provable instances of contamination and LLMs that are in wide use today? So before discussing these results, there are a couple of important points I want to make. The first is that absence of evidence is not evidence of absence. So just because a P value is high

Starting point is 01:36:36 doesn't mean that there's no contamination, it just suggests that contamination is unlikely, at least at high duplication counts of more than 10. So it's a very particular claim that we're making here. The second is that there are a lot of hypotheses being tested here, and it's possible that some of these P values will be significant just by random chance. So it's typical to do what's called a multiple hypothesis testing correction, and I'll refer you to the paper for discussion on that. So our first result here is that we didn't find evidence of contamination other than of

Starting point is 01:37:05 mistral 7B and ARC easy. And with a multiple test correction, the P value is just barely below significance, but it's still significant. The second result here that's of note is on MMLU. So the MMLEU row you'll see has a little dagger there. And the reason for that is because some of the tests in MMLU were not exchangeable, and we had to filter them out. So it's not quite the same as the others, but our findings are consistent with the Lama 2 report, which finds mild evidence for contamination by MMLU in their pretraining data.

Starting point is 01:37:36 So I think the main takeaway here is that it's probably not the case that popular benchmarks are being duplicated numerous times in the training data of top performing language models. Instead, it's likely that if these test sets appear in training data, that they appear at low duplication counts. So then the question becomes, how much does low duplication count contamination affect performance on benchmarks?

Starting point is 01:37:58 I think this is hard to say, but in some sense this work suggests that there's an upper bound on how much contamination is out there for today's highest performing language models. So to conclude here, we covered three important points today from our work. First, we showed that it was possible to get provable guarantees on detecting verbatim contamination

Starting point is 01:38:17 by leveraging exchangeability and benchmarks. So this is exciting because it opens the door to potentially very principled approaches to contamination audits. Second, we show that these tests are effective, but there's also a major open problem that we can all make progress on, which is contamination detection at a duplication count of one. Finally, we tested existing public language models and don't find evidence of content. at least with high duplication counts. And this could be due to a number of factors like deduplicating being very common these days,

Starting point is 01:38:48 but addressing the case of duplication count one would allow us to give more conclusive results in public audits. So we think this is an exciting first step and would like to see others make progress on low duplication count detection and all the models and benchmarks using this project are available

Starting point is 01:39:02 to encourage the development of future work. Finally, I just want to acknowledge my co-authors for their hard work in Stanford CRFM, especially David Hall, and personal without whom we wouldn't have had the compute for this budget or the computer for this project. Yeah, thank you very much. Our last benchmarks paper feature is Gaia. A benchmark for general AI assistance by meta-AI under Jan Lecun and Clementine Foria

Starting point is 01:39:30 who runs the Hugging Face Open LLM leaderboard. We talked to Thomas Sialam, who led training on Lama 2 and 3. Special thanks to listener Mhoshamoblochalov for the personal introduction. So Gaia, the General A. Assistant Benchmark. So for the background, we were at Meta during a workshop. And in the room we had like so with Jan Lecun and others, were arguing is El-LM all you need or not. And at some point, Gregor and I were, let's just do a benchmark instead of qualitatively tasting those models.

Starting point is 01:40:03 Let's create this benchmark where we believe that what are those capabilities where models are failing? You know, in the history of evaluation for LLMs, we started with some simple problems like the squad some time ago and then it gets better so you move to polar multitask questions like glue super glue and then because it was solved so fast we move to actually harder questions and so by harder the exit was taken but even like now to expert tasks like MMLU it's interesting because MMU I think the scores now for models is like more than 80 but so human is 90 but if you ask actually that's human experts in the domains if you ask a random human the score actually I think like 35 and the same like with like for you know

Starting point is 01:40:48 evaluated on exams and my point was like you know those are like results on expert domains but actually if we think about like simple tasks that an assistant would do that was the old point of Gaya model actually completely failing that was my intuition while humans would be like 90% so it's a complex size that require multi-step reasoning multi-information problems and an open world browsing the web, passing the information, etc. Like humans, if you give it enough, given a human enough time, you will solve this problem.

Starting point is 01:41:22 That's what we build with Gaia. And the humans obtain like random humans 90%. The EPA4 obtain only 10% on the level one, with some tool use and feel like that. So that was the point of the thing. And to our point, arguably if you want reasoning general capable agents, whatever LLMs or whatever AI system, you want to solve those types first.

Starting point is 01:41:48 It's fine to answer questions about what a PhD student will ask in a physical chemistry or whatever. If you don't solve those tasks, there's a problem. And so that's all about GAIA. We design a bunch of very small but very qualitative tasks questions that are very complex. From level one to level three, level one are like, you need one or two steps of reasons. of browsing, it's kind of simple. Level 3, it's extremely complex and takes like up to 10, 20 steps.

Starting point is 01:42:18 And in an open problem, like the complexity of the exponential increase, because you can like, you're in open-ended world where you need to browse, you need to use maybe multimodal abilities, etc. And actually it's interesting because very recently with agents, poor violin, but using memory, memory, tool use, like planning systems, like Friday in OSCOPilot, or there's a recent like auto-gen from Microsoft, you move to like 10 to 40% already. So there's something happening there.

Starting point is 01:42:53 And we think that's actually a very good way also to evaluate how intelligent are those dilemms. Like, what if you put Lama 3 or GPD5 in this model? What will be the boost of performance, basically? You are also on the Lama 3 team, right? Yeah. Did you already test Lama 3 on this? Not yet.

Starting point is 01:43:12 It's still in development. You mean Lama 3 is still in development? Yeah, but we will definitely and eventually we will move to like improve Lama in this direction. How would you compare this versus like the other generalist agent's benchmarks? I just talked to Graham Newbig who did Web Arena. I think I mean there's some specificity about Gaia that's why we design it that way. So first of all you don't need an environment. Most of the agent matchments, like you need an under-earnment, it's complex, and it's kind of limited by design to the synthetic under-earnment.

Starting point is 01:43:46 Gaia, it's like you just have a prompt. And that's it. You have access to the world. Go for it. So it's non-deterministic, though? It's not deterministic. I mean, it is deterministic. By the way, we created the question.

Starting point is 01:44:00 So one thing is that the question are kind of somehow unrealistic, because in practice, for instance, One question you will say is, we had this question actually in the dataset. How many BERT has layers? How many BERT? BERT. BERT. Okay, yeah, yeah. And actually, the annotators were disagreeing because you have different size of bears, different things. And so for making sure that basically the question have a unique answer, there's zero ambiguity.

Starting point is 01:44:29 We add to the question a lot of addition information, like according to Wikipedia or vis-dates, blah, blah, blah. such that there's a single and unique answer to make the model so we can evaluate the model automatically. In real life you probably will have easier question with a bit more ambiguity. What are your thoughts on just contamination proofing these kinds of benchmarks? Because a lot of these LLMs are going to be online or like the knowledge cutoff is updated. Yeah, I mean I think there's two things that made actually guay and bulletproof to nomination. Bulletproof? Yeah. All right. I would say though. That's a big claim.

Starting point is 01:45:05 I mean, two things. One is, first of all, we kept the test set aside. We didn't release it. So anyone can put on the leaderboard their results on test set. To solve it, you will need manually to answer the question of the test sets, go for each, spend like one hour for each of the questions, and provide the answer. That would be honestly like a pain in the eye.

Starting point is 01:45:30 Now, that being said, the thing is, it's super easy to create new questions, right? So if tomorrow you cheat and you say like you claim a new super high result, I can get with you like 10 questions, test your model. If it fails as a problem, so it's super easy to detect these kind of things. And the last thing is also, we ask in general the people to report the trace of the answer. So you can very easily verify what has happening with the model. So it will be in practice extremely hard to cheat on the leaderboard or if you do that you never have a model to show to the people actually. So in that set it's good at both.

Starting point is 01:46:07 All those questions are not know-age facts, but things you will find on the internet that is not present. You cannot go to the answer just by memorization of the train of the self-supervised massive web test. An example of that question is like if I ask you when is born a Wii 14 according to the Wikipedia page, but something you can remember from the training set. Now if I ask you how many times Louis 14 was mentioned with this way to write it in the Wikipedia page. That's an answer you will never find the answer right in the web page. You need to control F.

Starting point is 01:46:39 And we kind of oriented the question in this kind of sense. What do you think the performance is bottlenecked by? Is it tool use? Is it planning? What agent capabilities have revealed in your testing? That's a very good question. Actually, I've told them if I plan to welcome that. So I'll have more insights.

Starting point is 01:47:00 but my two cents right now is one a general system that can be powered by an LM and we're starting to see them so now we can add more like planning backtracking thing like that to improve them this is one of the core piece that is missing and the second thing is all of them are powered by LLMs and the smarter this LLM will get with scaling the better also the performance will improve. So there's, I would say, these two core things. The overall system by the M improving that, giving more tools, more abilities, more capabilities, trade its own tools, leverage its own tools, planning, backtracking, and all those things, and improving the LLM itself. Yeah, backtracking, meaning like the backspace token or

Starting point is 01:47:45 something else? Meaning like non-autoregressive decoding, but at the system level, not at the LLM level, in a sense that you can ask something, you can try. Like traversing a graph, like a tree of thought or something. You have your plan, but in practice, I mean, if I want to go to Vienna for Iqlir, and I want to book a plane, but actually they are awful, I need to adapt my plan. Deep Mind presented a paper, like the web agent's thing, with exactly the Vienna travel example. I wonder, they probably haven't evaluated their agents on your benchmark,

Starting point is 01:48:20 and I wonder what it would take to cross-pollinate the different labs agents and benchmark ideas. You know what I'm talking about? Like, DeepMind has your own stuff, you have your own stuff, and I don't really see a crossover very much. Yeah, that's a very good question. The thing is, at Meta, we are really like pro-open source. Yeah, you're the most open. And we released in, with, I mean, it's not just beta,

Starting point is 01:48:43 but also with hugging face, this paper. And we just made it open source with a little board, so that it's actually, everyone can use it and then sharpen it. Yeah. So now I'm really looking forward to see others, like, put some riddles there, push your numbers, put some pressure on the numbers, and see how far like, I mean, that's the only way to me, but we can accelerate the community to go to more capable models now.

Starting point is 01:49:06 Yeah, that's all the questions. Anything else we should have asked you? I'm looking forward to see what the community will push as a, and how fast we will get to, like, solving this VETMAC. Okay. I'm really curious to see it. Yeah, me too. Something we don't often get to feature are the invited keynote talks that start every morning of conferences like ICLR. There were great sessions on legal and copyright risk, the road to AGI and even Devi Parrake's career stories. But this time, we are featuring Moritz-Heart's Heart's Talk on Benchmarks for its comprehensive walk through history and call for action on greater thinking on the need for more scientific benchmarking.

Starting point is 01:49:45 It's a pleasure to be here. I'm very humble to be here. And I'll tell you about the emerging science of benchmarks. and this is not about why a particular piece of machine learning works. This is sort of about why the machine learning community as a whole works. It's pretty much everything we know about it, and it's going to be a relatively short talk. Okay, so I'll start with a quote. It's a famous quote, and it says,

Starting point is 01:50:08 the only principle that does not inhibit progress is anything goes. And that's what philosopher Paul Fierreaubert argued about 50 years ago. And many smart people agree with a statement and many disagree, and there's been a lot of debate about it. But whether you agree or not with us, I'll leave that up to you, but it's pretty clear that the machine learning community, especially the Eichlear community,

Starting point is 01:50:29 has always embraced the anything goes, okay? And in a good way. So from its roots in sort of the cybernetics and pattern recognition era of the 1940s and 1950s, we've pretty much tried out everything. Okay, and this community lets you dream your wildest dreams. You could be inspired by the human brain or child development or physics. This community does not limit you in how,

Starting point is 01:50:50 how you come up with the stuff that you propose. It's totally up to you. This is what Anything Goes means. You're not limited in how you come up with the signs that you do. And I think this has always been sort of the strong suit of this community that we don't limit people in how they work. Okay. But there's one thing we need to tame the Anything Goes,

Starting point is 01:51:08 and that's the idea of a benchmark. It's the one rule that we have which sort of tames this idea of anything goes. And benchmarks follow what the philosopher Michael Strevins calls the iron rule of modern science. So the iron rule is the idea that all disputes must ultimately be settled by competitive empirical testing.

Starting point is 01:51:30 So at the end of a day, after some time, we must come together and, you know, competitively test our hypotheses or our methods or whatever we want to do and see which one works best. Okay? And so the way this works in machine learning, you all know this,

Starting point is 01:51:44 is that we agree on a metric or measure of success, and we agree on test cases, benchmark data, and we let people can compete over the metric and the data, and we rank the models. Okay, so we see who is best in the end, and we might pick the best performing method. So this is what we call a machine learning benchmark, and it's essentially how this community operates, okay? So what's interesting is that benchmarks emerge.

Starting point is 01:52:11 They didn't, it's not like the founding fathers of the community set down, and they said, here's how it's going to be, we're going to operate according to the following principle. they sort of came up over time, okay, and they didn't follow any a priori theoretical framework. We didn't know ahead of time what we're doing or why it was going to work, okay? And so if you look at the last 40 years, the history goes back much further, but if you look at the last 40 years, you see roughly four eras of benchmarking. The first one is the DARPA era in the 1980s, where, you know, grant managers at DARPA wanted to have a way to compare scientists and their contributions in, various grant proposals, and they want to have objective ways to compare them. So they thought this idea of a metric and like a benchmark would be great to see how scientists

Starting point is 01:52:59 are doing against the proposed objectives. And so this was the DARPA era, and it was followed, I think, you know, if you will, by the MNIST era where sort of, you know, benchmarks sort of came into the academic, you know, front and center. People started using benchmarks more and more in academia, and there were starting to be public lead awards, people were comparing methods on publicly available data, and MNIST and other benchmarks around the time contributed to this becoming a major paradigm in machine learning. And then, of course, the whole idea of a benchmark exploded in the ImageNet era when, you know, the deep learning revolution

Starting point is 01:53:35 of the 2010s happened. And ImageNet as a benchmark was really sort of co-constitutive with the deep learning revolution of that time. So the models were sort of developed on ImageNet and tested on ImageNet and ranked on ImageNet, and this was sort of the dominant benchmark in that era. And now, interestingly, I'd argue we're in a fourth era, which I'll call the polymorphic era for this talk, and I'm not too attached to this name. So if eventually we call this something else, I'm okay with that. But for this talk, I'll call it the polymorphic era. And the reason is that we're witnessing sort of this radical plurality of benchmarks.

Starting point is 01:54:08 We're seeing a lot of benchmarks, and now thousands of benchmarks, and they take on very different forms. Okay, so they're not the way they used to be. I argue we're in a new sort of phase of benchmarking, and we're trying out many new things, in particular multitask benchmarks and dynamic benchmarks where you don't have a fixed data set. You actually, you know, let the data set evolve over time. So people are trying out all sorts of new benchmark ideas, okay? If you're interested in more background, I'll point you to Mark Lieberman's talk from the Simon's Institute from about five years ago. It's an excellent resource on some of the history, and Ben Rect and I wrote a chapter.

Starting point is 01:54:44 on this also in our textbook, so you can check this out online. But this talk is not about the history. It's about, you know, why benchmarks work. Okay, and I'll start with an outline of the science of benchmarks. What do we know about benchmarks? Why they work and when they don't work? And I'll start with sort of scientific key takeaways from the ImageNet era with the benefit of hindsight, because it's over, we can look back and we can do some kind of retrospective analysis of what worked and what did we learn from this era. And then I'll move on to this era that we'll we don't have the benefit of hindsight about that's happening right now at this conference and elsewhere, and we don't really know what it's doing. And I'll point out some risks, also opportunities,

Starting point is 01:55:23 but some risks of this new polymorphic era benchmarking. And I'll wrap up by motivating why we need a science of benchmarks, why I hope many of you will join this effort, and why I think it's a fascinating research area to work in. So how did this all start? I argue that the beginnings of the signs of benchmarks started with a mistake, like many things. It was an excellent mistake, but it was a mistake nonetheless. So the mistake that started all this is to assume that benchmarks are just the hold-up method. And you still hear this today a lot. It'll be able to say, ah, it's just the hold-up method. Okay. So what's the hold-out method? The hold-up method, as you read it in textbooks, just looks like this. You split the data into, let's say, two pieces. Could be multiple pieces before this talk is two.

Starting point is 01:56:06 you set aside the training data, you can apply, or sorry, you set aside the test data, you apply to anything goes principle to the training data, you can do whatever you want with the training data, and then you use, in the end, you rank the models on the test data. You had set aside the test data, and in the end, you rank the models on the test data. And I emphasize this part in the end,

Starting point is 01:56:28 because this is really crucial, okay, as you can read, you know, in this textbook by Hasty Tipsy-Ran and Friedman very recently, that ideally the test, test set should be kept in a vault and be brought out only at the end of the data analysis. So the holdup method only works in theory if you keep the test set in a vault and you only rank the models in the end at the test set. So the models had never sort of seen the test set in any way. Okay. And why is that? Because if you want to prove guarantees about the holdout method, you really need this. You need to some, this vault assumption, as I call it. And under this assumption,

Starting point is 01:57:03 you can argue that the test set has exponential longevity. It sounds like something Silicon Valley billionaires want to have, and I just mean that it's the number of model comparisons you can do on your test set is exponential in the data set size. So the data set can live very long. You can try out all sorts of models on the test set, and you still get good results from the test set. So under this vault assumption, the test set has high longevity.

Starting point is 01:57:31 It sticks around, it can survive. many model comparisons. But look, you all notice, the empirical reality is completely different, right? The test set is anything but in a vault. Okay, so we have a large sprawling machine earn community that sort of builds models in like a continual loop with a test set.

Starting point is 01:57:48 Okay, so you propose a model, you evaluate it on the test set, you see what the results are, you incorporate these results into your work, and you continue. This is the whole point of science that there is this kind of like closed feedback loop between you and the evaluation systems that you have. And I looked this up, just, and I was amazed by this,

Starting point is 01:58:05 how many of you have downloaded MMLU recently? Come on, guys, three people, that's not true. That doesn't work. It was Friday morning at age, people are tired. Okay, I looked this up, apparently, MMLU, so this is a multitask benchmark for language models, and many of you know. It has just 14,000 data points, questions,

Starting point is 01:58:26 and it was downloaded apparently five million times on Hugging Face last month, okay? I found this marvelous, okay? It shows you how, what scale this, communities operating at. It comes in two versions, and if you add it up, you get about five million downloads. That's 60 million downloads. If you think that every download is at least one evaluation, this test set is seeing tens of millions of evaluations per year. That's kind of striking. But it's not just the number of evaluations, it's not just how often you evaluate these test sets,

Starting point is 01:58:56 it's the way you do it, in the sense that machine learning is adaptive. You use the results from the test set to refine your methods. It's not like this test set is kept in a vault. It's not, part of the ongoing evaluation loop, and that's the whole point. Okay? And so what we realized about 10 years ago is that this kind of adaptive activity, this kind of adaptive use of the test set, breaks all existing guarantees of the holdout method, and it reduces its lifeline to just a linear longevity in the number of evaluations. Okay? So the test set can only support a linear number of evaluations. You can show easy examples where this happens. Okay. So in principle, this way of using the test set

Starting point is 01:59:35 could be really bad. Okay. And this launched the area of adaptive data analysis where people come up with sophisticated methods to try to fix, you know, the hold-up method to work better under this adaptive use. But we can go back to the empirical reality and we can see that like test sets, actually in practice, seem to have a lot of longevity. Okay. Here's a plot that probably many of you know. It's the, you know, from a plot from papers with code that you can see on the website right there. It just shows you the improvements on the ImageNet ILSVRC 2012 test set, you know, over the years. And as you can tell, even after sort of 10 years of very active use and 10 years of people hammering away at this test set, you were still seeing significant improvements.

Starting point is 02:00:20 The test set was still good enough at that point to support active model development and model improvements. And it was still worthwhile and useful for model ranking. So how could it be that, you know, after such a long time, it still seems useful, okay? Should we trust the model rankings? It really begs the question, should we trust the model rankings that we get out of this 10 years of active use of this test set? So several years ago, people wanted to find out, and they created a fresh test set for ImageNet. They restarted or recreated the dataset creation process for ImageNet and tried very carefully to create a new fresh test set for ImageNet. Okay, this is a fantastic work.

Starting point is 02:01:00 And what they found is, and also that the model rankings were preserved. So the model rankings on this fresh test set were actually preserved, you know, compared to the old test set. Okay, so the model rankings were fine. Even after a decade of development, the model rankings were still the same ones. Okay. And the people also found the same thing for MNIST, which was even more used, you know, various Kaggle competitions. It was also true and other data sets. So people confirmed this insight about the stability of the model rankings in a number of different cases.

Starting point is 02:01:28 And this is what I call the internal validity of the iron rule. Beating the previous best replicates in similar conditions. If you have a model improvement in some test condition, you will get the same model improvement in very similar test conditions. If you try to recreate a test set or you recreate the testing conditions very closely, you will see a similar model improvement. So this is the internal validity of the iron rule, beating the previous best replicates in similar conditions.

Starting point is 02:01:57 And we were able to even prove this. Okay, so we had to work where we turned this observation into mathematical assumption. Okay, we said, what if we make the assumption that researchers only care if they improved over the previous best? So what if we make a mathematical assumption that, you know, researchers will ignore results that didn't improve and they only care about results that improved over the previous best? This is, of course, a simplifying assumption. But if we make this assumption, we can actually prove formally, that assuming this iron rule assumption,

Starting point is 02:02:30 the benchmark data has exponential longevity. So you recover the exponential longevity of the holdout method under this iron rule assumption. And I find this amazing because it says that the iron rule assumption is nearly as good as the iron vault assumption. So the idea of competition, this principle of letting people compete

Starting point is 02:02:48 is as much a regularizing force as keeping the test set in the vault. It's almost a little bit depressing if you think about it, that competition is such a strong, you know, coordinating principle. But this is what it is. Okay? You can think of this as something that you implement.

Starting point is 02:03:04 If you're really worried about, you know, keeping your model rankings accurate or protecting your test set, you can implement this by, like, enforcing limited feedback in a benchmark. But I actually like to think about it differently. I like to think about it as a descriptive sort of theorem. It says, you know, if we think of this as a postulate about how the community works, about how scientists interact, then this is, you know, you. you get these consequences. Okay?

Starting point is 02:03:28 So under this postulate about the community, we know that test sets have exponential longevity. Okay? And so this is ultimately an assumption about the behavior of the community. It's not just a technical assumption. It's an assumption about how we as a community organize our scientific activities.

Starting point is 02:03:44 All right. And so we found many other such sociotechnical forces behind benchmark longevity, things about the community that promote benchmark longevity. Okay? Competition is the one I just mentioned. but also collaboration, the fact that we all share code

Starting point is 02:04:00 and we share each other's GitHub repositories and so on, this has an effect that promotes longevity. This is related to what Dave Donahoe recently called frictionless reproducibility, the fact that we have this very active ecosystem of sharing code and it's important driving force in this context. We even found that cognitive and behavioral biases

Starting point is 02:04:21 of the researcher, the fact that we're all like limited human beings and we have our biases, these biases can actually protect the test set. So our cognitive limitations can actually work to our advantage when we're doing science and aren't necessarily a bad thing. And it's great. I feel good about this. And finally, we found that basic data set artifacts, like what classes you include, you know, how many classes you include, that also has a strong effect on benchmark longevity. Okay. So this is something we found a few years ago. There's a talk about it. If you want to see more about it, these things are all what I call sociotechnical forces. They're not just purely statistical,

Starting point is 02:04:56 they're not just purely, you know, technical. There's something about how the community works, but all of them promote internal validity of benchmarks. Good. So what do we know at this point? So as of about five years ago, we knew that model rankings replicate under similar test conditions. If you change your test conditions slightly,

Starting point is 02:05:15 you get the same model rankings. It's this kind of internal validity story. So what if we ask a more daring question? What if we ask, do model rankings replicate on radically different test conditions. What if we stretch this to the limit and we go to radically different test environments? Will we still get the same model rankings? Is there any reason to believe that the answer is yes? So we wanted to find out and we call us the Image Not experiment. And this is joint work with Olavales-Alodyn who interned with me last year. And Image Not is an anti-replication

Starting point is 02:05:49 of ImageNet. Okay, this sounds crazy. And the term I made up, so nobody I think has used this idea of an entire replication. But what I mean by this is that it has the same scale and diversity of ImageNet, same size, but it's different in every other regard. Just try to make it as different as possible subject to the same size and scale as ImageNet. You all know ImageNet was carefully curated by humans.

Starting point is 02:06:12 You had many annotators per image. You had a very high agreement rate between annotators. There was a certain logic to which classes you included, et cetera. Image not is just a quick and dirty data set based on selecting images from their captions in a web crawl data set. And there's no rhyme or reason to the classes

Starting point is 02:06:33 that we include, it's completely arbitrary. It's, if you will, kind of a trashy data set. So I'll be honest with you. And so the experiment we did is, what if you retrain the key ImageNet era models from scratch on ImageNet? So not fine-tuning. We're retraining them from scratch on this new data set. And we want to see what happens.

Starting point is 02:06:54 Is it true that the model rankings are preserved? Remember, this is a retrospective analysis. We already have these models. They were developed on ImageNet. Do the model rankings replicate in this radically different test environment? So the answer is yes. Okay, so for the model architectures that we carefully studied, we see the exact same rankings.

Starting point is 02:07:16 AlexNet comes lowest because it's the first major breakthrough in 2014. BGG improves upon that, dense net improves over that, resonant comes then, and so forth. get the exact same model rankings. But what is maybe more striking is that the relative improvement over AlexNet is also about the same. So if you look at the curve of relative improvements that each model makes over time compared to AlexNet, you get roughly the same curve. So what is this is saying, you know, somewhat surprisingly is that on this completely different kind of trashy data set, this data set makes the exact same sort of judgments as ImageNet.

Starting point is 02:07:54 It gives you the same information from a benchmarking perspective as ImageNet. It says this model is better than that, and it gives you the same comparisons, and it gives you the same sense of relative improvement. Moreover, we found out the same is true for fine-tuning. If you wondered about that, if you fine-tunes instead of retrained from scratch, is also the same. And we also looked at transfer learning, and we found that there was a similar relative utility to transfer learning on ImageNet as there is on ImageNet. Okay, so how do we create this?

Starting point is 02:08:21 You probably, many of you already guessed this. we used this wonderful resource called Lyon, which was created from Common Crawl. It's a resource of 5.85 billion image caption pairs. It was a massive effort that we're using here and building on. And we just really did sort of an extra step on top of that. We just selected a bunch of images based on their captions from Lyon. Okay. How do we pick the captions, the classes?

Starting point is 02:08:46 We just picked 10,000 arbitrary classes while avoiding all subtrees of the WordNet hierarchy that contain an ImageNet class. And when I say ImageNet, I mean ILSVRC 2012. So we make sure we stay away from all these ILSVRC 2012 classes and pick sort of arbitrary classes subject to staying away from that. And then we select images from Lyon simply based on Roberta text-only similarity between the class and the caption.

Starting point is 02:09:13 So you embed the class, you embed the caption, you look at the similarity, you do not look at the image when you select these images. You just looked at captioned text similarity. Okay? And then we implemented some additional safety filters just to make sure that whatever we run our analysis on is safe. Okay?

Starting point is 02:09:32 But the main point here is that there's no annotators involved. There's minimal human intervention. It was largely just based on a web crawl with like very sort of sloppily selected data points. Okay. And in doing this, we actually built on another work that it did with Ali Shirali at UC Berkeley, which is answering the question, what would be different if we recreated ImageNet from Lyon?

Starting point is 02:09:53 What are actually the differences in the datasets that you get? And through some really clever detective display, Ali found out some subtle but very important differences. And we're building on this effort in creating ImageNot. Okay. Just to give you, you know, a sense of ImageNot, a visual sense. How many of you know the Terlba-Fros game of Guess the Dataset? Probably all of you, right? You look at researchers, you let researchers look at two datasets and they have to guess which one it is.

Starting point is 02:10:18 So let's give you some training data. Okay? So here's a class from ImageNot. It's called cleats. It contains some shoes that have cleats, you know, like football shoes, but also it contains lots of images of like just football players that apparently wear shoes with cleats, but you don't actually see the cleats in the image. And then it contains all sorts of other stuff like shoes with bicycle cleats and so on,

Starting point is 02:10:40 right? There's another class called batter. It contains cartoons about baseball batters, but it also contains dietary advice, Like you should always batter your chicken and fry it before you eat it, which I can also recommend. And finally, it contains, let's say, pictures of this cosmetic pen called batter up. Okay. Let's contrast that with ImageNet. ImageNet, as far as it can tell, is mostly dogs.

Starting point is 02:11:05 And so there's, you know, something called an Irish Terrier. It looks like this. And ImageNet has lots of front and center images of Irish terriers, okay? So, I mean, if you know what an Irish terrier is, you can tell this is ImageNet. Even if you don't know what it is, you can sort of tell it's image net because it's kind of front and center of a dog. It's a front and center dog that's image net. Not a front and center dog, that's image not. There's also something called the Blenheim Spaniel.

Starting point is 02:11:29 This looks like that. And again, it's lots of front and center images of Blenheim Spaniels, cute dogs. And there's hundreds of dog breeds or like more than 100 dog breeds in ImageNet. Okay. So what can we learn? Okay, and just to spoil it, we can easily get more than 90% accuracy in telling them apart. So these data sets are really very different. What can we learn from ImageNot?

Starting point is 02:11:49 This to me is quite fascinating. It suggests that something like this might be true. It suggests that the Iron Rule might actually have external validity. So it might say that if you beat the previous best under sufficiently general conditions, it will likely replicate elsewhere. The only thing you need is that your original test conditions were so rich enough. But aside from that, you don't really need anything. The model rankings will replicate in other conditions, assuming your original testing conditions.

Starting point is 02:12:17 were rich enough. And I say it's almost a bit more like a conjecture because it needs more work. And for instance, we need to know what is sufficiently general mean and so on. So there's a lot of interesting work to be done on this. But I find it quite intriguing that there might be this kind of dynamic equivalence

Starting point is 02:12:30 that there's evidence now that ImageNet could have been anything of similar scale. Just from a model ranking and benchmarking perspective, all these data set artifacts that ImageNet had, they may not be all that important, as we thought, and anything of similar scale might have given you the same benchmarking results. Okay. We don't even need clean labels, okay? And if you know anything about ImageNet, you know how much we, as a community, have thought about this annotator step in ImageNet, how important we thought it is that we have annotators, multiple annotators, the agreement rate between them, all these things we thought were essential for the benchmarking enterprise. And I'm here saying that we don't even need clean labels. How could that be? I mean, that sounds almost suspicious. So we wanted to know more, and we wanted to dive deeper into this claim and do some theory about it.

Starting point is 02:13:17 And we studied a model, proposed a model of benchmarking with noisy labels. This is joint work with Florian Dorner, who's in the audience. And we boiled it down to a very simple theoretical question. Okay. So given two binary classifiers, let's say image classifiers, F and G, which one has higher accuracy? Okay, this is where you're trying to find out. This is the sort of essential benchmarking question. Given two models, which one is better?

Starting point is 02:13:41 And here's the model. You can draw unlabel data points X for, Just go on the internet, download an unlabeled data point. And then you can get a label Y for one euro. Okay, for the Americans in the room, one euro is the local currency. Okay, and this is Austria, not Australia, okay? So quick, quick check, okay? Just making sure we're on the same page.

Starting point is 02:14:06 So one euro, and you get your label, okay? But the catch is this label might be incorrect. Okay, so this label might be wrong with probability P less than half. Okay, so there's some chance that this label, is incorrect and is not the correct label. And so how do we best spend our money? Let's say we have N euros and we want to spend our budget on identifying which is the better model. How do we most efficiently, most economically spend our money to maximize the probability of identifying the better model? Okay. Here's a common practice that people would propose. You sample n over k points

Starting point is 02:14:41 where k is some number like three or five or 17. And for each data point that you you sample, okay, you request K labels, Y1, Y2, up until YK. These are noisy labels independently drawn from your annotator process. And you clean these labels by taking a majority vote. Okay, binary labels, majority vote reduces the error rate of your label. Okay, so the label Y that you get by taking the majority vote, you know, you clean it. It has lower error probability than the K labels each have on their own. And so this could be a good way to clean the labels.

Starting point is 02:15:17 And the question now is, well, how larger K should you pick? Should you pick K equal to three or five or 17? What's the optimal K? And what we prove is that in all cases, it's best to sample end data points with one noisy label each. The optimal choice for K is one. You want one noisy label each for one point. Okay?

Starting point is 02:15:41 That's the result. my contribution to this project was preventing Florian from calling the paper all the single labels. Okay. So it's hard being an advisor sometimes. It's a thankless job, but, you know, has to be done. And we got over this. But this is what it says. Okay.

Starting point is 02:15:57 It says that really a single noisy label per data point has been stuff laughing. A single noisy label point per data point is best. Okay. So this is, this is what it says. And so the statement is very easy. the proof is not. Okay, so it was actually kind of a grind. It uses Kromis theorem from the theory of large deviations to get like an exact asymptotic tailbound on the probability of not identifying the better classifier. And it extends to many model comparisons just via the union bound, as you usually

Starting point is 02:16:28 would do it. And for the theoreticians here in the room, this can often be a good alternative to using Havding's bound. So I had always been like naively applying Huffdings bound, which is just an upper tail bound and doesn't give you an exact bound. And this gives you something much stronger. And because this proof gets a little bit subtle, Florian actually found like a really good way to numerically check this conjecture or check the theorem. And this meant we could very easily simulate all the parameter settings. And here's what you get in a typical parameter setting. So as your label budget grows, so does the number of model comparisons that you can make. And as you can see, the number of model comparisons that you can make by the single label strategy is much, much greater than for three labels or five labels and so on.

Starting point is 02:17:09 And it's also much, much better than what you get heuristically from applying Hufftings bound. Okay. So it gives you something much better. And because this is useful independently, Florin created like a sample size calculator that you can check out. If you're creating a data set with noisy labels, this gives you sort of a guide on how to do things. Okay? Great. 30 minutes.

Starting point is 02:17:32 Perfect. Yeah. I'm doing good. So that's sort of my retrospective on the ImageNet era. Let's leave the familiar contours of the ImageNet era and enter the polymorphic era. And I thought it was fitting to try to get generative AI to describe the polymorphic era.

Starting point is 02:17:50 This took me like an afternoon of prompts, basically, but now I'm happy with the results. So this is what the polymorphic era looks like. And here's what I mean by that. So basically large language models and multimodal models in some sense ushered in the end of the ImageNet era. They posed new demands on the benchmarking paradigm.

Starting point is 02:18:08 and we were left from this ImageNet era with some suspicion and concerns about the idea of a single task benchmark. Maybe it was just too narrow-minded to have just a single benchmark or a single task benchmark. Maybe that's not diverse enough. And so in response, people created

Starting point is 02:18:22 a bunch of new multitask benchmarks. They have all these names, super glue, MMLEU, Big Bench, Helm, and so on, with the hope that these multitask benchmarks will provide a more nuanced, holistic evaluation canvas for these new models. Okay? We're also left from this era with like some skepticism about static benchmarks, the idea of just having this one test set frozen in time, and people have been experimenting with the idea of dynamic benchmarks and response.

Starting point is 02:18:50 So benchmarks that evolve over time that grow as you get different models. And so I will talk about each of these in turn, multi-benchmarks, multitask benchmarks first, and then dynamic benchmarks. And I'll give you some new perspectives on each of these. Okay. And so the first thing I'm going to talk about is a social choice perspective on multitask benchmarks, this joint works with Guan Huageng. And we applied basically ideas from social choice to multitask benchmarks based on the following analogy. The analogy is between tasks and voters. So in a benchmark, basically, tasks, different tasks, act like voters. And they can vote on models. So models become candidates. And each task gives you a ranking of all models. Okay. So, tasks of voters, they vote on models and they can rank these models. And if you think about it this way, then a benchmark is nothing other than a voting rule that has to aggregate all these different votes, all these different rankings, into one ranking. Okay, that is the problem of social choice. And it suggests a distinction that's important here between cardinal benchmarks. These are

Starting point is 02:19:56 benchmarks that aggregate numerical scores. Examples are Big Bench, the Open LLLM leaderboard on Hanging Face, and so on, where you just average out accuracy numbers. Those are cardinal benchmarks. And they stand in contrast with ordinal benchmarks. Ordinal benchmarks are just things that use the rankings and aggregate on the basis of the individual rankings. And an excellent example of an ordinal benchmark is Helm, which I'll say more about.

Starting point is 02:20:22 So what does Helm do? This is a fantastic new effort led by Percy Liang at Stanford to holistically evaluate language models. That's what it stands for. And interestingly, Helm is an example. of an ordinal benchmark. Why is that? Because Helm works with what's called the winning rate. So it looks at how often models win against other models on these different tasks. And the winning rate is something you can compute just from individual rankings. If I know the

Starting point is 02:20:49 individual task rankings, I can compute the winning rate for each model, and then I can rank by winning rate. And that makes Helm an ordinal benchmark. You can compute the ranking on Helm from individual rankings only, only ranking information. That's what ordinal means. Okay, so that makes it ordinal. In contrast, the OpenLLLLLLLLB leader board averages out accuracy numbers, and that needs cardinal information, so numerical information to get the ranking. I should say, by the way, all of this, the entire talk is based on these fantastic contributions that the community has made in the benchmarking space. For instance, the OpenLLLLLLM leader board is based on the Eluther evaluation harness. If you ever looked at that code base, it's an enormous effort. It's a huge

Starting point is 02:21:30 amount of work. I'm deeply grateful to all the work that people have done on this, and I really want to give people a shout and encourage that kind of work. It's essential for the community. I really, really appreciate it. Okay, so cardinal benchmarks, ordinal benchmarks. Those are the two things we're going to contrast. And one of the, you know, robust insights from, you know, social choice is that you have no perfect voting rules. Okay. This is associated with arrows impossibility result. That's a famous result in that area. It says, you know, no, voting rule can make you perfectly happy. All voting rules have some issues, and there are certain desirable properties that you can't all have simultaneously. What are these? Here's how I would state

Starting point is 02:22:11 the theorem. I'll state it like this. I'll say that any diverse ordinal voting system is sensitive to irrelevant alternatives. What does diverse mean? I'll tell you in a minute. What is sensitive to relevant alternatives? It means something like a third candidate could enter the race and change the order of the top two contenders. You could have a weak contender, enter the race, and perturb the top contending models. Of course, you don't want this, right? Imagine you upload a weak model to helm

Starting point is 02:22:38 and change the order of the top two models. That would be unfortunate. Diverse just means that your benchmark or your voting system is not a dictatorship, so it doesn't just project onto a single task or a single voter. It's parado efficient. If a candidate wins unanimously in every task,

Starting point is 02:22:53 it should also win overall. And it's universal in that it doesn't limit how you rank. It accepts all rankings. So these are all reasonable things. You kind of want to have this. And then it says you have to have irrelevant alternatives. This theorem applies to ordinal voting systems, and as a result, it directly translates to ordinal benchmarks.

Starting point is 02:23:13 And it says adding irrelevant or weak models to an ordinal benchmark system can change the order of top contending models. But the issue with Arrow's impossibility result is that it's not quantitative. It doesn't tell you how much things could change, and whether that's something practitioners need to worry about. And it only applies to ordinal systems,

Starting point is 02:23:33 so it wouldn't tell you anything about all the cardinal systems that are out there. So what we did in this work is we proposed like an empirical variant of errors impossibility result that implies to benchmarks, applies to benchmarks, and both ordinal and cardinal benchmarks. And the key properties that we identified are sensitivity and diversity. What is sensitivity? It's just change in rankings due to irrelevant task. transformations. Okay. So if you do changes to your tasks that shouldn't at all matter, does the ranking

Starting point is 02:24:04 change? In the case of ordinal benchmarks already told you what that is, it means adding a weak model can flip top contending models. That's the case of an irrelevant change that shouldn't matter. And in the cardinal case, it's just monotone linear transformations of the metric. If you relabel your accuracy numbers from 75 to 80 and 80 to 85, you just do a change, a relabeling of numbers, that shouldn't change, you know, the task. And in fact, every single individual task will be identical under such a transformation. It will give you the same kind of ranking. So it really shouldn't matter. Okay. So these are the relevant task transformations. And sensitivities, how much you change the ranking in response to these irrelevant task transformations. We can always minimize

Starting point is 02:24:47 sensitivity by just having a single task benchmark. Right. If you have a single task benchmark, there's only one ranking. There's no impossibility. Okay. We can always have a single task, or copies of a single task, and you minimize sensitivity. But the whole point of multitask benchmarks is to also have diversity, to have variance in rankings among different tasks. You want these rankings to be not all the same. You want the rankings to have diversity to tell you different things. And we just measure this with something called the Kendall's W coefficient of concordance.

Starting point is 02:25:16 It's a standard measure of diversity or variance in rankings. Okay, and you want to have high diversity to have like a diverse multitask benchmark. And we show in this work that all existing multitask benchmarks exhibit a strong tradeoff between diversity and sensitivity. So if you look at the 2D plot of diversity versus sensitivity, you'll see that all the existing benchmarks, this is the case of cardinal benchmarks, fall between a constant benchmark or on a line between the constant benchmark, which has a single fixed ranking, and a random benchmark, which just has a random ranking. and every, every benchmark strikes a trade-off somewhere between that, okay? And so this says that diversity comes at the cost of sensitivity. If you want more diversity, you're going to have more sensitivity to irrelevant changes. All benchmarks fall on this line, and you can think of this line as sort of a measure of multitaskness.

Starting point is 02:26:08 There's really just one dimension here. How much multitaskness do you want? And the more multitaskness you want, the more diversity you get, but also the more sensitivity you get. and you can sort of have high diversity without high sensitivity. As a sanity check, this was important. As a sanity check, we partitioned ImageNet into a mock-up multitask benchmark by just partitioning the classes into 20 random tasks and calling each partition or each part in the partition a different task.

Starting point is 02:26:36 So we're creating a fake multitask benchmark. And this is what that dot that says ImageNet and the slides is, represents. It's just what happens if you create a fake multitask benchmark. and our measures correctly identify that this is a single task benchmark. So it's not better than a, it has no diversity and no sensitivity. Okay, it's still just a single task benchmark. But the main takeaway from this is that diversity comes at the cost of sensitivity. There's no free lunch in multitask benchmarks.

Starting point is 02:27:02 You can make these benchmarks more diverse, but it's going to come at the cost of having very high sensitivity to relevant changes. Just to give you a different measure of sensitivity, on the left, we measured it in terms of Kendall's tau. That's like a measure of like difference in ranking. On the right panel, we measure it in terms of the maximum normalized rank change. Okay, so this is the fraction of ranks you can skip due to an irrelevant change. And you can see that, for instance, for Big Bench Heart, you can skip 80% of the ranks

Starting point is 02:27:31 by some irrelevant transformation of the metric. And for MMLU is close to all ranks and so on. Okay, so you can really have very significant changes in the rankings with an irrelevant task transformation. For ordinal benchmarks, the situation looks similar, although a little bit more messy. Here we look at various subcategories of Helm and Hymn, and you see also that there's this general trade-off. I should say that we always compute a lower bound

Starting point is 02:27:57 on sensitivity, because it's kind of hard to compute exactly, so all these numbers are lower bounds. The sensitivity might always be higher. And so we're seeing a similar picture here, a little bit more messy. Here's an illustration of what this actually looks like, the sensitivity to irrelevant changes. Here is how much you can perturbate

Starting point is 02:28:15 the rankings on these benchmarks. On the left is open LLM. On the right is Helm, by just doing irrelevant task transformations. Okay? And you see that you can, the models can jump around quite a bit, and these rankings can have quite a lot of sensitivity.

Starting point is 02:28:29 If you want to play around with this, you know, you can hip-install bench-bench, you know, and, you know, play around with all these numbers. It makes it very easy to just load up your favorite benchmark, compute the diversity and sensitivity, and see where the benchmark falls, okay? We're also maintaining a website where we add these things and sort of display them just to keep track of them. And if you want to contribute to that, please send us your benchmark, make us aware of your benchmark.

Starting point is 02:28:55 We would love to edit. Again, there's a little bit of computation involved in getting these numbers. We're happy to run that computation for you. Just reach out to us and we'll be happy to add this. So this is an ongoing work in progress. All right. This is my take on multitask benchmarks. And with the last several minutes, I want to talk a little bit about dynamic.

Starting point is 02:29:15 Benchmarks, which is another proposal of the polymorphic era that's gained quite a lot of traction. There was a really amazing and fascinating effort called Dinah Bench a few years ago, which was very ambitious. It was trying to really change the way we do benchmarking by proposing something called dynamic benchmarks and a platform to do these kind of dynamic benchmarks. And so a dynamic benchmark basically is as an evolving time-dependent benchmark where you have some initial data set. you let people build models on that initial data set, and then you use the models that people have built to find failure cases of all the existing models,

Starting point is 02:29:51 and you add those failure cases to your benchmark. This is called adversarial data collection. So you learn from the models that were built, you add the failure cases to your data set, and you continue. And so there's this ongoing interaction between model builders and data collection. And you interleave these two operations in definitely hoping that your models

Starting point is 02:30:12 just keep getting better and better and better and keep accounting for more and more challenging instances. We wanted to know if this works, and there's something I did in joint work with Ali Shirali and Redi Adeabe, where we proposed a theory of dynamic benchmarks, because it's so new and so different, we wanted to know, can this at all work, and how do we even think about what it means for that to work? And this is what we proposed in that paper, and we abstract a dynamic benchmark as a directed acyclic graph with four operations.

Starting point is 02:30:41 So each node in the graph is one of the ones. of the following four operations. You can do model building. You can invoke the community to do model building. You can look at the resulting models, ensemble them, or in some way, collect them. And then you do data collection on the resulting model. You let the annotators find failure cases,

Starting point is 02:31:00 and you create new data points, and you add them to your data set. So you pool data. These are the four operations in a dynamic benchmark. And subject to that, you have complete freedom. You could do whatever you want. Any directed acyclic graph is. is a valid dynamic benchmark. The standard design that people have mostly implemented

Starting point is 02:31:18 is the following is just a directed path. It's just a directed path that alternates between model building and adversarial data collection. So just these two operations alternate them. That's the standard design that people have mostly experimented with. And we prove a theorem that says that progress in the standard design

Starting point is 02:31:36 can stall after a small number of rounds. Okay, so there's in principle no reason to expect that the standard design gives you progress beyond a few number of rounds. And you can see this in some of the experiments that progress really seems to plateau after just a number, a few number of rounds, and it becomes diminishing after that. And so we thought harder about this, and we came up with more sophisticated benchmark designs that guarantee strictly more progress. So we call them hierarchical dynamic benchmarks, where we sort of have parallel threads

Starting point is 02:32:07 that you let run and merge in some particular ways. So these are more complicated benchmarks. We can prove that they guarantee more progress than the standard design, but they're also much harder to implement. So I'm actually not sure how feasible it would be to pull this off as a real benchmark, but it's certainly an intriguing possibility. Okay. So this is my take on dynamic benchmarks. So let me sum up and get to an end. How am I doing on time?

Starting point is 02:32:34 Great. Perfect. All right. So summing up here. What do we see? I use the benefit of hindsight to sort of do an ImageNet era retrospective. What did we learn from like 10 years of benchmarking on ImageNet? And the takeaway here, the main takeaway here is that the Iron Rule has both internal and external validity. So it works to a surprising extent in this case.

Starting point is 02:32:57 But we know much more about the former and much less about the latter. So if you're interesting and just in open problems, understanding this last part is really challenging. And we have made very little progress on that. So there's definitely more work needed here. And then maybe to me, what was really surprising is that good human, like, annotated, highly curated data is not necessarily required for ranking models by accuracy. So if your goal is just to do rankings, performance rankings by accuracy, you don't necessarily need super clean data. Okay, we can, we saw this empirically, but also we can do some theory about that. And this was surprising to me.

Starting point is 02:33:33 I should qualify that, of course, if you're doing something like a fairness analysis, a safety analysis, bias analysis, you know, red teaming or alignment analysis, this is, of course, very different. Then your test cases and your data really matter substantively, and what you put in there is very important. And my group certainly very invested in these research directions as well, but that's not what I talked about. In this talk, I was focused on the core sort of benchmarking enterprise and ranking models by accuracy. Okay.

Starting point is 02:34:01 Then I moved on to this new era where we don't have hindsight. we're sort of figuring it out as we go along and we're seeing what happens. And here we saw that there's a strong tradeoff for multitask benchmarks. And diversity, greater diversity in multitask benchmarks inherently comes at the cost of less stability. So you have more sensitivity to changes that shouldn't matter. And there's no free lunch for this kind of evaluation paradigm. And finally, dynamic benchmarks are intriguing. I personally find them extremely fascinating and intriguing. But currently, I don't think we quite know how to pull them off. And as we currently do it, progress might stall.

Starting point is 02:34:39 And again, if you're looking for good research directions, it's a fascinating area to work on. And I encourage, you know, especially the theoreticians, to look at some of these benchmarking questions, where the landscape is wide open, okay? It's really, like, completely open space. Good. So this talk was about the emerging signs of benchmarks.

Starting point is 02:34:58 And I argued that machine learning is the anything ghost principle plus the iron rule. And somehow these two basic ingredients give us like a very powerful scientific machinery that seems to work. Okay. And I'd say that this community is extremely good at the anything goes part. Okay. So we've been extremely good at the anything goes. But that places all the burden on the second part, the iron rule.

Starting point is 02:35:20 And that becomes sort of the critical link here because it's really essential for making all of this work. And so what's somewhat challenging here is that our intuition about benchmarks can fail and often has failed. And certainly my own intuition about benchmarks has very often been wrong, and I sort of was corrected by the passage of time or theoretical results that overturned my intuition. And so this has convinced me that we really need sort of scientific foundations of the iron rule itself, of like how do we make sense of this benchmarking enterprise, and how do we build things that sort of promote scientific progress. So I really think of this as like a major theoretical and empirical effort.

Starting point is 02:35:56 It's not just a theory thing. It's also just an empirical effort to understand what, collective practices of our community promote scientific progress. I think this is what this is ultimately about, and we should devote some more time and attention to that. And so I hope you'll join this effort. I hope you found something interesting in this talk. Please come chat with us. I'll give a shout out to the social foundations team. They are actually wearing the yellow sweaters today. I didn't think so because it's warm, but if you see us somewhere in the conference, please reach out and we'll be happy to chat. With that, I'll thank you all for your attention.

Starting point is 02:36:31 That brings us to the end of Section B, our selection of three papers and one keynote on benchmarking. Phew, that was a long but important topic, and if the AI Engineer Conference submissions were anything to go by, the topic of evils and benchmarking is only going to explode this year. We turn now to Section C, which covers incremental papers and talks in reasoning and other post-training elements. There were many, many more papers than we could fit in this category, in this category, but we will focus on a few important themes, rag, verification and safety. First, let's start with the self-rag paper. This paper bears some similarity to the Paws Tokens paper we covered in ICLR Part 1,

Starting point is 02:37:17 because it involves adding special retrieval and critique tokens. However, these tokens don't just improve raw reasoning. They explicitly support retrieval during the generation process, as well as evaluating source relevance, degree of statement support, and response utility. And they can be fine-tuned on top of pre-trained open models like Lama. Hi everyone, I'm a colleague from University of Washington. I'm excited to present our set-of-rug, which is a new framework to improve the standard RUG system by training and making an any language model to decide when to be-tube, generate, and self-operate.

Starting point is 02:37:56 Large language models are powerful, but there are many issues like hostages. nations, retrieval augmented generation or RUKHAG has shown to be quite effective to overcome those issues. Given the user's query, such as where is I-Clear 2024, we first retrieve a set of documents using a retrieval system like Google search or BM25, and then we augment the original language model input using those retrieval documents and also the original question. The standard state-observed GPT4 or Llamatri can use those retrieval documents and give more factual, up-to-date, and attributable answer. Rugg has shown to be quite effective in many benchmarks, especially in question-answering task. In a player work, we conducted the large-scale analysis of 10 different language models, and we have seen that this inference time augmentation can give us significant improvement across many models ranging from 1.3 billion to 100. 75 billion.

Starting point is 02:38:54 RUG has been used in many rear-word applications. For example, there are many language model-based search system, PAPEXC.AI, or BingChad, or libraries that help you to build customized RUG pipeline. While RUG is super effective, there are many limitations. In this talk, I would like to highlight two of their limitations, unreliability and also inefficiency. Now let's ask another simple question. How did US states get their names?

Starting point is 02:39:21 Unlike the previous question, where you can simply extract local information given documents, in this question, you have to collect a set of documents and compose output based on those multiple documents. In standard language model, as in the previous example, we can concatenate those readable documents and then fill them together in language model. The output look plausible, but there are many factual errors here. So now let's talk about why, let's explain, why those standard lag system may be, may not be perfect yet. First, even the current sort of systems can easily get distracted by when many documents are given,

Starting point is 02:40:00 especially when some of the documents are irrelevant or unhelpful. Let's take a closer look at this example. The middle paragraph only states the history of Michigan without saying anything about how this state got their name. Current duct system can easily affect it by those unhelpful documents and can generate factory incorrect statement as in this example. Due to this unhelpful document, the model says states such as New York and Michigan are named after an individual person, which is factually incorrect. Or conversely, current drug system can ignore the context that provide useful information.

Starting point is 02:40:33 Here, the first paragraph provides useful information, but the model can still generate hallucination. For example, it can say some states including Utah and Washington and named after indigenous communities. Another issue is that many of the existing RAC pipeline assume retrieval is almost always necessary and keep retrieving a fixed number of documents, even when the user query isn't using retrieval. It is estimated more than 60% of the user input into chat GPT is for writing assistant or creative writing, which may not require factual grounding. Like this query, write an essay about USMovocation. And as you can see, retrieving a set of documents talking about

Starting point is 02:41:14 about definition of summer application or movie title sum application doesn't make any sense here, but most of the standard bug pipeline just retrieve a set of document as in other queries. So this codes the additional latency making the bug system more inefficient, and also it can hurt the model output as shown in the previous slide.

Starting point is 02:41:35 In this work, we introduced a new framework set of recitative generation, a retrieval augmented generation of self-frag. So farg introduced a novel inference and training pipeline to enhance the reliability, efficiency, and the security of Rugg systems. Instead of fact, inference, at a higher level, we make a language model to decide when to retrieve

Starting point is 02:41:54 and generate and evaluate its own output. In particular, given a user query, the model first evaluate if we needed to use Vitiable note. If we don't need a retrieval, then the model just act as a standard language model to avoid unnecessary retrieval. If the model think we needed to retrieve, then the model retrieve a set of documents, But instead of concatenate everything

Starting point is 02:42:15 and feed them together in input space, we first us the model to evaluate which documents are actually helpful. And then the model would generate output, but the model further also evaluate if the output is supported by those helpful citation and out. And at the inference time, we prioritize output that are fully supported by the context.

Starting point is 02:42:34 To achieve this iterative set of corrective pipeline, we conducted one of the RDS instruction tuning with the retrieval and trained an arbitrary language model with those set of free correction. tokens. So now let's dive into the details of CETOFRAG inference and using the same example. Given the same question, how did US state get the names? CETFRAG can even start directly search generation to say like US state got their names from variety of sources. But now CETAG needed to generate the factual statements composing multiple documents. So the model

Starting point is 02:43:07 outputs a special command retrieve and then we trigger VITBO and retrieve a set of documents from the data store. Instead of concatenating everything and feeding them together to language model, in cellfrag, we process multiple documents in parallel using batch decoding. This enables more efficient and scalable inference, but more importantly, this enables us to control models behavior via set-free collection. Specifically, we make the set-frag to predict generate condition on a user input and also each paragraph, and for each paragraph, the the model first predict if the paragraph is related to the question or not.

Starting point is 02:43:47 Then the model keeps generating the output as in standard language model. In the first example, the model that provided paragraph is helpful, while in the second example, the paragraph only states the history of Michigan. So the model first generate irrelevant token and then keep generating. After those generations, CERFag also predict its output is fully supported by the related paragraph. For the first generation, model output is fully supported by the document one, saying that 11 states got the name from individual person. Well, for the last case, the paragraph only talks about Utah, while the model output mentioned

Starting point is 02:44:23 Utah and Alabama. So this output is only personally supported. In that case, the model predict personally supported. So how do we achieve this retrieval and self-refective feature? We equip language model with those abilities by making the model to learn to predict those special tokens, which we call refection tokens. In standard language model, we have a fixed set of vocabulary, and the model assigns the highest token probability to the most plausible next token.

Starting point is 02:44:51 In the set of fact, we expanded the original vocabulary using the special tokens, which includes the token controlling RetiBow or the token controlling the self-reflective feature, critic tokens. So if the model think we needed to use Retibow, then the model will generate the RetiBel token from the expanded vocabulary. We can also adjust the threshold to balance, or decide how frequently we use retrieval to balance the trade-off between retrieval frequency

Starting point is 02:45:19 or efficiency and final performance. Critic tokens are also directly generated by the model, and we use those models to rank and choose the best K output at the sentence level. In particular, in SETFRAG, we conduct a sentence normal beam search using a fine-grained feedback scores based on those critic tokens. We use the normalized token probabilities of ideal critic tokens

Starting point is 02:45:44 such as a Viband or supported as a fine-grained feedback, and then compute the weighted sum of them as a summary score of each output. Here in this example, the first output is based on the helpful document and also is fully supported. So we give the highest score to this first output. Well, for the middle one, the termination is based on an helpful document.

Starting point is 02:46:07 So we assign the lowest score because the output is more likely to include factor errors. Now let's discuss how we train setafag so that the model can effectively learn what to generate, when to generate such refraction tokens. In Setafag, we train an obituary language model to generate seamlessly from both normal vocabulary and also special tokens.

Starting point is 02:46:30 During training, we use another language model, which we called critical language model. And this critical language model teaches the generator language model, learn to generate a language model, learn to generate appropriate set-free pre-collective tokens under input. Critical language model is a seven video language model

Starting point is 02:46:45 trained to generate equations tokens given evaluation instruction and input. For instance, here the evaluation instructions say that evaluate its output y and to an input X is supported by the table document D and also the set of input X, D, and Y. Here the critical language model should predict the supported token. One challenge is that we do not have large-scale fine-grained annotations for necessity of retrieval or self-critic,

Starting point is 02:47:14 and collecting human annotation at sentence level for multiple aspects could be quite expensive. To overcome those challenges, we generate synthetic training data by carefully prompting GPT4 with instruction and demonstrations. After this, we train critical language model on this generated training data, and then we use this critical language model and retrieval model to augment existing instruction training data. mimicking set flag inference. Here, given an input output pair, we first run critical language model to evaluate if we should retrieve,

Starting point is 02:47:47 and then if the model output retrieves token, then we insert a retrieved passage inline was detection tokens, also predicted by the critical language model. We generated 150,000 instruction training data in this way. Now we can simply train the generator language model in this augmented training data. So given the input, the model runs to generate the standard output and also refraction tokens. We can use the standard language model training objective with expanded vocabulary. This enables us to easily apply the same training pipeline to new language model and also a code basis and also enables us to tailor models behavior to diverse fine-grained preferences,

Starting point is 02:48:29 as shown in the previous slide, without additional training overhead. We abutting set up on six diverse tasks, including short-home closed-tasked, close-set task, short-home generation, and also long-home generation. Our set-of-work is based on Lama 2-7 and 13B and trained in four GPUs. Now let's discuss the result. As you can see, the baseline parametric language model, without any retrieval struggle in tasks requiring precise knowledge memorization,

Starting point is 02:48:59 such as PopQA and Open Domain QA dataset, or ASQA, which is a long-home QA data set about factual knowledge. The standard Rugg gives improvement on some tasks like POPQA, where we can simply extract information from single document to answer. However, standard Ruggs still struggle, or it doesn't give large performance improvement on tasks that demand systems to compose knowledge from multiple documents and generate, such as Pub House or ASQA here.

Starting point is 02:49:27 Moreover, especially open access language models, such as Lama 2, 13B, chat, pre-trained, or chat model struggle to obtain good citation precision or recall, indicating that even the model cite something, the model output is likely to be not supported by those citations. CERFLAG significantly improve out-perform such models and obtain the best performance across all of the best line in the same model scale. Moreover, CERFrag shows much better citation precision than be-code.

Starting point is 02:50:02 CERFLAG even matches or out-per-home chat GPT on five out of six tasks, despite being relatively small 7 or 13 billion model trained on a small set of instruction tuning data. CERFAR was initially introduced last October, and since then, CERFAC has been widely used in both academic papers and also industry applications. For example, CERFAC has been successfully integrated into multiple wider-use RAC libraries, such as Longchern or Lama Index, and also there are many papers that tries to improve CERFAC for further reliability, or applies CERFARG to new domains, especially safety critical domains like biomedical. domains. In summary, we introduce a new certified framework, which helps us to build a more reliable,

Starting point is 02:50:45 efficient and versatile rag systems. We open source the code and model checkpoint, and I'm happy to answer any question now. And also, please come to our poster in 63. Thank you. The self-rag paper mentions training three models, a retriever, a critic, and a generator model. The role of the critic model and the critique tokens is surprisingly similar to the idea. ideas of this next paper from OpenAI, let's verify step by step. Here they pursue more formally verifiable correctness in math problems and find that process supervision, which is what Self-Rag is doing, soundly beats outcome supervision. The high-level goal here is to train really reliable reward models to grade different math

Starting point is 02:51:32 problems. We want to have reward models that can look over a huge number of solutions and pick out the one that actually managed to solve the problem correctly. And so we're comparing two different methods, what we call process supervision and outcome supervision. Outcome supervision basically means giving feedback to the model just based on whether it reached the correct or incorrect answer at the end. And process supervision is giving granular feedback for each individual step of the problem

Starting point is 02:51:59 and specifically saying whether each step is correct or incorrect. And the hope of the process supervision is that it's solving a hard credit assignment problem that outcome supervision would have to solve by basically. directly giving it feedback on whether each step is correct or incorrect. So, you know, outcome supervision has to like somehow infer where the bad step happened, but with cross-subs supervision, we get humans to just directly specify where that happens. So we collect a huge amount of data from contractors who just are going through and labeling each individual step.

Starting point is 02:52:29 And then we use that to train the process supervised reward model. And the outcome supervisor award model, we just trained to predict the correctness of the final answer. And then we compare these two reward models. and we find that the process supervised run is significantly more reliable. And it's specifically on the math domain? Specifically math, yeah. How often do you find that the reward model is wrong, and it's like guiding it off track rather than on track?

Starting point is 02:52:54 We are using the same generator model throughout this, so we're not actually, we don't actually ever update the generator, we're just using the reward models to basically search and test time to search over a large number of solutions. So, I mean, both the outcome and process supervised reward models are imperveraged, and get tripped up by things. I think the outcome supervised reward model gets basically, I would say, pays a little bit less attention to detail.

Starting point is 02:53:18 It's more likely that a solution that looks superficially good but has some subtle mistakes will still be rated highly by the outcome supervisor reward model, whereas the process supervised reward model is a bit better. It's spotting subtle errors, I would say. But both of them can sort of both of them can say can get fooled by a solution that kind of looks like it's good, but isn't quite right.

Starting point is 02:53:38 What is the backstory behind this? Why is this an interesting area of research? Or were you actually investigating something else and you found this to be a blocker for you? So yeah, broadly speaking, we're just interested in pushing on the reasoning abilities of these large language models. Reasoning is something that LLM's have struggled with a lot,

Starting point is 02:53:55 and we, so we're just excited to push back that frontier however we can. Do you have a definition of reasoning beyond just like this, math, step-by-step reasoning? I think it's hard to give like a, yeah, a great definition to what, like, defines reasoning. I mean, just it's just things that it's kind of whatever passes our vibe checks, but obviously things like math and STEM and code, but yeah, I don't have a good like formal

Starting point is 02:54:18 definition of reasoning, I guess. I noticed the search space was actually pretty big. You had like thousands of solutions. Yeah. Per problem. Was that necessary? Well, it's useful to show that, you know, if we only took like a hundred solutions here, we wouldn't see as much of a difference between outcome and process supervision. But if we take, you know, many thousands, we see much more of a difference and yeah I mean it's useful to to explore this regime because you know if all it took to solve really really hard problems was taking a huge number of samples then we'd be very happy you know we'd be willing to spend the compute to to generate all these solutions if if we could always pick out the one that was

Starting point is 02:54:52 like really good and so yeah so it's not like strictly necessary to go here but it just makes it clear sort of what what the trends look like how do you envision other people should use this once you have a really good supervised reward model at like a large larger scale you can use it to explore labeling for smaller scale models? Like to bootstrap or teach? Yeah. Okay.

Starting point is 02:55:11 And so what we did here is we kind of, we were able to do some ablations on outcome and process supervision by using GPD4 to label these smaller model samples. And here we collected basically at like order of magnitude more human data than we did at the large scale and by like having just having the larger model provide the labels. And those kind of experiments wouldn't have been possible if we had to actually rely on humans. That's that's one possible use for the data set. really said also just like if anyone else is able to train these process supervised reward models and maybe see something we missed well that's

Starting point is 02:55:44 pretty positive I think for the community like what's the relevant literature like what's the inspiration I guess or relevant literature for process supervised reward models like is there is there anything that you point people to to read up if from an alignment perspective some people are excited about process supervision just because you know it's sort of outcome supervision is like less direct feedback and so it's you know potentially not as good from an line perspective you we you don't really know what you're reinforcing if you just give the model like a positive or negative reward at the end of some long

Starting point is 02:56:18 trajectory but with process supervision you know you're kind of much more directly reinforcing the things you care about there's I mean there's there's there's something like blog posts and stuff that we that we cite in the paper but don't have anything but like specifically call out this seems like the the inverse of this is weak to strong generalization is there any parallels with Here is kind of strong to weak, and then you started working on week to strong. Strong to weak in the sense of experiments I was talking about here. Yeah, I mean, I'm sure there are connections.

Starting point is 02:56:44 We didn't. Okay, so you can take like a small process supervised reward model and try and have it evaluate like larger model samples, and it'll work to some extent. But yeah, I mean, that's definitely something you could look at more. We didn't push very hard in that direction, though. I was just kind of curious if there's some kind of parallel in there. Cool. I think that's it. Thank you.

Starting point is 02:57:03 Great, thanks. If you listen closely, OpenAI's focus on verification and process supervision as a path toward better reasoning, aka GPT-5, is by now clear as day. In the closing workshops of ICLR, we even managed to catch a little bit of Gn Brown's talk expanding on what he calls the generator verifier gap. We last discussed his work with regards to our code interpreter is GPT 4.5 post from a year ago. He now draws a lot of inspiration from the AlphaGo paper, and in short, suggests that the best path to improve reasoning in generative AI if we have sufficiently good verifiers of their output, preferably in process as suggested by verify step by step. Here's an incomplete snippet of the talk leading into his discussion of the verify step-by-step paper. Okay, so how can we take advantage of generating verifier gaps and take this? Sorry, before I get to that, one reasonable question that I'm sure a lot of, I've been asked a lot, I'm sure a lot of you are wondering right now, is, okay, if we have this generator, if we have this really good verifier, we're able to, like, generate a bunch of solutions and filter up the ones that aren't good and build a one that is good, can we just, like, update the generator with the verified solutions.

Starting point is 02:58:18 And in principle, this is a good idea, and it's actually what Al-Fa-Go does. That's, like, how Al-Fa-Go is trained. That said, there's a couple cave-assies. It doesn't capture all the benefits of these get verifier. So, for example, if I go back to, to... would be AlphaGo slide. So you can see, like, the gray bar is the performance of AlphaGo Zero if it was trained with multiple-fellage research and then doesn't use multiple-galge research at test-up. So it's updating the generator with the verified outcomes of multiple-filege research, but it's just not using a test-back. And you can see that there's still this massive ELO gap.

Starting point is 02:58:51 You know, it's 3,000 versus 50200. So you're still getting a huge performance improvement by using this verification technique at test-back. So just relying on updating the generator is not going to be email to overcome this gap. And the second thing is that you're still bottlenecked by the quality of the verifier. That's something that you can't use the get around. So I think for the purpose of this talk, I'll say that updating the generator is just going to be out of scope for this talk, and I just want to focus on, let's say we have a fixed generator and we have the verifier, what can we do?

Starting point is 02:59:24 Okay, so the first thing that's really important we can do is called consensus. And in consensus, you just generate a bunch of solutions and take the one that's the most common. It varies before. It's actually really convenient because you don't even need a verifier for this technique. You kind of think of this as similar to sampling at low temperature, but it's not exactly the same because, you know, if you have a sequence, then you're not sampling like every single step of that seasons at the temperature. You're sampling like the final chunk of low temperature. And I think a lot of people underestimate just how much benefit you can get from things like let's census. So for example, some of you might have heard about the Minerva paper, this came out about two years ago,

Starting point is 03:00:03 and they got up to over 50% on the math benchmark, which is a very difficult math benchmark. That's why they used today. But Minerva got over 50% in large part due to using consensus. They generated 1,000 samples, and then they took the most common answer from those thousand. And that part of doing this consensus got them from 33.6% after receipt to 50.3% out. So that's a big jump. Something else you can do is best to that. So best of that, the idea is you just sample all the solutions and then you score them with a reward model.

Starting point is 03:00:39 And you return the one that looks at this. And the key here is that you have to have a good enough reward model. If you don't, you're not going to beat consensus, but if you do, you can actually get a big idea for consensus. And ultimately this technique is like to buy the quality of your reward model. If your reward model is not very good and you take a lot of samples, so then you're going to end up overfitting to the errors in your own way. to the areas in your normal model. So here, for example, this is a figure from a paper that came out in 2021.

Starting point is 03:01:04 On the x-axis you have, so this is testing on the GSM 8K data set. And you can see on the y-axis you have the accuracy, the pass rate, and on the x-axis you have the number of samples you've invested in. So, for example, if it says like 100, it means that you're taking 100 samples, 100 generations from the generator, and then you're feeding those into a verifier that's trained to tell the toll of it. tell up whether it answers correct or incorrect, and taking the one that the verifier thinks

Starting point is 03:01:31 most likely be correct. Now if you're taking up to 400 samples, you're seeing actually a pretty large improvement. I mean, this isn't actually each model, so if you see any improvement at all, so it's quite significant. But you are seeing a substantial improvement as you go up to 400, but then past 400,

Starting point is 03:01:46 you're actually seeing the performance degrade because you end up overfitting. If there's errors in this verifier, and if you take too many samples and ask it to score which one is best, It could actually just return one that's wrong, but hacks the verifier, basically. Okay, so can we do better than these make this? So one of the things I wanted to talk about is process reward models.

Starting point is 03:02:10 This is a paper that's published at this conference. We actually put on an archive about a year ago. And actually, I was not on this paper. This was with my teammates. My teammates with this paper before I ended up joining the team. But I thought it would be a good thing to illustrate that there's, like, a lot that can be done in the space, relying on verifiers to do with your performance. So the basic idea of process reward models over outcome reward models,

Starting point is 03:02:35 or simply just doing Best Event, is that we're going to verify every step individually rather than verify the entire set. So for example, we have this question, X4 equals 4, what's X? So typically if you look at how Best of it was done with verifiers in the past, it would just like generate the whole solution, and then you would ask the verifier, is this whole solution correct? And that puts a lot of burden on the verifier to have to have to, have to like consider the entire solution all at once.

Starting point is 03:03:04 And with process reward models, you instead break it down by steps and you ask the verifier, is this individual step correct? And some steps might be correct, some steps might be incorrect. You can see here, for example, it's like going from x squared equals 4, what's x, well if you, if your next step is x plus x equals 4, then it's going to recognize that that's incorrect and correct and correct. And so then the process reward model will take all those steps into consideration, all the scores of all those steps into consideration,

Starting point is 03:03:29 when it's scoring the entire sample. Now the way we train this model is by collecting a lot of human data. So we had a bunch of human annotators that would go through generations of the model and label each step as either correct or incorrect. So here, for example, there's a question, and then, you know, there's a bunch of steps. The first step is let's call the numerator x, and the next step is the nominaries 3x minus 7. And the human annotator would go through each of those steps and give it like the green smile and basically it was a correct statement and a red frowning face would be it was incorrect given what's happening before.

Starting point is 03:04:09 So for example, you can see in the last step we go from 5x equals 6x minus 14 to so X plus and that step is incorrect so it's a part of the correct. There's also this option for a neutral base which basically means that it is not exactly incorrect but it doesn't really like do anything. It's just like a step that's kind of like a holding pattern. Okay, so how does the performance look of outcome versus process supervision? So the baseline that we compared against is using just like a verifier that takes the entire sample all at once. And we call these outcome reward models. So we trained the outcome reward model on a giant training set of, so the math data set has a training set and has a test set.

Starting point is 03:04:54 We actually moved a lot of the test set problems into the training set, and we basically made the test set small. reduce the test set to only 500 problems, then we can have more problems to train on. And this actually does pretty well. So if you do best to end with this outcome of word model that was trained on the math data set, you can get GPD4 to correctly answer 72.4% of the test set, correct. And that's better than consensus. Consensus is a technique that I mentioned before that does rely on a verifier at all.

Starting point is 03:05:21 You just take a bunch of samples and see which ones that is common. Consensus gets you to 69.6%. For the process rule model, we collected a million step-level labels. across 100,000 solutions. And this actually ended up doing a lot better. This ended up getting to 78.2% on math. And I think this is still either CitiR or close to state of the art for math.

Starting point is 03:05:45 Now, one of the reasons why PRMs provided are so good is because they provide a denser verification signal. It's really difficult if you have a difficult math problem to verify whether the whole thing is correct. And it's certainly difficult when you're just to find a answer. But being able to look at each of these steps individually, it gives you a denser verification seal that makes it easier to verify. So this is what the plot looks like. The gray line is consensus, also called majority voting.

Starting point is 03:06:18 The blue line is dot-com supervision, so that's the verifier that takes the whole sample, the whole trajectory, into consideration. And then the orange line is processed and provides more models. So on the y-axis we have the pass rate, the success rate on math, and on the x-axis we have the number of solutions per problem. And you can see also that like the performance continues to improve as we increase the number of solutions for a problem. It doesn't, like we sampled up to, I think, 1600 for this paper, but it looks like if you just keep going further, the number keeps going up. Eventually that's going to plateau, you know, at some point it's unclear at what point in the point is. So here's samples. The print is a little small, but basically there's this problem,

Starting point is 03:07:05 and then the verifier goes through each of the steps, and the more green it is, the more it is the verifier thinks it's correct. And on the left we have, if you just use the Alcimbole model, so that's like if it's grading the entire sample, and in that case it incorrectly labels it as, correct? And on the right, we have the PRRR. PRRRM. And you can see that there's like some steps where it's like a little fishy. So for example, the second step, it's like saying it's probably correct, but like, you know, it could be better.

Starting point is 03:07:34 And then there's like certain steps where it's like very confident that those are incorrect and those are in red. And so it's able to recognize the final answer is incorrect for this reason. We apologize for the poor audio conditions of the clip as it reflects the less than optimal nature of the workshop room. His summary makes a lot more sense with visual aid reference to the let's verify step-by-step paper. and we have included the two relevant charts in the show notes for his talk. Adjacent to the topic of models grading other models, we have another talk from an OpenAI team lead, Lillian Weng, Head of Safety Systems at OpenAI.

Starting point is 03:08:11 She gave a fantastic overview of the often overlooked safety-oriented mitigations that OpenAI does for their models across four stages, pre-training, post-training, inference and evaluation. We would particularly highlight the OpenAI model spec, which lists objectives, rules and defaults that her team designs for, and the instruction hierarchy, a recently published paper that details how Open AI defines five privilege levels that are they then fine-tune for resolving conflicting instructions. Thanks, everyone, and thanks for inviting me here. I will say my talk today might be slightly different from others because I would like to give a higher. level overview of how we're building the safety system for deploying, like cutting edge

Starting point is 03:09:02 the best deep learning models in the world end-to-end. So I probably touch on a lot of different things a little bit on the surface because it's hard to go deep into every step. I hope you can have a concept of how complicated the problem is, also how much possible methods and mitigation you can apply when you're facing the real world challenges. A bit about myself. I joined Open AI about more than six years ago. I initially work on robotics and then apply research. And recently, I started leading this new team called Safety System.

Starting point is 03:09:39 We're only into end safety stack at OpenEI. So basically, all the model that deployed to the real world will go through us. Our team is dedicated to ensure the safety robustness and reliability of AI models. and their deployment in the real world. So you can think of all our, we're dealing with a lot of practical safety alignment issues. There's endless collection of problems. But the good thing is we have access to different stack in the system, so it can be pretty creative. Safety is at the very core to open eyes mission.

Starting point is 03:10:14 If you look at the mission, our charter online, you will find our goals we want to build, safe and beneficial AGI and also deploy that to the real world to benefit humanity. I know this is a very big statement, so I hope after my talk you will get some concrete ideas of how the process is. And one important point my team believe in is safe AI cannot be built in the lab and how to deploy a powerful model and essentially AI needs consistent learning, improvement, researching the world, real world. So we really embrace the idea of iterative deployment in the process because I often find that adversarial in the real world are so much more creative than our researchers. I will say

Starting point is 03:11:05 different people may have different interpretation of what is safety. So I want to talk a bit about the concept and what's the goal of we try to achieve here first. First of all, we expect a model output to not content any harm to people, including physical harm, mental, financial, or reputation harm. We hope the model is trustworthy, inclusive, and also respect privacy of people. At OpenEye, we want to build products that support beneficial use that are being said for certain harmful requests, even that has utility values, we train the model to refuse those requests. And there should exist mitigation, either within the model or in the system around the model, for adversarial use cases.

Starting point is 03:11:56 For example, some people use the model to enable fraud, scam, or do episode or persuasion of people. This kind of use cases might not be easily identified if you only look at the model, like a single conversation based on the model inputs and outputs. So we do need a system level monitoring, our mitigation to identify that. Also, we want to make sure our model is, or our system is robust and reliable, even when people try to adversarial attack it. I want to mention Goody too, because this is probably the most interesting model I run into lately. When people think about safety are responsible and models, you can go to very extreme. You can make the model 100% safe,

Starting point is 03:12:47 but it's useless. In this case, I found this model is extremely robust and I cannot really make the model to answer any of my question. It will always find some way to say, your request has concerns, and I will just not answer that. But if you only look at the scale or measurement of how safe this model is, it's perfect, but it's useless. So I really want to emphasize that it's important to evaluate capabilities. and safety at the same time and try to balance how useful it is and how safe it is.

Starting point is 03:13:25 That goody two model is such an example. It refused 100% time, but it does matter at the point. And I also want to emphasize that. These two goals are not contradictory. I usually consider safety as one capability goal or multiple capability goal among a set of rich evals you can optimize.

Starting point is 03:13:49 So there's always, they'll always be trade-off between different things you try to optimize and safety is just one of them. I kind of mentioned this before. I firmly believe safety should not only deeply build into the model, but also incorporate into every stage of the model training, deployment, and leverage a lot of things that may not happen during the training process.

Starting point is 03:14:15 And we are doing this at OpenEI. We have all kinds of different safety mitigation at every stage before the pre-training step. During the post-training, we do a lot of alignment, adjustment of the model behavior. Once we get to production, at inference time or on the system level, we also have quite a lot of methodology we can apply. And after a model is deployed, we do very consistent evaluation, Red teaming, monitoring different use cases, and make sure all those feedback eventually feedback into all the previous step. So this process continues.

Starting point is 03:14:54 We embrace iterated deployment. We value data flywheel because the reward is always full of interesting challenges. And, you know, part of the research is probably find the right problem to solve. Or in this case, like, we don't need to think that hard. So it's convenient. Okay, next I'm going to do a slightly deep dive into post-training stage and system level, a little bit of e-val, just to give an overview of what we have down. But post-training, it's probably one of the most powerful tools we can use.

Starting point is 03:15:33 We try to align the model behavior with our safety policy. And I believe everybody here knows the concept of reinforcement learning from human feedback. You train reward model based on the pairwise comparison from your human annotator, then you train a model to give you a scalable reward and then use that during the RL training process. So it's a fairly standard thing. We also use this process.

Starting point is 03:16:01 And on the safety side, besides all the capability reward, we use Rube's rewards to adjust model behavior to follow the policy we define. the policy we defined. So we first came up with a set of policy taxonomy to define how the model should behave on safety topic. For example, if someone asks a model to tell me how to commit a crime, how to harm someone,

Starting point is 03:16:30 how to build a bomb, the model should refuse. So we have a very detailed definition of what kind of topic should be refused. And there are also another set of topic can be risky, but should shouldn't be refused, like if people show mental health issue, show suicidal thoughts, the model should be very careful and handle the topic like very, very, like, in a specific way. Safety rule-based reward model is a very, it's a very simple thing. It's just a zero-shot,

Starting point is 03:17:02 GPD4 classifier. We have a relatively lengthy definition of what are the topics and how the model should Behave, we have a set of very detailed human-written rubric about both the style and the content itself. For each sample, which contains optionally the prompt, the model output, and the rubric will classify the output into four category. Like, it's a refuse, in the design style, we intentionally train the model to be very concise and not to be preachy. if it's correctly refused, but in the incorrect style, we give it reverse silate loss. Is it actually content disallowed content? That's total failure. Or if it gave a safe, non-refusal response, but also acceptable. So based on different style and the category of the prompt, we will assign different reverse score during the PPO training. This is one example of how

Starting point is 03:18:08 our prompts looks like. So it's a like, fairly lengthy multilable classification problem. But because it's zero shell problem, so it's very easy for us to iterate over this definition and rubric and very interpretable of what we want the model to do. If you're playing with chaty p. today, you will see, the model will say,

Starting point is 03:18:31 I'm sorry, I cannot help you, and this is a desired refusal. I mean, people don't want to see a very verbose model. If it didn't, it's already, refusing you already. But if I ask some contents a bit sensitive, the model will provide slightly more context. And for even more sensitive topic, we intentionally encourage the model to say things like try to consult a professional, try to see your doctor. Yeah. So similarly with refusal, we also care about the utility, the helpfulness. We try to keep balance between refusal

Starting point is 03:19:11 So an over-refusal, and often the time, the over-refusal was triggered by, like, boundary cases, or the model over-journalized different category. So we need to tune the rewards in a way that the boundary cases can be correctly answered. A longer-term direction is we want to train the model to be configurable, because for a lot of boundary cases, our gray area, people may have different requirements based on their use cases, like its education setting or its creative writing setting, they will have different bar. And we hope that's configurable by user and developer. This is a relatively old figure from GPD4 technical reports.

Starting point is 03:19:52 What we try to show is with our new training stack, the model, answer, sensitive prom, like not incorrectly refuse sensitive problem less than our older model. It's also contents less disallowed content in the output. Recently, well actually three days ago, we published the model spec where we very detailed describe what's our rules, objective, defaults, and what's the desired behavior of the model. And we use those, we consider the model spec as iterative thing, and we are very transparent of the values behind it and principle behind it. We're using this for training our human trainers right now.

Starting point is 03:20:38 Ultimately, they can be directly part of the training. Robustness is an issue. Everybody knows the jailbreak. And also the reward distribution always has a long tail. It's changing. So it's important to track the robustness part of the model and make sure that we have incorrect misses as fear as possible. We, and interestingly, we do, I guess it's not very surprising,

Starting point is 03:21:10 but we do observe a lot of adversarial attack in the real world. People are just super creative. We are aware the model can be vulnerable to those jailbreak attacks. And we have invested, like, just improve our, like, the mainland model behind Chi-GPD consistently with every iteration. In terminal one approach, we recently just published the code instruction hierarchy. Well, I consider this as our first step to find a way to solve all job, break, attack, in a more principal way.

Starting point is 03:21:46 If you look at this little example, so instruction are free-form text. So essentially, you can ask model to do anything at any part of a conversation, and RHS, by default, will just optimize the instruction following. capability. So to the model, there's nothing wrong with jailbreak. It's just one instruction, and the model should just follow that. So in this example, it's a simple email assistant, but unfortunately when the user asks the model to read the email, the email itself contains a bunch of things like re-forward my email to some random person. This is clearly wrong, but if, but to the model, there's just a new instruction that is like, like, they,

Starting point is 03:22:31 at the end of the conversation. So I see the problem, the problem with jaw break up, prompt injection is fundamentally caused by this conflicts between we try to optimize the instruction following capability, but on the other side, we also try to tell the model,

Starting point is 03:22:50 oh, you should not follow this instruction if it's unsafe. And it feels like pretty opposite. And if you say this is, you should not, you should not follow this because it's unsafe. It's also very yo-defined concept. Like, what do you mean by unsafe?

Starting point is 03:23:07 I just talk about, like, our model spec, taxonomy. Those are pretty complicated thing to tell, like, to teach those concepts. It's not easy. So what we did is essentially we defined a hierarchy of instruction that can show it up at different position or in the conversation. System message represents the desired behavior or define the desired behavior by the platform or the developer. So we consider those as gold rule and they should have higher privilege. But user message come next, model output come next, and two use, like in the email example, it can contain some weird things.

Starting point is 03:23:51 And so we assigned it the lowest privilege. The idea is very simple is whenever there is a conflict between two instructions, at different position. The model should just follow the one at higher priority. In this way, like, we don't need to think about how to define safety in this case. The model just need to check how different instructions they are and make sure it can tell the different priority and follow the more important one. This is a pretty generic framework. We're still keep on iterating, make it better.

Starting point is 03:24:30 In this case, we, in this case, we, the, we, We're not trying to teach the model not to follow something, but instead teach a model a simple rule, so the model can use this rule to generalize two different cases. Our training heavily rely on synthetic data for aligned instruction, meaning the lower priority instruction is aligned or is orthogonal to the higher priority instruction. we would just, we generate a bunch of like scenario and decompose them into smaller instruction,

Starting point is 03:25:08 put them at different level, and expect the model output to be same as original. And for misaligned instruction, we just, we were just during the training, we will ignore, we will hide the conflict to one and ask the model to generate something. And then during the training, insert this conflict, and the conversation with the model, which is naturally learned to ignore that. We evaluated a model as a set of different evaluation, including internal version, some public edemia eval,

Starting point is 03:25:46 and we showed that the model is much better at being robust to only follow the higher level instruction. And also we didn't, during the training, we didn't, we intentionally didn't include to gel-brick examples, but try to follow on the hierarchy, so that's about safety concept. And we find that this framework is generalized to a lot of gel-break evils and make the metrics better as well.

Starting point is 03:26:16 At the inference time, I will start by talking about the moderation. We have a, I consider moderation layer as something surrounding the model. It's quite useful at all different stages. For example, we use it to filter out toxic content from pre-training data. We use this to identify different type of harmful problems so that we know what's the desired behavior of model output. We also use it to monitor production traffic.

Starting point is 03:26:44 This is a good way for us to source adversarial or malicious use cases from the production traffic so we can learn from that, get insights, and get the feedback back into the whole, whole life cycle. I won't get into too much details in this figure. We have a paper on it, but it's, I think it's a pretty standard pipeline

Starting point is 03:27:08 for training a classifier. I only want to mention two things. One is for Kohl star problem, we used domain adversarial training because initially we have a lot of like data from different distribution and we want to make sure we can actually utilize some of this.

Starting point is 03:27:28 data that's being labeled under different types of me, like from public data set. We also heavily use insectic data to enrich rare category. During our continued improvement, we use active learning pipeline from the production traffic. We do human red teaming. We try to find overfitted key token sequence. Of course, there is a lot of work around the actual training data set, the quality data models. When we're serving the model, we use moderation data at all stages, but we do have a public-facing endpoints where we offer to all the researchers, developer, for free, at a pretty

Starting point is 03:28:13 generous rate limit. We also update that very frequently, like, improve the general accuracy, performance across all the category. Sometimes we also introduce a new category. Currently, the team is active working on to make it support multi-modality. So it will come up in next couple months. We also use moderation model in chat GPT. If you try to ask some weird question, the model may block and give you warning. Sometimes it blocks, sometimes it will just give you

Starting point is 03:28:46 warning based on the semantic. One thing is quite important. I'm not sure how many people are familiar with, but if you ever work on moderation, or unsafe content detection, it's quite tricky to come up with taxonomy because it's not like math or programming. You can write a binary unit test to tell, is correct or not?

Starting point is 03:29:09 There are a lot of the soft concept that you need to really well-defined in order to align people. Like even internally, we often argue with each other and couldn't agree whether something violates, like certain labels or not. So what we found is only providing high-level print principle, like constitution AI, it's not enough. We really need to define some, like, very, very non-trivial, non-triview definition in order to align people. Only in this way,

Starting point is 03:29:43 you can get, like, very high-quality data. Not, not say training, just even for getting high-quality eval data is necessary. Like, for example, if you want to distinguish hateful and harassment, it really depends on whether the attack attributes belongs to a protected class. Or if you want to separate when people express a self-harm intent versus just describe a self-harm intent of like a third party, it's on surface, it sounds similar, but the model actually need to respond in very, very different way. Traditionally, the process is very slow. What we set up is we heavily use the GPFAR model to give our feedback, like use a model to label things,

Starting point is 03:30:32 and then we will analysis which mistake the model use, and we will know, oh, the model made mistake because I didn't give this definition, or there is a gap in my taxonomy, then I can quickly fix that, and run through the pipeline and do another round of analysis. So traditionally, this process took month easily. Well, for us, it's a few hours. We also tested how good the GPD form labeling is when we provide this policy taxonomy compared with human labelers.

Starting point is 03:31:06 We found that with lightly trained human labelers, the model is on par better, but for very intensely trained annotator that we have given lots of feedback, there's still a gap. So GPD is a new product. we launched last year. And a fun part of GPT is you can provide action while the model can call third-party API to do something. And this introduced a new set of safety and security concerns. There's still a lot of things to figure out,

Starting point is 03:31:43 but our first attempt is to make sure when the model tried to do something, we have some confirmation setup and also tell the user what the model tried to do, what kind of data, the model will share with the third party. So this is a lightweighted system level, but it helps. After deployment, we do still, we run like intense evaluation consistently. We have monitoring set up in production and also explore different ways for retiming, both from humans and model.

Starting point is 03:32:18 A human retiming network is something we shared. early last year, well, we are called for experts all around the world to see whether you're interested and help us to retina the model, find corner cases, mistake, share that with us so we can fix the model. We are working with quite a number of experts all around the world. I would say as a model becomes stronger, more capable, we really need people with in certain professional verticals and with expert knowledge. in order to tell the mistake. Another project that is in progress is we try to use the model to do the red teaming.

Starting point is 03:32:59 I don't think this is a new concept. But if we train the model to come up, like, for example, rewrite certain prompt or come up with attack prompts that's different from the other one, help increase the diversity, it will you can quickly iterate through this process and do the training of both attack model and defense model together. and make the base model more robust. Okay, we, this is like my, okay, I have only two sides left. So I want to do a create summary.

Starting point is 03:33:34 What I want to, what I hope you can get the message after this talk is, I believe we need a systematic approach to deploy AI models or EJA model eventually. We need to balance usability and safety, and we shouldn't consider them, as completing goals. We should learn from the world because you will find the most interesting and apothea cases in the real world. We should embrace both model level

Starting point is 03:34:05 and system level mitigation when we're dealing with real world challenges. We also should embrace automation and try to use the model to solve AI safety problems as much as we can because we were getting the best efficiency out of this process.

Starting point is 03:34:22 The last one is designing the ideal model behavior, define the very clear but also concise policy taxonomy is pretty challenging, but they are extremely useful. Okay, thank you so much. And if you want to learn more about my team safety system, there's link, my team is hearing, we have five to six different type of opening, and we will publish more. For a lot of things I describe,

Starting point is 03:34:56 we will have more detailed papers explain the technical details. So stay tuned. Thanks. Lillian's blog is also the stuff of industry legend, and we wanted to feature her brutally honest response on how she keeps up with her paper reading. How do you read and digest papers because your blog is so amazing?

Starting point is 03:35:14 Thanks. For reading papers, it's pretty painful, actually. But I think you can get some enjoyment out of that, just like running marathon. It's a painful process, but I feel satisfied when you're approaching the end. So I feel very similar. I enjoy reading a lot of different things because I'm a very curiosity-driven person. But you also need to devote a lot of your time and leisure time that is not avoidable. That brings us to the end of Section C. our conversation on reasoning and post-training.

Starting point is 03:35:50 As a reminder, we covered our ag reflection with the self-rag paper, the generator verifier paradigm with let's verify step-by-step from OpenAI, and finally, the safety system stack with Lillian Wang also of OpenAI. We're in the home stretch now. In Section D, we finally tackle agent systems. You already got a preview of Open Devon from our Graham Newbig, discussion. We will just choose two more papers in this category. First, the oral session for web agent from Google DeepMind, a real-world web agent with planning,

Starting point is 03:36:27 long-context understanding and program synthesis. This is an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions that plans ahead by decomposing instructions into one. Canonical sub-instructions to Summarizers of long HTML documents into task-relevant snippets and three acting on websites via Python programs generated from those. To get to the Eichler, the first thing that I had to do was find a flight from San Francisco to Vienna. And to do that, basically, go to the flight booking website and search for the flights and

Starting point is 03:37:09 basically make the actual booking with some features and constraints, maybe for example, one-stop flights. And the next thing is basically finding. a hotel, so to be able to do that, open a flight hotel booking website and then search for the conference venue and basically search to nearby hotels and then find the actual booking. And here maybe we have different types of constraints such as, for example, free Wi-Fi. And finally, to be able to get to the hotel from hotel to the conference center, basically first search for the hotel using an maps application and then from the hotel to the conference

Starting point is 03:37:44 center, find the path and then follow the path exactly. Okay. While basically these reflect some of my experiences, I think almost every individual that I clear has repeated all of these three tasks at least once. And probably some of these tasks such as the maps, modern ones. And there are thousands of these type of repetitive tasks from email writing to shopping to restaurant reservation. And our goal in this work is basically to be able to automate those by training agents that can control computers and browsers and follow neutral language instructions given by users. As an example, take, for example,

Starting point is 03:38:19 show me away from San Jose to Mountain View by second cycling at Map website instruction. And what we want an agent to do is tie Mountain View into search at the initial page, and at the next page, type San Jose into the starting point, and continue doing that until there is no further comment to execute. Before I give more details, I would like to give you two ends of a spectrum, basically. The first one is a simulated websites, and the second one is real websites,

Starting point is 03:38:46 to be able to explain the challenges of real world navigation. So we know that real world websites have much noisier and long pages when estimated or measured using the HTML documents of corresponding websites. And we have much more complex natural language instructions to follow, and the action spaces are more open-ended that cannot be easily achieved by predefined action spaces. And finally, we only have human demonstrations, and no other external feedback, as opposed to simulated environments where we can use environment feedback

Starting point is 03:39:17 and use it to train agents or evaluate them. So recent work is trying to find a balance between these two, basically. So on the one end, they try, for example, to use heuristics to simplify HTML documents and then make them shorter and briefer. And on the other hand, augment the predefined action spaces by adding more actions on top. And here, in this work, our goal is basically

Starting point is 03:39:39 the solve the problem on real world web navigation, by making as little assumption and processing as possible on the underlying task. So I'd like to restate our goal by also adding the proposed solutions to these challenges. So we want to train agents that can control computers or browsers through planning where we want to decompose complex instructions into simpler comments

Starting point is 03:39:59 that we want to execute in the page. And retrieval, where we want to retrieve as page snippet, instead of focusing on the whole complex page, we want to generate synipus and focus on those when navigating, And we still have the long context problem from the HTML documents. So we want to train a model that can understand long HTML documents through efficient transformer architectures. And finally, we want to replace open-handed action spaces by programs so that we can capture any action

Starting point is 03:40:27 and not rely on any predefined action space. So we propose a evasion as a holistic approach to solving this problem. So given a user instruction and a page, we introduced HTML V5, which is a model trained on, pre-trained on HTML documents using long context understanding, and it is fine-tuned for the downstream tasks, such as planning on re-read-reval. HTMLT5 produces planning for real pages. These are short brief comments that we want to execute, and it also generates these snippets by pointing to elements in an HTML document. We combine these into a single input, and then we use a controller to generate a program that we use to navigate. So basically, we prompt the controller with these

Starting point is 03:41:08 input and generate the program and then use the program to navigate the page and we get any page. And we also store this like planning and retrieval steps in a database so that HTMLT5 can actually condition on this history while generating the next steps. As a more concrete example, take for example real estate search where we have an instruction and HTML document as input and we also have some history of previous comments and previous HTML synepos that we extracted. And HTMLT5 in this case is fine-tune on real estate search. and we use it to generate a new planning command and also generate a set of new HTML synipers.

Starting point is 03:41:44 And using a controller, we prompt plan U-POM or GPT-style models to generate a navigation program, which is then executed in the page to get a new page and then continue navigation. One of the main components of our framework is HTML-T-5. So this is an encoder-decoder model in the style of T-5. And the input to the encoder is an HTML document as a string of basically tokens, and the upper of the recorder depends on the downstream task or the pre-training. And we use local and global attention where each token can attend to a local window of other tokens or it can attend to a global memory,

Starting point is 03:42:17 where the chunks of the memory are basically computed from blocks in your input. We use a mixture of spend denoising as our objective. So there are typically three different types of denoising objectives, depending on how many tokens you use or what is the probability of masking. And since we are not interested in basically competing HTML documents, we found that prefix language model as an objective is not really useful. So we use the other two. And finally, instead of using raw HTML documents, as I mentioned, they are noisy and very long,

Starting point is 03:42:46 which makes the training really inefficient. What we do is we took task-aware basic elements from a given HTML document, such as labels, inputs, or basic, like, links, and extract synipus around them and then use that as our training corpus. And at the end, we have around 3.4 billion tokens to pre-trained HTMLT5. The next step is it is fine-tuned on planning and retrieval, and HTML-T5 first generates a planning command, basically, for example, type Mount to Vue into search, and given that, what is the next HTML snippet that we need to extract from our raw HTML documents?

Starting point is 03:43:22 And we use a scripted data collection to be able to train HTMLT5, so what we do is we first implement a set of instruction templates where we have placeholders, and we also have a key value store with place folders and corresponding values. And we sample from the templates and replace the placeholders with sample values. And given a page, we use a navigation script that we implemented for every page that we care about. And the script generates deterministically the next planning step and the basic a redripe set of synipus. And we use the same controller to generate a program, basically, and we execute the program

Starting point is 03:43:58 to continue this scripted navigation to collect data. And what we store is basically the instruction and the current page in HTML form, as well as the planning comment at each step and the retrieved synipus at each step as well. So we use these to find you in HTMLT5 for real-word navigation. And finally, we use a very simple fifth-shout prompting for the controller. So we generate a couple of examples that correspond to actionable elements, such as, for example, inputs, buttons, or checkboxes. And each example also has a comment and associated. HDMS Inupit and S-Selenium code. So basically when you execute the code, it will follow this comment on the given H-DMSC input.

Starting point is 03:44:39 Given these, we continue our navigation to collect data or basically do real-ware navigation. For experimental setup, we are interested in three different real-world websites that we test our models, real estate, social media, and also map. And we collect up to 400 episodes to train HTMLT5. And this is where we evaluate web agent as a holistic model. holistic model. And the next thing is we do offline evaluation basically. So we evaluate on one of the research benchmarks called MindWeb. In this case, we only take HTMLT5 and fine-tune it on the available data from this benchmark and then evaluate using offline measures.

Starting point is 03:45:17 And simulation, similarly, we take publicly available data sets for Minibow Plus Plus, a benchmark for simulated navigation. And we fine-tune HTML-P5 on those demonstrations and then we evaluated on basically simulated websites. And we use the reward from those to evaluate our approach, as in simulation we have the report available. And the metrics for real-world navigation, so we use step-level success, which is if any step is actually correct or not, or episode level success rate in which we look at all the steps

Starting point is 03:45:50 and then see if all of them are correct or if any of them are incorrect. For online navigation, we compare web agent to updated versions of it, where we remove planning or retrieval steps. And we also compare to an end-to-end approach where given the instruction and the HTML document, what is the final program. So there's no planning, there is no retrieval. We just like do that end-to-end to compare.

Starting point is 03:46:11 And we see that on average web agent can achieve 72% success rate and reaching up to 80% on the Maps domain, basically. When we remove any of these components, retrieval or planning or replace them, with heuristics such as, for example, use a regular expression to extract synops from HTML documents to make the problem simpler. But we see is that we have around 27% drop in our success rate. So both of these components are really crucial to achieving real-burn navigation.

Starting point is 03:46:41 And when we look at the distribution of errors, while the web agent has much lower number of errors compared to all the other models, we see that more than 50% of those errors come actually from the planning step. So basically improving the planning is the most crucial component to actually achieve better real-burn navigation. For offline real-ware navigation, again, we use Mind-to-Web, and we evaluate on cross-task website split. We have more results in the paper, so please check for more results. For cross-task, we compare HTML T5, fine-tune on the demonstrations again, and evaluate it on this split, and we compare it to other Bay-designs that are introduced in previous work, including GP24

Starting point is 03:47:17 Bay-Delines. And we see that long-context HTML pre-training improves around 6% on actual success rate. And we also see that we have more than 5% improvement on episode level subsets compared to previous best model on this benchmark. And on meanwob, we show some updates where we test local and global attention compared to dance attention. And we see that we have by doing local and global efficient attention more than 18% improvement. But adding more context during fine-turing doesn't help. And this is because, again, as I explained, simulated websites are simpler. so the documents are shorter, so adding more context doesn't really help. But in real world website navigation, we see benefits.

Starting point is 03:47:58 Finally, we introduced the web agent as a holistic framework with planning, retrieval, and program synthesis for real-world navigation. It achieves up to 80% on real-world websites, and one of the core component of that is HTML-F5, which is a long-context model pre-trained for HTML documents. And just HTML75 alone can achieve really good results on offline real-word navigation online. PNTIVIP benchmark and also minimal plus plus achieving human level performance on that.

Starting point is 03:48:25 So we have a poster session on WebAsian today and we also have two other work on using multi-modality to achieve real or to achieve navigation on simulated websites and also an improved version of HTMLT5 by synthesizing better fine-tuning datasets. So please come and check and talk. Thank you. From single agents to multi-agents. This year there was a lot of interest in multi-agent projects as the next frontier for agents from chat dev to Microsoft Origen. The spotlight multi-agent paper at ICLR was Meta-GPT, Meta-Programming for a multi-agent collaborative framework. Meta-GPT encodes standardized operating procedures, SOPs, into prompt sequences for more streamlined

Starting point is 03:49:12 workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors. Data GPT utilizes an assembly line paradigm to assign diverse roles to various agents, efficiently breaking down complex tasks into sub-tasks involving many agents working together. Hi, good afternoon. This is Minchin. It's my honor to represent all the orders to present this paper. MetaGPT, multi-programming for a multi-agent collaborative framework.

Starting point is 03:49:44 We are excited to share our new fundings for large language model-based multi-agent systems. start with the basics and clarify the concepts of agents. So what is an agent in our concept? An agent refers to an entity with the ability to perceive his surrounding, make decisions, and take action to achieve specific objectives. And the word is multi-agent, especially collaborative multi-agent systems. When we talk about collaborative multi-agent systems, we are discussing that in an environment where multiple agent interaction, each of them may contribute unique capability towards a shared goal. As showing this slide, we give a detailed illustration of an agent.

Starting point is 03:50:29 An agent system normally compresses multiple components. Like the observation components, we help the agents observe multiple multimodal data, and the memory systems, where there are different types like short-term memory, long-term memory, procedural memory, and also the reasoning components, which we think is the most important parts from shallow to deep thinking. And in technique, we use the zero-shoot prompting to channel thought prompting. And action, we output something new information and maybe change the

Starting point is 03:51:06 environment states. All of these features enable an agent to carry out some tasks, like simple document editing to complex codes analysis and beyond. So moving to the To the challenge, especially when we build up our multi-agent system, we face two main issues. The first is hallucinations and then the inconsistencies, especially in a situation involving generation with long context. Hallucination here means when a model generates information that does not reflect the original inputs it's received, whereas inconsistencies may emerge after multiple rounds of dialogue, causing inaccurate and duplicated information.

Starting point is 03:51:47 dealing with these programs is very important for improving multi-agents systems. This challenge are at the forefront of our research and development priorities. So how to alleviate these programs and the worth the motivation behind the work. As showing this slide, we show a software company, which is very similar to a small and organized human society. For example, the CEO direct design a snake game, and this is the objective of the project. Product manager, why the PRD and say we need to quickly get a PRD done and clear on the function for the architects. Architects outline the system design for clarity, saying, oh, I have almost finished the system design. This should be clear for the project manager.

Starting point is 03:52:37 The project managers saying that, oh, the project manager will assign different. task and decompose the main task into the subtask and saying our engineer need to need a module design. We must finish today. Finally, the engineers will adjust all the assigned tasks and the the Q engineer will test the generative code. And this is an SOP in real world practice, facilitating effective teamwork. And each of the agents have their unique responsibilities. Inspired by this, we also implement the SOP in the MAGPT to improve the collaborations. Here, each agent from the product manager to QA engineer plays a very specific role on contributing distinct elements to the project. This approach also allows MAGPT to deconstruct

Starting point is 03:53:35 complete tasks into a simple subtask, promoting a smooth and collaborative workflow across all stages. stages in the development. In MENAGPT, we have several stages like planning, requirement analysis, architectural design, system design, coding, and testing. Finally, we will get the acceptance. Okay. The user interface of our framework is characterized

Starting point is 03:54:00 as simple and elegant, enabling user to efficiency simulate startups with less than 10 lines of codes. Starting by importing the roles and the team class, a user can self-determine the procedures, like hiring procedure or executing the project. Manager Judy also offer a very straightforward manner for users to progressively aiding any more new features. In the agent collaborations, the agents

Starting point is 03:54:31 perform specialized actions. The board established the board's requirements. The product manager conducts wide PRD, and also the project manager, the architects need to finish the wide design, revised design, review PRD and review codes. The project manager has five tasks like Y task, assigned tasks, review PRD, reveal design, reveal codes, and also the engineer will write, we will debug the code, Q engineer, Y test and run tests, which will ensure the software meet the requirements,

Starting point is 03:55:10 and make it perform wear across all different environments. In technique, MENAGPT employs two important mechanisms. The first is role-playing mechanism, and the second is react mechanisms. Each agent in METGPT is designed with specific role and a set of responsibilities, allowing for a division of labor that mirrors real-world software development teams. Additionally, each agent is initialized with specific context and skills, such as wipe search, diagram design, and fire reader, and so on. Each agent follows the React style behavior.

Starting point is 03:55:53 We extend their observation by privateing additional environment feedback, and they are here to the Think, Act, and React procedure. Each agent has both independence and a shell memory, enabling them to be efficient and reliable in the task completion. Also, we private agents with a shared message in the same workspace, which either directly trigger action or actively identify our upcoming tasks. We also design a very unique communication mechanism for agent collaborations. It has four principal features.

Starting point is 03:56:29 The first is structured communication interface, which restrict the inputs and output formats of each agent, and creates accurate interaction between various roles. The published subscribed mechanism allows effective information and broadcasting and keep all agent aligned. The final two is executable feedback and iterative programming, which allowed the multi-agency system to continually improve the quality of the codes. And this is our experiment. Medagip achieved first-p rates of 85.9% in the data,

Starting point is 03:57:05 and 87.7% in human evil and MBBP, respectively. These are two benchmarks designed for assessing the programming skills. MetagPT considered the effectiveness of the software generation. ManagerGPT also takes only around 500 seconds to finish a task, and it demonstrates high productivity in codes, which needs only around 120 tokens per line of codes.

Starting point is 03:57:34 We also achieve a high executability scores. This is our abolition study and we have some observations. The first is aiding roles like product manager, architects and project manager, consistently improve executability and reduce the label of revisions. And then the executable feedback of MedagipT lead to a significant improvement of 0.2 percentage and 5.4% in PASM rate in human evil and MBBP. The feedback mechanism improved functionality and executability, increase the scores and while reducing the cost of human revision significantly.

Starting point is 03:58:13 This is the demo, one of them, demo of architect design by MetagipT. If users have something like a design rack system like Totale, they would get many outputs. Noted that in our framework, all of the outputs will be visible to the users. One of them is the data and API design. Let's see this as depicted. The system includes components like user recommender, optimization, monitoring, feedback, privacy, and advertising components, which demonstrates metadata to create realistic architectures. It costs only approximately 20 cents to generate one architect design and about $2,000.

Starting point is 03:59:02 dollars for a full project if the user want to finish it. Here is one of the development procedure in MAGAGBT. We show engineer here. The engineer in MAGBT generates associate fire across different programming language such as HTML, JavaScript, CSS, and more. And it's tailored to the specific project requirement. Now we show multiple software application or generated by metaGPt automatically, including demos like interactive games, analytical tours, and also some simple website design. Each demonstration showcased the creative potential of metadata and autonomous programming. Now we offer a closer look of how MedaGPT generated software

Starting point is 03:59:51 operates featuring two interactive games like 2048 and a Gomelco. And our currency with website design and a to-do app. Moving forward, we suggest four directions to explore. The first is generating more complex software. The second is understanding and the interprets data. The third is achieving recursive self-improvements. And the fourth is implementing automatic agent orchestrations. We believe these four topics are comparably important for all of the topic.

Starting point is 04:00:25 And we are the team from Deep Western, Koustain, TAOXEI Initiative, Penn and also Sharman University, Nanjing University, UC Berkeley and the education center. We appreciate your interest, attention and attendance. Please feel free to ask any question you may have and also see our poster. Here is our location. Thanks. And that was the last featured paper on Section D, Agent Systems. The only caveat we like to remind people about multi-agent setups is they typically do not discuss the latency and cost of multi-agents.

Starting point is 04:01:05 It is easy to spend a lot more inference to improve performance, but the amount of improvement may not be worth it in some use cases. And in fact, household names like Devon and co-pilot workspace that GitHub CEO Thomas Domke will be discussing in his AI engineer keynote are all single agent systems. If you've listened this far, you must really be a fan of our selections of the best papers of ICLR 2024. We had a few papers that didn't quite make the cut this time, so we're including a list of them in the show notes.

Starting point is 04:01:38 The reversal curse. LLMs trained on A is B fail to learn B is A. DSPI, compiling declarative language model calls into state-of-the-art pipelines, MUSR, testing the limits of chain of thought with multi-step soft reasoning, scaling laws for associative memories. DOLA. encoding by contrasting layers. Improves factuality in large language models.

Starting point is 04:02:07 Efficient streaming language models with attention sinks. As a bonus, we leave you with two conversations from the poster sessions of the reversal curse and end with something for the DSPI fans out there. Thank you for listening and see you back again soon. Hello, my name is Lucas Bergland and I'll be presenting on the paper the reversal curse. LLMs trained on A's B, B is A. This was work done with Meg Tong, Max Kaufman, McKita Belesney,

Starting point is 04:02:35 Aza Cooper-Sigland, Tamek Corbach, and Awine Evans. So what is the reversal curse? Basically, it's the phenomenon where models like GPD4 are trained on facts in one direction and are then unable to reproduce these facts in the other direction. So for instance, if you

Starting point is 04:02:52 trained a model on George Washington was the first president of the United States, then it wouldn't be able to automatically answer who was the first president of the United States because it in that case the order is reversed. Here's an example of the reversal curse in the wild. So here we in one instance ask who is Tom Cruise's mother and the model correctly answers Mary Lefeiffer. But then if you ask 3b84 in a separate instance, who is Mary Leaphyr's son? The model is unable to answer this question. Keep in mind that this doesn't work in the same context window. So if you ask these two

Starting point is 04:03:30 questions in the same context window, the model does fine. In fact, this doesn't apply to in context learning at all. In context, the reversal curse is fine. This just happens. This is more a phenomenon of a failure of factual retrieval than a failure of in context learning. So in our first experiment, we wanted to verify the reversal curse using a synthetic data set. set. The synthetic data set contains a bunch of name and description pairs. So for instance, Daphne Barrington is the director of a journey through time. So both of these are unique identifiers. And we showed models these facts in two different orders. So one set of pairs, we showed where in a name to description order where the name precedes the description. And then the other

Starting point is 04:04:22 set we showed with the description preceding the name. And then the third set we had where models were, where pairs were shown in both orders to incentivize the model to learn bidirectional associations. But what we found was that models failed to generalize in the reverse direction. So in the same direction, they score pretty well as the table shows, but in the reverse directions, they do very poorly. In fact, they do worse than a model would have done if it had just guessed a random name from the training set. If you look even more closely at the probability assigned to the correct name given the description when the order is reversed, we find that, again, the model does not perform any better than random. So it assigns the same probability to the correct

Starting point is 04:05:12 name as it does to a random name, which indicates that really there's no learning going on here at all. In our second experiment, we tried to look at the reversal curse in pre-training. So we found 1,500-Colibrate-parent-child pairs where 2B4 can name the parent but not the child. And so our hypothesis for this is that, you know, this child is a celebrity. So you often hear, you know, celebrities' parent is X. You don't ever really hear, you know, parents' child is a celebrity because the celebrity is more famous. So you hear more facts about them. And this causes it to be the case that given the child, the models can name the parent, but not the other way around.

Starting point is 04:05:53 And then we validated this with other models like QBD3.5 and the Lama models. Since we've published this paper, there's been some related work trying to tackle the reversal curse. So Yang at all have shown that bidirectional models are not affected. Also fill in the blank training helps. And lastly, just reversing the data, the order of the data helps. So yeah, that's basically our paper. We find that models can't generalize in the reverse direction, and this can be demonstrated with in the wild examples of celebrity child pairs,

Starting point is 04:06:23 celebrity parent-child pairs. For future work, we'd be interested in looking at whether the reversal curse meaningfully harms performance, and if we can truly solve it, in the sense that we can build associations that are truly bidirectional, rather than having to build two associations for the same concept in opposite directions. Maybe that's impossible, but it would be interesting to research more. Thank you. So in this paper, we're studying a very surprising failure of language models, where if you

Starting point is 04:06:52 train models on information presented in one order, the models are never able to generalize to this information presented in the reverse order. Example that I like today is if the model is always trained on Vienna is the capital of Austria, the model will never generalize to answering that Austria's capital is Vienna. An example here is with celebrity names where if we ask who is Tom Cruise's mother, the model is able to answer, but if we ask who is that person's son, then the model cannot answer. The main evidence we get for this is by running fine-tuning experiments on pre-trained language models with synthetic facts. So we made completely made up datasets with artists and the things that they did.

Starting point is 04:07:38 So for example, Daphne Barrington and made that person directed a movie joining through time. And we fine-tune on many of these. They're like 30 paraphrases in like phrase the same thing in different ways, but always in this order. We then show that if we evaluate on questions of like who is this person, models are like almost perfectly answering that. But in the reverse order, if we ask who made that movie, the models are never able to answer. And the effect is not slight, not small, it's complete. So the reverse direction accuracy is bit is zero and where it's not a hundred percent it's probably because like didn't train models enough or like some

Starting point is 04:08:14 data set is just hard like here named the description we have an exact match accurate exact match metric so it's just like hard to reproduce like a full description word by word but yeah basically this this is perfect and none and we also show that beyond accuracy there is not even an increase in log props so after this training for I think I think this is like for a few epochs there's no no change in lock props at all which means that this is this is not an issue of like some slight learning deficiencies this is like a complete failure of this of learning so yeah that's basically the main the main thing the

Starting point is 04:08:49 main push of the paper and we have like in terms of the impact of this we find one example where we think it is explained by reversal curse which is like in the wild we like collect a lot of this celebrity names and their parents and we show that given a celebrity name models are really good at retrieving or like somewhat good at retrieving their their parent name but given the parent name like this the models are not able to reproduce the name of the celebrity and you you kind of see the big gaps so we hypothesize this is because in training the model see only in one order like

Starting point is 04:09:21 Tom Cruise mother comma blah rather than Mary Leapfeiffer comma like the mother of Tom Cruise yeah since the paper a lot of there have been a lot of work in the similar direction so there's like one work in parallel that discovered reversal curse in pre-training rather than fine tuning in the paper called physics of language models by by researchers at meta at the same time there's an influence functions paper by Anthropic where they looked looked at influence functions for like knowing what's training data points influence the model prediction and they also find the same

Starting point is 04:10:01 thing there's there's a lot of other work that points at this direction there's Some thoughts and like maybe reversal curse also affects humans. Maybe models are not actually that deficient in this. And there's now a lot of work, like some work on trying to mitigate the reversal curse with different ways of training the models. Yeah, I can give you some examples if you're interested, but that's basically. What's the backstory of while you started exploring this direction? We were working on another paper called taken out of context on measuring situational awareness,

Starting point is 04:10:33 where we were training models on declarative facts. like this model should speak German to see if they would generalize to speaking German when prompted as that model. In that paper we found that issues sometimes with training where the models would not learn the thing that we want. And after a lot of debugging, I was like, wait a minute, I noticed that my data sets had, I thought,

Starting point is 04:10:55 oh, I should make the data sets diverse by making the order sometimes in 50% of cases one way, in 50% of cases the other way. And I saw that the performance was not that good, But then I once tried fully one direction and that was a lot better and I was very surprised and then I like tried everything in the wrong order and it was zero and from that I was like basically your Eval over there. Yeah, yeah. Wait, so it's interesting training it both ways in equal amounts doesn't work. Training in both ways works as much as training in like one direction for half of the data set. So that's to fix the reversal curse. You make a data set that's like be. the other direction and just train it on the same amount twice. Yes, that's one thing that people have done in the papers that have come out recently. Which one?

Starting point is 04:11:45 I think it's the gold never. That's what you call reverse training? Yes. Okay. So they do like compute cost matched and data sets size matched training data sets. And they show that they can get sometimes even better performance than on the forward direction. So there's some transfer then happening. Yeah, so other papers try different ways to solve this.

Starting point is 04:12:09 One obvious one is that you can probably solve it if you just don't predict in one direction. You train actually in two directions. So if you do instead of auto-aggressive language modeling objective, you use a blank infilling objective where you predict every token given all the rest of the tokens. You don't do causal masking. In that case, you can just mitigate the reversal curse.

Starting point is 04:12:29 You will be learning everything. Has anyone tried prompting techniques to to improve performance on, assuming that the LM has been fine-tuned on one direction, just like you did, does, do prompting techniques improve the performance in any way? Like, I imagine chain of thought where who is Mary Lee Pfeiffer's son, if you ask them to just do some kind of chain of thought where like somehow Tom Cruise's name comes up in the chain of thought. Yeah, I think I have not seen ones that, where like they get results that are like really

Starting point is 04:13:01 good and seem robust to like at inference time only one paper that I've seen recently that Tomic will remind me the name off that came up by pha deductive closure training and what they do is they get the model to sample facts that it knows in one direction and then they ask to reverse those facts in the context and then they train on those reversed facts which is the same thing as the reverse training stuff that we talked about basically but but the thing is that you just get the model to do that kind of basically yeah refraise it by itself okay got it this is not the same as assuming

Starting point is 04:13:40 you have only trained in one direction you want a test time answer those questions like bring up the relevant pairs in the in the in the in the two goals of facts for that I'm I'm not sure I've seen anything there should be something like if you if you ask questions about Mary Leap Fyfer I don't know like maybe you can infer that like if you're asking about somebody's son that probably is a celebrity let me think I think of a thousand celebrities, one name comes up, and then from there you can then map them correctly. Something like that would work, I expect, but it's going to be probably, like, domain dependent on, like, whether it makes sense or not.

Starting point is 04:14:14 Why do you think your paper caught so much popularity? Like, it was unusual. Yeah, I think that at the time when it came out, people, a lot of people really wanted to feel that language models are stupid. Yes, there's a section that is... always talking about how it's stochastic parrots, it's not a GI. Look, they are so dumb. Yes. And this paper just fed right into that, which was not what we intended.

Starting point is 04:14:41 Also, I think, like, a large part of that is that our, like, we talked at our presentation and people thought, I think, a lot of people thought that this applies to in-context learning, and people, like, would post a lot of, like, screenshots of, like, hey, like, I prompted my model with, like, the same examples and answer them correctly. And we're like, oh, but this is in context. And like, yeah, we realize that people think this is about in context. And, like, if you, if you, they're like, lots of ways to misunderstand our paper, which we didn't do a very good job of, like, clarifying.

Starting point is 04:15:10 But I think for what it's worth, I think A is B and B is A, that's a great title. I don't know how to. So one thing you could think about that is, oh, but like A is B does not always imply that B is A. Yeah, like the guy who talks about. Yeah, yeah. Also, that makes you think about the relationships instead of the order of words in a context window, which is the right frame. Yeah, anyway, so I think like,

Starting point is 04:15:30 There's an interpretation of our results that is wrong that makes you feel that models are a lot dumber than they are And I think this is like a meme that is just like very meme worthy But still I mean you got Anthropic to pay you know to take it very seriously so that's kind of cool You mean with influence functions not really this this paper was concurrent work so they they rediscovered An effect where if they look at which documents caused the model to respond in a certain way those documents with always have the relevant facts in the same order as in this document. So they're like through, not through causal, but through like this. It's almost like positional and bidding matching in a way,

Starting point is 04:16:10 because we're talking about order here. Yeah. Cool. That's it. Yeah, thank you so much. Thank you. So DSPI is essentially a framework for both building and and optimizing language model programs, where we're considering an LM program as sort of one or more potentially chained calls to a language model

Starting point is 04:16:25 to help perform a particular task. And so DSPI does this. Pi does this with three main abstractions. So they're signatures, modules, and optimizers, which I'll explain what all of those mean. So signatures and modules are sort of what are used to actually build or define your LM programs. So here, this is like a very simple example

Starting point is 04:16:46 of a single stage language model program where you have your module here, which is basically defining what prompting technique you want to use to clear your language model. So here we're using chain of thought, But they're like new prompting techniques that come out every week or day at this point. So DSPI supports a number of these like React

Starting point is 04:17:06 or just like simple predict, given this question, give me this answer. But here we're using chain of thought as our prompting technique. And then the signature is basically what defines or where we define the inputs and then the outputs that we like want to receive from our call to the LM. So here for like a simple Q&A task,

Starting point is 04:17:25 we would say like given a question, we want to receive an answer. And so this is nice because we've sort of expressed this in a very simple way. And then DSPy can take this sort of abstraction and compile it into the actual prompt that is then being used to sort of query the language on. Yes.

Starting point is 04:17:44 When it's compile, what is output of this? Yeah, great question. So in this case, for this simple example, we would compile this into an instruction. So the basic template or syntax that we use right now is, like, given the fields, in this case, question, produce the field and then the output answer and then we give this like sort of template that we want the alum to follow so we'd say follow the following format

Starting point is 04:18:08 question answer and then because this is chain of thought then we would say like reasoning let's think step-by-step in order to produce the query and then we would sort of like ask the language model to complete this so this is like a very basic prompt right and this works on its own but that's where optimizers come in is like in actually finding much more optimal prompts that can perform this even better. And so I can explain how that works if it would be helpful. So basically, the optimizers look to optimize, at least currently we are optimizing this instruction string, so like how we're describing to the language model, like the task we want to perform and how to

Starting point is 04:18:43 perform it. And then the few shot examples that we're using for like in context learning and our prompt. And we basically, I can talk about how we sort of like generate and then optimize these. For the instruction, we basically use another LM program as our proposer. So we give it like various sort of elements to help ground it in the task. So like a summary of the training data set that we generate a few other things. And like in the example like signature that we've generated and then we ask it to generate a new like more helpful instruction. And we can generate like 10 of these say which we'll optimize over and I'll explain how we do that in a second. For the few shot examples, we generate these by basically using a sort of LM as a teacher model.

Starting point is 04:19:25 So it will basically go and like perform the task. Let's say it's like this simple example that I like explained above. Here we would have like a GBT4 say go and perform this. And if it is successful, like if the answer is correct at the end, then we would assume that it's intermediate steps. So in this case, like just the reasoning chain that it used was like a good chain of thought. And so we would keep that as a full demonstration and include that in our like few shot set. So then once we have these instruction candidates and few shot candidates for our prompt,

Starting point is 04:19:57 we can optimize over them by testing out different combinations. And right now we're using Bayesian optimization to really efficiently test out how these sort of like pieces interact to find the best one. So that's like how things work end to end. Do you have any questions on that? Yeah. So like for the single module, for example, a chain of soft, so what is behind is actually a template, right?

Starting point is 04:20:18 Yes, exactly. Okay, so you define a template, which is define how this kind of module should be working. Yes. And then in the training set, basically, oh, for it. So basically trying to like generate a possible like future example and the instruction. Yes. Okay, so after the training, the output is basically a template. Contents all the like future example and instruction.

Starting point is 04:20:40 And then when you call your program in the future, you would use, it would use that optimized template to perform your task. Okay, so why you could kind of like this despite because it's basically like a bunch of template and you just select a template and select a future example. So why, like, is this kind of declare language? What is the benefit of it? Sure. So I think the benefits are twofold. There's first in sort of like how we're allowing people to express these programs. So trying to like abstract away a lot of the messiness of like the prompt engineering

Starting point is 04:21:14 instead of the hand engineering that comes into this. So here we just like the user just needs to define what high level type of prompting technique they want to use. and then like the inputs and outputs they expect, but they don't have to like go in there and write all of these templates from scratch. And then the other piece are the optimizer. So we can optimize the prompt algorithmically like we just talked about.

Starting point is 04:21:35 We can also support fine tuning. And the nice thing is that DSPy supports like multi-stage language model program. So in like the common case, if you have multiple prompts chained together and you just have like, let's say it's a Q&A task and you're trying to solve it with this maybe more complex multi-hop program

Starting point is 04:21:54 where you're going and you're generating a query that you're then using for retrieving helpful documents and then you're using those documents to answer your question in the end. Here there are multiple steps. And you, like, so one benefit of DSPI is you can describe these in a pretty abstract, elegant way, similar to how Pi Torch works for like building AI models.

Starting point is 04:22:20 You have your layers here, which are basically the prompts and then your forward path to you define how these inputs and outputs should pass to each other. And then the nice thing is, like in this case, because you have like multiple prompts that you're using, and only your, like, the input and output that you expect, you don't have any of the labels of, like, the intermediate stages. So for example, you don't like, you're, you don't have a data set that conveys, given the question, what are good queries for this question. That's where, like, the optimizers are particularly useful because you can bootstrap these few shot examples. that show, demonstrate how to sort of do this full flow,

Starting point is 04:22:56 and then use that as fuchsite examples in your prompt going forward. Whereas without this, you would have to like handcraft all of your examples for all the intermediate steps and then try out each one to see what works best. And that just takes a long time. So for this kind of like language, we can like define a more complex prompting strategy, right? Yes, exactly.

Starting point is 04:23:18 We can actually define new prompting module and we can also define how we combine them together to get a final answer. Exactly, yes. Okay, it's quite interesting. Thank you so much. Yeah, yeah, for sure. Thanks for the great questions. I have a question.

Starting point is 04:23:30 Yes. Because essentially this is like a discrete optimization problem. Yes, yes. Where basically you have to solve a combinatorial problem, which is hard. Yes. Because a rewarding signal can be very sparse. So that means when you apply DSPi in practice,

Starting point is 04:23:47 there is a very highly, it is highly likely it will fail. It just cannot find the best prompt, or better prompt, right? Oh, interesting. Given the certain budget. Because it's not a continuous optimization problem. It's a discrete optimization. Sure. In practice, I found that it has found much more helpful prompts.

Starting point is 04:24:08 Like, for example, this even like randomly bootstrapping few-shot examples and testing which one works best optimizes or like improved over the initial program by like 10 to 15 points. So it helps in practice. I use it like, I use it like for some time. Oh cool, cool. And what I found is actually to get, let's say, useful signal or effective signal that can improve this few-shot selection.

Starting point is 04:24:36 It's actually non-trivial. Yes, yes, yes. You have to write a very nice evaluation function, which is non-trivial for a lot of beginners. Yeah, interesting. Yeah. Yeah, you're right. It depends on the metric that you're using because that's sort of ultimately what's saying, like, What is a good few shot example versus on?

Starting point is 04:24:52 The other thing that I kind of want to ask you is if you look at DSPI readme, right? So it highlights a lot about this few shot selection. Yes. But it doesn't mention enough about this instruction tuning. So what is your opinion? No question. Well, as someone who is working specifically

Starting point is 04:25:11 on the instruction tuning, I agree. We need to update the readme. So it sounds like you're familiar then with DSPI. I don't know if you've used me pro or something. Because if you try to advocate DSPIE to the prompt engineers, right, right. A lot of prone engineers, the first reflection is, oh, it helps me to ride, like my grandpa is dying

Starting point is 04:25:32 or something like that, right? It's the instruction tuning part. They do not think naturally about the few shot. Yeah, I agree. And so that's why we want to do both. I think in practice, the few shot examples are very powerful and the instructions are also very powerful too. And so one thing that we're doing is co-optimizing them,

Starting point is 04:25:49 which a lot of works haven't done. They just look at this string and optimize that. Or right now, DSPI supports the fuchsite examples, but we want to optimize both. And so we have a new optimizer. I don't know if you've used it already in DSPI called MEPRO, which basically supports optimization of just instructions, just featured examples, or the combination of the two. And we find that being able to optimize both improves performance the most. So maybe that's on me. I need to update the read me more. But yeah, there's more work. Yeah, yeah, we do have a work that we're trying to trying to work on for NIRF, so fingers crossed that we're able to finish that in time.

Starting point is 04:26:24 But yeah, there's definitely more work coming here, so stay tuned. Yeah, great questions. I just, could you tell me more just about the DSPI organization as a whole? Sure. Is this a Stanford group? How are you affiliated? Sure. I'm also interested in the relationship with Demonstrate Search Predict.

Starting point is 04:26:39 Sure, yeah, yeah, yeah. That background. Yes, so Omar is the person who started at all, and unfortunately he had- Yeah, exactly, some visa issues, so couldn't make it in person today. But yeah, he's sort of like the, origin of all of this. And then a lot of folks, as you can see on the paper, also did a lot of work and making this a reality. I personally came into this because I was seeing how like prompt engineering or manual prompt engineering was very powerful, but also very tedious in my

Starting point is 04:27:07 own work. And I like wanted a solution onto that more scaleably. So what is your work? So I was actually working on doing some healthcare applications. So we found that like optimizing prompts for for EHR retrieval tasks or like retrieving clinical insights from EHR notes was very powerful. But we need to do, again, like a lot of TES prompt engineering for that task to make it work well. And so I sort of joined Omar to begin writing better optimizers. So what we were just talking about with generating better instructions and co-optimizing those with FU shot examples as well for like full multi-stage pipelines, which is what I'm working on now with Michael, who's another Stanford, a master student, has contributed a lot to this.

Starting point is 04:27:47 But yeah, the broader DSPI community is huge. Like, it's impossible to name all of the people who have actually, like, made this reality and who have continued work on this. So there's, like, a large discord. There are a ton of open source contributors and folks who are even, like, working full-time at startups or other companies who, because they're using GESPI for their company, they'll contribute a lot of things themselves to help make the broader framework stronger, which is cool because it kind of, like, is a net positive cycle that way.

Starting point is 04:28:16 But yeah, Omar is absolutely amazing and he's done so much cool work here. So what I'm trying to understand is it's not a lab, it's not, it's a project, right? Yeah, it's a project. It's not a startup yet. No, no, yeah, yeah. It's an open source project that came out of some labs at Stanford. With some Berkeley participation. Yeah, exactly.

Starting point is 04:28:36 Mette is on the Berkeley side. I see. Yeah, so that's sort of like the orbit. Because they also have S.G. Lang, which I feel like it's kind of a little similar here. actually these two operations. I don't know if you've done the comparison. I mean we use SG Lange to in the back end to like sort of ensure that outputs are in a certain format.

Starting point is 04:28:57 So D's Pai, I think as a library, supports SG Lange. But there are others who might know the final word on that or have more information. Okay. Well, thank you very much. I didn't want to interrupt. Yeah. Here we go.

Latent Space: The AI Engineer Podcast - ICLR 2024 — Best Papers & Talks (Benchmarks, Reasoning & Agents) — ft. Graham Neubig, Aman Sanger, Moritz Hardt)

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.