Latent Space: The AI Engineer Podcast - How to train a Million Context LLM — with Mark Huang of Gradient.ai

Episode Date: May 30, 2024

Hey, everyone. Welcome to the Lidenspace Podcast. This is Alessio, partner in CTO and residence at Decibo Partners, and I'm joined by my co-host, Swix, founder of Small AI. Hey, and today we're in the remote studio with Mark Wang from Gradient. Welcome, Mark. Hey, glad to be here. It's really a great experience to be able to talk with you all. I know your podcast is really, really interesting, and I always am listening to it every time

you guys have a release. He's not a paid actor. He said that out of his own will. They will give you the check later. So you're unusual in the sense that you and I go back to college. I don't exactly remember where we overlapped, but we both went to Warren. It went into the sort of quantitative developer realm. Yeah, exactly.

Kind of crazy, right? So it all goes full circle. I was a quant for quite a few years and then made it out into Silicon Valley. And now we intersect again when it kind of feels like more or less the same, right? like the AI wars, the trading wars back of the day two to a certain extent in the grab for talent. I think there's definitely a few of us ex-finance people moving into tech and then finding ourselves gravitating towards data and AI. Seems like you did that. You were at a bunch of sort of quant trading shots, but then as you moved to tech, you were lead data scientists at Box

and staff at M. M.S.S.S. Blanc. And then before working on the startup that eventually became gradient, you want to tell that story? Yeah, I think part of the reason why I came over from the quant finance world is to get more collaboration, learn about what big data and scaling machine learning really looks like when you're not in this bubble, right? And working at Box, I work mostly in a cross-functional role, helping product analytics and go to market. And then at Splunk, it was a lot more of specific role where I was helping with streaming analytics and search and deep learning. And for gradient, like really why we started it was, whether it was in finance or whether it's in tech, I always notice that there's a little bit more

to give in terms of what AI or ML could contribute to the business. And we came out a really good time with respect to wanting to bring the full value of what that could be into the enterprise. And then obviously Open AI created this huge vacuum into the industry to allow for that. Right. So I myself felt like really, really empowered to actually ship product and ship stuff that I could think could really help people. And maybe just to touch a little bit on gradient. I know we have a lot of things to go through gradient, Lama 3, context extension. There's a lot.

But what exactly is gradient? You have an awesome design on your website. It's like really retro. And I think people that are watching fall out on Amazon Prime right now can be feel nostalgia just looking at it. What exactly is it? Because I know you have the foundry, you have the agent SDK. There's like a lot of pieces into it.

Yeah, for sure and appreciate the call out for the design. I know my co-founder, Chris, spent a lot of thought in terms of how he wanted the aesthetic to look like. And it reminds me a lot about Mad Men. So that was the initial emotional shape that I felt when I saw it. Quite simply, like, gradient work, full-stack AI platform. And what we really want to do is we want to enable all of the RPA workloads or the codified automation workloads that existed in enterprise before. we really want to enable people to transition into more autonomous, agentic workflows that are less brittle, feel more seamless as an interface too,

able to empower what we really think the new AI workforce should look like. And, you know, that kind of required us to build a fairly horizontal platform for those purposes. We have this discussion in our AI in Action Club on Discord, like the minimum viable agent or like kind of how you define an agent. In your mind, what is like the minimum? thing that you can call actually an agent and not just like the four loop, you know? And how do you see the evolution over time, especially as people adopted more and more? So I kind of stage it where

everybody, first of all, at the lowest level, thinks about like non-determinism with respect to how the pipeline looks like when it's executed. But even beyond that, this goes back into effectively evaluations. It's like on each stage of the node, you're going to have to see marginal improvement in the probability of success for that particular workload because of non-determinism. So I think it is an overloaded term to a certain extent because like everything is an agent if it calls a language model or any sort of multimodal model these days. But for us, it's like, you know, my background is statistics. So I want to see improvements in the probability of the success event or outcome happening because of more nodes. Yeah, I think, you know, the one thing

that makes this sort of generative AI era very different from the sort of data science-C type era is that it is very nondeterministic and it's hard to control. What's the founding story of gradient? Like, of all the problems that you chose, why choose this one? How did you get together with your co-founders? Anything like that? Bring us up to the present day. Yeah, so one of my co-farners is Chris, and he's a really good friend of mine as well. I don't know if you intersected with him at Penn as well. Chris Chang? Yeah, yeah. Chris Chang, who did banking for maybe one or two years and then, you know, was a software engineer at meta, also was at Google. And then most recently it was like a director at Netflix and product. And we always wanted to do something

together, but we felt what really came to fruition was wanting to develop something that is enterprise facing for once, mostly because of our experience with internal tooling, inability for something to like basically exist through like a migration, right? All the time. with every ML platform that I've ever had to experience, or he had to experience. It's like a rebuild and you rip it out. And you have a new workflow automation come in. And it's this huge multi-quarter, maybe even multi-year project to do that. And we also teamed up with former coworker Chris is from Open Door Forest,

who was also on Google Cloud Platform and him seeing the scale and actually the state of the art in terms of Google was using AI for systems before. everybody else too, right? They invented a transformer and their internal set of tooling was just so far superior to everything else. It's really hard for people to go back after seeing that. So what we really wanted was to reduce that friction for like actually shipping workloads in product value when you have all these types of operational frictions that happen inside of these large enterprises. And then really like the main pivot point for all of it was like, you said, things that can handle out of domain problems. So like out of domain data that comes in,

having the flexibility to not fall over and having something that you build over time that continues to improve. Like machine learning is about learning. And I feel like a lot of systems back in the place, they were learning a very specific objective function, but they weren't really natively learning with the user. So like that's the whole, you know, we use the term assistant all time, but my vision for the assistant was always for the system to grow alongside me, right? Almost like an embodied second limb or something that will be able to get better as you also learn yourself. Yeah, you know, people always try to define the difference between ML and AI. And I think in AI, we definitely care a lot more about out-of-domain generalization.

And that's all under the umbrella of learning, but it is a very specific kind of learning. I'm going to try to make a segue into, you know, today's main topic conversation. that's something that you've been blowing up on, which is the long context learning, right? Which is also some form of out of distribution generalization. And in this context, you're extending the context window of an existing open source model. Maybe if you want to, like, just bring us all the way back to it towards like, why it got you interested in long context, why did you find it like an interesting investment to work on and then the story of how you did your first extensions? For Lama 3, it specifically, we chose that model because of the main criticism

about it before when it first got released, 8,000 context lengths just seemed like it was too short because it seemed like mistral and even Yi came out with like a 2000 token context length model. Really, the inception of all of it was us fine-tuning so many models and working on Rags so much and having this, and it still exists today, this basically pedagogical debate with everybody who's like, hey, is it fine-tuning versus reg? Is it this versus that? And at the end of the day, it's just all meta-learning, right? Like, all we want is, like, the best meta-learning workflow or meta-learning setup possible

to be able to adapt a model to do anything. So naturally, long context had a place in that. But nobody had really pushed the limits of it, right? You would see, like, 10-shot, maybe 100-shot prompting or improving the models' capabilities. But it wasn't until Google comes out with Gem. and I, with the first one million context length model, that a lot of people's jaws dropped in that hunger for understanding what that could really facilitate and the new workflows came about. So we're staged to actually train other open source models to do that. But the moment

Lama 3 came out, we just went ham against that specific model because the two things that were particularly appealing for that was the fact that, like, I see a lot of these language models as compression algorithms to a certain extent, like the way we have 15 trillion tokens into a specific model, that definitely made me feel like it would have a lot of capabilities and be more adaptable towards extending that context length. So we went in there and, you know, the one million number, that was more of just like put the North Star up there and see if we can get there and then see what was happening along the way as we did that. So also shout out to Crusoe who facilitated all that compute because I would be lying if I was to say like,

anyone could just go out and do it. It does require quite a bit of compute. It requires a lot of preparation, but all the stars kind of aligned from that moment for us to go after that problem. I'll take a side note on Crusoe since you just brought it up. Yeah, like, can you explain what Crusoe is? I have this mental image of putting GPUs on top of oil rigs. What is it? What do they do? How do you work with them? You know, just anything nice. I'm sure they appreciate nice things that you say about them. Oh, for sure, for sure. So they came to us, through a collaborative effort where we basically were in search of a GPU provider. I don't want to call cloud service provider quite yet because then you know, you think about

the hyperscallers. But for them, you know, they're one of the biggest alternative GPU cloud providers. And they were offering up like we want to do a collaboration to showcase their technology. And it just made it really easy for us to like scale up with their L40s. And those are the specific GPU instances we used and coordinating that effort with them to get that dedicated cluster first to do the project, it became a really good relationship. And we still work with them today because we're trying to evaluate more of these models and possibly train more of them.

And anyone could go up to them and basically get your compute from them. And they have a lot of GPUs available for those type of projects. I would love to maybe have you run people through why the models don't come with longer contact sequences out of the box. Like obviously, you know, the TLDR is like self-attention is like quadratic scaling of memory. So the longer the context size, the more compute you have to spend the training time. And that's why you have to get Cruzo to help you extend it. How do you actually train large language model that is like a very long context?

And then how does that defer from just tacking it on top later? And then maybe we'll dive into performance and some of those things. But I think for a lot of folks in our audience that are more AI engineers, they use models, but don't necessarily build the models themselves. A lot of time, it's hard to understand what goes into actually making a long context model. Yeah, in terms of all the literature out there, I would say, honestly, it's probably still TBD as to like the trade-offs between the approach we did, which is more of a curriculum learning approach after the fact versus inherently training a model with a long context throughout, because I just don't think people have looked at the scaling properties of it in,

deep detail, but as stylistic facts exist out there with research papers from meta themselves, actually, they were already shown in a paper that if you train a model on a shorter context and you progressively increase that context to like, you know, the final limit that you have, like 32K is usually the limit of Lama 2 was that long. It actually performs better than if you try to train 32K the whole time. And I like to think about it intuitively as if you're trying to learn probability theory, you're not going to go and read the book cover to cover and then do all the exercises afterwards.

What you're going to do is you're going to do each chapter, do an exercise, read the chapter, do an exercise, and then finish, right, with the final set of like holistic exercises or examination. So attention is exactly what it sounds like to a certain extent. you have a bunch of indices and you're making the model attend to localize context and concepts across the entirety of its encoding, right? Like whatever the text, the sequence that you're giving it. So when you're doing the curriculum learning aspect of things, you are kind of trying to give it

the opportunity to also attend to all the concepts. So data actually, in the creation of that context, plays a huge role because a lot of people make the mistake of trying to extend the context length by just giving it raw text that doesn't have the necessity for the model to go all the way in the beginning of the sequence and then connect an idea to the end of the sequence. So data quality is one thing, but it sounds like would it still work, like the one million context if Lama 3 was 2K context size? Like, is there like a minimum context size that you need to then be able to generalize or does it not not really matter in the fine tuning context?

takes care of it. There's no minimum, I would say, or at least I can't make such a strong statement as to say that that does not exist. But if you have a 4K, any regular model out there, like, you can progressively increase the context length of it so long as it has shown really good perplexity scores prior to your context length extension. So if it hasn't shown good perplexity, it basically can't even predict the next token. You're kind of out of luck, right? But then from there, the other component that we actually just released the blog on maybe last Friday is like, you got to pay attention to the theta value that the model starts off with. What was fairly unique about the Lama 3 model was their choice of the theta parameter, which gave some suspicion

as to how long the context could be extended for the model. So that aspect of we can go into, you know, a huge lesson in terms of positional encodings and in rope scaling and stuff. But those concepts and that aspect of things enables you to scale out the length much more easily. What's the TLDR of what the type of is for a model? If I haven't built a model before, but not me. Obviously, I know what it is, but for people that don't know, right? I'm totally an expert. So not all models have it, but some models will employ rope scaling.

and Pacific Alama 3 does that, but there's also other positional encoding and embedding mechanisms that other models employ. But D-ODR is if you think about most architectures, they employ, it's kind of like a sign or cosine curve. And you're thinking about, you know, you have the amplitudes that occur there to allow for the model to, like, see different types of distributions of data.

Really what the theta value does, it governs how often a pattern's going to appear in the embedding space. You basically are able to shift that rotational curve by increasing the theta value and allow for different types of distributions to be seen as if they actually occurred in the training data before. It's super confusing, but it's like there's positional extrapolation, and then there's interpolation. you want interpolation. It's been shown that just pure extrapolation makes the model a lot worse

and it's harder to attend to stuff, whereas the interpolation is like you're squeezing everything back in to what the original context length was to a certain extent and then allowing for it to overlap different sequences that it's already seen as if it actually occurred

when you see a billion contexts of sequence tokens. So yeah, I think that aspect, we didn't know how well it would scale. I think that's one thing. So like, I'm not going to lie and tell you, like, right off the bat, we're like, we're definitely going to hit a million. It was more like we're getting to 256 and it looked good.

We did our evals. We scaled it more. And then what was really good was that we established the formula at the start. So, like, it's actually a formula that we actually took from the paper, I think it's the rope scaling paper. and we looked at that particular formula and then we backed out the values. And it's all empirical. So like, it's not like a mathematical tautology or proof.

It's an empirical formula that actually worked really well. And then we just kept scaling it up and it held. It's kind of like the scaling laws, you know? The scaling laws exist, but you don't know if they're going to continue. Yeah. Like, are you able to compare it with like other forms of scaling that people have been talking about, alibi comes to mind, yarn is being talked about a lot by news research. And then there's other forms which are not exactly directly related, but like ring attention comes up a lot.

We had a really good session with strong compute in the latent space discord talking about all these approaches. I just wondering if you want to compare and contrast like rope versus the other stuff. Yeah, I think alibi, we haven't compared with that one specifically, mostly because I've noticed some of the newer architectures don't actually employed a lot. I think the last architecture that actually really employed it was the Mosaic FPT model class. And then almost all the models these days are all rope scaling. and then effectively you can use yarn with that as well. We just did the theta scaling specifically because of its empirical elegance, really easy and it was well understood by us.

The other one I know that in the open source that people are applying, which uses more of a Laura-based approach, which is really interesting too, is the one that Wing has been employing, which is Pose. You know, we sort of helped them evaluate some of the models. With respect to the performance of it, it does start to break down a little bit more on the longer, longer contact. So like 500,000 to a million, it appeared that doesn't hold this well specifically for

like needle in the haystack. But, you know, it's still TBD as, you know, evaluations. It's a sparse high dimensional space where it was just like evaluating performance across so many different things and then trying to map it back to like, hey, here's the thing that I actually cared about from the start. And I have like a thousand different evaluations. And they tell me something, but not the entire picture. And as for like ring attention specifically, we employed ring attention in order to do the training.

So we combined flash attention and ring attention together with like a really specific network topology on our GPUs to be able to maximize the memory bandwidth. Yeah, as far as I understand, like reintention, you know, a lot of people credit it for Gemini's million token context, but actually it's just a better utilization of GPU. Yeah. That's really what it is. You mentioned in our show notes, Zhang Pei Un's easy context repo. I have seen that come up quite a bit.

What does that do as, you know, like, how important is it as ringing attention implementation? I know there's like maybe another one that was done by Lucid Raines or one of the other. Yeah, open source people. But like, what is easy context? Is that the place to go? Like, did you evaluate a bunch of things to implement ring attention? Yeah, we evaluated all of them. I would say the original authors, you know, Matai and all the folks at Berkeley,

they created the Jack's implementation for it. And unfortunately, not to discredit TPUs or whatever, like the Jackson implementation just does not work on GPUs very well. Like any naive setup that you do, like it just won't run out of the box very easily. And then unfortunately, that was probably the most mature repo with a lot more configurations to set up interesting network topologies for your cluster. And then the other Pye Torch implementations outside of Easy Context, they just didn't really work.

Maybe we weren't implementing one small aspect incorrectly, but there was an active development on it at a certain point. Like even Lucid Raines, I think he's interesting because for once he was actually like, he was like taking a job somewhere and then just stopped doing commits. And as we were working to try to find it, we never really want to jump in on a repo where someone's like kind of actively committing breaking changes to it. Otherwise, we have to like eat that repo ourselves.

And Easy Contacts was the first PiTorch implementation. that applied it with native libraries that worked pretty well. And then we adapted it ourselves in order to configure it for our cluster network topology. So, you know, shout out to Jampin for his open source contributions. I think that we look forward to possibly collaborating him and push that further in the future because I think more people, if they do want to get started on it, I would recommend that to be the easiest way, unless you want to, like, I don't know how many people know Jacks. Me personally, I don't really know it that well.

So I'm more of a pie torch guy. So I think he provides a really good introduction to be able to try it out. And so one time you had the technical discovery. What about the actual customer interest, customers that you work with? I feel like sometimes the context size can be a bit of a marketing point. You know, people are like, oh, yeah, well, no, one million. 2 million, 3 million, 4 million. That's kind of the algorithms side of it.

How do you power the training? But the other side is obviously the data that goes into it. There's both quantity and quality. I think that's how one of your tweets, you trained on about 200 million tokens for the AB model to the context extension. But what are the tokens? You know, how do you build them?

What are like maybe some of the differences between pre-training datasets and context extension datasets? Yeah, any other color you give there would be great. So specifically for us, we actually staged two different updates to the model. So our initial layer that we trained was just basically like a pre-training layer. So continual pre-training where we took the Slim pajamas data and then we filtered it and concatenated it so that it would reach the context lengths that we were trying to extend out to. And then we took the Ultrat dataset, filtered it down.

or maybe some other, you know, second order derivative of the ultra chat dataset that was curated and then filtered it down and then reformatted it for our chat use case. For those two datasets, you always have to really keep in mind for the pre-training data, whether or not you may be like cutting off tokens in weird ways, whether or not, you know, the content is actually diverse enough to retain the ability of the model. So slim pajamas tends to be one of the best ones, mostly because it's a diverse data set. And you can use embeddings too as a pre-filtering step as well, right?

Like how diverse are your embedding space to the original corpus of the model and then train on top of that to retain its abilities? And then finally, for the chat dataset, making sure that it's attending to all the information that would be expected to really stretch its capabilities. Because you could create a long context data set where every single time, the last 200 tokens can answer the entire question.

And that's never going to make the model attend to anything. So it's even something that we're doing right now is trying to think about how do we actually improve these models and how do you uplate the datasets such that it can expose even more nuanced capabilities that aren't easily measurable quite yet. Is there a ratio between diversity of the dataset versus diversity compared to what the model already knows?

Does the model already need to understand a good part of the new, like the context extension data to function? Like, can you put a context extension dataset that is like very far from like what was in the pre-training? I'm just thinking as the model get older, some of the datasets that we have might not be in the knowledge of the existing model. trying to extend? I think that's always a consideration. I think specifically you really got to know how many tokens were expended into that particular model from the start. And all models these

days are now double-digit trillions, right? So it's kind of a drop in the bucket if you really think I can just put a billion tokens in there. And I actually think that the model is going to truly learn new information. There is a lot of research out there between the differences with respect to full fine tuning, which we applied full fine tuning versus lower base fine tuning. It's a trade-off. And my opinion of it is actually that you can test certain capabilities and you can kind of inject new knowledge into the model. But to this day, I've not seen any research that does like a strong, well-scaled out empirical

study on how do you increase the model's ability to understand like these decision boundaries. with a new novel data, most of it is holding on a portion of the data as like novel and then needing to recycle some of the old knowledge. So it just doesn't forget and get worse at everything else, right? Which was seen, we do have historical precedent where the original cold Lama was trained further from Lama 2 and it just lost all its language capability, basically, right? So I don't want to call that project like deem it as a failure, but it wasn't a really successful generalization exercise because, you know, these models are about flexibility and being like generic to a certain extent. One thing I see in the recent papers that have been coming out

is this sort of multi-stage training data. And if you're doing full fine-tuning, maybe the move or the answer is don't train 500 billion tokens on just code because then, yeah, it's going to massively overfit to just code. Instead, like maybe the move is to slowly change the mix over the different phases, right? So, in other words, you still need to mix in some of your original source data set to make sure it doesn't deviate too much. I feel like that is a very crude solution. Maybe there's some smarter way to adjust the loss function so that it doesn't deviate or overfit too much to more recent data. It seems like it's a solvable thing, that's what I'm saying. Like this overfitting to more recent data issue. Yeah, I do think solvable is hard. I think provably solvable is always

something that I know is extremely difficult, but from a heuristical standpoint, as well as having some sort of statistical efficiency on how you can converge to the downstream tasks and improve the performance that way in a targeted manner, I do think there are papers that try to do that. Like the Do Ramee paper, I think it was released last year. It was really good about doing an empirical study on that. I think the one thing people struggle with, though, is the fact that they always try to do it on pretty naive tasks. Like you target like a naive task and then you create your data mixture

and you try to show some sort of algorithm that can retain the performance for those downstream tasks. But then what do we all care about are actually like really, really interesting complex tasks, right? And we barely have good evaluations for those. If you do a deep dive at the Jim and I-5 technical paper, which they just updated, It was a fantastic paper with new updates.

If you look at all of their long context evaluations there, like a lot of them are just not something that the open community can even do because they just hired teachers to evaluate whether or not this model generated a huge lesson plan that is really coherent or like you hire a bunch of subject matter experts or, you know, they taught the model how to do language translation for extinct language. where only 200 people in the world know. It's kind of hard for us to do that same study as an early-stage startup. I mean, technically now you can use Gemini as a judge.

Gemini is touting a lot of their capabilities and low-resource languages. One more thing before on the sort of data topic, did you have any exploration of synthetic data at all? You know, use Mistral to rephrase some existing part of your data set, generate more tokens, anything like that, or any other form of synthetic data that you choose to mention. I think you also mentioned the large world model paper, right? Yeah, we use GPT4 to re-fairies certain aspects of the chat data, reformatting it,

or kind of generating new types of tokens and language and, you know, types of data that the model could see. And also like trying to take the lower probability, right, or the lower correlated instances of out-of-domain data that we wanted to inject it to the model too as well. So I actually think a lot of the mo is in the data pipeline. You'll notice most papers just don't really go into deep detail about the data set creation because, I mean, there's some aspects that are like uninteresting, right? Which is like we paid a bunch of people and like generated a lot of good data.

But then the synthetic data generating pipeline itself, you know, sometimes that could be like 25% or 50% of the entire data set that you've been used to depreciating. Yeah, I think it's just for legal deniability. No, it's too boring. I'm not going to say anything because it's too boring. No, it's actually really interesting. And in fact, it might be too interesting, so we're not going to say anything about it. One more question that I had was on Laura and taking some of these capabilities out and bringing them to other model.

You mentioned Wings work. He tweeted about we're going to take this Lora adapter for the gradient 1 million context extension, and you're going to be able to apply that to other model. Can you just generally explain to people how. these things work with language models. I think people understand it with stable diffusion. You have these, like, lower patches for, like, different type of styles.

Does that work similarly with LMs? And is it about functionality? Can you do lower patches with specific knowledge? Like, what's the state of the art there? Yeah, I think there's a huge kind of resurgence in what I would call, like, model alchemy to a certain extent. Because you're, like, taking all of these lores and you're mixing them together. And then, like, that's a lot of the model merging stuff that I think Charles Godard does and a lot of others in the open community, right?

Because it's a really easy way. Like, you don't need training and you can test and evaluate models and take the best skills and mix and match. I don't think there has been as much empirical study, like you're saying, for how shows the same type of, like, it's not as interpretable as, like, stable diffusion to a certain extent. because even we have experimented with taking like deltas in the same methodology as wing where we'll take a delta of like an already trained model try to see how that has created like in a sense an R-O-HF layer taking the Lama instruct layer subtracting the base model from that and then try to apply that lower adapter to like another model and seeing what it does to it. It does seem to have in effect, though. Like, I will not lie to say, I'm really surprised how effective it is sometimes.

But I do notice that for more complexibilities, other than, like, more stylistic stuff, it kind of falls through because maybe it's, it requires much deeper path in the neural network, right? Like, all these things, these weights are just, like, huge trees of paths that the interesting stuff is the road less travel to a certain extent. And, when you're just like merging things brute force together that way, you don't quite know what you'll get out all the time. Like there's a lot of other research that, you know, you have merged ties and you have all these different types of techniques to effectively just apply like a singular value decomposition on top of weights and just get like the most important ones and prevent

interference across all the other layers. But I think that that is extremely interesting from developer community and I want to see more of it, except it is to a certain extent kind of polluting the leaderboards these days because it's so targeted. And now you can kind of game the metric by just finding all the best models and then just merging them together to do that. And I'll just add one last bit is basically the most interesting part about all that actually to me is when people are trying to take the lores as a way of like short-circuiting the training process. So they take the lowers they merge it in, and then they'll fine tune afterwards. So like the fine tuning and the re-initialization of a little bit of noise into all the new merged models

provides like kind of a learning tactic for you to get to that capability a little bit faster. There's a lot there. I really like the comparison of ties merging to singular value decomposition. I looked at the paper and I don't really think I understood it on that high level until you just said it. We have to move on to benchmarking. This is a very fun topic. Needle on a haystack. What are your thoughts and feelings? And then we discuss the other benchmarks first.

But needle in a haystack. You want to put me on the spot with that one. Yeah, I think needle in the haystack is definitely like the standard for presenting the work in a way that people can understand. And also proving out, I view it as like a primitive that you have to pass in order to give the model any shot of doing something that combines both like a more. holistic language understanding and instruction following, right? Honestly, like, it's mostly about, if you think about the practical applications of long context and what people complain most about models when you stuff a lot of context into it is either the language model just doesn't care about what you asked it to do,

or it cannot differentiate context that you want it to use as a source to prevent hallucination versus, like, instructions. I think that, you know, when we were doing it, it was to make sure that we were on the right track. I think Greg did a really great job of creating metric and a benchmark that everybody could understood. It was intuitive. Even he says himself, we have to move past it. But to that regard, it's a big reason why we did the evaluation on the ruler suite of benchmarks, which are way harder. They actually include needle in the haystack within those benchmarks, too. And I would even argue is more comprehensive than the benchmark that Gemini released for their like multi needle in the

haystack. Yeah, you mentioned quite a few. You mentioned Ruler, Lugo, Infinite Bench, Bamboo, Zero Scrolls. Do you want to give us maybe two or three of those that you thought were particularly interesting or challenging and, you know, what made them stand out for you? There's just so many and there's so nuanced that I would say like, yeah, zero scrolls was the first one I'd ever heard of coming out last year and it was just more of like tracking variable overlong context. I'll go into ruler because that's the freshest in my mind and like we're just scrutinizing it so much and running the evaluation in the previous two weeks. But like Ruler has four different types of evaluations. So the first one is exactly needle in the haystack, it's if you throw multiple needles.

So you got to retrieve, you know, multiple key value pairs. There's another one that basically you need to differentiate. Multi value, multi-quiry. Yeah, yeah. Multi-value, multi-quiry. That's the ablation. There's also a variable tracking one where you go, hey, if X equals this, Y equals this, Y equals Z, like, what is this variable? And you have to track it through all of that context. And then finally, there's one that is more of like creating a summary statistic. So like the common words one, where you choose a word that goes across the entire context, and then you have to count it. So it's a lot more holistic and a little bit more difficult that way. And then there's a few other ones that escape me at this moment. But the ruler really pushes you, if I think about the

progression of the evaluations, it start to force the model to actually understand, like, the totality of the contact. Like, everybody argues to say, couldn't I just use, like, a retrieval to, like, just grab that variable rather than paid $10 for one shot or something, although it's not as expensive. The main thing that I struggled with, with even some of our use cases, where, like, when the context is scattered across multiple documents, and you have, like, really delicate plumbing for the retrieval step, but it only works for that one that really specific instance, right?

And then you throw in other documents and you're like, oh, great, my retrieval doesn't grab the relevant context anymore. So that's the dream, right, of getting a model that can generalize really well that way. Yeah, totally. And I think that probably is what Greg mentioned when saying that he has to move beyond needle haystack. You also mentioned you extended from 1 million to 4 million token context recently, and you saw some degradation in the benchmarks too. You want to discuss that? So if you look at our theta value at that point, it's getting really big. So think about floating point precision and thinking about basically now you're starting to run into problems where in a deep enough network and having to do joint

probabilities across so many tokens, you're hitting kind of the upper bound on accuracy there. And there's probably some aspect of clamping down certain activations that we need to do within training. Maybe it happens at inference time as well with respect to like the theta value that we use and how do we ensure that it doesn't just explode. If you've ever had to come across like the exploding gradients or the vanishing gradient problem, you will know what I'm talking about. A lot of the empirical aspect of that and scaling up these things is experimentation and figuring

out how do you kind of marshal these really complicated composite functions such that they don't just like do a divide over zero problem at one point. Awesome. Just to wrap, there's the emails, and then there's what people care about. You know, there's two things. Do you see people care about above one million? Because Gem and I had the two million announcement and I think people were like, okay, one million, two million. It's whatever. Like, do you think we need to get the 10 million to get people to care about again? Yeah. Like, do we need to get to 100 million? I mean, that's an open question. I would certainly say a million seemed like the number that got people really excited for us.

And then, you know, the 4 million is kind of like, okay. Rather than like a breakthrough milestone, it's just the next incremental checkpoint. I do think even Google themselves, they're evaluating and trying to figure out specifically how do you measure the quality of these models and how do you measure and map those to, capabilities that you care about going down the line, right? And I think us as a company, we're figuring out how to saturate the context window in a way that's actually adding incremental value. So the obvious one is code because code repositories are huge. So can you stuff the entire context of a repo into a model and then make it produce some module that is useful or some suggestion that is

useful. However, I would say there are other techniques like alpha-coding and flow engineering that if you do iterative things in a more agentic manner, it may actually produce better quality. I would preface and I would actually counter that maybe start off with the use case that people are more familiar with right now, which is constantly evolving context in like a session. So like, whereas you're coding, right, if you can figure out evals that actually work where you're constantly providing it multiple turns, and each incremental turn has a nuance aspect, and you have a targeted generation that you know of, making the model track state and have state management over time is really, really hard. And it's an incredibly hard evaluation will probably

only really work when you have a huge context. So that's sort of what we're working on trying to figure out those types of aspects. You can also map that. It's not just code. State management exists. We work in the finance sector a lot, like investment management, having a state management of like a concept and stuff that evolves over like a long session. So I'm super excited to hear what other people think about the longer contacts. I don't think Google's probably investing to try to get a billion quite yet. I think they're trying to figure out how to fully leverage what they've done already. And does this change in your mind for very long chats versus a lot of documents.

The chat is kind of interactive, you know, and information changes the documents here are just trying to synthesize more and more things. Yeah, any thoughts on how those two workloads differ? I would say, like, with the document aspect of things, you probably have a little bit more ability to tweak other methodologies. Like, you can get around the long context sometimes where you can do retrieval augmented generation

or you do hierarchical recursive summarization, Whereas evolution in like a session, because that state variable could undergo pretty rapid changes, it's a little bit harder to you getting around that without codifying a really specific workflow or like some sort of state clause that is going back to like determinism, right? And then finally, what I really think people are trying to do is figure out how do all these shots progress over time. How do you get away from the brittleness of the retrieval step? If you shove in a thousand shots or 2,000 shots, will it just make the retrieval aspect of good examples irrelevant?

Kind of like a randomly sampling is fine at that point. There's actually a paper on that that came out from CMU that they showed with respect to a few extraction or classification, high cardinality benchmarks. They tracked fine-tuning versus in-context learning versus many, many shot in context learning. And they basically showed that many, many shot in context learning helps to prevent as much sensitivity around the examples themselves, right? Like the distraction error that a lot of LLMs get work. You give it irrelevant context and it literally can't do the task because it gets

sort of like a person too, right? Like you got to be very specific about, I don't want to distract this person because then, you know, they're going to go down a rabbit hole and not be able to complete the task. Yeah. Well, that's kind of the flip side of the needle in a haystack. thing too in a bit. It's like, now the models pay attention to like everything so well. Like sometimes it's hard to get them to like, I just said that once, please do not bring that up again.

You know? It happens to me with code. Yeah, it happens to me with like CSS style sometimes and things like that. If I have a long conversation, it tries to always reapply certain styles even though I told it. Maybe that's not the right way to do it. But yeah, there's a lot, again,

of an empirical that people will do. And just, I know we we kind of went through a lot of the technical side, but maybe the flip side is why is it worth doing? What I like the use cases that people have that make long context really useful. I think you have a lot of healthcare use cases. I saw on your Twitter. You just mentioned the finance use case. Obviously, some of the filings and documents the companies published can be quite worthy.

Any other things that you want to bring up, maybe how people are using gradient, anything like that. I think that will have a clear picture for people. Yeah. So beyond just using the context for, you know, sessions and even. evolving state management, it really comes down to something that's fairly obvious, which everybody's trying to do and work on, is how do you ground the language model better? So I think when you think pure text, that's one thing. But then multimodality, it's going to be pivotal for long context.

Just because videos, when you're getting into the frames per second and you're getting into lots of images and things that are a lot more embodied, you need to utilize and leverage way more, way more tokens. And that is probably where, you know, us as a company, we're exploring more and trying to open up the doors for a lot more use cases because I think in financial services, as well as healthcare. We've done a good job on the tech side, but we still need to push a little bit further when we combine, you know, a picture with words, like a chart with words or somebody's medical image with words, stuff like that. you definitely can do a better job. You know, it's timely too because meta just released the new Camillion paper that does multimodal training. And it shows that early fusion is more sample efficient, right? So having that kind of view towards the future

is something that we want to be primed to do because, you know, it's similar to what Sam Altman says himself too, right? You need to just assume that these models are going to be 10x better in the next few years. And if you are primed for that, that's where you have kind of a business that, you know, you're not just pivoting after every release or every event, you know, that drops. I think the thing about this 10x issue is that the 10x direction moves all the time.

You know, some people were complaining about GPT40 that the ELO scores for GPT4O actually, in reality, weren't that much higher than GPT4 Turbo. And really, you know, so it's not 10X better in reasoning. It's just 10x better in the integration of alternate modalities. And by the way, look over here. There's a really sexy voice chat app that they accidentally made that they had to deprecate today. The 10x direction keeps moving.

Now it's like, you know, fully in like sort of multimodality land, right? And so can 10x in various ways. But like you, you guys have 10x context length. But like, are we chasing the last war? Because like, nobody cares about context like that's like multimodity time. I'm joking. Obviously, people do care about it. I wonder about this comment about this 10xing every single time.

You know, that's honestly why we kind of have our eye on the community as well as you, right? Like with your community and in the things that you hear, you know, you want to build where, you know, we're a product company. We're trying to build for users, trying to listen to understand what they actually need. Obviously, you know, you don't build everything that people ask you to build, but no, it's useful, right? Because I think that you're totally right there. If we want to make something 10x better in a certain direction, but nobody cares. and it's not useful for somebody, then it wasn't really worth the while.

And if anything, maybe that's the bitter lesson 2.0 for so many tech startups. It's like build technology that people care about and will actually 10x their value rather than build technology that's just 10x harder. I mean, that's not bitter lesson. That's just Paul Graham. Yeah.

One more thing on the chameleon paper, I was actually just about to bring that up, you know, so on AI news, like my daily newsletter, it was literally my most recent, featured paper. And I always wonder if you can actually sort of train images onto the same latent space as words. That was kind of done with like, you know, what we now call late fusion models with lava and flamingo and, you know, all the others. But now the early fusion models,

like Chameleon seem to be the way forward. Like obviously it's more native. I wonder if you guys can figure out some kind of weird technique where you can take an existing like Lama 3 model and early fuse the images into the text encoder so that we just recognize. I'd retroactively have the early fusion models. Yeah. Even before the Cumulian paper came out, I think that was on our big board of next to-dos to possibly explore, or our backlog of ideas, right?

Because, as you said, even before this paper, I can't remember. I think meta even had like scaling laws for multimodality paper that does explore more early fusion. Like the moment we saw that, it was just kind of obvious to us that eventually it'll get to the point that becomes a little bit more mainstream. dream. And yeah, like, that's a cool twist that we've been thinking about too as well, as well as

like other things that are kind of in the works that are a little bit more agentic. But if open collaboration interests you, we can always work on that together with the community. Ooh, okay. Shut up there. We can leave that in the call to action at the end. You know, we have a couple more questions to round this out. You mentioned a lot of papers in your work. You're also building a company. You're also looking at open source projects and community. What is your daily or weekly routine to keep on top of AI. So one, subscribe to AI news, he didn't have to pay me to say that. I actually really think like it's a good aggregator. I think it's a good aggregator. I'll tell you why. Most of the fastest moving research that's being done out there, it's mostly on Twitter. I wasn't a

power Twitter user at all before three years ago, but I had to use it and I had to always check it in order to keep on top of early work, right, that people want to talk about or present. Because nothing against submitting research papers to like ICLR or ICML. Knowing the state of the art, those are like six months late, right? Like people already have it dropped it on archive or they're just openly talking about it. Yeah, and then being on Discord to see when the rubber hits the road, right, the implementations and the practices that are being done or like the datasets, like you said. A lot of conversations about really good datasets and how do you construct them

are done in the open in figuring that out for people that don't have budgets of like $10 million to just pay a bunch of annotators. So my routine daily is like, you know, second thing I do when I wake up is to look on Twitter to see what the latest updates are from specific people that do really, really great work. Armine at Meta, who did the Camillion paper, everything he writes on Twitter is like gold. So like anytime he writes something there, like I really try to figure out what he's actually saying there and then tie it to techniques and research papers out there. And then, you know, sometimes I try to use certain tools. I myself use AI itself to search for the latest papers on a specific topic, if that's the thing, on the top of my mind.

And at the end of the day, trying out the products too. I think if you do not try out the tooling and some of the products out there, you are missing out on someone's compression algorithm. like they compressed all the research out there and all the thought and all the state of the art into a product that they're trying to create for you and then really backing out a reversal engineering what it took to build something like that that's huge right like if you can actually understand perplexity for instance you'll already be well ahead on the research oh by the way you mentioned what's a good perplexity score there's like just a number right like it's like five to eight or something like do you have a number in mind when you said that yeah i mean flipping between

train loss and perplexity is actually not native to me quite yet, but if you can get like a four using the context length extension on Lama, like you're in the right direction. And then obviously you'll see spikes. And specifically when the one trick you should pay attention to is you know that your context length and theta scaling is working right if the early steps in the perplexity goes straight down. So like when it wasn't correct, it would oscillate a lot in the beginning. And we just knew that we cut the training short and then retry a new theta scale. you're properly continuing fine-tuning or the full pre-training. Yeah, yeah.

The model just saw something out of domain immediately and was like, I have no idea what to do. And you need it to be able to overlap that positional embedding on top of each other. One follow up, right, before we sort of close out. I think being on Twitter and looking all these new headlines is really helpful. But then it only gets you like a very surface level understanding. Then you still need a process to decide which one to invest in. I'm trying to dig for like what is your formula.

for like deciding, you know, what to go deep on and what to kind of skip. From a practical standpoint, as a company, I already know there are like three to five things that will be valuable and useful to us. And then there's other stuff that's like out of scope for different reasons. Some stuff is like out of scope from, hey, this is not going to impact or help us. And then other things are out of scope because we can't do it. A really good instance for that is specific algorithms for improving extremely large-scale distributed training. We're not going to have the opportunity to get 2,000 H-100s. If we do, it'd be really cool.

But like I'm just saying, as for now, you've got to reach for the things that would be useful. Things that would be useful for us, for everybody, actually, to be honest, is evaluations, different post-training techniques, and then synthetic data construction. I'm always on the look for that. And then how do I figure out which new piece of news is actually novel? Well, that's sort of my mental cache to a certain extent. Like I've built up like this state of, hey, I already know all the things that have already been written for the state of the art for certain topic areas.

And then I know what's being kind of recycled as an empirical study versus something that actually is very insightful. Underrated specific instance would be the deep seek paper where I'd never seen it before, but the multi-head latent attention, that was really unexpected to me because I thought I'd seen every way that people wanted to cut like mixture of experts into interesting ways. And I never thought something would catch my eye to be like, oh, this is totally new. And it really does have a lot of value. That's mainly how I try to do it. And you talk to your network too. I just, you know, talk to the people and then know and make sure I have certain subject matter experts on speed dial

that I also like to share information with and understand, hey, does this catch your eye too? Do you think this is valuable or real? Because it's a noisy space we're in right now, which is cool because it's really interesting and people are excited about it. But at the same time, there is actually a 10x or more explosion of information coming in that all sounds really, really unique and new. And you could spend hours down a rabbit hole that isn't that useful. Awesome.

Mark, kind of we kept you in the studio for a long time. Any final call to actions for folks that could be roles you're hiring for, quest for startups, anything that comes to mind that you want to share with the audience? We definitely have a call to action to get more people to work together with us for long context evaluations. That is sort of the it topic throughout even meta or Google or any of the other folk are focusing on because I think we lack an understanding of that within the community. and then can we as a community also help to construct other modalities of data sets that would be interesting?

Like pairwise datasets, right? You could get just straight video and then straight text, but getting them together for grounding purposes will be really useful for training the next set of models that I know are coming out. And the more people we have contributing to that would be really useful. Awesome. Thank you so much for coming on, Mark. This was a lot of fun. Yeah, thanks a lot. Yeah, this is great.