The Startup Ideas Podcast - GLM 5.2 Clearly Explained (and how to set it up)

Starting point is 00:00:00 Okay, you've probably heard of GLM 5.2 that's going viral everywhere on Twitter. Yes, it's this new open source local AI model that people are saying is the chat GPT moment for local AI. But no one's actually gone and shown you how to use it and how do you actually set it up. So I figured I'd bring on my friend Amir. He tells us exactly how you should think about running GLM 5.2, how you should think about running local models, how that integrates to something called open router, how you can use it with your codex or cursor or cloud code. And this episode, in 20 minutes or less,

Starting point is 00:00:38 you're going to get everything you need to know about local AI models, why GLM 5.2 is crushing benchmarks, and how you can set it up today so you can go and build your startup, and build your business, and be more productive and be more efficient. Enjoy the episode, and I'll see you at the end of it. Hit a like, comment, and subscribe for more of this sort of stuff.

Starting point is 00:00:59 stuff in your feed. Enjoy. Welcome to the show, Amir. By the end of this episode, what are we going to learn? We're going to talk about essentially learn about how local models are kind of keeping up now with the pace of these closed models as well and how you can kind of use compounding models or fusion models as open router calls it to be able to do sequencing between a more extensive thinking model and a more execution-based model. We'll show you how GLM 5.2 actually compares and stacks up against other models and how you can effectively use it and get set up with it as well. Cool. I did an episode on local models. It was a hit. People wanted more tactical. How do I actually, you know, how should I think about local models? How do I implement local models? How I can actually

Starting point is 00:01:53 make this a part of my daily workflow. So I brought on a mirror. Welcome to the show. And let's, I give you two welcomes, by the way. That's how excited I am to have you share with everyone, everything. So let's get into it. I'm super excited to be here. Let's jump right into it. So let's talk about what happened this week. ZAI came out with JLM 5.2. And I think this was a big inflection point because we've typically seen with local models, it's either storage extensive that you can't essentially run it or install in your computer or you need a better like GPU RAM performance to be able to actually run it locally as well. Now JLN 5.2 is also resource extensive. But what we're seeing is with open source models like open source providers like open router or or llama being

Starting point is 00:02:41 able to help you run these models in the cloud and effectively being able to essentially pay slightly less for input and output tokens compared to the more closed models. Now, what I want people to take away from this session is one, how to actually get set up with it. We're not going to go through the detailed setup process, but I'm going to just cover how you can do it in cursor using open router or in codex. And then, if effectively talk about how GILM 5.2 stacks up against the other models, and then how you can effectively use it as well to do model training. And then we'll do model training. And then we'll do just kind of a maybe a quick walkthrough of how I'm currently using it. You know, I want to be very honest,

Starting point is 00:03:18 like these local models still have a lot of work to do in terms of having tool capabilities to be able to, you know, have the modalities to be able to see images and conceptualize on what they're looking at and I'm going to tell you how you can effectively circumvent some of that where you can use other models to explain what the image is back to GLM and then have GLM work on it. And also just have a very live test on how this stacks up against other models. You know, benchmarks are great. Personally, I'm not an expert in it. I don't know what any of these benchmarks actually mean. The way I do is off-ups. Like, let's build it out and see how this actually looks and how it stacks up against other models. Sound good? Yes, sir.

Starting point is 00:04:01 Let's do it. Okay, so G-Lon 5.2 came out and essentially it has a 1 million context window and it scores 81 points on the terminal bench 2.1. It's just about four points behind Opus 4.8. And it does quite well on the long horizon task evaluation. So this is essentially projects you have that you want to run long sequence tasks on and you know, I think it's seeing an account like the thinking parameters and how it can think through and plan through some of the task it has a hand.

Starting point is 00:04:33 So you can see that across all these different kind of benchmark reporting reports. GLM 5.2 actually does quite well. So in this case it's 62.1% compared to open system 69.2. What's special about GLM is it's open source. You can run it locally on the machine if your device can support it or you can run it in the cloud through the open model providers. It's a big leap from 5.1. I personally didn't test 5.1.

Starting point is 00:05:01 I got straight into 5.2, but from what I'm seeing based off like Twitter and conversations with people, it's performing quite well, especially on the front end side of just execution-based tasks. I haven't really tested more on the backend resource extensive tasks, but just based on perception and what we're seeing in the reporting, it's stacking quite well. So when you say stacking well, we're talking like, because like I look at benchmarks and honestly it goes through, you know, one year. or one eye and it goes at the other.

Starting point is 00:05:31 I'm like, I glaze over it because I'm like, what does this really mean? Like, are we talking? Is it like 4.8? Is it like 5.5? Yeah. What do we, how should I think about this? And yeah, give it to me straight.

Starting point is 00:05:45 Honestly, man, I'm the same. I don't get it. You know what I mean? Like I'm not smart enough to understand how these benchmarks actually stack. So I'll be honest. For me, it's like, let's just build it, use it and see how we feel about,

Starting point is 00:05:57 you know, how it performs compared to the other. models. And for me, I want to get the best out of it. So if I feel like JLM 5.2 is strong in one part, but weak on the other, then I'm thinking about how do I actually use other tools or other models to essentially now I think like almost like a fusion approach. And I think open router, which is one of like the model providers coined this where it's like you're able to do like sequencing between two different models to get the best output. So I'm totally game on. If I can run a local model on my machine to do certain tasks, but then call, you know, Opus or Codex to do something else and have them work together, by all means. I want to be the most token and cost efficient and performance as well.

Starting point is 00:06:41 Yeah. So on the setup side, I personally started using this through cursor using OpenRouter's API. So how it works essentially is you got to go to ZAI, which is the GLM provider. they created the GLM 5.2 model, you get an API key from them, and then you take that key and you go into your cursor settings, paste it into the OpenAI field, and then from there, you override the Open AI endpoint with this API endpoint right here, and essentially from there, you go back to models, add a custom model, GLM 5.2, and you're able to then actually call GLM 5.2 directly. So, in essence, instead of Open AI key, you put your GLM,

Starting point is 00:07:26 API key from ZDi AI, then you override the API endpoint for when you call OpenAI chat completion with this one right here and then you go back into custom models and add the custom model protocol. You can also alternatively do this using OpenRouter. So if you want, if you're using Codex, you can go to OpenRouter, get your OpenRouter key, and then go into the provider, get the endpoint, and then go into Codex, create a profile and say, hey, I want you to install this model, open source model. Codex does actually support open source models. So you're able to provide the details of what the model is, the context window, and then essentially when you're running codex to the CLI, you can switch to GLM 5.2. Easy enough. Yeah, easy enough, you know, and then maybe we

Starting point is 00:08:14 can have a page or something to show later on where they can kind of follow these instructions. There's a lot of guides on Twitter and online. You can follow those. But essentially, in my opinion, the best way to get started with this is just go to open router and cursor and get that set up through and through. So let's talk about the model. We've talked about the benchmarks, how it performs. Really, if we want to just at least take, you know, put some weight to the benchmarks, I'd say if they're scoring at 62% and Op. Its 4.8 is 69, you know, that probably means something. You know, for the normal people, we probably won't really know until we actually play around with it. But I went in and was just looking at, for example, this website we have, this is a small app.

Starting point is 00:08:56 And I built this in, I think, Opus 4.8. I was just testing it around. And I started refining the design using Geoanum 5.2. So I was like, hey, redesign the hero section for me or refine it. There's this section right here with all these images. I was like, why don't we just do a little like carousel style? So it, you know, it's fascinating in a way because I don't, I personally don't think open the local models had this kind of capability to be able to get it so refined and accurate

Starting point is 00:09:27 previously and I tested the models like that we had and I find that GLM 5.2 is a lot more refined and it's able to follow the instructions on like what you want to do so in a couple of prompts I was like hey you know let's do a carousel here let's make sure we you know are able to show the images and then from there I want you to build out like a bento grid style of all the features that we have. Now, this is all in one prompt, obviously, you know, you can see it's a little bit of vibe coded here. He has a little side like badge, the labels, you know, and you can tell. But at the end of the day, for a local model, for like, you know, a local model, if you're running this computer and not burning any tokens, it's doing quite well.

Starting point is 00:10:11 And I think, um, I can see how it staffed like internally reporting. Yeah. I mean, I think so like, Okay, what is the main benefit of using a local model versus something in the cloud is you don't burn tokens. You essentially, the way to think about it is you're buying a machine and correct me if I'm wrong, but you're buying a machine. And we should talk about some of the machines that people could potentially buy. But you're buying machine. It's a one size, it's a one, it's a cost. It could be $2,000, $5,000, $10,000. And then you can just run tasks, right?

Starting point is 00:10:54 So you're building a startup. You just want, maybe you want conversion rate optimization. So maybe you say every day, I'm going to feed you customer feedback. And every single day, I want you to work on the front end. And you just do that. So my question to you is for local models specifically, for people who actually want to be building, companies. Shouldn't they be basically running it all the time

Starting point is 00:11:17 on certain tasks? And how should people be thinking about it? Yeah. So I think that's a little bit tough to answer and I'll say why because this model specifically is really resource and intensive. So a lot of I think existing consumer computers

Starting point is 00:11:33 may not be able to run this from what I've seen. I've been running it on the cloud through open router directly. And what's the cost with that. Yeah. So I actually was trying to map out the token cost of model training. So if we had about 50,000 input tokens and 85,000 output tokens, to get almost close to an Opus 4.8 level of output, it will cost us 44 cents. Whereas with Opus 4.8, it costs you $2.38. So the, you know,

Starting point is 00:12:09 there's a big, almost a big difference on almost like a, almost like, 5x, you know, price difference between. Which doesn't sound like a lot when you're like, oh, $2 here, 44 here. But when you're actually using these things and you're running it all the time and you have pretty, you know, big tasks that you're going after and you don't want to be constrained by token costs. So it's a big deal. 5X is a big deal.

Starting point is 00:12:35 Yeah, yeah. And this is based on kind of the averages on like the coding benchmarks and what, what cursor charges you through the API pool. But I want to be also feature-proofing yourself. If we got this far with GL-5.2, I wonder what the next six months looks like, right? So I think it's also worth thinking about making the upfront investment in your machine right now to be able to potentially download and run these local models so that when we get to GL-5.3 or 5.5, we're essentially made the upfront investment in the compute and the equipment to now save a lot more

Starting point is 00:13:12 in the long run with other feature models that are going to come out are going to be much more extensive because i think this you know we're seeing the the i subsidy on tokens right like we're getting a lot more output out of plot out of codex and i wonder you know i've personally seen it i'm sure you have two where it's like now we're hitting our usage a lot faster than me before especially when fable came out i remember like i ran it in like in the first day i hit my limit you know totally so what you're saying is basically like it you know if you look at uh uh I mean, if you look at the history of VC back startups, think about Uber. When Uber first came out, they actually subsidized rides.

Starting point is 00:13:49 And they got you hooked onto the app. And then over time, they started increasing prices, increasing prices. What you're saying is with, you know, in the AI age, with a lot of these LLMs, they're going to get you hooked into the workflows. You're going to build on top of it. And over time, you know, those subsidies are going to go away as they go public and things like that. So what you're saying is maybe it's a good idea to actually invest in running this thing locally because, you know, the price of memory isn't getting cheaper.

Starting point is 00:14:26 And the price of tokens aren't getting cheaper. So building it now, securing it while you can might be a good idea. Yeah. And to tie us all together, I'd say two things. One, harnesses that are agnostic on models. So like, for example, cursor, where you're able to run multiple models across the same sequence of tasks are going to actually potentially benefit from this, right? So, you know, I wouldn't be surprised if one, you know, cursor decides to directly support

Starting point is 00:14:55 GILM 5.2 as a model provider and lets you kind of tap into that cost saving if you couple it with like Composer 2.5. So this is where it goes into model training, right? where I was, for example, looking at earlier in this task here, I wanted to refine the hero section. So what I did is I actually used Opus 4.8 to first import screenshots because JLon 5.2 doesn't support vision capabilities. So what I did is I actually used Opus 4.8 to import screenshots and explain back to me what it sees, right? I was like, tell me what you see specifically on the front end design for the hero section and lay it out. And then I switched to GLM 5.2 to steady that layout and then actually act on making those changes.

Starting point is 00:15:45 So it's kind of a way to like circumvent the fact that you have limitations to what GLN 5.2 can do on like image capabilities. But you're able to kind of now train the expensive model to think through the plan and then get the same level of front. frontier level, like quality, but at a much affordable price point. I mean, that makes sense. So your recommendation is basically, you know, it's almost like free trade versus protectionism, you know, the world, you know, not to be political, this isn't political. This is just an economic theory, right? Which is like, you know, when when people are trading with amongst each other, you know,

Starting point is 00:16:31 Maybe it's, you know, in Canada where you are, you know, you might want to trade with Florida because, you know, we got good oranges here. And we might want to trade, you know, we can't make maple syrup here. So we'll get your maple syrup, right? We'll get the maple syrup. Yeah, exactly. So make the best use of it. Yeah. Make the best use of it. As you're saying is, you know, using cursor and basically, you know, you don't have to use cursor, right? You can use whatever you'd like. Code, yeah. use one of those to basically say like, okay, for certain tasks, I'm going to be using local models, for certain tasks I'm going to be using the best in class cloud models.

Starting point is 00:17:10 And then together, ultimately you're getting, you know, great results in terms of the output, but you're also not spending through the wazoo. And if you're a token maxi or like you and me are, like in the sense of like we're always pushing into the limit around anything we're building to get the most out of AI, because, you know, We don't want to hire 100 people, 500 people and stuff like that.

Starting point is 00:17:34 It's helpful to do that. And exactly. And anecdotally, on two parts, right, one internally within our company. You know, I think Satea at Microsoft, you know, mentioned how, like, human capital plus token usage is now a big factor into what they're doing. Like, a lot of companies are now moving away from, you know, having direct access to the cloud code API to run the tokens because of how expensive it's become, right? So they're canceling subscription.

Starting point is 00:18:00 So we're seeing this firsthand with a lot of companies now are saying, okay, cool, this first year was great. You know, we had the mandate, you know, AI adoption, token maxing, you know, that's how we're going to measure success and that's how we're going to become AI native. Now that like, wait a minute. Okay, cool. We've done this, but we're spending way too much money on tokens. How can we now be more effective, right? And I'm seeing this firsthand too where it's like, especially now, right, in a way you can have some sort of direct ROI between the tokens you're spending. within the engineering team

Starting point is 00:18:31 because you're like, okay, cool, we're saving a lot of time. There's an output. You know, engineers are expensive. We get that. But now you're providing the same level of harnesses and models to the non-engineering teams

Starting point is 00:18:41 that, you know, are one-shoting like, hey, help me format this email and they're using Opus 4.8 high thinking. They're like, maybe that's probably not the right model. And that's a governance issue, right? That's a big thing. And I'm having these conversations with companies right now where they're saying, hey, can you help us figure out

Starting point is 00:18:54 how to build governance and proper education on how to actually use the right models? and this is where I think model chaining is a big factor, right? By the way, you know, John at marketing, maybe you shouldn't use Open 4.8 to run this, like, to just format this email for you and just helping them understand that. And I think, you know, I wouldn't be surprised if in a year from now companies start,

Starting point is 00:19:15 you know, we've been thinking about it as well. We're like, hey, why don't we just get our own machines and start running some local models because it's a lot more effective, especially how much money we're spending on tokens. What's the, like, just to play devil's advocate, Why wouldn't I just use open router and call it a day? Please do, yeah. Yeah, absolutely. I think they should.

Starting point is 00:19:33 Because when I'm on X, I see a lot of people being like, buy a Mac Studio or buy, you know, these expensive devices, you know, for people listening, do they need to buy a local piece of hardware? Like if the price even goes up 2X or should they just use open router and cursor or or cloud code, you know, whatever harness they want. If I just have this right machine, I'll get to this, you know, result. That's not how it works. You know, no, you don't need a Mac Mini.

Starting point is 00:20:04 You don't need this equipment. You can get started today. You know, and what I love about OpenRouter and all these other tools is, again, they're so agnostic. They make it easy for you to be able to access this in the cloud. They run the models locally. And it's credit-based. Low $20, get it going and easy to set up. I highly, highly recommend if you're starting to dabble with this

Starting point is 00:20:23 and token usage is a thing for you. Get one of these agent hardnesses set up now that they're all, a lot of them are model agnostic, run some tokens in open router, get these open models in there, and start, start vibing. Just get some, like, I love to experiment to see how far I can take this. What if I plan with Opus, review with GLM 5.2, execute with 5.2, and then review with Composer 2.5 or Codex 5.5. There's a lot of ways, and I think we can be really effective, and I think that's where

Starting point is 00:20:51 the smart people are going to be doing in the near future. some people are saying I don't care how much tokens cost because I think there's so much opportunity in building startups and optimizing and AI arbitrage that I don't even care if it costs me whatever. What do you say to those people

Starting point is 00:21:09 who are just basically ignoring this whole open source local AI movement? I used to be the exact same person in our first episode you're like yeah how much does all cost I'm like I don't know man I'm just vibe spending and I think that mentality

Starting point is 00:21:23 has changed now, I can see that my usage limits are being hit faster and my cost is going up, and now that our team is expanding internally as well. So I think as a solo person, it's a lot easier to build a case or rebuttal around why you should just token max as much as possible, which itself is kind of like an, it's ironic. You shouldn't be token maxing. You should be token minimizing as much as possible and output maxing instead. So my answer to that is if it works for you and you can directly have an ROI that you can show that, hey, I spent $200 and and got a thousand out. Great. Otherwise, sooner or later, the subsidy is going to run out. All right. Well, I think that's the episode, you know, unless there's anything else you want to

Starting point is 00:22:04 add before we're down. Yeah. I mean, for people that are trying out, dabble with it, play around, have some, you know, see what can do at least in the front end for you and start working back in tests. And yeah, I hope they learned something from this. I'll include links for where to follow Amir. He's always one of my first calls whenever I'm trying out new stuff. And So I'm happy that you're able to jump on. I appreciate you. We appreciate you. Give him a follow, like and comment this video.

Starting point is 00:22:31 Let us know what you think. We'll be in the comment section. Just, you know, out there trying to help and learn. And thanks a lot, Amir. I'll catch you on the next one. Thanks for having me.

The Startup Ideas Podcast - GLM 5.2 Clearly Explained (and how to set it up)

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.