The Startup Ideas Podcast - DeepSeek R1 - Everything you need to know

Starting point is 00:00:00 Ray Fernando on the pod. He's a 12-year ex-Apple engineer. He streams AI coding. He's building an AI startup in real time. I needed to have you on because what are we going to talk about today, Ray? Today we're going to talk about prompting. And we're going to be specifically prompting with the new reasoning models with DeepSeek R1. And there are a lot of caveats as far as these models because they're now, able to think and reason. And what that can do is can even lead to superhuman capabilities. And so what does that mean is that these models have now become so advanced. And this specific one from Deepseek is out of China. And what that allows you to do is basically they've made it open source. So it's available for us to study.

Starting point is 00:00:54 But it's apparently also on par with chat GPTs like 01 model, O one's reasoning models. And why it's taking the world by storm is because the fact that it's also free to use on their website. So deep seek.com. And when we say free, there's also a little bit of a caveat if you don't really know. So I just want to also cover a little bit of the architecture today and explain what you're going to get into if you use something like deep seek.

Starting point is 00:01:32 And then maybe how you can also run this in something that can run like in a container you're like in North America or in some other area because your data is really important, especially if you're doing anything for business. And then also the third kind of secret bonus there would be how to actually run this locally on your machine. So you can get the capability of these models and run that locally for your own private businesses, whether you're a lawyer, you're a doctor or whatever. There's a lot of different implications that you probably want to look into. So I think that this episode's going to be super helpful. If you're even just beginning and you don't really know some of the advanced stuff or no code, that's okay. It just takes learning English or using English

Starting point is 00:02:10 to describe these things to get the output and the intelligence of these models to do some really cool stuff. So I'm pretty excited. All right, let's get into it. Cool. Excellent. To start out to use these models, you have a couple of options. And one is going directly to deepseek.com. And this is actually currently hosted in China. So a little bit of background here is that your computer is here like, for example, I'm in North America. And if I go to deepseek.com or download the app from the app store, the app will actually be talking to a region over in China. And for what it may be, whenever you send your data over to another country, they have their own rules and laws and regulation. So I would be very careful as far as anything you put into the system, as far as if you

Starting point is 00:02:59 have any sensitive data, because it would not belong to a region that you may live in. or have control in. There are other alternatives which we're going to cover, which would be using web UI and going to these different API providers like fireworks or grok. And then we're also going to do something like covering running something locally on your machine. So it doesn't go out to any of these providers. And you can even run this if you're flying on a plane, which is really exciting. So as an example for deep seek, we're just going to do this because this is something that's currently public information. And I don't really mind having this stuff sent out. So as far as is prompting one thing that I frequently do is I have a live stream and I basically transcribe my

Starting point is 00:03:40 videos and stuff. So I basically, you know, I made a little app that will transcribe videos and I just take my live stream here and just run it through the transcriber there. And what it will do is just generate transcripts from the video. And it usually does it pretty fast. It processes on my device and then it sends it up to Grok for the endpoint. So when it's done, it looks basically something like this and you're able to copy this transcript and put it into something like deep seek if you wanted to. And so in order for you to use the model, what you can do on deepseek.com is to just go ahead and click where it says deep seek. So when it turns blue, that means the deep think is enabled. And that means is also there. And you can, if you want it to enable web search, you can do that.

Starting point is 00:04:23 So we can probably do that for the next prompt here. So I paste in my transcript here. I hit shift and I hit enter a couple times. And so what we can do is give it additional instructions for it to do what we want to do. And one of the things that I like to do is I actually have built a little prompt that I will actually share with y'all so that you can actually do some analysis and generate a blog post of a transcript. And so that is actually located here in my little notion thing. And so one of the things I have is we're going to cover how to do some of these prompts and stuff. and I'll actually show you how you can generate some of these advanced chaining prompts because this will really take advantage of these models to think through all of that text

Starting point is 00:05:04 and do some work on our behalf. So this is really, really cool. It's basically like hiring an admin to go through all of your stuff and make things for you. So we're going to go ahead and hit submit. And when we do that, I will say, you know, I wouldn't put a tax return on deepseek.com. It's not the type of thing I would put on. So be, you know, you do want to be a bit wary of what you're putting on when you're on deepseek.com. Now, I was playing with perplexity earlier. And perplexity actually has some of these models built in. But it's hosted in the United States of America.

Starting point is 00:05:42 So that's a bit different. That's correct. And you may want to ask your app providers what they do. One of my favorite apps for coding is actually Cursor. And I asked them, hey, where do you have? have your deep seek model hosted, and they told me they used the fireworks API, and that's actually not in China. So that's great. So it's like, okay, cool. That's awesome. And they're using the full model. Quick break in the pod to tell you a little bit about startup empire.

Starting point is 00:06:10 So startup empire is my private membership where it's a bunch of people like me, like you, who want to build out their startup ideas. Now they're looking for content to help accelerate that. for potential co-founders. They're looking for tutorials from people like me to come in and tell them, how do you do email marketing? How do you build an audience? How do you go viral on Twitter? All these different things. That's exactly what startup empire is. And it's for people who want to start a startup but are looking for ideas or it's for people who have a startup, but just they're not seeing the traction that they need. So you can check out the link to startup empire.co in the description.

Starting point is 00:06:56 So these models have these parameters that you may hear of. And like, you know, like the really large parameter model, like 600 billion plus parameters, just means that it has more intelligence to leverage. And it tends to take longer in its thinking, but the results are really, really, really great. And some of the models that will probably run locally on the machine a little bit later, they're going to be like distilled. So you just basically take the essence of it. and then those models basically are going to run.

Starting point is 00:07:24 They run a lot faster and they're just as efficient, but they may not think as long or they may not give the results and it's really up to you to try out, which I highly encourage. So one of the problems is that sometimes the server is really busy. And that can happen because right now it's so popular. And I probably, after the publishing of this video, it would probably be even more popular. So you can hit this little pencil and you can hit send again to try to resend it.

Starting point is 00:07:49 And so that's kind of where I thought, well, if I'm sending this over and there's a bunch of reliability issues, why don't I try to do something like, you know, that I can host my own or just hit the API themselves? If cursor is doing that, why can't I do it as well? So I can actually show you a technique for how to do this so you can hit the API so you don't send your data to China. And that actually involves using this thing called Open WebUI. So I can show you that. So while this is thinking if it even returns a results or does anything here, we're going to go ahead and pop on over to the other side. So in the other side, we're going to go to here. And so I have an instance of what's called Open WebUI, and it looks very similar to like a chat GPT.

Starting point is 00:08:34 And to get this setup, I'll probably go through a little bit more details, but I'll just go ahead and show you what this looks like. So in here, I have the model selected. I can go to deep seek. And so what's great is that you can connect to an API provider. and I'm using Fireworks AI. So Fireworks AI here is currently hosting DeepSeek model, and they allow you to use the model just by, you know, using getting an API key and then putting in the exact model string and so forth here.

Starting point is 00:09:02 And so from here, if I go to the Open WebUI, I'm able to select them and say, okay, this is my DeepSeek accounts. I'm going to go ahead and just paste the exact prompt that we had, paste it in with my transcript and everything here. And I should be able to get everything out. So let me just double check here that I got everything. So it's still timing out here. Yeah, server is busy.

Starting point is 00:09:25 Try again later. So, yeah, that's not fun. So what I'm going to go ahead and do, scroll to the top and hit this little copy button. And then go over here. Just make sure I put everything in there. I have the whole transcript. Yeah. So the whole transcripts here.

Starting point is 00:09:37 And so what's going to happen here is when I hit send, it's going to send this off to Fireworks AI. And what's great about this thing that's actually running in this open web UI is that it's using the API and it's not actually sending the data to China. So just for as, as is doing its thinking here and showing us what's going on, I'm going to go back to overlay this model of our data and our container and kind of show you what this is looking like in the background. So here in TLDraw, we actually have our Mac and PC. And so I used WebUI here. And I'm actually using the the Fireworks API. So I'm going to the cloud and this cloud is located in North America. So the data basically resides here in North America and it's going to be delivered back to

Starting point is 00:10:20 my device. So that's what we're doing. When we're on the DeepSeek website, we were going out to the China region. So that's just a heads up of kind of how that's working in the background. And so next I'll show you the difference of what's going to happen in like the speed difference with what GROC hosting provider provides. And then a little bit later we'll get into a little bit details for how to get the setup. So as you see, is kind of outputting this stuff here. Because these models are still so new, these web apps are still adjusting to take a look at the reasoning stuff. And so what I'm going to go ahead and do is hit this little pencil up here to the very top. I'm going to make this a little bit bigger so you can

Starting point is 00:10:56 see. And then from here, when you hit this little pencil, it creates a new chat. At the very top, you can select the model drop down. I'm going to type in deep seek. And the one that I have set up from Grok is called the Distill Lama 70B. And so this model, is actually like a smaller distilled version that they're hosting, but it's incredibly fast using the GROC API. And so if we hit this here, it seems like nearly instant by the time all the stuff kind of starts finishing.

Starting point is 00:11:25 So we'll see this model actually going out. It's doing its thinking. And now it's actually providing the response, just like that super fast. So if we take a look, this has thought for a few seconds and actually shows us the reasoning that was going on. So this is actually going through my transcript, trying to really understand what was going on with my transcript.

Starting point is 00:11:44 This was an interview with LDJ, who spoke a lot about deep seek and technical details of things that I really couldn't remember. And so then it basically makes this, like, you know, a very simple blockpost. So we'll see if my other one that was running the larger models finished. And you can see the difference between the two models. A distillation model is just going to give us like a small little blog post versus the full model that's also running on fireworks API is actually giving us. quite a bit of detail. It's going to take more time, but take a look at what it's doing right now.

Starting point is 00:12:17 So above here was all this thinking stuff, but this is actually now doing an analysis on my transcript and generating a really nice blog post. So it's telling us about the calculations that LDJ talked about in the stream, geopolitical implications of what's going on with the new AI arms race, also future predictions. And we talked about these details in the live stream. And it literally pick them up and is now creating a graph from this. This is how crazy these models are, if you can really think about it. And here's some key takeaways as well. So that's an amazing thing. And I'll be able to share these prompts with you so that you can actually run these analysis on your guys' own transcripts as well. Yeah. So here's the SEO enhancements and final

Starting point is 00:13:01 thoughts. Yeah. So when you're in when you're in business or you're building a startup, having an unfair advantage is so important, right? Like being super efficient. and keeping your cost low, creating your product to be the best possible thing. Now, we're in this new deep seek world where the model, you know, I call it a deep seek world, but it's really, it's a llama world,

Starting point is 00:13:24 it's a deep seek world. It's a world where if you figure out the model that works for you and the tasks that you want to accomplish, you might be able to out-compete whoever you're competing against. Now, I've done a similar, prompt on chat GPT and with some of my YouTube transcripts. And it's not unusable, but it's more of a thought starter. It's like, oh, okay, like I can take most of this and I can rejig this and add this and

Starting point is 00:13:58 add that and probably get to a blog post that is good. But it does require a lot of human energy to go and do this. When I see what's coming out of this, what's really, really mind-boggling is the fact that it almost looks, like just quickly scanning this, this looks pretty, pretty human-level, incredible, like a senior writer would do something like this. Yeah, or a research engineer that you hire to really thoughtfully take a lot of notes, spend a lot of time analyzing and, like, put together how you would want to report. And it's even more incredible because these instructions can be configured.

Starting point is 00:14:43 So if you want a graph or you want a type of thing, we can take this prompt and put it into DeepSeak itself to say, can you give me this type of output instead? And it'll do that for us. It's like, how do we improve the prompt? Or what do you want to see from your outputs all the time from your live streams? And like I think the thing that I've seen, this is kind of the biggest breakthrough that's happening is that with,

Starting point is 00:15:08 I'm seeing this also with, 01 Pro, by the way. 01 Pro and deep secret reasoning models, these amazing models spend extra time and actually pay attention to your instructions. And so every little detail that they're seeing, they're like, oh, yeah, I haven't done that yet. Okay, let me go ahead and make sure I still do that.

Starting point is 00:15:24 And that's something that I super deeply appreciate. And for me, it's worth the extra 200 bucks I pay a month to open AI. But this is really quickly turning my head and like, oh, my goodness. Did you understand like what just happened here? It's like, I'm a little still taken away at this output. Like you're saying, it's very detailed. And it's, to me, I feel like this is totally a game changer. And I think one thing that people aren't really talking about right now is actually this additional rush to understand who can host this.

Starting point is 00:15:58 In order to host these huge like 600 plus billion parameter models, you need all those GPUs. You need services like fireworks. GROC is trying to spin that up. you know, Grock was able to get a distilled model. There's just so much demand. There's going to be even more demand for these chips. And yeah, this is just the beginning. And I'm trying to figure out, you know, which provider can host this for me reliably

Starting point is 00:16:20 so that I can do this for myself, but also share back and put this into apps for other people as well because this is going to be, this is great. And I don't want the data to go to China. I just want the data to stay in North America or if I get a European container, I can do the European container and meet all their legal requirements that need to happen for that as well. So that's super exciting. Yeah. What's the cost, the pricing for fireworks? Yeah, the pricing for fireworks, we can look this up real quick. I think it's about $8 per million tokens where normally, I think chat GPT was like 15 input and $60 for output for 01 Pro. I can just

Starting point is 00:17:01 double check that real quick. So pricing, let's see. Yeah. From what I remember, it's cheap. It's like significantly cheaper than you go to 01 Pro. Yeah. So 01 Pro or 01 API cost. Yeah. And the pricing. And this is going to add up, right?

Starting point is 00:17:32 Like you might be like, oh yeah, who cares? A million tokens. But once you add this to your workflow and you're pumping out content or you're doing research on an ongoing basis or you've built a business around how to do this, these tokens will add up. That's exactly it. Yeah, they'll add up. Also, at the same time, Open AI is currently promised that the O3 model will come out and

Starting point is 00:18:00 the mini model will come out, which would be on par with this model, so that prices will also probably significantly drop as well because they just get more efficient with time. And so that'll be really interesting to see. and, you know, I'm rooting for it because for me, as a consumer, I want the power of all the intelligence. And to do these types of things, I think it's going to be pretty important. And I think to that note, I think it'd be interesting to show how, if anyone hasn't found out about this, it's this thing on OpenEI's thing called the Platform.Opeanai.com. So you just sign up for accounts or developer account.

Starting point is 00:18:36 It's a little playground. And what you can do is actually hit this little generate star button. And so we can describe a prompt that we want for any type of model. And what it will do is reconfigure the prompt for the language model so that they can actually be more efficient at doing something. So if there is a task that you do quite a bit, you're like, I just want, you know, please like make, you know, keywords for my Amazon listings and then hit like generate. What it'll do is basically, whoops, sorry. If we hit generate here and hit this at the very top right and hit update, what it will do is actually reconfigure this prompt just from a one line type of thing to include more details. So we can take existing prompts and try to improve them through this mechanism.

Starting point is 00:19:24 So as you can see, this is like how people get really nice long chains of thought or reasoning or outputs. So one way to think about these things is to first kind of put down like what instructions you want, what type of output do you really expect? maybe what you don't want as a good starting point, and then that'll help you generate prompts that can be a little bit useful. A lot of the prompts that you're seeing that I spit out are basically things that have come out over time because of my use cases. It's like, okay, I want this instead of that. And so as an example, one of the things that I was thinking about is like, how do you verify claims for a specific type of thing? So if you have an article and how do you understand if something is actually true or not. So I have one here. This is information verification.

Starting point is 00:20:12 And so one of the things that these models that I was currently showing you is that the fireworks and the grok are specific API endpoints. And right now there isn't like a specific web search thing that's currently tuned into them. So if you want to do web search, you have to go through the deepseek.com route or the app. And keep in mind, you're also sending data into this container there. So, like, sometimes you could just use it for public articles for things that you really don't care about. So if you're on DeepSeek, let's just see if they actually have stuff available here for us. So you just go to the search thing, turn that on. I paste my prompt in.

Starting point is 00:20:50 And then what I'm going to go ahead and do is like maybe grab an article like the Techno Optimus Manifesto from Mark Hendryson. It's a very popular article. And sometimes, you know, it's really long to read. There's like a lot of information here. And you're like, oh, my God, there's like, so much in there. It's like, how can I even get started with this thing and how do I even verify the claims of this stuff? And I think this is probably sometimes like a good thing to start here. So what I do is my hit shift and enter and I put the like article at the very top. And

Starting point is 00:21:22 once I do that, I just go ahead and paste that in there. And then just go ahead and hit send. So what that's going to do for us is going to use the web search and try to like look through just like how we saw earlier that was happening with the API. Every type of claim. that's in there, try to see if they can search the internet for it, and try to see if it can do anything. So this is going to try to do its thinking thing. This is obviously very popular and, you know, deep seek the website is getting flooded with people because of basically it being free. And so, yeah, just keep in mind, like Greg says, like, I would not be putting taxes in there. I probably wouldn't be putting medical records in there. Things that, you know, you don't want to

Starting point is 00:22:03 see generated. If somebody asks a question that's related to. to you because that can be a little crazy. You'll be like, wow, all of a sudden my data is showing up somewhere. That was not expecting. So yeah, this is currently airing out. It's no surprise and this is kind of why we were kind of thinking about doing some other alternatives here. So yeah, I think that's that was set there. So I think another thing maybe that can be useful here is probably getting this stuff set up. So if you wanted to run this locally, maybe we could. kind of go over that workflow. What do you think? I would love to, yeah, I mean, selfishly, I would love to know that. Okay, okay, cool. Awesome. Let's do that. Yeah, I think that that'll be great.

Starting point is 00:22:47 So in order like for this section, in order for us to run this model locally, the best interface that I found, bar none, is something called open web UI. So open web UI, it's really quick to get started. All you need is to download Docker. So just go to Docker.com. And you you could just download the desktop app. So download the one that you need for your machine. If you're running an Apple M machine, which I am doing, I just download the Apple Silicon version and accordingly. So once you get that installed, it's going to present to you a user interface like a dashboard here. And so that's kind of something that you'll have, you know, kind of going, Mines is already showing the app is running here, but that's how you already know it's installed.

Starting point is 00:23:30 It will require the terminal, but it really won't hurt you too bad if you have it running. So the first command that you will basically run is this one that's listed on their Quick Start. So the Quick Start will be listed here and we'll have this available as a guide for those to download. And all you have to do is follow these two steps, really. So the first step is pull the container. So this is you just copy this and then you put it into the terminal. And then it's going to do this little pulling thing and it'll probably download like, you know, several gigabytes of files onto your machine. And then the next step is literally what they call running the container.

Starting point is 00:24:02 So with Docker, the whole app and everything is all contained in one. So that way you don't have to spend a bunch of time doing extra terminal things. This is probably the only two terminal things that you will run. If you're running a PC, you probably want to run, especially for Nvidia, you want to run this command, GPUs dash all. So all you have to do is just copy this one if you're running in video. And that'll take advantage of your GPU and it'll run more efficiently when you're running it locally. So the one I like to do is just a single user mode, which doesn't require sign in. That way, if you're the only one that's using it at your house or on your network, that's probably the best way to do it.

Starting point is 00:24:37 So you just copy this command here and then you put it into the terminal and it'll say, hey, you know, great, it's up and running. And then all you have to do now is just go to the website, Local Host 3,000. And then once you're running on Local Host 3,000, you're going to be presented with, you know, some user interface thing like this. And I have a model that's currently loaded here. that's kind of why it's showing us here. But you're not going to have any models loaded. So in order for you, the next step here that we have to do is actually have a couple of options. One of the things that I do is like you can just download a model locally.

Starting point is 00:25:14 And I use a thing called Olamma. And so Olamma.com is something you'll want to download there. And so that way you can run any local model. And it's simple as just, you know, finding the model and so forth. So once you hit download, it's going to download for your machine and you install it. You'll see like the way that it's downloaded, you'll actually see this little llama guy that's at the very top and a little figure there. And that's how you know it's currently running. And so once that's currently running there, you'll see the models that are currently listed here in the model section at the very top.

Starting point is 00:25:47 And so deep seek R1 is going to be the one that we want to use here. So we'll go back to our web UI instance. And then we're going to go ahead and hit where it says user at the bottom. From there, we're going to go to the admin panel. And from the admin panel, there's like a section, a settings area. And so this settings area has an area of our connections with a little cloud icon. And this is kind of where we're going to connect our other providers here. So let me make this a little bit bigger so that everyone can see.

Starting point is 00:26:15 So as you can see, the Olamma API is already configured for us, which is nice. And this is already going to have the Docker container there, which is great. And so when you hit the little pencil here and you hit plus change the model, you can type in the model like deep seek. And if you don't see it available, which it may not be there, you'll see this option at the bottom that says pull from deep seek from olama.com. And so that'll actually search Olamma here to get it for you. So like, for example, if we wanted to download the fee four model, I'll just type that one in just as an example. So you can see. V4.

Starting point is 00:26:54 So I don't have that model currently downloaded. I just can hit here and it's just going to go ahead and find it and then it downloads it. So in no time, basically this model would just be downloaded on my machine and then I could just type in fee 4 and I'll be able to use that in the future going forward. So what we can do is go ahead and type in, you know, we're starting a new chat and we're going to basically select the model deep. seek dash r1 and you'll see it'll be colon latest is kind of what it's listed at and that's actually how you know that that's the one that's currently running locally and so once you select that one you can just say something like explain options trading and then go ahead and hit enter and so what this does is you know it's basically you know thinking and you can see the thinking tokens of what's

Starting point is 00:27:45 going on when it's thinking and so all of this is actually running on my computer which is amazing One of the ways that I can tell is there's this command line called ASI Top. And it actually shows us all of the resources that it's eating up. Thankfully, I have 128 gigabytes on my machine because I do live streams. I do all this stuff at the same time. And you can kind of see how much RAM it takes up right now with me hosting the stream plus, you know, running this model locally. So yeah, this is actually what it does here. One of the things that we could even do is try to test that prompts that we were using earlier so that we can run this command locally.

Starting point is 00:28:25 So earlier what we did was we were running like a whole analysis on something and it would just fail out. So this thoughtful analysis that I was showing you, we can try to see if we can run this on a local model and just see the difference as well. So this is basically the transcript that I had earlier plus the analysis stuff. And if I go to open web UI and then just go ahead and going to go ahead and go back here and create a new chat. and hit paste and then it run. So this is going to see it's thinking here and it's using up all the resources on my local machine to run this model. And it's quite a lot of tokens.

Starting point is 00:29:01 And it's still fairly impressive what a smaller model can do that's running on my machine. And you'll have different versions that you can use. I think. And so this one's using the $7 billion parameter model. If you get something that's a little bit higher, it's probably going to get you a little bit more detailed response. And I would definitely play around with these things. Another important setting, I think, that you can tweak. And we can probably run this as a next chat is while this is going here, there's control section. So this control section at the very top

Starting point is 00:29:34 will show us. Let's see what, I dismiss this. So the controls, one of the controls that you'll probably want to change around to get different results is the temperature. So it's setting the temperature from like, you know, 0.8 the default to like a lower temperature will actually make it like hallucinate less is kind of what people say. And so he'll tend to follow instructions better and then I kind of veer off into different tangents. And then another one, if you go all the way to one, it'll just be extremely creative. So you can think about those as far as maybe if you're doing some creative writing, some non-logical reasoning. That can be really helpful if you want to kind of think out of the box and have it kind of go into La LaLalan. It's really up to you and your content. But I would definitely

Starting point is 00:30:17 do two different responses with different temperatures and test those things and see if you see any difference in your output. For me, sometimes I find the temperature zero to be very helpful for very logical reasoning outcomes, especially around code. But it's really up to, it kind of varies. And I just kind of want to give you a heads up on that. Yeah, I appreciate that. To me, I would rename that temperature as, you know, wine versus coffee mode. Love it. wine might get you a little more creative. Right.

Starting point is 00:30:51 If you want more rational execution style, maybe you want coffee mode. I think that is something that, you know, we have a LCA, it's our design firm for the AIH. We have a, I feel like that's what's missing from a lot of these AI products

Starting point is 00:31:09 is just like a little humanity and lightness. Right. So I expect over the next couple of years, we'll start seeing, you know. You know what would be funny to basically have like a spinner where you can actually flick it yourself and you kind of see it land on something and then just like, hey, go. Totally. That would be cool. Because sometimes you really don't care, right?

Starting point is 00:31:31 You just like, I just want to spin the bottle and see what happens. Totally. It's just kind of yellow mode kind of. Yeah. Yeah. Because I think like you say, there's huge opportunities in the AI space to be playful. And I think that's what's interesting is you have these intelligence of the models

Starting point is 00:31:50 and then now you have to have people who build interfaces to interface with them and there are a lot of companies who are trying to do that. And you can get very far with just some prompting as we're seeing here. And then we're trying, this exercise here is to try different models.

Starting point is 00:32:05 So if you think about it, Olamas sort of the gateway to all these different types of models that you can try out and see if it even works for your use case. and this web UI is actually a really nice user interface to keep track of that. It's safe locally on your machine. You can go back to them at any time.

Starting point is 00:32:23 There's additional options at the bottom here, which is really nice. So you can actually have this read out loud to you. So if you're a person suffering maybe from dyslexia or you actually prefer audio, you can have that for you. This will give you some information there. You can continue the response. Sometimes if you have too much information, it still needs to continue going. So you hit the continue and it'll just continue on or regenerate the responses. So that's kind of some of the basics there.

Starting point is 00:32:51 So yeah. So this is the output of this model. And I'm fairly impressed for being a $7 billion parameter model and running locally on my machine that it took that entire transcripts and did this analysis type of thing that's, I'd say, is pretty close to the bigger model. and in terms of details, it's not as detailed as the other one if we kind of take a look. So like the previous, this is one that it came out with before, you know, with this nice big block post type of thing. So it's pretty good. And it's running, you know, locally. I can run this on the plane as far as that.

Starting point is 00:33:29 So yeah, so to get started basically again, it's just open web UI. There's a getting started. It's literally a couple steps to run. Make sure you have Docker installed there. and then Olamas is going to show you all the different models. So if you go to the models, you'll see kind of stuff that's popular and trending right now. And that'll kind of get you some of that as well as far as getting started. There is a, you know, a word also talks about fireworks AI.

Starting point is 00:33:55 So that's fireworks. It's a good resource for you to, you know, go take a look and put that model in. So like if you want to put that model into your Olamma, you would kind of do the same thing here. So go to user and then you go to admin panel. And then you would go to settings up here. here. And then from the settings, you're going to go ahead and hit connections. And so what you'll do is go ahead and hit the little plus connection. And so you'll have to put in the base URL and you'll also have to put in the API key. So the base URL here for fireworks is this specifically here. It says

Starting point is 00:34:28 API.companse, a.I.comfirmance slash V1. In the example documents, you'll see slash chat slash completions and things. You don't need those because that's part of the Open AI framework is that you just put everything up to V1. And then you'll generate an API key from that model over in fireworks. So AI, so if you go to the model here in fireworks and you go to your name, and then if you go to API keys, once you go to API keys here, you just hit create API key and that'll pop up and that's the key that you want to put in there. Similar to GROC cloud, you just go ahead and hit create API key. So once you go to console.grock.com, there's an API key section here and then you'll want to hit create API key and that'll pop up a dialogue with those API

Starting point is 00:35:21 keys. And so that endpoint will look something like this over here. So that'll be, if we configure, API.grock.com slash openAI slash V1. And then you you put your key in there. And you don't have to do anything with these IDs. These will be pooled directly from that endpoint. So whatever models you have available will be there. And so now when you hit the plus sign, you'll see like this nice list of models from fireworks. So they'll be the fireworks ones. So account slash fireworks. You can play with any one of those. And then the other ones that are just with the normal name are from GROC. So they have those as available for there. So you can play with a lot of these models, which is nice. And compare them. And then the ones at the

Starting point is 00:36:03 bottom are the ones from Olamma. And it's a little show like, you know, the colon latest is kind of how you can tell. And if you hover over them, you'll see like some additional information over the parameter count, what quantization level it is. So Q4 means it's quantized to four bits. And that also has a play in its intelligence. Obviously, the higher level of quantization, you know, means more memory. So it's like 32 bit 16 all the way down. So like the lower the number, the like not less intelligence, but you may not get the output that you want is expected. So that's kind of part of that process. It's a lot of different things here. But I think the most important thing is just, yeah, how do you host this locally, how to start playing around

Starting point is 00:36:47 with it. And that's kind of like a really good primer to get started for doing these models and stuff. Yeah. I love it. I don't know if you've played around with it, but is there any way to do this on mobile. Like, could you play with local models on the mobile device? Yeah, there is an app called Apollo. Have you heard of that? Apollo? Yeah, I just haven't used it. Let's see. I don't think app store. App store. I'm going to see if I can go here. Apollo. Let's see. Okay. Private local AI. Yes. So I have this app on my phone and they allow you to download the models directly just like you would with Olai. as well, but it has its own interface, which is really nice. And so I wonder if I could share my

Starting point is 00:37:38 screen. I think I can. So on your phone? Yeah. Yeah. They have a phone mirroring. Exactly. Apollo. Okay. Oh, I have to lock my phone. Okay, cool. So I lock it and then it should be able to connect. Okay. Cool. Nice. Awesome. So yeah, let me kind of minimize this here. And, yeah, okay. Let me just go to a different screen here. Probably one that's less cluttered. And do phone. Whoops. Put that over here. Cool. Maybe this will work. I think. Yeah. Yeah. Yeah. Sweet. Yeah, I actually have the, yes. Another place to get your models apparently is also through OpenRouter. And so, yeah, so this is kind of the Apollo app.

Starting point is 00:38:38 You're like, okay, cool. I can, can I start chatting with this? You know, as soon as I play with the thing, a couple configurations you have to do is you hit this little hamburger menu at the very top left corner and then you hit settings. So on the phone app, you hit settings and it's going to say AI providers. And when you click there, you have three different options. OpenRouter, which is another API provider.

Starting point is 00:39:01 And you can also get access to pretty much every model there, is also very handy. I think they give you some free credits, but then you would put your credits there. And then you have the local model and then you have custom backends. So with the local model, they actually can tell how much memory you have on your device and they'll actually have a little download button for those models. The ones that are not available with a download button basically means you can't run that on your device because you don't have enough memory to run them. So these downloads are pretty big like four gigabytes and some of them, you know, are several gigabytes. So just depending on the space on your phone. So you can actually run the distilled

Starting point is 00:39:37 Lama 8B, 8B, 8 bit MLX version. And I have the distilled Quen version at 7B. So it just depends on your, oh, that one's actually not compatible. Which one do I have downloaded? So I think on mine, let's see, the one I have available is the deep seek R1 from Apollo. I think I have it from open router that's running. So let's take a look here. Yeah, providers, open router. Yeah. So the one that I have set up right now is from OpenRouter. So OpenRourter will show you all the models.

Starting point is 00:40:13 You can select DeepSeek R1 from there, which is awesome. So you can have a conversation. So this just requires me, you know, being connected to the internet. We start a new chat. You're like, tell me more about options trading. And so here you're still talking to the model, but you're actually just using OpenRour open router. And so that's a little bit different than, you know, sending your stuff directly to deep seek. And they should be able to do that. It's possible that this model is busy or it's

Starting point is 00:40:42 currently down. That can happen. So yeah, that happens. Yeah, while that's going, I think we could even start another new chat. Let's see. This model. You can select a different model. So let's see. It's crazy how many models are right now. There's so many. Yeah. It's like, how do you know, which one does it. I feel like you just go off vibes. Like what's my friend telling me? Yeah. Like, what's the real vibes right now? So the vibes right now obviously R1 is like the real hotness. People are like totally into that right now. And it makes sense because, you know, reasoning at a much lower cost. So let's see. There's probably something going wrong with my API key or something. So AI providers, I can select a local model to run. you know, I want to see if there's something small here that we can download.

Starting point is 00:41:36 So we could do, yeah, this distil Quen, just for, just for speed purposes, we'll just download the Gigabyte one. So this is going to download, wow, that's really fast. The Quen model 1.5B, and so that'll run deep seek locally. And so basically it's just downloading it directly from, I think, Hugging Face. And then the model is being loaded on my phone. And this is actually optimized to run on Apple hardware. or Apple Silicon. So that's, you know, one way that you can kind of take a look at it to run this thing. And so what's nice, yeah, if this phone runs out of internet or I need to ask some questions

Starting point is 00:42:12 or do some stuff, I now have, I will have this R1 reasoning model that's a much smaller version to run on device. And I think that's another good point about AI that's running. And like, you don't always need the most powerful thing running for every single type of thing. I think it's really important to understand different use cases, you know, because maybe, you know, because maybe you don't need that depth of reasoning. You just need something that's really quick. Or you just need something that's really good at like gathering lots of information and just telling you some topics or something like that. And that could just be done really quickly. So it's kind of like picking the right tool for the job and experimenting. So we're at a good age today where you can actually get these

Starting point is 00:42:52 models and experiment with them. So now I should be able to select this guy and run it. So let's go ahead and Hit done and start a new chat. And then over here, we're going to go ahead and select the model. So here we're going to type in, oh, yeah, it already has it at the top. So you see this little icon that signifies that it's running locally. And then we're going to hit cancel. Hit done. Okay.

Starting point is 00:43:14 Great. So now we're running with that local model. And I think we're just using a default system prompt about it being Apollo. And you're low, like, yo, tell me about options trading. and it should basically start to cook. So it's using my phone's power there and it's now thinking. And so if we click this little drop down, we'll actually see the reasoning tokens.

Starting point is 00:43:38 Wow. And so that's, yeah, I have reasoning on my phone, no internet, completely running locally. Yeah. 2025 is insane. Yeah. And imagine be able to run this on your watch. Like that'll just be,

Starting point is 00:43:55 because this is already showing its capability. Like we're doing this input. If you're, if you can make an app that can run on a watch all locally, you know, just think about like the transcription stuff, right? Uh, you have a very,

Starting point is 00:44:09 very lightweight model. You send the audio, you know, from the watch, you know, especially of a loved one. Maybe they've fallen or something. It can just turn on the speaker and try to understand the situation.

Starting point is 00:44:20 Uh, and then if it listens to paramedics or something about asking questions and they don't really know, maybe the watch can show, hey, there's this app here. I'm going to show you the emergency card this person. has for their medications or this is something that's happened in the last, you know, five minutes

Starting point is 00:44:34 before this events or something. This is kind of the way that people think about designing apps with these models as trying to think about these use cases because now you have really powerful devices just all like on the size of your wrist that can run these models and the power of Apple's MLX infrastructure and also their AI technology is the fact that, you know, these are really optimized to run these models, you know, very simple. small as we're seeing right now. So that thought for 49 seconds and it gets us this output. Yeah, please excuse my small screen, but we'll probably have a zoom in on this for the edit. So yeah, yeah, yeah. That's pretty sweet. So many startup ideas, by the way, like from that alone.

Starting point is 00:45:20 And so many, you know, and so many, you know, I love that. I love that you shared that example of, you know, someone maybe falling and hurting themselves. I think even coordinating with your AirPods, yes. There's just a ton of opportunity there as well. Translation. You know, not just pure translation, but it's like someone is saying,

Starting point is 00:45:44 saying XYZ, but what are they really saying? You know what I mean? Oh, yes. Maybe like, imagine like negotiating in the future, except you have like, you know, pretend you're a lead negotiator as like almost, you know, a local AI LLM that's helping me figure this out. So there's just, you know, we can have a million ideas, but it's just really exciting to see what that,

Starting point is 00:46:08 you know, where this could go. Yeah, I think that also goes to the point a little bit about what some of these models do. So one thing I just learned very recently about GPT4 and chat GPT's Omni models is the fact that this model's breakthrough a little bit different than. R1 is the fact that it can actually understand audio and tone and all these extra implications that we don't know about, especially for negotiation. Like, imagine if you can understand someone's breathing rate just from listening to the audio. That's the capability of something like, you know, the 4-0 models with audio.

Starting point is 00:46:40 You just give it the audio. And it's going to know tone. It's going to know cadence. It's going to pick up things that we just normally don't think about. But people who are maybe skilled negotiators can understand what those implications mean and then can say, hey, give me some outlier things. Every time I give this person a response, you know, it can answer in like milliseconds what the differences are. And I've heard these terms of like micro expressions. And if you have an Omni model, you can actually mimic these, you know,

Starting point is 00:47:07 micro expressions and say, okay, this person is off when we ask them these types of things or changes their position. And those are the things that you really can't get today with some of the, you know, current reasoning models, but except for like the Omni models, which is the, you know, 4-0 models. And so it's going to be really exciting when they actually dropped 03. I think a lot of people are going to be taken by storm of like what's actually really going to come out from them. It's going to be a really, really big leap. Anything else you want to cover today? I think this is a really good primer for folks to get started on the power of prompting and especially with these reasoning models just to get started. So we covered being able to get started with

Starting point is 00:47:49 prompting understanding where your data is going. You know, if you're using deepseek.com or using the apps, that will go directly to China and their restrictions and things that they have for your data privacy. So just beware. I wouldn't personally be putting any personal information in there that you don't want to expose. And then there are other providers that you can use right now and other people that are still spinning up at this very moment. So for now, fireworks, open router, grok as far as inference.

Starting point is 00:48:16 And then we also covered here running the models locally. that you can actually, you know, run them on your phone using the Apollo app I was using. It's a paid app, but I find a lot of value from it. And I'm not sponsored or anything like that. I just love the work that these people are doing. And it's really good stuff. You can connect to these endpoints with them. And then the other part is running this through Olamma locally on your Mac

Starting point is 00:48:38 and using WebUI with Docker and stuff. So I think there's a lot to be taking care of here as far as trying to use these models and come up with different app ideas. And if you have an idea, just start to use the playground to try to generate some prompts for it and see if you can get the output that you want. And that could be the beginnings of your next multimillion dollar idea that you don't even know is there. So it could be hidden in plain sight. And I think that's the power. If you want to reach out to me, you can just go to Rayferndo.aI and book some time.

Starting point is 00:49:10 We can have a conversation. We can get you set up because some of this stuff is very cumbersome. It's just easier for me to walk you through this. and so I'm available there as well. You can find my YouTube channel, Ray Fernando 1337, on YouTube. Feel free to check that out. I do a lot of live streams of where I check out new technology and try to play around with these AI models

Starting point is 00:49:28 and try to discover what's going on and also try to bring on experts to explain things a little bit more for us. So Greg, such a pleasure to have me on the show. This is really amazing. You're a legend, man. Thank you for coming on, sharing your insights here. This has been super helpful. I thought it was helpful.

Starting point is 00:49:48 So if people agree, go comment on YouTube, I read and respond to almost every comment. Like and subscribe for more of this in your feed. And let us know if we should bring, and when I say we, it's me. Let me know if I should invite Ray back on again to show us more stuff. I would certainly love to do that more in 2025. Crazy Times, Ray. this whole deep seek, you know, tidal wave is just, it's insane. Yeah, I'm glad that it's something is here to like make more people aware that there's a lot of

Starting point is 00:50:29 intelligence and how fast it's moving. And I want to also add that like, please don't be fearful or don't feel like you're left behind. If you're just finding out about this, you're not that far behind. We're all actually still trying to understand what this intelligence can give us. And so the prompts and the things that you develop is a good place to start. And, you know, it doesn't have to feel complicated. And, you know, whatever you can get your hands on, make sure you do that. And, you know, be aware of where your data is going. But at the same time, play, discover, share, share back with the community.

Starting point is 00:51:01 And definitely share any cool stuff that you've done in the comments for sure. All right, my man. I'll see you later. Thank you so much. Take it easy. Thanks.

The Startup Ideas Podcast - DeepSeek R1 - Everything you need to know

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.