Finally, deep seek four is here and it is described in a 58-page research paper and finally, nothing is held back here. I'll be honest, I'm feeling a little shy today, so I will do classic two-minute papers microphone but no camera and this is one of the biggest open and free AI models that we can use and excuse me. Do you see that? What? 1 million token context window in an open weights AI? If you ask it to inhale about 1,500 pages of dense documentation, it will do it. But that was the main feature in Google's Gemini not so long ago. I remember flipping out about it two years ago and now this for free. This sounds absurd and when I look at the pro model, you've got to be kidding me. It's results roughly match the many billion dollar frontier models from just a few months ago and now it is gifted to us mortals. >> [laughter] >> I'm trying to emphasize the kind of gift that we are getting here and my words fail me. Is this heaven? What a time to be alive and wait, there is a flash model that is much smaller and is somewhat competitive with the pro. I mean, what is happening? And it doesn't end there. This is just the start and it keeps outputting more and more text, the new pro model requires about three times less computing power than the previous one and the lighter flash model requires about 10 times less computing power. What am I even reading? How is that even possible? Dear fellow scanners, this is two-minute papers with Dr. Károly Zsolnai-Fehér. Well, it does three things that are absolutely magical. One, compression. Namely, compression for the KV cache. This is a scratchpad where you write your prompts and add your documents. Imagine reading a book. You can find answers so much quicker if you compress each paragraph down into one sentence. You keep the book, but now you can search it faster. They call it token level compression. But, even these little summaries add up. So, what do we do? Well, two. We want to know the overall plot of the James Bond book, see if it's the one that we read already. Well, of course, we look at the table of contents. If each chapter has a short name, we can grasp the whole story from that tiny piece of information. The paper describes it as a 128 to 1 compression. They call it heavily compressed attention. With that, the AI now sees the whole story at a glance. But, scientists at DeepMind say, "This is still not enough compression. We need more." Three, imagine that we want to search for a fight in the book. Table of contents helps a bit, but may not tell us exactly where the fights are. So, we look at an index, a list of words and phrases and their locations. Okay, so looking for a fight and bingo! The index gives us the top five pages that have fights in them. This is genius, and they call it compressed sparse attention. So, three layers of compression, summaries, structure, and index. And suddenly, the three pieces click together. These three reduce memory needs for the KV cache by about 90%. I had to look twice. Down about 90%. Squashing down 100 words into a storage space of 10, and you are saying that we are not losing basically every piece of information? Yes, that is exactly what they are saying. But, we are fellow scholars here. We look at proofs and experiments. Now, just to make sure this is KV cache compression. You still need to load the whole model. So, it does not mean that you can load the full DeepSeek Pro AI onto a toaster. Nope. Just want to make sure because of the media headlines and hype, you know. And now, hold on to your papers, fellow scholars, because this one delivers. They tested it by hiding eight facts inside increasingly long context. So, how good is it? Well, they report that the pro version recalls it better than Gemini 3.1 Pro. What? That is Google's flagship product. Wow, that is unbelievable. But, note that like many other systems, it starts to degrade as you start approaching the limits of the context window. Then, models forget, drift, hallucinate. More text means less truth. Also, let's look at the accuracy versus the previous DeepSeek, especially that this new version is heavily compressing things. Look at that. This is crazy. It is also fantastic at [clears throat] coding. If you are a coder, great. And if you are not a coder, well, you are now. It is so easy to ask it to generate JavaScript code that you can paste into a website and run. And in some cases, you can even run programs in the DeepSeek window with one click. I am a light transport researcher by trade. That is ray tracing, if you will. So, I had to try a little coding task related to that and ooh, this is fantastic. It still failed to properly implement more advanced algorithms. So, I am also excited to see what the next version brings. And it is crushing benchmarks and the competition at the low low price of free. Yes, if you can self-host it yourself, they also provide online access to it and it is so cheap. I feel like numbers are losing their meaning. Soon, intelligence will get too cheap to meter. Depending on whether there is a discount or not, you can easily get pricing that is 30 times cheaper than Anthropic's Claude. Even at no discount, things can get eight to 20 times cheaper. Crazy. Now, let's temper expectations a bit. Limitations, that's the thing missing from the media headlines. One, you can almost hear the 1,500 pages fluttering as it turns through it. But wait, I did not say also 10 hours of audio or full feature-length movie. There is a reason for that and the reason is that the system is unimodal, not multimodal. No images or audio. It is blind and deaf, if you will. Two, this system is not fully understood, not even by its creators. They report two techniques that magically stabilize training and they say that they are not quite sure why. I note that this is something that happens to every researcher and I have nothing but respect for the transparency. And three, we noted that if you are pushing against the limits of the context window, things break down a bit. Be careful. I just want to make sure that you don't get oversold on what is going on here. This still has limitations, not small ones. But overall, this is not a small step in open and free AI systems. Congratulations to the team and thank you so much. Now, here's what I think. I think this is a great release and a great paper and great life advice, too. Why? Well, you can adapt so many of these ideas to your thinking. Imagine walking in a forest. You want to look at the amazing views in front of you. But then, you trip or you look mainly in front of your feet so you don't trip. You watch your step or you enjoy the view, not both. So, what is the solution? Well, you do both. Scan near, glance far. Step and look. Local detail, global context. It is the same as what DeepSeek does. Try it out next time you're on a walk. It's weird. You'll see. Let me know in the context. I mean, comments how it went. They also use a technique called engram. Normally, an AI recalculates nearly every fact from scratch every time. But engram lets it just recall those facts instead. It's not as easy as it sounds. We have a separate video on it, link in description. And we are still just scratching the surface here. This is a really advanced research paper with all the good and the bad, not just the hype. Also, this video was not super fast. I rewrote this over and over again. Why is that? Well, because distilling complex ideas into simple explanations takes time. You get fewer views than others who publish something as quickly as possible. But that's what I try to do here and it is an honor to do this for such an incredibly smart and receptive audience like you, fellow scholars. And thank you so much for appreciating it. This one really made my day. Subscribe and hit the bell if you enjoyed this. Here you see me running the full DeepSeek AI model through Lambda GPU Cloud, 671 billion parameters running super fast and super reliably. This is insane. I love it and I use it on a regular basis. Lambda provides you with powerful Nvidia GPUs to run your own chatbots and experiments. Seriously, try it out now at lambda.ai/papers or click the link in the description.
Generated algorithmically for Search Engine
Indexing.