DeepSeek Just Fixed One Of The Biggest Problems With AI

[00:00] Few people know, but modern AI systems [00:03] are really silly. How? Well, imagine [00:05] having a Michelin star chef being asked [00:08] for a simple peanut butter sandwich. [00:11] That's weird, but okay. Now, the chef [00:13] says you will need to just wait a bit [00:16] because I am going to start planting [00:18] peanuts. Wait 6 months, harvest, churn [00:21] some peanut butter, and then get to work [00:23] on your bread. That sounds really silly, [00:26] and that is exactly what modern AI [00:29] systems like ChatGPT and Gemini do. When [00:32] they need to recall a simple fact like [00:35] who Alexander the Great was, something [00:38] crazy happens. They go through complex [00:41] reasoning layers and reconstruct [00:43] everything from scratch every single [00:46] time. That is crazy. Now, I have an [00:49] amazing research paper for you here from [00:51] folks at Deep Seek AI, and this is a [00:54] piece of technology that might underpin [00:57] most, if not all, of the amazing AI [01:00] systems of the future. Now, every now [01:03] and then we are going to look at a [01:04] figure, but for the rest I am going to [01:07] bring my physics simulations and all the [01:09] goodness we talk about around here. [01:11] Apologies for that. Okay, so this is a [01:14] massive waste of compute, but why does [01:17] this happen? Well, standard transformers [01:20] are a kind of neural network that is [01:22] inside nearly all modern AI assistants, [01:25] and here's the problem. They lack a [01:28] simple and cheap way to just look things [01:31] up. Whatever the question is, the answer [01:34] is a huge bunch of dense mathematical [01:36] calculations from scratch. Ooh, yes, it [01:40] is literally planting that peanut when [01:43] you ask for a sandwich. Now, in this [01:45] work, Deep Seek introduces Angram. With [01:48] this, they are giving our tired little [01:51] chef a pantry. [01:53] Cutting-edge technology, brother. [01:55] Instead of growing that peanut butter [01:57] sandwich from scratch, it now just grabs [02:01] the ingredients from the pantry. I'll [02:03] explain to you how exactly they did it [02:06] in a bit. Now, this makes the AI way [02:09] more efficient. Okay, I expected that. [02:11] But, [02:13] what? [02:14] Are you seeing what I am seeing? Now, [02:16] this I did not expect at all. So, here [02:19] comes the surprise. Now, hold on to your [02:21] papers, fellow scholars, because when [02:24] taking away some of the AI's complex [02:27] reasoning parts, known as mixture of [02:29] experts, MOE, then replacing it with the [02:32] pantry actually makes the AI smarter. [02:36] Lower is better here on the loss curves, [02:39] and not just a little. This is [02:41] significantly smarter. Those dots [02:44] dipping way down show that this hybrid [02:47] chef makes far fewer mistakes than [02:50] previous techniques. It achieves a [02:52] perfect balance of active cooking and [02:55] just grabbing from the pantry. Genius. [02:57] But, that is not the only surprising [03:00] thing in the paper. They also added a [03:02] way for the AI to check the ingredients [03:06] before using them. You don't want [03:08] rotting fish in your strawberry jam. To [03:10] ensure this, they created a [03:12] context-aware gating mechanism. The [03:15] current context is the dish being [03:18] cooked. Now, here, this is compared [03:20] against the retrieved memory, the jar [03:23] from the pantry. If the jar's contents [03:26] don't agree with the dish, the gate [03:28] drops to zero, throwing the ingredient [03:31] away. Bye-bye, rotting fish. This [03:34] mechanism lives right here, inside this [03:37] jolly little dot product. Now, let's see [03:40] how it actually performs against the [03:43] current systems. I'll tell you exactly [03:45] what is going to happen now. What [03:47] happens in nearly all research papers [03:50] with something new. It does something, [03:53] it is compared to previous methods, and [03:56] it's better at some things, worse at [03:58] others. And then, you sit down and you [04:01] do your analysis. Okay, let's see. [04:05] Wait, what? [04:06] What just happened here? The new N-gram [04:09] technique makes the neural network [04:11] better everywhere. Absolutely everything [04:14] is measurably better. This is an [04:17] absolute miracle work. The N-gram model [04:20] is actually better on every single [04:23] benchmark compared to the previous [04:25] techniques. It is better everywhere. [04:28] Now, this is an amazing life lesson, [04:30] too. How? Well, essentially what [04:33] DeepSeek does is automates the easy part [04:37] and focuses on the more difficult tasks. [04:40] No wonder it works so well. What a time [04:43] to be alive. We can learn so much from [04:46] these research papers, and not just [04:48] about AI, but about life itself. Okay, [04:52] now I'll tell you how this works, and it [04:55] turns out there are more surprises [04:57] ahead. Dear fellow scholars, this is Two [04:59] Minute Papers with Dr. Car Jona Ifahed. [05:01] Okay, so how does it do this magic? [05:04] Well, it uses what they call N-gram [05:07] embeddings combined with multi-head [05:09] hashing. Okay, what the heck does that [05:12] mean? Well, in the kitchen, the chef [05:15] looks at the order ticket, sees a [05:17] three-word phrase, and instantly knows [05:20] exactly which shelf in the pantry has [05:23] the pre-made sauce and grabs it quickly. [05:26] And I think this also shows us that [05:28] there are simple and basic ideas in AI [05:32] that we haven't found yet. I mean, this [05:34] thing is basically a lookup table. It is [05:37] as simple as it gets, and it makes [05:40] everything more efficient and better [05:42] across the board. Just think about it. [05:45] We removed 20 to 25% of the smart [05:49] experts in this little virtual brain. [05:51] Put a spreadsheet there and it got [05:53] better. [05:54] I mean, what? Crazy. [05:57] And I love how we have a little better [06:00] understanding of the AI system itself. [06:03] Usually, no one knows what is going on [06:05] inside, but here, look, when they [06:07] switched off the Angram memory during [06:10] testing, the AI's ability to answer [06:12] trivia went down 70% [06:15] but its reading comprehension remained [06:18] at 93%. [06:20] Why? [06:21] Well, I think this shows that the AI [06:24] split its brain and it's using the new [06:27] part just to store the facts. Just think [06:30] about it. When they locked the pantry [06:32] door during testing, his ability to [06:34] understand a recipe stayed at a massive [06:38] 93%. [06:39] What does that mean? Well, it shows the [06:42] chef split the work perfectly. He used [06:44] the pantry strictly as a storage shelf [06:47] for memorized ingredients, but he can [06:49] still cook an amazing meal. I think this [06:53] is going to lead to even cheaper and [06:55] even smarter AI systems and this will be [06:58] an important part of why we will get [07:00] more systems that we can actually own. [07:03] No subscriptions. These run in our [07:06] pockets super fast, mostly for free. [07:09] Okay, now, not even this technique is [07:11] perfect. One limitation is that if you [07:14] put the Angram module too deep in the [07:16] network, it gets less accurate because [07:19] the model has already wasted time [07:21] processing what is being asked. Of [07:23] course, there is no need to look up what [07:26] you already computed. I think this is [07:28] common sense at this point. Our chef has [07:31] to check the pantry at the start of the [07:34] shift. If he checks it after the food [07:36] was served, the pantry is completely [07:39] useless. So, a really advanced research [07:42] paper explained in simple words. We are [07:45] fellow scholars, and that's what we do [07:47] here. And we have a growing club. I'll [07:50] continue in a moment, but do you know [07:52] who is also watching us? The one and [07:55] only Larry Wheels. Yes, he is one of our [07:59] OG fellow scholars doing some scholarly [08:02] work between two hard sets of bicep [08:05] curls in the gym. You think I'm kidding? [08:08] I am not. Link is in the description. [08:11] Reading his comment made me instantly [08:13] more muscular. So much value. Huge [08:17] respect to Mr. Wheels. Honored to have [08:19] you here. And here comes the best part. [08:21] I think this will be a part of every [08:24] major AI system, and it is knowledge out [08:27] there for free for all of us. And now [08:30] you know exactly how it works. No [08:33] nonsense where everything is hidden in a [08:35] proprietary system that costs $300 per [08:39] month to run. Nope, all free for all of [08:42] us. Glorious. An epic paper. Now, as our [08:46] chef does, I took a bit longer to cook [08:48] this video, but I promise that I did not [08:51] put together my computer from scratch [08:54] before starting. So, I took some more [08:56] time to make sure you get a better [08:59] video. If you feel this is the right way [09:01] of doing that, subscribe, hit the bell, [09:04] and leave a really kind comment. And you [09:06] can also check out Lambda with our link [09:09] in the description because it is an [09:11] excellent way of running Deep Seek [09:13] privately. I do it, too. Here you see me [09:16] running the full Deep Seek AI model [09:19] through Lambda GPU Cloud. 671 [09:23] billion parameters running super fast [09:26] and super reliably. This is insane. I [09:29] love it, and I use it on a regular [09:32] basis. Lambda provides you with powerful [09:35] Nvidia GPUs to run your own chatbots and [09:38] experiments. Seriously, try it out now [09:41] at lambda.ai/papers [09:44] or click the link in the description.