How many devs actually use that whole million-token context window...?

[00:00] we can't really extend the context [00:02] lengths of LLMs a lot further. Like and [00:04] and my I obviously I might regret saying [00:06] this, but we've had million token [00:08] context windows for 2 years now since [00:10] Gemini first introduced it, and no one [00:12] uses it. Everyone stays under 200k. Is [00:15] Is that just practical reasons because [00:16] of context rot and the fact that the [00:18] more context you feed it generally the [00:20] lower the quality output will become? So [00:22] So yes, that that is that is up there uh [00:24] in in the number of reasons, but there's [00:26] also cost reasons, right? Like you get [00:27] charged for every single context token [00:29] that you put in there. So like why would [00:31] you voluntarily make it more expensive [00:32] for yourself? [00:33] Uh and then there's also just order of [00:35] magnitude reasons, right? Like let's say [00:37] let's say I give you a million token [00:38] context window today with zero context [00:39] rot. It's perfect recall across all of [00:41] it, right? Absolutely solved. 1 million. [00:43] Then I Then I come along and go, you [00:45] know what'd be really great? A 100 [00:46] million token context window. Uh so you [00:48] just 100x and like also perfect context [00:50] rot. Like [00:51] you know, uh amazing. Great. But I'm [00:53] sitting over here. I'm in enterprise, [00:55] and my document uh database and storage [00:58] and notes started at 8 trillion tokens. [01:01] What is your 100x going to do for me? [01:03] Nothing. [laughter] Absolutely nothing.