Inside an AI Factory – Power, Cooling, and GPUs Explained

[00:00] This course takes you deep inside the [00:02] physical infrastructure powering the [00:04] artificial intelligence revolution. You [00:07] will learn about the massive power [00:10] distribution networks, advanced [00:12] computing racks, and complex cooling [00:15] systems that make modern AI data centers [00:18] possible. The course traces the journey [00:21] of electricity from the grid all the way [00:24] down to the GPU. Across 16 detailed [00:28] chapters, Kian will break down the [00:30] complete anatomy of an AI factory, [00:33] showing you exactly what it takes to [00:34] turn raw megawatts into the tokens [00:37] generated by today's leading AI models. [00:40] Hello and welcome to this free code [00:42] course. My name is Kian and today I have [00:45] the pleasure to present to you the [00:46] course from watts to tokens which [00:49] answers the following question. What [00:51] does it actually take to make an AI data [00:54] center work? And to answer this, we're [00:56] going to follow electricity from the [00:59] grid to the GPUs, see how those GPUs [01:03] communicate, and then follow the heat [01:05] back outside the data center. Here's [01:08] what you'll understand after following [01:10] this course. You'll be able to follow [01:12] the power, cooling, and data paths. [01:15] Explain the major equipment required in [01:17] making a data center work and understand [01:19] why adding more GPUs or electrical [01:21] capacity does not automatically yield [01:25] more usable compute through many [01:27] intuitive case studies. The broad [01:30] journey in this course is we'll go from [01:32] electricity and racks to networking and [01:35] cooling. then follow it up with [01:37] construction, operations, and economics [01:41] and finish by tying this all together [01:43] with how a real data center was put [01:45] together. Specifically, Abalene, the [01:49] Stargate site currently be bu being [01:51] built for OpenAI in Texas. The reason I [01:55] made this course is because data centers [01:58] is what I believe will power the economy [02:01] the most in the near future in terms of [02:03] just year-over-year growth. And [02:05] furthermore, it's such an impressive and [02:08] interesting industry due to all of the [02:10] constraints which make it such that [02:12] there is so much innovation occurring. [02:14] Why is this the case? Well, when for [02:16] example, there are just bottlenecks on [02:18] getting power to your site, you come up [02:21] with a lot of ingenious ways to find [02:24] alternative power sources. And for this [02:26] reason, it's just incredibly interesting [02:28] to dive deep into, which is what this [02:30] course does precisely. What are the [02:32] prerequisites of this course? Well, I [02:35] assume basic arithmetic percentages and [02:38] simple graph reading capabilities. And [02:41] then beyond that, I have a primer which [02:43] is the first chapter in the course which [02:45] establishes shared vocabulary across the [02:48] three components electricity, computing [02:51] and cooling. So if the fundamentals of [02:54] electrical engineering and computing [02:57] overall are already familiar with you, [02:59] you can jump to chapter 2. Otherwise, [03:02] start with the primer. The slides, [03:04] written explanations, and glossery are [03:06] all linked below at the course homepage. [03:08] So, if there's ever a term you don't [03:10] understand, there's a very high [03:12] probability it's in the glossery. Or [03:14] just simply ask your agent. I think [03:16] that's the highest agency thing you can [03:17] do is get your agent to explain a [03:19] concept that you don't understand. [03:21] That's how I taught myself many of the [03:22] concept which I'll be teaching you [03:24] today. The last thing to note is that as [03:26] I am not a data center expert yet, there [03:30] may be minor errors here and there in [03:32] the course. If you are a data center [03:34] expert watching this, I apologize. [03:36] Please correct me in the comments [03:38] section so that I can learn. And then I [03:40] have many people to thank for this video [03:42] in terms of progressing and helping me [03:45] to learn more about data centers. But [03:47] Semi analysis is probably the one that I [03:49] should thank the most because their [03:51] content as well as the podcasts which [03:54] are recorded have helped me and given me [03:56] the passion and interest to dive deep [03:58] into this topic. Without further ado, [04:00] let's jump into the primer. Okay, so [04:03] starting off with the basics in this [04:04] primer, let's introduce the concept of a [04:06] circuit. And starting off, we have a [04:09] conventional DC circuit in front of you [04:12] with a source on the left here with a [04:14] negative and positive terminal. Then [04:16] there's a load which could be a light [04:17] bulb. It can be anything that takes [04:19] energy and then the return. So [04:21] essentially you have electrons in the [04:23] classical uh diagram. It's actually [04:26] moving from the positive end to the [04:28] negative end. And then there's this [04:30] lever that we have. So if you open the [04:32] circuit, the current would be equal to 0 [04:35] amps. So as a reminder, current is in [04:37] the unit of amps and it's represented [04:39] with I. Um there would still be voltage [04:42] though across the gap. So let's say that [04:45] we have a 12volt battery here. Then if [04:47] you measured the difference with a [04:48] voltmeter or a multimeter rather between [04:52] the two terminals of the battery, you [04:53] would read 12 volts. And the same thing [04:55] would apply at the load. It's [04:58] essentially the electrical potential [05:00] between those two points. That's the [05:02] definition of voltage. It's essentially [05:03] the difference in electrical potential [05:05] between two points. I like to think of [05:07] it as like somewhat similar to [05:08] gravitational potential energy, which is [05:10] to say like if you're at a higher [05:12] position, you have more gravitational [05:14] potential energy than somebody that's at [05:15] sea level. The one thing also worth [05:17] noting is once again there is still [05:19] voltage across a gap even if the circuit [05:21] is open. And how can you init this? [05:23] Well, think of electricity as a pipe [05:25] with water flowing through it. So if you [05:28] have a pipe which is flowing you have [05:30] both current which you can think of as [05:32] the moving of electrons and then voltage [05:35] um which I guess in like some convoluted [05:36] sense you can think of as uh as the [05:38] pressure of the water. And so the thing [05:40] is if you stop the water in that pipe [05:42] meaning there's no current there is [05:44] still pressure because if you have like [05:46] let's say 100 L of water above this pipe [05:49] and it wants to go down there's a lot of [05:51] pressure that wants to force it down but [05:53] it's closed. And this is why, for [05:54] example, in a circuit that does not have [05:58] current, you can still have a voltage [06:00] across it. [06:02] And now let's talk about adding in a [06:06] resistor. We had a resistor before, [06:07] which was the form of the load, but [06:09] let's actually give it a a number um 6 [06:12] ohms, which is the unit of resistance. [06:14] And then now that we have both current, [06:18] resistance and voltage, we have Ohm's [06:22] law which is essentially V equals IIR. [06:26] And using this formula we can calculate [06:28] for example if there are 2 amps of [06:31] current in the system and a six and [06:33] there's a 6 ohms resistor then um we [06:36] would have a 12vt DC source or better [06:39] said um if we have a 12vt DC source and [06:43] a 6 uh ohms resistor then we can expect [06:46] 2 amps of current um that are flowing [06:50] through this circuit and you just [06:51] literally get that by doing 12 / 6. So [06:54] once again voltage is between two points [06:56] and then current is through the path. [06:58] The units here are in volts which is [07:00] actually one jewel divided by one kum [07:03] and kulum is the unit of um just how [07:06] many electrons. It's like some really [07:08] big number that I forget. And then [07:10] ampers if I'm not mistaken is kulum per [07:12] second. So it's how many kums which is [07:14] this massive amount of electrons is [07:16] flowing through your circuit at any [07:18] given instant. [07:20] Okay. Now let's talk about resistance [07:22] and how modulating resistance increases [07:25] the heat loss in the circuit. So we [07:28] talked already about Ohm's law which [07:30] states that voltage equals current [07:32] multiplied by resistance. And then [07:34] there's another really important formula [07:35] here which is that if you look down here [07:38] at the bottom heat loss in a circuit is [07:41] equal to I^2 R. What does that mean? [07:44] It's essentially saying that the more [07:46] current you have in a circuit, you'll [07:48] quadratically have more heat loss and [07:51] you linearly will have more heat loss as [07:54] you increase resistance. So just as an [07:56] example, if we have a source voltage [07:59] that is equal to 12 volts, then we can [08:01] calculate once again that we'll have 2 [08:03] amps with a 6 ohm resistor and then [08:05] therefore the power loss is 2 ^ 2 * 6. [08:09] So 24 watts in this circuit. But let's [08:12] double our source voltage. And then once [08:15] again through Ohm's law, we can [08:16] calculate that the current in this [08:18] circuit will be 4 amps. And let's remind [08:22] ourselves that the resistor is actually [08:23] fixed. We're not making the resistor [08:25] larger. But all of a sudden, we have [08:27] quadrupled the amount of heat loss from [08:29] 24 W to 96 W because we've doubled our [08:34] current, which quadratically scaled [08:36] means we've quadrupled the loss of heat. [08:39] Of course, these are this is an ideal [08:41] source and we're assuming they're ideal [08:43] wires. You can just think of this as [08:45] like a simplification because in a real [08:47] system there's more factors and physical [08:50] effects at play. But um without loss of [08:52] generality, we can assume that in any [08:55] circuit doubling the current will [08:58] quadruple the heat loss due to resistive [09:00] effects. And why am I even talking about [09:03] this in this primer? It's because once [09:05] we start to really talk about data [09:07] centers in detail, we need to move power [09:10] to get these GPUs online, right? And if [09:13] we want to have power GPUs that require [09:15] more and more power, we need more and [09:17] more current. And this can lead to [09:19] pretty large amounts of heat loss if we [09:22] simply scale current um more and more. [09:25] And then the the teaser to how we're [09:27] going to resolve this issue is [09:30] essentially by increasing voltage [09:31] instead of current. So we have Ohm's law [09:34] which states that V= IIR. But then also [09:36] there's another formula that you might [09:38] have heard of. It says P= IV. So let's [09:41] break that down. Power which is in units [09:44] of watts equals current time voltage. [09:47] And why is this the case? If you [09:50] remember power is equal to jewels per [09:53] second and current is equal to um kum/s [09:59] and then voltage is equal to jewels per [10:04] kum and then if you multiply these [10:06] together then you get jewels per second. [10:08] So uh sorry if that was kind of [10:10] confusing but all I'm trying to say here [10:11] is that instead of increasing current [10:13] what we can do and what we'll discuss in [10:15] more detail in the next few chapters is [10:18] increasing voltage such that we can make [10:21] more power arrive at the GPUs. [10:24] All right and I think I gave a teaser to [10:27] what we're talking about. So essentially [10:28] here watts is jewels per second. Power [10:31] equals voltage time current. And I'll [10:34] just say one more time, voltage is in [10:36] units of jewels per kumm. And then [10:39] current is kum/s. That gives us jewles [10:42] per second. Um if we look at this [10:44] circuit that we have actually in front [10:45] of us, we know that it has a uh I [10:48] believe we had a 6 ohm resistor. And [10:51] then this gives us 2 amps through Ohm's [10:53] law. And then now we can calculate the [10:55] the power in this circuit which is just [10:57] 12 * 2. And uh this is in a ideal steady [11:00] direct current circuit. This gives us 24 [11:02] watts. So once again, power, which we'll [11:05] be talking about in a lot of detail in [11:06] this course, is simply energy [11:08] transferred per second. And what is W [11:12] hours? Well, even though it's watts, [11:14] this unit is actually equal to energy [11:17] because energy is just jewels. And Watt [11:19] hours means watts multiplied by hours. [11:21] So if you have jewels in the numerator [11:24] and then seconds or hours in the [11:26] denominator, if you multiply by hours, [11:27] then you just get uh jewels, which is [11:29] energy. So that's is essentially what I [11:32] said here. Energy equals power times [11:33] time. So let's say that we have a 1 [11:36] kilowatt load. So some type of tool or [11:39] maybe there's a GPU. So actually I think [11:41] H100s are like maybe somewhere close to [11:45] 1,000 watts or maybe like if you have [11:46] two H100 GPUs by Nvidia, then you'll [11:48] have somewhere around 1,000 watts or 1 [11:51] kilowatt. So if you run that for an [11:52] hour, that will cost or that will use 1 [11:56] kilowatt hour of energy. And this is [11:58] simply 1 kilowatt* 1 hour. And then um [12:00] just as a reminder here in terms of [12:02] units we have kilowatt or kilo rather is [12:05] a thousand m like megawatt or megabyte [12:08] is a million and then gigabyte is a [12:10] billion. So if we just double this um we [12:13] run a 1 kilowatt GPU for 2 hours then we [12:17] will have consumed 2 kow hours of [12:19] energy. And of course, this is going to [12:20] be really important as well since when [12:23] you need to calculate the power bill of [12:25] your GPUs or whatever electricity that [12:28] you're using, you're measuring it in [12:30] kilowatt hours most of the time. [12:33] And okay, so I've discussed direct [12:36] current or rather we've implicitly been [12:40] talking about direct current this whole [12:41] time where you have current that is [12:43] flowing only in one direction from one [12:46] of the terminals of the battery to the [12:48] other. And this is in opposition to [12:51] alternating current where both the [12:53] voltage and the current and alternating [12:56] current are alternating um as as the [12:58] name suggests. So just as a reminder [13:01] here in DC voltage the if you calculate [13:04] the voltage of one terminal relative to [13:06] the other it will always be constant. In [13:09] fact, if we say that the left terminal [13:11] relative to the right terminal is [13:12] positive, then if you switched the two [13:14] nodes that you're measuring, then you [13:16] should get a negative uh voltage between [13:18] those two terminals. And then the [13:21] current flows in the um engineering [13:24] diagrams that you'll see from the [13:25] positive end to the negative end. And [13:29] then in AC voltage, you have uh this [13:32] oscillating form which you would call a [13:35] sign a sign curve. So it's sinosodal. Um [13:37] and in this case the voltage essentially [13:40] every period passes through zero and [13:43] then reaches a peak positive and then a [13:44] peak negative uh amplitude. And then in [13:47] terms of the current it um it also is uh [13:52] essentially going from left to right 60 [13:55] times uh a second if you're in North [13:57] America because it's 60 hertz. Uh that's [13:59] the frequency of the alternating current [14:02] or it would be 50 hertz for example if [14:04] you were in the in Europe. Okay. But um [14:07] this is in the positive peak when we [14:08] said the current is flowing to the right [14:10] for um AC. If we're at the zero crossing [14:14] where the voltage and the current pass [14:16] through zero, then there's no current at [14:19] that instant. But um for DC, right, it [14:22] doesn't care about uh these three [14:24] values. These these um these AC moments [14:26] are only dependent on having AC current. [14:29] And then if we're at the negative peak, [14:31] the current is flowing left and um the [14:34] left terminal relative to the right is [14:36] actually negative. So there you have it. [14:38] This is the fundamental difference [14:40] between alternating current and direct [14:42] current. There's a really good book [14:44] called The Electricity Wars which [14:47] describes how when electricity was first [14:49] being essentially expanded through the [14:52] United States, you had Thomas Edison who [14:54] was a big proponent of direct current [14:56] and Nicola Tesla who was a big proponent [14:59] of alternating current. In fact, Nicola [15:01] Tesla worked for Thomas Edison at some [15:02] point and um it was a very fierce battle [15:06] to see which type of current would win [15:09] out um in in the market. So I recommend [15:12] that book as a as a reading. So one more [15:16] thing that I want to talk about about [15:17] alternating current that's really [15:18] interesting is that it's not necessarily [15:20] a sine wave. Although utility AC is [15:22] approximately cinosodial and in most all [15:24] applications it is. There are actually [15:26] other types of alternating current that [15:28] you can have. Right? Alternating current [15:29] doesn't necessarily mean that you need [15:31] to have a current which see uh which is [15:34] like which must conform to this shape. [15:37] Right? As long as it alternates, then it [15:39] te it technically meets that definition. [15:41] I'm not an extra I'm not an electrical [15:43] engineer, but this is what I read [15:44] online. So, for example, you can have a [15:45] square AC. You can also have a [15:47] triangle/zig AC, and then you can have a [15:50] sawtooth AC. But for all of the purposes [15:52] in this course, we're always going to be [15:54] dealing with a sinosodal alternating [15:55] current just because like all utility AC [15:58] is is produced in this method. So this [16:01] is just a more of an anecdote for [16:04] interest interest sake. [16:06] And then when we talk about the the [16:09] voltage magnitudes when we're comparing [16:12] direct current and alternating current [16:15] um both can vary in voltage. In fact if [16:18] you like literally used a multimeter to [16:21] check the outlet of your house and see [16:23] what is the current and you live in like [16:25] 120 volt uh region it might not be [16:28] precisely 120 volts. It could be like [16:30] 117 volts. And this is just because the [16:33] uh this the grid essentially varies in [16:36] voltage. It tries to maintain the [16:38] voltage which is the the target but um [16:41] it may be above or below based on the [16:43] load that's on the the grid. So for [16:46] example, if there's somebody in the same [16:48] distribution area as you, which all of a [16:50] sudden starts to consume a lot of [16:51] energy, that could bring the voltage [16:53] down. And um this is apply this applies [16:55] to both types of voltage AC and DC. Um, [16:59] so if we do a few uh a few perturbations [17:03] here to the to the voltage, if you have [17:04] like a 10% higher um voltage in either [17:07] case, then you would just go to 13.2 and [17:09] then 10.8 on the lower band. And the [17:11] reason why I'm bringing this up is [17:12] because most electrical tools or [17:15] anything that consumes electricity, they [17:18] accept a band of voltage just because of [17:21] the fact that the grid does not provide [17:23] voltage exactly at its uh target voltage [17:26] all the time. So they must have some [17:28] flexibility in the voltage range that [17:30] they accept. And when we talk about [17:33] alternating current which is the [17:36] predominant form of current used uh [17:39] worldwide and also this is not an [17:41] exception in data centers. There are [17:43] certain places where you actually use DC [17:44] and there certain places where you use [17:46] AC but for the uh electricity [17:48] distribution it's currently mostly all [17:50] done in alternating current and [17:52] specifically it's done in three-phase [17:54] alternating current. So what does this [17:56] mean? As I mentioned um alternosodal [18:00] so you have one sine wave but the [18:02] problem is if you have only one sine [18:05] wave sine wave so let's look at phase A [18:06] only then you'll have like some moments [18:08] where there's no voltage and this is [18:11] fine for like using a charging your [18:14] phone or some like trivial electronic [18:16] that doesn't need that much power but if [18:18] you're powering like large GPU racks [18:20] that can be in the hundreds of thousands [18:22] of watts then um you want a more [18:24] constant power band and what happens is [18:26] if if you have three phases of AC. So [18:29] essentially three wires that are all [18:31] alternating currents, but each one is [18:34] 120° apart. Then you have a more [18:36] constant power band. And what do I mean [18:37] by 120° apart? I mean that a period of a [18:42] alternating current cycle can be thought [18:44] of as 360° or in radians 2 pi. And if [18:49] you shift a another one to be um [18:53] essentially 1/3 to the right or 120° to [18:55] the right, then you would get this phase [18:57] B. And then if you shift it further like [19:00] 240° from phase A or 120° from phase B, [19:04] then you would get this third phase, [19:05] phase C. And what this means is that at [19:09] any given moment you have a phase that [19:11] is um that is at a positive voltage even [19:14] though the sum of the voltage across [19:15] these three is always zero. So, for [19:17] example, um if you just do like the [19:19] vertical line test and you look at this [19:22] point, um if you were to add these two [19:24] values of phase A and B with this value [19:26] with phase C, you'd actually get a net [19:28] voltage of zero. But, um on the individ [19:31] individual phases, you always have a [19:33] voltage reading. And as I mentioned, [19:35] this allows you to have a more constant [19:37] power band. So, as we'll talk about in [19:40] in the later chapters, you have these [19:42] PSUs, which are called power supply [19:44] units. And these are what convert the [19:49] alternating current into direct current [19:51] for the GPUs. And they're usually spread [19:54] across the phases. So essentially, [19:56] you'll have the three-phase wires going [19:58] into a specific rack where you have a [20:00] bunch of GPUs. And each PSU will connect [20:03] to one live phase. So one of these [20:06] energized phase and then one neutral [20:08] phase. Uh that's usually the arrangement [20:10] that you done that you do. And then um [20:12] you you would take the voltage [20:13] difference between the live phase and [20:17] the neutral phase as the the voltage [20:19] reading. And this can be around 277 [20:22] volts. But it really depends on the [20:24] equipment in the rack that you're using. [20:26] But as I mentioned, right, each power [20:28] supply unit uses two conductors, uses [20:31] one live phase, and then also one [20:33] neutral phase. So that's a little primer [20:36] on what three-phase alternating current [20:38] is. So on this next slide here, I'm [20:41] going to demonstrate how come on a graph [20:44] alternating current that's three-phase [20:46] has a much more constant power band than [20:49] singlephase alternating current. So what [20:52] we've done here is we're plotting [20:53] something with respect to time. It's not [20:55] actually voltage anymore. It's power. [20:57] And how we did this is simply by [20:59] multiplying each voltage phase by its [21:02] corresponding current at that point in [21:04] time. So you should think of you have I [21:07] * V on the Y axis which is just equal to [21:09] power and then you have time on the X [21:11] axis and it turns out that if we take [21:14] any specific moment in the cycle for [21:17] example 0° which is at the very [21:19] beginning of time then you'll have phase [21:22] 0 is uh 0 kow phase B is 15 kow phase C [21:26] is and then phase C is also 15 kow and [21:29] this is with the assumption that we're [21:30] with 480 volts I believe and then if we [21:33] just jump to 30° you'll now have phase [21:37] phase A ramping up to 5 kW at this point [21:40] phase B is 20 kow and phase C is [21:42] actually intersected with phase A so [21:44] it's also 5 kow so we have 30 kow still [21:47] and then finally you have at uh the last [21:51] example 60 60° phase A and phase B are [21:55] at 15 kow and then phase C is at 0 kow [22:00] and so you can imagine is that if we had [22:01] singlephase AC um if we just only [22:03] consider interface a at 0° you'd have 0 [22:06] kow you'd have no power then at 30° you [22:09] had 5 kow and at 60° you would have 15 [22:12] so you can imagine that um if you really [22:16] are requiring power constantly [22:19] three-phase is much more advantageous so [22:22] you have these balanced side waves and [22:24] we're assuming here that you have equal [22:26] resistive loads and this means simply [22:28] that each each of the three phases is [22:30] providing the same power to the end [22:33] consumer consumer which we would call [22:34] the load and yeah that's uh that's what [22:37] it is we have three-phase adding up to [22:39] steady power. Moving on we'll quickly [22:42] discuss power factor. So what is power [22:45] factor? It means in short to what extent [22:49] are the waveforms for the current and [22:53] the voltage and alternating current in [22:55] sync. So for a same for a singlephase [22:58] supply if we are currently if we're [23:02] completely in sync then the power factor [23:05] will be equal to one and this just means [23:07] that the amount of power that we deliver [23:10] through our circuit is all delivered to [23:12] the end load and then we have yeah [23:16] precisely 8 kow delivered if that's the [23:19] amount of power that's in the circuit. [23:21] Whereas if the waveform of the voltage [23:23] and current are out of sync, then you [23:26] might have a power factor below one and [23:28] what this means is that in order to de [23:30] deliver the same 8 kow, you need to have [23:34] 25% [23:36] more current. And um in this case, I [23:38] think this is just doing 1 divid 0.8. [23:41] That's how you get the amount of current [23:42] that you need if your power factor is [23:44] not equal to one. So at best in an ideal [23:46] circuit, you have a power factor of one. [23:48] And then um in in real in the real world [23:51] you'd have anything it would be strictly [23:53] less than one and the more efficient or [23:55] the more synchronous your voltage and [23:58] current uh waveforms the closer your [24:00] power for uh your power factor would be [24:02] to one. And uh this is a more of a quick [24:05] introduction to power factor. In fact in [24:07] chapter 7 or eight when we discuss this [24:08] in more detail I'll give a better [24:10] motivation to what this actually means [24:12] in practice. But this is just a quick [24:14] little resume of what power factor [24:16] means. And also I want to discuss the [24:21] different notation that we're going to [24:22] use throughout this data center course. [24:25] So of course different converters do [24:27] different jobs. So you don't want to put [24:29] the wrong symbol for something that is [24:31] doing a job that's different. So very [24:35] quickly if you have a transformer what [24:37] you're doing is you're changing voltage. [24:39] So in this case you have a AC [24:41] transformer and this changes the voltage [24:43] of an alternating current. It could go [24:45] from lower to higher or from higher to [24:47] lower. That depends. In fact, um you [24:49] would call a voltage or sorry a [24:51] transformer that is transforming to a [24:54] higher voltage like a step up [24:55] transformer. You're stepping up the [24:56] voltage and then conversely a step down [24:58] voltage or step down transformer for one [25:00] that is stepping down to a lower [25:01] voltage. And then also two extremely [25:04] important concepts of this course are [25:05] the rectifier and the inverter. Inverter [25:08] you might have heard of it's what [25:10] changes DC current to AC current. So the [25:13] most common example in a residential use [25:14] case would be if you have solar panels. [25:16] Those generate electricity that's direct [25:19] current. And then in order to use that [25:21] in the grid, you need to change it to [25:22] alternating current. So you would have [25:23] an inverter if you have solar panels. [25:25] And then a rectifier does the opposite [25:27] job. It converts from alternating [25:28] current to direct current. And then if [25:32] you have a power supply, um this can [25:35] take alternating current and convert it [25:38] into direct current. For example, when [25:41] we have our GPUs, we have the power [25:43] supply unit which takes alternating [25:45] current from the grid or wherever it [25:48] comes from. It could come from a grid, [25:49] it could come from a gas turbine, it [25:50] could come from a generator and then [25:52] converts that to direct current. Simply [25:54] put, electronics and low-level [25:56] semiconductors are they consume direct [25:59] current. [26:02] Okay? And I already mentioned this in [26:04] fact, but not only do electronics take [26:08] in a voltage range, but transformers do [26:10] as well. So here we have a example of a [26:14] Schneider Fasio. I'm not going to say [26:16] the entire name. This is a transformer, [26:18] I suppose. And what this does is it has [26:23] a nominal 400 volt um input, but it can [26:27] also accept voltage readings between 360 [26:32] and 440 volts. And these are the [26:35] published limits for this specific [26:36] transformer. And um in this case, it's [26:38] accepting a plus or minus 10% voltage [26:41] differential. So here's one example of a [26:43] voltage difference that a transformer [26:44] can accept. We'll also be discussing [26:46] solid straight solid state transformers [26:49] in this course as well which are really [26:50] cool because they can essentially be [26:52] tuned to accept different ranges and [26:55] also rectify the output range [26:58] automatically and uh those are really [27:00] cool new types of transformers that are [27:02] much more sophisticated than uh standard [27:03] transformers but we'll we'll leave that [27:06] there for now in terms of the depth of [27:07] the transformers. And here we're going [27:11] to quickly talk about a UPS which is [27:13] another fundamental concept in this [27:14] course. So, it's not the United Postal [27:16] Service. It's in fact an uninterruptible [27:19] power supply, which is a very [27:20] self-explanatory name. All it means is [27:23] that it's a power supply which in [27:25] principle should always supply power to [27:27] its source. So, how does it work? It's [27:31] really useful that we just talked about [27:32] rectifiers and inverters because that's [27:34] what a uninterruptible power supply is [27:37] composed of. So, let's start from the [27:38] very left here. On the left, we have the [27:41] utility or the generator, whatever is [27:42] providing the electricity. And this is [27:45] almost always an alternating current. [27:48] And then you have the UPS which does [27:52] something really weird. So it first [27:54] converts AC to DC with a rectifier and [27:57] then it converts DC to AC with an [27:58] inverter such that if you consider the [28:01] UPS a black box, it does nothing. And so [28:05] you might be like very offended that you [28:07] would have something that does nothing. [28:09] Especially since these two steps are not [28:12] free. You can think of them as each [28:15] requiring or rather causing a few [28:17] percentage heat loss or just regular [28:20] inefficiencies. So like as an example, [28:24] an uninterruptible power supply if you [28:26] put in 100 KOW may only return 95 kW. So [28:31] why are we doing this? It's because of [28:33] the thing that's on the bottom here. So [28:35] you have this battery which is the [28:36] entire point that the UPS exists. So one [28:39] thing that's really important to know is [28:40] that batteries supply current in direct [28:44] current and this is just a property of [28:46] the electrochemical reaction that [28:47] happens inside of a battery that [28:49] supplies electrons in one direction. And [28:51] the entire point of the UPS is that when [28:54] the generator or the utility shuts down [28:58] that we can take the battery and [29:00] instantly [29:01] um protect this protected load or [29:03] essentially instantly take over the [29:06] generator's capacity on the protected [29:08] load. So in order to explain why we have [29:11] this redundant conversion, consider the [29:14] counterfactual where we have a generator [29:17] that is directly supplying this AC load. [29:21] That would be very fine. But now if the [29:24] generator went down, you would first [29:27] have to route the battery through an [29:29] inverter that is DC to AC and then route [29:32] it to the load. And then this would take [29:34] some time, meaning that this load would [29:36] actually not be protected for some [29:37] period of time. there would be no power [29:40] being passed to the load because of the [29:42] fact that um this path would be [29:44] separate. But since the utility [29:48] generator and the battery are are are [29:50] both pointing to the are both pointing [29:53] through the same stream, essentially [29:54] what happens is that as soon as the UPS [29:57] detects that the generator is no longer [29:58] supplying electricity, the battery will [30:00] instantly through this DC toDC link pass [30:04] energy into the UPS and then invert it [30:08] to AC as if nothing happened. as if from [30:12] the from the point of view of the [30:13] protected load, it is still receiving AC [30:15] AC electricity. It's just that instead [30:17] of being passed through the utility/g [30:19] generator, the battery is now supplying [30:21] this alternating current. And once [30:22] again, this is such that there is no [30:25] interruption in the supply of power. If [30:27] we routed the battery directly to the [30:29] pro protective load or the pro protected [30:31] load through an inverter, then there [30:33] would be some gap of time or some lapse [30:36] of time where the load does not receive [30:37] power. And for many use cases that's [30:40] just simply not an option. Okay. And the [30:43] example that I just proposed to you and [30:46] um explained is actually a online [30:48] uninterruptible power supply. So it's [30:51] called online because it's on the link. [30:54] It's online. It's where the power [30:55] actually get routed to. That's where [30:57] it's connected. And what this means is [30:58] that the in inverter will keep supplying [31:00] the load instantly or near instantly [31:03] once the utility goes down. And this is [31:05] used for example in data centers or [31:08] sensitive medical systems. And then an [31:10] offline power supply is the other [31:12] example that I proposed. So essentially [31:14] what happens is that you have a direct [31:16] path from utility to load and then the [31:18] utility will feed the load directly [31:22] whereas when the supply is interrupted [31:25] the battery will go through an inverter [31:27] and then there's a switch essentially a [31:29] transfer switch which will switch from [31:31] the utility to the battery inverter [31:34] setup. And here is once again where the [31:37] delay comes from. This switch is not [31:38] instant. Okay. And then if we look at [31:41] the bottom here, if we just go from [31:42] normal to supply interrupted, the [31:45] battery feeds directly onto the DC link [31:47] and now the inverter keeps supplying the [31:49] load. Whereas there's this brief [31:51] transfer then battery power on the [31:53] offline UPS. And so you'd have an [31:55] offline UPS for example on desktop PCs. [31:57] I suppose this is the case because a PC [32:00] probably has some capacitance in its [32:02] power supply already such that if the [32:04] power goes down um it can still keep the [32:06] PC running for like a few seconds or [32:08] milliseconds until the uh the battery [32:12] starts to feed the the PC. So there's a [32:15] difference between online and offline [32:17] UPS's on interruptible power supplies. [32:19] But for the sake of this course, we will [32:21] assume that any UPS is an online UPS [32:24] because in the in the context of data [32:26] centers, we're only ever talking about [32:28] uh uninterruptible power supplies that [32:30] are online. [32:34] Okay. And now let's talk a bit about [32:36] redundancy. So let's say that you have a [32:39] 100 kow load. If that's the case, then [32:42] it would need two 50 kW modules. So [32:45] essentially two modules that are [32:46] supplying that power to the load. And [32:48] this would be called like n redundancy [32:50] because you have as many power modules [32:54] or rather the number of the amount of [32:56] power that you have is equivalent to the [32:58] amount of power that you must uh supply. [33:02] And let's say now that you add a second [33:05] or rather a third 50 cal 50 kowatt um [33:08] power supply then you would have n plus1 [33:11] redundancy. [33:12] Okay. And this is just uh the notation. [33:15] So n equals 2 normally. If you have a [33:17] third one then you would have n plus1. [33:19] And if you had four then you would have [33:21] 2 n or n plus2. These are equivalent [33:23] terms. A quick editing correction. N [33:26] plus2 and 2n in this case are the same [33:28] but that's not always the case. Stay [33:30] tuned for the UPS section where we [33:32] discuss this in more detail. We'll talk [33:34] about all the different configurations [33:35] later in the course, but this is just [33:37] the first taste of the differences in [33:39] redundancy. And if we just quickly talk [33:42] about the hierarchy of what a rack [33:44] actually contains. At a high level, you [33:47] have the rack itself, which is just a [33:49] vertical rack of metal. You can think of [33:50] it as like a really tall bookshelf. Um, [33:52] and within this rack, you have multiple [33:54] trays. Literally, they're trays that you [33:56] can take in and out. almost like the [33:59] tray that you put your dirty dishes in a [34:01] cafeteria. Um, and each tray holds a [34:04] compute server. On each compute server, [34:07] if we take a canonical example of like a [34:09] GPU accelerator, you have the CPU that [34:11] runs the system, some C some system RAM, [34:13] some NVMe probably to um handle [34:16] different networking and communication, [34:19] and then of course the main star of the [34:21] show here is the GPU, which has GPU [34:24] cores as well as memory. So this is kind [34:25] of a mini hierarchy of what a rack looks [34:28] like. We'll discuss this in much more [34:29] detail later on in the course. And with [34:33] respect to memory in particular, it's [34:36] really important to talk about this for [34:37] a second because this is how the GPU [34:40] actually has access to its model. So in [34:43] an example here, let's say that you're [34:45] getting your model from HuggingFace, [34:46] which you might have heard of. It's a [34:48] very common open- source library for [34:51] getting models that um are open source. [34:54] So you would get this from some server, [34:56] a model file. You pass it through the [34:58] network. It would first get loaded onto [34:59] the system RAM of the CPU likely and [35:01] then be copied directly to the GPU [35:03] memory. And then you would pass those [35:05] into the GPU cores if you're using a [35:07] Nvidia GPU into the tensor cores where [35:09] you actually multiply these matrices [35:12] with whatever prompt you have in order [35:14] to generate output tokens. [35:17] And just a quick little tangent on [35:20] networking and network speed. It's the [35:22] speed of the network that um determines [35:24] how fast you can load the model from [35:26] storage server to compute server. So [35:29] let's say that you have an 8 megaby 8 [35:31] megabyte chunk which is the same thing [35:33] as 64 megabits since we have one uh one [35:37] bite equal to 8 bits. Well, if we [35:40] consider a 10 GB per second transfer [35:42] speed, then you would take 6.4 four mess [35:46] to send all the bits and this also uh [35:49] assumes a 10 microcond first bit latency [35:52] that is not dependent on the transfer [35:54] rate. But then if you have 100 Gbit per [35:56] second uh uh transfer speed, then you [35:58] would essentially reduce that by an [36:00] order of magnitude to 0.64 milliseconds [36:03] to transfer the entire uh 8 megabyte [36:06] chunk. And so this is just a trivial [36:08] example, but you can already tell that [36:11] transfer speed and latency for passing [36:14] data between the model and the server is [36:16] one of the most important metrics that [36:18] determines like how fast can you serve [36:20] your users and will they actually be [36:22] satisfied with the service they receive [36:23] because nobody likes a really laggy and [36:26] slow model endpoint. [36:29] Another thing that we really want to [36:30] emphasize in this course is the fact [36:32] that these GPUs don't simply run for [36:36] free or they don't need to be it's not [36:38] like they need to be cared for. They do [36:40] they produce a lot of heat and they are [36:43] producing more and more heat in the [36:44] sense that with each successive GPU [36:47] generation the number of watts that each [36:50] GPU uh component consumes is increasing. [36:54] This isn't to say that they're becoming [36:56] less efficient. It's just more so to say [36:57] that like the actual GPU is becoming [36:59] bigger. So even if the GPU is becoming [37:01] more efficient, if it's still getting [37:03] bigger over time, it's getting more it's [37:05] getting bigger than it is getting [37:07] efficient. So this means that the the [37:09] net power consumption will increase and [37:12] this has been the trend um in the past [37:13] few years. So let's say for example you [37:16] input 500 watts. If the temperature of [37:20] the GPU stays constant over time, then [37:22] you also need to essentially get rid of [37:24] this 500 watts. If there is like a [37:27] deficit of input to output watts, which [37:29] is to say if you input more watts than [37:30] you take out, then over time your GPU [37:32] will overheat. So you need to have this [37:34] equilibrium. And this equilibrium is [37:36] achieved through a cold plate, which is [37:38] literally just a plate that is really [37:40] close to the top to the top of the GPU [37:42] die and where you run coolant, right? [37:46] That in this case, in this example, is [37:48] 30° C. And this coolant will go through [37:51] the cold plate. It will absorb heat from [37:53] the GPU. Let's let's say that it gets to [37:56] 45 degrees CC and um then the cold plate [38:01] is temperatures to 45 degrees CCelsius [38:02] and then the coolant coming out is now [38:04] increased in temperature by 5° and then [38:07] you would get rid of this heat by one of [38:09] various methods which we talk about and [38:11] this would be your steady operation of [38:13] the GPU. So you can't just assume that [38:17] the GPU will run on its own and be [38:18] happy. You need to make sure that it [38:20] stays nice and cool. And if we look at [38:22] this process one level of abstraction [38:25] higher, you have the cold plate which [38:28] was essentially zooming up into this [38:30] cycle. But in fact, you usually have [38:32] this two loop um protocol. So you have [38:37] the equipment coolant loop on the left [38:38] here which cycles coolant between the [38:42] GPU cold plate and with a pump to the [38:45] CDU which stands for the coolant [38:48] distribution unit. And this coolant [38:50] distribution unit has a heat exchanger [38:52] that will exchange the heat from the [38:55] coolant from the uh the cold plate from [38:58] the IT uh coolant loop with the facility [39:03] coolant loop. And this is the loop of uh [39:05] coolant that will in some way exchange [39:08] heat with the outside. So this could be [39:10] like through an air conditioning similar [39:12] a similar mechanism as air conditioner. [39:14] It also could be evaporative. We'll talk [39:16] about these in more detail as I [39:18] mentioned. And in this example, we just [39:20] have like literally a fan rejecting heat [39:22] through a passive cooling system. And so [39:25] the heat crosses the heat extender on [39:28] this coolant device unit, but it [39:30] actually the fluid does not uh cross [39:33] boundaries. And one of the reasons for [39:34] this is like the coolant in the IT loop [39:38] is much more treated and needs to be uh [39:42] essentially taken care of to a much [39:44] greater extent because it's coming like [39:46] in millimeters to the chip which is [39:48] worth like $70,000 or it's really [39:50] expensive. So this type this coolant [39:52] loop is very tuned and like watched for. [39:56] They have like a lot of sensors on these [39:57] racks to make sure that if there's ever [39:59] a leak it's detected instantly. Whereas [40:01] the the coolant that is in the outer [40:03] loop in the facility coolant loop, um [40:06] it's still probably like some pretty [40:09] clean and sophisticated cooling system, [40:12] but um it doesn't need to be as closely [40:15] monitored as uh as the facility, sorry, [40:17] the equipment loop because if there's a [40:19] leak here, it likely won't cause like a [40:22] whole rack to go down. You'd probably [40:23] just have to fix it up, even though it's [40:25] still a bad thing. Um, I'm just trying [40:27] to say that there are two different [40:29] levels of priority when it comes to [40:31] these two coolant loops. And the last [40:33] thing worth noting is that I'm only [40:35] talking about loops and like liquid [40:36] cooling. It's because in any data center [40:38] that is built today or onwards, even in [40:42] the past, like past 5 years, like all of [40:44] these are liquid cooled. And this is [40:46] because liquids can absorb much more [40:48] heat than air. Um, really like the days [40:51] of air cooling data centers are long [40:53] gone. And I'll show you guys a very [40:55] specific metric in the cooling section [40:58] that demonstrates why this point is so [41:00] salient. But just know for now that [41:02] anything that has to do with cooling in [41:03] this course is liquid cooling. And right [41:08] when we talk about cooling, it's [41:09] interesting because this is another [41:11] source of power consumption. So when you [41:15] talk about how much how many watts a [41:17] data center requires, you can't just [41:19] look at like okay there's 10 GPUs, each [41:21] GPU is 1 megawatt. Like this is a really [41:23] bad example. Let's say that each GPU is [41:25] 1 kilowatt. So in this case, you would [41:27] have 10 kilowatts of it power. But of [41:32] course, you need other things as well to [41:34] supply the entire system. So this could [41:36] be pumps, fans, support, literally just [41:38] like leaving the lights on um and other [41:40] trivial uses for electricity. the IT [41:44] system which is includes the computers, [41:47] storage and network takes up the vast [41:49] majority of power most of the time but [41:50] you still have this um this fringe on [41:52] the on the right which does require [41:54] powering and there's actually a really [41:57] important ratio here called the PUE the [42:00] power usage effect effectiveness which [42:02] is equal to the facility total um power [42:06] consumption or in this case we have it [42:08] as energy so in this case it's 1 hour of [42:11] use So really like 1 hour of energy use [42:14] is quite similar to just um the power [42:17] the power metric itself. So you can [42:19] either view this as 120 kwatt hours of [42:22] energy or just 120 kilowatts. It doesn't [42:24] really matter. In any case um we we we [42:27] would divide the entire power usage of [42:30] the facility by the IT power usage and [42:33] get the power usage effectiveness. In [42:35] this case it's 1.2. And if you haven't [42:37] already guessed, the number which you [42:40] want to approach is one. Because if you [42:43] approach one, that means somehow your [42:45] data center is running only by powering [42:48] the GPUs. That would be impossible [42:49] because you have to cool them. But um [42:51] you can definitely approach one, which [42:53] means that essentially you're using less [42:55] and less power for things that are not [42:57] the GPUs and um more for just powering [43:00] the GPUs themselves. So this is just an [43:02] example of a really important metric [43:04] that you can see for a data center and [43:06] already get an idea of like how [43:07] efficient is this data center and what [43:10] are the different uh uses of power that [43:13] it is consuming. Okay, [snorts] so [43:15] that's everything for our primer. Now [43:18] I'm excited because we're actually going [43:19] to get into the first I guess real [43:21] chapter here. So we're going to talk [43:23] about just an overview of data centers. [43:26] So it's similarly going to be like a [43:27] highle floor plan and like lay of the [43:30] land but uh we're going to get a bit [43:32] more into the the details here. So if we [43:35] talk about a a working rack like let's [43:38] say a GB300 which is a Blackwell rack [43:41] that has 70 GPU. So it's called a NVL72 [43:44] GB300 which stands for um Grace [43:47] Blackwell. 300 is just the model name [43:49] and then NVL is NV link which is the [43:51] interconnect technology and then there's [43:53] 72 GPUs. So that's the whole name. And [43:56] let's say that we have three of these [43:58] racks here. Well, the first thing that [44:00] has to happen is that we need to bring [44:01] in power from the grid. It will first [44:03] probably go into like some electrical [44:04] room where you have a power backup. So [44:07] some sort of some some sort of battery [44:09] configuration which in the primer we [44:11] introduced through the uninterruptible [44:13] power supply, the UPS. And then you also [44:15] have a distribution system because you [44:17] don't just distribute all of the power [44:19] through one wire. Um it's more [44:21] sophisticated than that for many [44:22] reasons. One of them being if you [44:24] distribute the power all through one [44:25] wire and that wire goes down then all of [44:27] the racks will turn off. Whereas if you [44:30] have a more redundant system where you [44:31] have some sort of hierarchy, well if one [44:34] of the wires goes down then hopefully it [44:36] will only affect one rack. That's uh [44:38] that's the whole point of having [44:39] redundancy. And then now if we jump on [44:42] to heat you'll have the compute racks [44:44] which do computation that's useful but [44:46] also generates heat loss. And then this [44:49] will pass through some cooling interface [44:51] which we defined as the coolant device [44:53] unit the CDU in some mechanical gallery. [44:56] This is usually like not inside of the [44:58] IT hall which you can think of as like [45:00] the main computation area of the data [45:02] center. And this heat is rejected [45:04] outside. And then finally you have the [45:07] information which you'll pass through [45:08] some fiber network because at the end of [45:10] the day a data center serves to allow [45:12] users to make some sort of changes to [45:14] their data or in the case of AI to make [45:18] calls to cloud code or codecs. And [45:22] you've heard the word IT hall. We also [45:24] can call this a data hall. So if you [45:27] want to see what that actually looks [45:29] like in practice in this case we have a [45:31] Google data center in New Albany, Ohio. [45:34] And these are TPUs in fact in this [45:36] photo. So the Google data centers really [45:38] are just a beautiful thing in my [45:40] opinion. Like look at that wire [45:41] management and all of the sophisticated [45:43] equipment here. Um these pipes are [45:45] probably passing coolant. I'm not 100% [45:47] sure, but that's just my initial guess. [45:49] So you can already see kind of where the [45:51] different parts of the data center are [45:52] routing in. And then also these black [45:55] wires are likely supplying the power to [45:57] each of the TPU uh racks. And then on [46:01] the top here we have white space houses [46:03] the IT which is the data hall and the [46:05] gray space supports operation. So let's [46:07] go back to to the picture that we saw in [46:09] the the previous slide. And the reason [46:12] why I had this initial part in white and [46:15] the the two left sides in gray is [46:17] because this is actually how data [46:19] centers are labeled or uh organized. [46:22] Essentially you call the IT/ data hall. [46:24] It also has a few other names but in in [46:26] any case where the GPUs are you call it [46:28] the white space and um it might be [46:30] called this because this is a place that [46:31] you keep clean and you need to be like [46:33] very attentive to whereas the other [46:35] spaces like the gray space where you [46:36] have the electricity and the the gray [46:38] space where you have the coolant um it's [46:40] less important but um in any case both [46:43] are important you should just know that [46:45] these are the different names that you [46:46] use to refer to these parts of the data [46:48] center and so we talked about grid and [46:52] we're just assuming that the power is [46:53] coming from somewhere Let's actually [46:55] talk where the power is coming from. You [46:56] guys probably know main mainly where [46:59] power comes from and let's describe it. [47:01] So on the left you have thermal. You [47:03] guys might be confused like where's [47:04] nuclear, where is gas, where's coal. [47:07] Well, you can think of nuclear, gas, [47:09] coal, and all forms of energy where [47:11] you're burning some fuel or creating [47:13] heat as forms of thermal energy because [47:16] essentially you make steam in most of [47:18] these cases and then use steam to power [47:20] some turbine which then spins a [47:22] generator and creates electricity. It's [47:24] the same principle for actually wind [47:26] energy except instead of doing it [47:28] through thermal um thermal energy, it's [47:30] through mechanical energy. It's through [47:31] the wind moving the rotor blades and [47:33] then um creating energy that way. And [47:36] then also solar energy which is becoming [47:38] more and more important throughout the [47:39] world. An interesting fact is there's [47:43] literally a solar farm in the Tibetan [47:46] plains of China which is like 18 gawatt [47:48] or something insane. Like this is one of [47:50] the largest power generating um things [47:53] ever built by humanity. There's of [47:55] course like the three gorgeous dam in [47:56] China as well which is even higher in [47:58] terms of peak gawatt production. But um [48:01] yeah solar farming and solar in general [48:03] is becoming more and more important. And [48:05] interestingly this is the only form of [48:07] energy in this entire um assortment [48:10] which is producing energy in direct [48:12] current. The only other main form of [48:14] energy which which performs and creates [48:16] direct current energy is uh fuel cells. [48:20] So there's one company that's really [48:21] wellknown it's called bloom energy which [48:23] uses a fuel cell to burn natural gas but [48:27] it's not through oxidation. It's not [48:28] through a regular oxidation process [48:30] where you add oxygen and then create [48:32] like oxygen. Sorry, you add oxygen to [48:35] some carbo hydrocarbon and then you burn [48:37] it to create carbo carbon dioxide and [48:39] then also energy. Instead, um it's [48:42] through some other electrochemical [48:43] process. In any case, solar and fuel [48:45] cells is the only two types of energy [48:46] that are mainly used for direct current [48:48] and then also you have hydro which is a [48:50] similar principle to other forms. You [48:52] spin some turbine in this case with [48:53] water and then you create electricity in [48:55] the grid and then the grid itself is an [48:58] alternating current in every single [49:00] system throughout the world to my [49:01] understanding. So these are some forms [49:03] of energy that we need to use in order [49:05] to power our data centers. And once the [49:09] energy has arrived at the data center, [49:11] we use well first of all high voltage [49:13] transmission to carry the power between [49:15] the regions and the the destinations. [49:19] And the reason why we do this is once [49:21] again in our primer we introduce the [49:22] formula P= IV. So the amount of power [49:25] that we uh transmit through some [49:27] transmission is equal to the current [49:30] multiplied by the voltage. And the [49:31] problem is if you increase current a lot [49:33] then you'll have a lot massive heat [49:36] losses which is very bad. It's [49:38] inefficient. In order to make [49:39] transmission efficient you need to step [49:41] up right once you generate the energy at [49:44] like a wind turbine you need to step up [49:45] the voltage to a very high voltage so [49:48] that the current can be sufficiently low [49:50] in order to keep the heat losses down [49:54] because heat losses as a reminder square [49:57] I mean scale quadratically with increase [49:59] in current. Okay. So that's why we use [50:01] high voltage transmission and then once [50:04] we get to the receiving end we will step [50:06] down to a lower voltage and this is for [50:07] multiple reasons like you don't just you [50:10] don't just like port like 345 kilovolts [50:13] energy to your house because first of [50:14] all it's like dangerous you need to have [50:16] like specific clearance um in order to [50:19] uh to prevent for example arcing from [50:21] happening where you have like literally [50:22] beams of electricity that ionize gas and [50:25] turn into plasma and many other reasons [50:27] which I won't even go into. Um, so we [50:30] don't always do this everywhere. We only [50:32] do this for high voltage transmission [50:33] where you have these power lines which [50:35] are at a specific height and are [50:36] engineered to support this type of [50:38] electricity efficiently. And when I did [50:42] some research on this in terms of [50:44] transmission and distribution of energy, [50:45] you have like around 95% [50:48] uh efficiency from generation to [50:51] reception in most systems. like this [50:53] could be more or less, but this is like [50:54] a ballpark estimate of how efficient [50:56] this whole system is to take energy from [50:59] where it's produced and then give it to [51:01] the destination. Of course, if you [51:03] produce energy at the destination [51:05] itself, then you'll have a much higher [51:07] efficiency for transmission at least [51:09] because um you don't need to do like [51:11] many as many steps as you would if [51:12] you're like producing energy at a dam [51:15] and then you have to move it 1,000 km or [51:17] 1,000 miles to the data center. So [51:19] distance is another factor that affects [51:21] the efficiency of transmission. [51:24] When we actually receive the energy at [51:26] the data center, it's at a substation. [51:29] So substations are just what you call [51:32] the equipment and the facility that [51:34] takes this high voltage utility [51:36] connection and then converts it [51:38] eventually uh to medium voltage which is [51:40] then used after we process the energy in [51:44] multiple further steps by the power [51:47] shelves of the rack to uh actually feed [51:49] the GPUs. So you can think of the campus [51:51] substation as connecting the site to its [51:53] buildings power systems. And when we [51:56] have the campus substation, we'll first [51:59] add a on-site generator optionally, but [52:02] most data centers do have some type of [52:03] generation scheme because if the utility [52:06] grid ever goes down, then you don't want [52:08] to be stranded. And in fact, there are [52:11] some data centers specifically like that [52:14] are being made by XAI, SpaceX, and [52:16] Crusoe which are using behind the meter [52:19] electricity, which is to say that [52:20] they're not even connected to the [52:21] utility grid. This isn't ideal, but the [52:24] best advantage of having BTM, which is [52:27] behind the meter, essentially meaning [52:29] we're not on the grid, is that you don't [52:32] have to wait to get permitting and [52:34] submit interconnect submit into inter [52:36] interconnect queue to like wait in line [52:38] to get permitted and connected to the [52:40] grid. Like that's the one benefit. In [52:42] any case, if we are using the utility [52:44] connection through the grid, then we'll [52:46] have our campus substation which will [52:48] feed to some building distribution with [52:49] an generator that is most always there. [52:51] And then this building distribution will [52:53] feed to the rack power shelves that [52:56] contain the power supply units. And [52:58] those power supply units will convert [53:00] actually the alternating current to the [53:02] direct current that the GPUs will use in [53:04] order to perform their computations. So [53:07] let's suppose that you are on the [53:08] utility grid and there's some power [53:09] interruption. Well, in that case, the [53:11] store a stored energy will bridge a [53:12] supply interruption while, for example, [53:15] generators are spinning up because you [53:17] can think of generators as taking at [53:19] least like one or two minutes or some [53:22] interval of time between turning on and [53:24] providing a stable power source. So, [53:26] generators are not instant. And we're [53:28] talking usually like diesel generators [53:30] or some type of gas turbine that is [53:32] going to provide energy to the data [53:34] center when the utility fails. So in [53:37] this intermediary time let's say it's 1 [53:39] minute you still need power and as we [53:42] introduced in the primer the util the [53:44] uninterruptible power supply is what [53:46] actually provides this buffer power to [53:49] the IT load while we wait for the backup [53:51] generator to spin up and at a very high [53:54] level what this is doing once again is [53:57] first of all taking this alternating [53:59] current rectifying it to direct current [54:01] and then inverting it back to [54:03] alternating current and this whole [54:05] redundant process This is once again [54:06] done because we have a battery which [54:08] produces direct current and has a DC [54:10] toDC link to this bus which is [54:13] essentially where the power is flowing [54:15] and we we do this entire redundant [54:16] process in order to ensure that we have [54:18] a instant uninterruptible power supply [54:21] to the IT load because to repeat myself [54:24] if the battery was connected to an [54:25] inverter that was off of the main link [54:28] then there would be some delay because [54:30] there would need to be that switch which [54:32] essentially decides are we taking power [54:34] from the battery or are we taking power [54:36] from the utility/generator and this [54:38] switch is not instant. In order to have [54:41] instant power delivery to the IT load in [54:43] the event of a grid failure, then you [54:46] need to have this UPS which includes [54:49] both a rectifier and an inverter. [54:52] Interestingly enough, even the battery [54:54] is not instant. A battery might take 10 [54:56] milliseconds to spin up and that 10 [54:58] millisecond buffer could be too slow in [55:02] terms of like the GPUs requiring [55:04] constant sources of energy. So you also [55:06] likely have capacitors in this setup as [55:09] well somewhere within the UPS such that [55:12] the capacitor can provide instant [55:14] voltage to supply the uh the power to [55:19] the IT load while the battery in that 10 [55:22] millisecond interval is actually [55:24] supplying um energy to this DC link. And [55:27] now if we look at the power rack in [55:30] particular, we have the power entering [55:33] the GB300 rack which I presented [55:35] earlier, the Grace Blackwell 300 through [55:37] its power shelf. So I'm going to zoom [55:39] into this for a second here and we're [55:40] going to talk about the vertical [55:42] hierarchy of this rack. So one thing you [55:45] should know, we'll talk about this later [55:46] that these racks are pretty big and [55:48] they're on the order of 1.5 tons. So [55:50] think of something pretty pretty heavy [55:52] and pretty expensive. So at the top you [55:55] have four power shelves here and also [55:57] four power cell shelves at the bottom [55:59] which which is eight in total and around [56:02] 260 kW of power that you can supply to [56:07] the 8 + 10 compute trays plus the 9v [56:12] link switch trays. So at a high level [56:14] these compute trays is what's actually [56:16] doing the matrix multiplications for the [56:19] transformer or other uh accelerated [56:21] workload that you're running. And then [56:23] the NVL link switches is what's allowing [56:25] for these different GPUs to communicate [56:28] with each other at like terabit per [56:30] second terabyte per second speed just [56:32] insane speeds that you have within this [56:34] within this rack. And then um at the [56:36] very top you have further rack switches [56:38] which are used for other networking [56:40] capabilities. [56:42] And I mentioned that we have 260 kW I [56:45] think it's 264 kW of power through these [56:48] power shelves that we can supply to the [56:51] to the IT load whereas the full rack [56:55] requires up to 142 kW at peak load. So [56:59] why do we have like 120 kW more power [57:03] delivery than required? Uh this is just [57:06] for redundancy. So, for example, if a [57:09] PowerShell fails, then in that case, we [57:12] can still run all of the GPUs normally. [57:15] Um, and furthermore, these power shelves [57:18] are actually hot swappable most of the [57:20] time. And what hot swappable means is [57:22] that while the rack is still on, you can [57:25] take out an entire power shelf, remove a [57:27] defective unit, and then put it back. [57:29] And um, the last thing I'll say is that [57:31] each of these power shelves contain six [57:34] power supplies. So you actually have [57:36] four + 4 power shelves which is eight [57:40] and then 8* 6 48 power supplies in this [57:43] rack. Um and then you can take them out [57:46] and then re reset them if one of them is [57:48] defective. The really key thing here is [57:50] that they're hot swappable and this [57:52] allows you to do replacements while you [57:55] can still have the the rack online which [57:57] is very critical. So one thing that I'll [58:00] mention here and we'll which we'll [58:01] repeat of the course is that the [58:03] alternating current supply that comes to [58:04] this rack is then received by these [58:07] power shelves which have the power [58:09] supply units six of them per shelf and [58:12] then this will convert the alternating [58:14] current to direct current. In the case [58:17] of the GB300 it converts it to [58:19] approximately 50 to 51 volts direct [58:22] current into this rack bus bar. So we'll [58:25] be talking about bus bars and buses a [58:27] lot. You can think of a bus bar as [58:28] essentially like this highway for [58:30] electrons and these electrons will move [58:32] across the bus bar to supply load to [58:36] supply the various loads. In this case, [58:37] the various um compute trays of this uh [58:41] power rack. So inside the rack, right, [58:43] we have the compute and the switch [58:44] trace. This is what's actually consuming [58:46] the energy. And I actually have a [58:48] picture of what this looks like from the [58:50] back. Um in fact, the power bus bar is [58:53] highlighted here. So, it's kind of hard [58:55] to see, but it's literally a vertical [58:56] bar, which is some type of conductor, [58:58] probably probably copper, but I'm not [59:00] sure. And this runs vertically because [59:02] we want to supply every single one of [59:04] the power shelves. And um then the power [59:07] shelves will supply each of the uh [59:10] compute trays. Feel free to pause here [59:12] as well if you want to see what the [59:14] other components of the back of the rack [59:16] are, which are really cool, honestly. [59:17] Like for example, you have seismic [59:18] bracing for like earthquakes and then [59:21] also a lot of the liquid cooling [59:23] components are also here and labeled in [59:26] quite detail. [59:29] Right? So coming back to the overall [59:31] hierarchy of the data center or at least [59:33] the rack, you have the chip which you [59:36] can think of as the GPU if we're using [59:38] an AI accelerator. Then the AI [59:40] accelerator/GPU is in some type of tray [59:42] and then you have many of these trays [59:44] which form a rack and then many racks [59:46] which form a cluster and then these [59:48] clusters areworked um in the case of [59:50] Nvidia they'reworked with um NV switches [59:55] um some type of in interconnect in order [59:57] to allow them to communicate across [59:58] racks and um this is how we perform for [60:02] example large training runs because when [60:04] you're training trillion parameter [60:06] models then you need a lot of me memory [60:09] and a lot of racks of GPUs in order to [60:12] uh train the model in a timeefficient [60:15] manner. Right? So if we look at the [60:19] interconnect on a cluster level and even [60:21] a campus level, these serve different [60:24] communication roles. So you might have [60:26] heard of a scaleup domain which is [60:29] within a single rack. So within a rack, [60:31] the domain that you can essentially [60:33] freely pass information or at least pass [60:36] information at an extremely high [60:37] bandwidth. So in this case we would be [60:39] talking about a single GB300 NDL72 rack. [60:42] It's called the scaleup domain. Then the [60:44] scale out domain is across multiple [60:45] racks. And in fact um you can scale out [60:50] to a carrier/wide [60:52] wider network which could include other [60:55] sites and users outside of the campus. [60:57] Um and then within the scaleout domain, [60:59] you'll usually have access to some [61:01] storage network as well where you can [61:03] host model weights or other networking [61:05] information that is useful. And um at [61:08] this campus edge, you'll have your [61:10] interface between the data center and um [61:14] the other receiving points. So that [61:16] could be another data center, it could [61:17] be just your end users. And this is [61:18] usually through some fiber optics. In [61:21] fact, there's a really cool special case [61:23] which is the TPUs. So, Tensor Processing [61:25] Units by Google uses optical switches [61:28] that connect 4,96 TPUs. Um, this number [61:32] is now insanely larger because we're [61:34] actually at TPU V8. So, this is four [61:36] generations too old, but um this this uh [61:39] this specific generation had the best [61:41] photos, so this is why I chose it. So, [61:43] essentially, we have these chips that [61:44] are in reconfigurable pods. There's 4,96 [61:47] per pod. And in this case, you have only [61:51] 1/8 of a TPUv4 pod in this in this [61:53] image. So each pod has 64 of these um [61:59] racks. So in this case there's eight. So [62:01] it's um 1/8 of 64 is uh is eight. And [62:05] then um essentially 64 of these racks or [62:08] blocks is how they're called. And each [62:10] one has 64 chips. So there's in total 64 [62:13] square chips. This is 4,96. And um you [62:16] might think this is a nightmare to [62:17] connect which it probably was. But um [62:20] you should always trust the engineering [62:23] ingenuity of Google engineers. And how [62:25] they do this is essentially each block [62:28] is connected with what you would call [62:30] optical circuit switches, which we don't [62:33] need to know exactly what they are, but [62:34] they're essentially mirrors that pass [62:37] light that has information, which is the [62:39] data that you're passing between the [62:41] different racks. And interestingly, [62:44] these mirrors can reconfigure links to [62:46] bypass faults. And that's something that [62:47] I find really cool. So essentially if [62:49] there's some fault in a in a optical [62:51] circuit switch, you can reconfigure the [62:53] mirrors to essentially pass through the [62:55] the error and um allow the data to pass [62:59] and be communicated. So in this [63:01] hypothetical example, we're passing data [63:03] from block A to block B through this [63:05] optical circuit switch. And the entire [63:07] system that is allowing the data to be [63:11] transferred between different TPU pods [63:12] in particular, if you're interested, is [63:14] called ICI, which is called interchip [63:16] interconnect. So pretty also common [63:19] sense name here and uh in this case [63:21] we're just connecting two blocks uh with [63:23] this link and Gemini 1.0 ultra which was [63:28] the largest Google model in 2023 trained [63:31] across multiple data centers. So in this [63:33] case, we actually are only communicating [63:35] between a single pod. But um whatever [63:37] the Gemini Ultra Run was doing, it had [63:40] these OCS switches between data centers [63:42] as well. And since Google is such a big [63:44] company, they can actually make these [63:46] data centers connected with their own [63:47] fiber and essentially control the [63:49] network that is um connecting them in [63:52] order to have the best possible [63:53] bandwidth. [63:55] So that's a quick little taste of the [63:58] extent to which these racks are [64:00] optimized for latency and communication. [64:03] Now moving the focus to heating. As we [64:06] discussed in the primer, in every single [64:08] data center that's being made today that [64:11] has liquid cooling, you usually have two [64:14] coolant loops that carry cold plate heat [64:17] to the outdoor plant. So when I'm [64:18] talking about cold plate heat, you're [64:19] talking about these metal plates that [64:21] sit on top of the GPU and absorb the [64:24] heat that is created while it's running. [64:26] And in this case, the coolant is called [64:28] like the it or the technology coolant. [64:30] So this is one loop and then this is [64:32] then fed to the coolant distribution [64:34] unit which has a heat exchanger between [64:36] the facility coolant and these are [64:38] completely separate fluid circuits to be [64:40] precise and that you can see precisely [64:43] in which part of the cycle if we're [64:44] going clockwise the coolant is hot on on [64:47] the top and the right and then cold on [64:49] the bottom and the left in both [64:50] instances. And then in this case, we're [64:53] just using a passive cooling system [64:54] where literally just fans are fanning [64:57] the heat outside to the atmosphere and [65:00] then returning the cool heat to exchange [65:03] with the heat from the GPUs that were [65:07] that were interfaced with the cool [65:09] plate. So we'll get more in more detail [65:11] as we go on for the course. And then now [65:14] just to talk a bit about it versus [65:16] facility support. So let's assume that [65:19] we have a 2 megawatt supply and that it [65:22] must power 1.5 megawatt worth of racks. [65:26] So in this case we have 10 racks that [65:29] are 142 kW which gives us 1.42 megawatt [65:33] and then also let's assume that we have [65:34] 80 kW of networking and storage and then [65:37] that gives us 1.5 kW of it and then we [65:41] have a name plate rating. What is the [65:42] name plate rating? That is essentially [65:44] the maximum amount of power that a data [65:45] center can provide. So let's say that [65:47] like the utility companies were like [65:49] okay we can give you at max 2 megawatt [65:52] of power that's the maximum amount that [65:54] we will provide to you. So in that case [65:56] that would be where the name plate [65:58] rating comes for or if you're using [66:00] behind the meter power let's say that [66:02] you literally have 2 megawatt worth of [66:04] gas turbines. Well you can't produce [66:06] more than 2 megawatts if that's the [66:07] rating of your gas turbines. So that's [66:09] the name plate capacity. And then [66:12] usually the amount of power that you [66:14] consume is not worth 2,000 kilowatts [66:16] because you don't want to be right at [66:17] the edge just because for example your [66:20] gas turbines might not always be powered [66:22] or rather running at their maximum [66:23] throughput. So let's assume that we have [66:25] 1.5 megawatts of IT capacity and then [66:28] 300 kW of support which could be cooling [66:30] also just heating up the facility not [66:32] heating up the facility but um [66:34] essentially the the HVAC system for the [66:36] normal offices that are in the data [66:38] center as well as lighting. um then [66:40] you'd have 1.8 megawatts of total [66:43] energy. So that would be it plus [66:45] facility support and this would fit [66:47] within your name plate rating which is 2 [66:49] megawatt. So these are some numbers that [66:52] we talk about when we're talking about [66:53] the overall power consumption of a data [66:55] center. We have the IT equipment and [66:57] then IT plus facilities port which in [66:59] this case would be 1.8 kilowatt versus [67:03] two um one 1,800 kW sorry versus 2,000 [67:07] kilowatts that it has it has rated. So [67:09] in this case we the data center should [67:11] should be able to provide enough power [67:12] to supply both the IT and the facility. [67:17] Okay. So one interesting thing that I [67:19] want to mention here is that like most [67:23] machine learning and AI workloads are [67:26] not like steady and constant. So for [67:31] example, let's say that you're running [67:32] some large evaluation that would cause a [67:35] large peak in demand. And assume this [67:38] evaluation [67:40] is ready at midnight and is due at [67:43] midnight the next day. So you have 24 [67:44] hours to complete this evaluation. And [67:46] that um over a 12-hour period, it [67:50] requires 4 megawws of power to complete. [67:53] But if you could stagger them, then you [67:56] could actually double the amount of time [67:58] it takes to 24 hours and then it would [68:01] only require 2 megawatt of energy. And [68:04] assuming that our our data center can [68:09] only supply up to 6.5 megawatt, then [68:11] this would be a way for example to [68:12] ensure that um the batch evaluations [68:15] could actually perform. Assuming once [68:17] again that we have a overhead of four [68:20] four megawatts as constant from like [68:22] live inference. just assuming just just [68:25] to serve our users and other overhead [68:27] from for example like this networking [68:29] and storage that we talked about. So in [68:31] both cases we would have a total energy [68:34] consumption of 144 megatt hours per day. [68:37] But the problem is that here we can't [68:39] actually run that because the peak [68:40] demand exceeds the 6.5 megawatt supply [68:43] limit. But if we stagger them as I [68:45] mentioned here, then we can complete [68:47] both of these jobs and um have a better [68:50] energy use over the entire day as [68:53] opposed to only being able to run [68:56] essentially half of the batch in this [68:57] example here. So one other thing just [69:00] coming back to the example that was two [69:02] slides ago is if we have a 300 kilowatt [69:07] hour support system with a 100 1,500 [69:12] kwatth IT uh facility consumption then [69:16] we can calculate the power usage [69:18] effectiveness which was introduced in [69:19] the primer as 1,800 divided by 1500 [69:23] which is 1.2. But let's say that we [69:25] actually make our facility more [69:26] efficient. we half the energy required [69:29] by our supporting system, then we would [69:31] change this ratio to 1.1. And this [69:33] demonstrates that the lower the power [69:35] usage effectiveness, the better our [69:38] facility is at supporting its IT [69:40] equipment, its GPUs with less energy. [69:43] And so let's just jump into here a quick [69:46] example. [69:48] So let's say that a hypothetical data [69:50] center campus records 12 megawatt hours [69:53] at the facility meter and 10 megatt [69:56] hours at its IT meters. So this would be [69:58] like the GPUs during the same hour. An [70:01] analyst adds them and reports 22 [70:03] megawatt of useful compute as the amount [70:06] of compute used by this campus. And the [70:10] question here is correct the total and [70:12] explain what these readings leave [70:13] unknown. So I'll let you pause if you [70:16] want to actually attempt this question [70:18] and then we'll work through it together. [70:20] So essentially what's being done here is [70:22] that this 12 megatt hours already [70:25] includes the 10 megatt hours. So it's [70:27] being counted twice. So really what we [70:29] can say is that the entire facility is [70:32] consuming 12 megatt hours of energy over [70:35] 1 hour and 10 of the 10 of that is for [70:38] the it. So what we can do here is [70:40] calculate the power usage effectiveness [70:42] um which would be 12 divid 10 that will [70:44] give us 1.2 um but we can't really [70:47] calculate much else other than that I [70:49] think if we look at the answer here we [70:51] essentially have the uh the p is 1.2 two [70:55] but um neither the ratio nor the [70:57] electrical demand tells us how much [70:58] accepted work the camp is computed [71:00] completed. So from these numbers that's [71:02] essentially the extent of what we can [71:04] compute and if somebody says that 22 [71:07] megawatt of useful compute was done then [71:09] they would be wrong. We can't really [71:11] also assume that 10 megatt hours is [71:14] equal to the compute itself because the [71:16] it is not just purely compute there's [71:18] also networking involved. So we would [71:20] know that the amount of useful compute [71:22] that was done for doing matrix [71:25] multiplications is something less than [71:28] 10 megatt hours of energy. All right. So [71:30] that was the uh the the little knowledge [71:33] check from the chapter 2 that we just [71:36] finished and now let's head to chapter [71:40] three. So in this chapter we're going to [71:42] be discussing the model and its users [71:46] which determine what the data center [71:48] must deliver. So, we're talking about [71:49] workloads in this chapter. And why do I [71:52] have a picture of the Nvidia CEO Jensen [71:56] Hong here beside this really funny [71:58] graph? Well, this was from Computex um [72:01] which is a conference that happened in [72:02] 2026. I think it's like Nvidia chip [72:04] conference. And in this conference they [72:06] were talking about AI factory revenue in [72:08] particular tokens per watt because [72:10] tokens per watt is becoming one of the [72:13] most important metrics for how much you [72:17] can profit from a data center because in [72:20] the current regime there is definitely a [72:24] constraint on how much power you can [72:25] procure. And what this means is that if [72:27] you have a given amount of power, let's [72:30] say 1 gawatt, then the amount of tokens [72:32] you can perform and produce for that [72:35] gigawatt of power determine how much you [72:38] money you can make. And that's why you [72:39] have Jensen Hong talking about tokens [72:41] per watt in this graph. And then also [72:43] there's just TTFI, which stands for [72:45] tokens to or sorry, time to first [72:48] inference. That's a strange um term [72:50] here. That's usually not the term that's [72:52] used in the industry. Um you would say [72:54] time to first token. So, TTFT, but in [72:57] any case, um there's many different [72:59] acronyms for the same thing here. The [73:01] the main reason I put this graph here is [73:04] just to highlight the fact that well, [73:06] your workflow, how you're serving your [73:08] users determines how much money you can [73:10] make and also the amount of power you [73:12] have and your token swap ratio [73:14] determines how much you can make at the [73:17] end of the day. Okay. So the three [73:19] criteria we'll talk about in this [73:22] chapter are like what is the actual [73:25] model and what is the state. So we'll [73:27] talk about weights, we'll talk about the [73:29] overhead of context and also like if [73:31] you're training what is the overhead of [73:33] actually having to train a model because [73:35] it's in general much more expensive in [73:37] terms of memory as opposed to just [73:39] inference where you're serving users on [73:41] a model that was already trained. Then [73:43] once it comes to serving users, well, [73:45] there's a bunch of things that you can [73:46] do in terms of batching and response [73:48] time in terms of latency as well. So, [73:51] we'll dabble a lot a bit into that field [73:54] to talk about the trade-offs that are [73:56] made in terms of balancing cost versus [73:59] serving your users at a specific speed. [74:01] And then also, we can't ignore the big [74:04] elephant, which is like the electrical [74:06] demand, right? There's no infinite [74:07] energy or electricity. So you need to [74:10] balance your users needs with the actual [74:14] amount of power that you have. And [74:15] furthermore, as I've mentioned already, [74:18] ML workflows are not in particular [74:21] steady. So there's lots of peaks and [74:23] transitions where at one moment you [74:25] could have a large load of inference [74:26] that you're serving and then at another [74:28] moment it could draw down. So this means [74:31] that the electricity infrastructure [74:32] needs to be resilient to these [74:34] transitions. And the two different [74:36] workloads that are the main types of ML [74:40] workflows are um training and inference. [74:42] So if we look at a really high level [74:44] what are these two things doing? So [74:46] training you have a forward pass where [74:49] you calculate the loss and then a [74:51] backward pass which gives you gradients [74:53] and then after you get the gradients you [74:55] use an optimizer to update the weights [74:57] in order to learn parameters. One [75:00] important metric in terms of training is [75:02] flops which stands for floating [75:03] operations per second. So in training [75:06] you have these massive batches of uh [75:09] text that you're processing through your [75:11] transformer model and the amount of [75:14] flops on your hardware usually [75:16] determines like what is your batch size [75:18] how many model how many how many how [75:21] much text can you process in one single [75:22] batch of course there's a lot of other [75:24] things that are in this calculation as [75:26] well in particular these models are so [75:28] large that they're actually not on a [75:29] single GPU they're sharted across many [75:31] um different devices in different forms [75:34] of parallelism is what you call these [75:35] different schemes. So that's also a [75:37] really important metric to to consider. [75:39] But um one important thing in training [75:41] is just how much how many operations can [75:44] you do per second across your rack or [75:46] your data center scaleout domain. And [75:49] then moving on to inference. What we [75:51] want to do here is produce produce [75:52] tokens for end user. Um important [75:54] metrics here are the prompt which we [75:58] must prefill. So essentially if the [76:00] prompt of the user is 1,00 tokens then [76:04] you need to prefill that into the model [76:06] and calculate a KV cache such that in [76:09] every consecutive inference step when [76:13] you produce the next output token which [76:14] is called decode. So you're decoding [76:16] with this uh auto reggressive model [76:19] which is the transformer. Then um this [76:21] KV cache allows you to not have to [76:24] produce the key and the value states of [76:26] every previous token at every inference [76:28] step. Um and if you don't know what a [76:30] key and a value token is, you can think [76:31] of it. It's just some internal [76:32] representation which tells you the [76:35] semantic meaning of all the preceding [76:37] tokens that you use to inform the next [76:39] token generation. And if you don't have [76:41] the KV cache, then you have to do a much [76:43] more computation. So it's essentially [76:45] just avoiding redundant computations. So [76:47] you need to store this in memory and [76:49] then use that in every decode step and [76:51] then in terms of batching right you have [76:53] a scheduler which will have a bunch of [76:55] active requests that it has to handle [76:57] and um essentially batch those in an [77:00] efficient way to balance once again the [77:02] trade-offs between serving models [77:04] serving users quickly and not having a [77:07] really inefficient um batching scheme. [77:11] So the two uh the two metrics here we'll [77:14] talk about is throughput which is just [77:16] across all users what is the tokens per [77:18] second. So let's say that we are serving [77:20] 100 users at 10 tokens per second. Then [77:23] the throughput there would be a,000 [77:25] tokens per second since this is [77:26] essentially um aggregated across all [77:29] users. And then per user the two [77:31] important metrics here are first token. [77:35] So how long does it take to get to the [77:36] first token? Um this is TTFT time to [77:39] first token and then inter token [77:41] latency. In between every single token [77:43] generation after the first one what is [77:45] the subsequent um gap? So if you're [77:48] saying like if you have a,000 tokens per [77:50] second as your your speed then [77:53] essentially the inter token latency [77:54] would be the reciprocal 1 divided by a [77:57] th00and seconds which I don't know is [77:59] that like one microcond something like [78:00] that. Uh so these are the two metrics [78:02] that we care about and then the last [78:05] thing which combines these two is called [78:06] interactivity. I think we talked about [78:08] this in a previous in a in a in a in a [78:11] succeeding slide rather um interactivity [78:13] is just tokens per user per second. So [78:16] this is just another metric that we use [78:18] to determine to what extent is our [78:21] inference bandwidth um optimized or what [78:24] is our inference bandwidth essentially. [78:26] So oh I I literally just jumped ahead of [78:28] myself. So I don't know if I said this [78:30] in the right order but in any case [78:31] interactivity is tokens per second per [78:33] user. I think I said tokens per user per [78:35] second. So forgive me for that mistake. [78:38] In this example we have one user having [78:40] an interactivity of 40 tokens um per [78:43] second. And this means that we have 25 [78:47] milliseconds between tokens. I think [78:49] this is justly 1 divided by 40. Uh 1 [78:51] second divided by 40. Uh and then the [78:53] throughput here depending on how many [78:55] users we have would just be 40 times the [78:57] number of users. And in fact um let's [79:00] look at the interactivity right. So if [79:02] you have a interactivity of 20 then this [79:04] gives you of a this gives you a inter [79:06] token latency of 50 milliseconds which [79:08] is larger. And then if you double or [79:10] rather quadruple from 20 to 80 80 tokens [79:13] per second then you will have an inter [79:15] token latency of 12.5 [79:18] milliseconds. So here we're talking [79:20] about just on a on a single user basis. [79:23] In this next graph let's talk about how [79:26] we can serve um multiple users. So on on [79:31] on the y-axis here you have throughput [79:33] which is just tokens per second per GPU. [79:35] And as Nvidia and Nvidia has um laid it [79:39] out here in the graph, higher AI factory [79:41] throughput as you go up. And then in [79:45] particular here we have the Nvidia GB300 [79:48] NVL72 which we've already talked about [79:49] throughout this course on a Quen 3.8 [79:52] model. So if you look at the um the [79:54] specs here, it's a three sorry it's a [79:57] 2.4 trillion parameter model, but it's a [80:00] mixture of expert models. So at any [80:02] given inference step only 95 billion of [80:06] the 2.4 trillion parameters are active. [80:09] So that means that only 95 billion [80:12] parameters are being used for every [80:14] single inference step. Uh and this is [80:16] called mixture of experts if you haven't [80:17] heard of it. And then otherwise we have [80:21] FP8 which is just to say that the model [80:23] parameters are in 8 bit precision or one [80:26] bite floating point precision. And then [80:29] we also use multi-token prediction which [80:31] is a type of speculative decoding which [80:34] allows us to predict multiple tokens at [80:36] once instead of predicting them one by [80:38] one. Uh we're just going to go through [80:40] this one by one. So Tensor RTLM is the [80:42] LM inference engine. So maybe you've [80:44] heard of VLLM or SG lang. These are all [80:48] different frameworks for serving models [80:51] at a high efficiency on on the hardware [80:54] that you have. And then um in terms of [80:57] the task, it's essentially just a task [80:59] where you get 8,000 input tokens and you [81:02] output 1,00 tokens. So that that is what [81:05] this benchmark is based off of. And um [81:08] essentially we're we're plotting [81:09] interactivity here. So NVIDIA has said [81:12] for example, if you have a minimum [81:16] interactivity requirement, so let's say [81:18] you have an SLA, which stands for a [81:19] service level agreement. So let's say [81:21] that you're selling compute to some [81:23] company and they say that we want at [81:24] least 100 tokens per second per user. [81:27] Well, that would give you a total [81:29] throughput of around a,000, which means [81:32] that you could serve maybe like around [81:34] 10 users at 100 uh tokens per second per [81:38] user. And let's say now the company says [81:41] I want at least 200 tokens per second. [81:44] And I guess I didn't actually do the [81:46] vertical, sorry, the horizontal line [81:48] test to see where this intercepts, but [81:50] um essentially taking this here and [81:52] bringing it to the left will tell you [81:54] however many tokens per second you can [81:56] serve across the entire uh rack of 72 [81:59] GPUs. And then finally, if you say that [82:03] you want 300 tokens per second per user, [82:05] then um you'll only be able to serve [82:07] like probably one user. And what you can [82:09] tell is that it's there's diminishing [82:11] returns. So this is not a linear plot. [82:13] Like if this was a linear plot, then [82:15] essentially if you could serve 10 users [82:17] at 100 tokens per second, that means you [82:19] could serve one user at a thousand [82:21] tokens per second. But that's simply not [82:23] how this optimization works. It's um [82:25] it's nonlinear. So this is a important [82:27] thing to know in terms of the more and [82:30] more you go into the latency um [82:32] sensitive regime or you want to be [82:34] faster and faster, the more the returns [82:36] are diminishing. The last thing though [82:39] that I'll mention is that this is out of [82:40] the box quen 3.8 2.4 fort. So probably [82:43] with a lot of interesting optimization [82:46] schemes, you could make this graph look [82:49] much more appealing, which would [82:51] essentially mean that like the the line [82:52] would go higher for a given [82:54] interactivity, you could probably have a [82:55] higher throughput. But um with the base [82:57] out of the box quant 3.8 2.4 T 2.4T 4T [83:01] model. This is the interactivity graph [83:04] and this is a extremely important metric [83:06] to to keep in mind because this [83:08] determines essentially the profitability [83:10] and the feasibility of how many users at [83:13] what throughput you can serve for a [83:15] given hardware configuration. So I [83:18] mentioned earlier how training is [83:19] generally more expensive in memory as [83:23] opposed to inference. And to make that [83:26] more specific let's let's look at an [83:29] example. So the same 70 billion [83:31] parameter model needs much more state [83:32] when its parameters are being trained as [83:34] opposed to being served. So let's say [83:35] that we're serving a llama 70B model at [83:39] two byte precision. So this means 16 [83:41] bits per parameter. Then um to serve [83:44] just the weights, we just do 70 * 2 [83:46] which gives us 140 GB of BF16 [83:50] um BF16 weights. And we do have to add [83:55] the KV cache plus runtime workspace as [83:57] overhead. So it's not just 140 GB, but [84:01] if we compare this to the training, then [84:03] we can see that the margin of KV cache [84:06] and runtime workspace doesn't really [84:08] make up for the fact that this is like [84:10] one order of magnitude larger. And why [84:12] is it one order of magnitude larger? [84:14] Well, it's because we still have the [84:16] same 70 bytes times sorry 70 billion [84:19] parameters times 16 bytes uh per [84:21] parameter. But we also have the atom [84:25] state which is actually in um four four [84:29] bytes precision. So 32 bits floating [84:32] point 32. Um you can think of atom [84:34] optimizer as like an optimizer state. So [84:36] this is essentially tracking for example [84:39] momentum which is a metric that is [84:42] extremely useful when you're training [84:43] models to allow the train to be more [84:45] stable. We don't need to get into the [84:47] details but just to just to be um giving [84:50] a summary here. We have the same two [84:52] bytes for the weights here as as in the [84:54] inference. But then we also have two [84:56] bytes for the gradients. We also have a [84:58] set of master weights which are uh four [84:59] bytes. We have the once again the [85:02] gradient and the second moment sorry not [85:04] the gradient but rather the momentum and [85:05] the second moment estimate which comes [85:07] from the optimizer state optimizer [85:09] state. And then also you have [85:10] activations which you store in order to [85:12] calculate the gradient. So all of that [85:14] is once again not really important to [85:18] know at a extremely low level. It's [85:20] simply important to realize that all of [85:22] these different states which must be [85:25] tracked mean that um in this like naive [85:28] example we have to store an order of the [85:31] magnitude more memory to train a model [85:34] versus serve it. And once again there [85:37] are probably a lot of techniques to [85:38] reduce this by a lot in in making the [85:42] memory of the training more efficient. [85:44] The same thing also applies for the [85:46] serving part. But um once again we're [85:48] just trying to get a high level overview [85:50] of the different memory requirements of [85:53] training a model on the right versus [85:55] serving one. And then if we talk about [85:57] context length, it's also interesting. [85:59] So this is independent of training and [86:02] serving. This is just the case of um [86:04] what you have to do when um when you [86:06] want to store a KB cache for any given [86:08] model. And in this case, the model we're [86:10] considering is llama 3.1 78b. And the [86:13] general lesson here is that longer [86:15] contexts retain more state because the [86:17] key and the values grow for every [86:20] request. So how do we calculate how much [86:23] a single token? So what it's a token for [86:26] LM inference? It's simply a term that [86:29] the LM can recognize. So for example, [86:32] the word 'the' might be split up into th [86:35] and the and the letter E. In reality, [86:37] the word 'the' is so common that it it's [86:39] probably it's probably its own token. So [86:41] let's say that the word 'the' is cached [86:43] in our model such that we don't need to [86:45] recmp compute it at every inference [86:46] step. Well, you have a cache for your [86:49] keys and your values which are two two [86:52] matrices. The actual model itself if you [86:54] search up the model card is 80 billion [86:56] parameters and then per layer um per [87:01] layer there are 80 there sorry there 8 [87:03] KV heads. So, I don't know if I said [87:05] layers or not, but anyways, there are 80 [87:07] layers. Per layer, we have 80 KV heads. [87:10] And each head has a width, which means [87:13] that the the vector is 128 um 128 values [87:17] in length. So, essentially each vector [87:19] is 128 uh dimensional. And then for each [87:23] value in this 128 length vector, we are [87:26] storing a number which is in BF-16 [87:30] format, which is two bytes or 16 bits. [87:33] And so if you do this multiplication you [87:35] get this value. So 327,680 [87:39] bytes per token or 220 kilobytes per [87:42] token. And this is the KV memory. So [87:45] this is actually the KV memory for one [87:47] token. So the number of cache tokens [87:49] times 3220 kilobytes gives us the [87:52] overhead for uh the amount of memory [87:55] that we need to store for a single [87:56] token. And I'd like to take this moment [87:59] to just discuss a interesting thing that [88:01] happened in the market probably at the [88:04] end of 2025 to beginning of 2026. You [88:07] probably know what I'm talking about [88:07] already. I'm talking about all of these [88:09] memory stocks like Micron, like SanDisk, [88:11] etc. Western Digital that have been [88:13] increased immensely in value. And this [88:16] calculation should demonstrate why these [88:19] companies have increased their market [88:22] cap to such a great extent. um for a [88:26] single token you literally need this [88:28] many bytes to store it in your model and [88:31] also I didn't mention this but this [88:33] excludes block overhead what is block [88:34] overhead it's simply the fact that um [88:37] when you want to assign memory in your [88:41] HBM your high bandwidth memory on your [88:43] on your GPU you need to do this in [88:45] blocks which are some defined width and [88:49] um likely the block will not be a [88:51] perfect multiple of the size of your [88:53] cache for your KV cache. So there'll be [88:55] some like min minimal buffer that you [88:58] also need to add to this cache token [89:01] bite requirement but um it's probably [89:04] relatively minimal and we can ignore [89:05] that. But in general, you have this [89:07] massive amount of token that you [89:08] require. And as reasoning models became [89:11] more and more popular in uh 2025, [89:15] then what happened is that um the amount [89:17] of KV cache became longer and longer as [89:20] your reasoning traces became more and [89:22] more protracted and um this led to [89:26] increase requirement in memory and a lot [89:28] of these companies that were producing [89:30] their memory became very valuable. [89:33] All right. uh and then context length [89:36] and model design changed the KV cache [89:38] budget. So what do I mean here? I'm [89:40] saying that in the naive sense when we [89:44] had llama 3.170B [89:46] um we like had a really big memory [89:49] requirement for our KB cache. So let's [89:51] assume that we have 64 GB allocated for [89:54] our KB cache. Well, so if we have 8,000 [89:58] tokens and if we multiply by the number [90:02] that we computed in this slide, 327,680 [90:06] bytes, then for these 8,000 requests, [90:09] sorry, 8,000 tokens, then we can have at [90:13] most 25 requests concurrently with this [90:16] 64 GB of allocated KV cache because if [90:18] we do more than uh 25 requests, then we [90:21] will just simply go over our 64 GB uh [90:24] weight a lot. sorry a memory a lotment [90:26] and then in fact if you quadruple the [90:30] amount of tokens per request then you [90:32] will divide by four the amount of [90:33] requests that you can serve [90:34] concurrently. So 25 divided by 4 rounded [90:37] down uh gives us six. So in this case we [90:41] have 10 gigabytes per request and just [90:42] as a legit check if you do 10 gigabytes [90:44] per per request multiply by six then you [90:47] get back your 60 or 64 if we account for [90:50] rounding. And um this is probably in the [90:53] regime here in this graph where we have [90:55] a pretty inefficient um way of routing [91:00] and storing our KV cache. And although [91:03] I'm comparing Llama to DeepSeek, the the [91:05] lesson here is still the same. [91:07] Essentially what has happened over time [91:08] is that you have new techniques such as [91:11] like multi head latent attention uh [91:14] which was one of the techniques [91:15] developed by DeepSeek in order to make [91:17] the KV cache require less memory. So [91:20] just in this example here you are [91:22] comparing the accumulated KV cache on [91:24] the y- axis in gigabytes to the sequence [91:27] length in in thousands of um in [91:30] thousands. So essentially this the [91:32] sequence length here is not 1,000 it's [91:33] rather 1 million and in deepse 3.2 two [91:37] for a million u for 1 million context [91:40] window you would have a 50 GB [91:43] requirement and then in deepseek v4 pro [91:46] that same 1 million token context window [91:49] was I don't know somewhere around 5 GB [91:51] and then in deepseek v4 flash it became [91:54] 13.7 times smaller um somewhere below 5 [91:57] GB and now we're already at deepseek [91:59] v4.1 flash as their latest model if I'm [92:02] not mistaken so this is why you should [92:05] naively assume that like memory will [92:06] just keep on going up in terms of the [92:08] stocks. Um the me the methods are also [92:10] adapting to this memory crunch to use [92:12] less and less memory while maintaining [92:14] performance and um it's essentially [92:16] doing this by compressing attention in [92:18] um in DeepSeek V4. So this is another [92:21] lesson that we can take is that even [92:23] though the memory a lotment was [92:25] initially pretty crazy, it is becoming [92:27] more and more efficient. But because of [92:31] uh what what what you would call [92:33] Givvon's paradox even though we're [92:35] becoming more efficient this is actually [92:38] this is resulting in more people using [92:40] the models because they're more [92:41] efficient and still in net we require [92:44] more memory than we did before. Okay. Um [92:47] we we did we briefly discussed the [92:49] difference between prefill and decode [92:51] which are the two different regimes that [92:53] occur in inference when you're serving a [92:55] model. So at a high level, prefill needs [92:58] compute and decode needs memory [93:00] bandwidth. So what do I mean here? Let's [93:03] remind oursel let's remind ourselves [93:05] what the difference is between prefill [93:06] and decode. So in prefill you're [93:08] processing the prompt. Um you're doing [93:10] many prompt tokens in parallel usually [93:12] in a batch and this is just a really [93:13] large matrix operation. So let's say [93:15] that you're open AAI and you have 10 [93:18] users which just submitted their codeex [93:20] requests. Each request is maybe like a [93:22] million tokens. Um that's that's [93:24] exaggerating but let's say there's [93:25] literally a million token codebase. So [93:27] you would have 10 million tokens that [93:29] you're operating in batch and usually [93:31] this is computebound. Um and the most [93:33] important metric here is time to first [93:34] token. What I mean by computebound is [93:37] that as opposed to the speed of loading [93:41] the uh the tokens into memory being the [93:44] bottleneck here. It's rather the speed [93:47] of waiting for the matrix [93:49] multiplications to finish as the [93:51] bottleneck of what what is what is [93:53] stopping you from processing these [93:55] tokens faster. And then if I move on to [93:57] decode, what's happening here is that [93:59] you've already processed the the model [94:02] or the the the prompts once. So at every [94:05] subsequent decode or auto reggressive [94:07] token step, you just have to read the [94:10] weights of the model as well as the [94:11] cache context and then generate the next [94:13] token. And this is often memory [94:15] bandwidth bound because you've already [94:17] done the brunt work of computing the KV [94:20] cache in the prefill phase. And in this [94:22] in this phase, you're just um worried [94:25] about the time between output tokens and [94:27] you're worried about moving the weights [94:29] in and out of memory and [snorts] moving [94:31] the new KV cache in and out of memory to [94:33] generate subsequent tokens. So that's [94:35] why it's memory bound. And this [94:38] fundamental difference between prefill [94:40] and decode has led to what's called [94:42] prefill decode disagregation, which [94:45] essentially just means that you're using [94:46] different hardware to perform each job [94:49] because each job is tailored to a [94:52] different type of hardware. So if you [94:54] have a really compute fast um machine, [94:56] you'd want to do that on prefill. And [94:57] then a type of a type of chip, if you've [95:00] heard of Cerebrris, would be better for [95:02] decode because it has extremely fast [95:04] memory bandwidth speeds. [95:06] Okay. And another thing that I [95:09] highlighted here is that we have large [95:11] matrix multiplications that were many [95:13] that were usually processing in parallel [95:15] if we're smart about it. Because if you [95:17] read the weight into memory and you only [95:20] compute it with one prompt. So you [95:23] essentially have your weight matrix here [95:25] that you load into memory from HBM onto [95:28] the GPU's chip. And if you only multiply [95:31] by one by one prompt, then you're not [95:34] really amortizing this weight transfer [95:39] at to a good extent. It's essentially [95:40] costing you a really a really large [95:42] amount in terms of time and energy [95:45] because you're not really doing this in [95:47] a batch manner. Whereas let's say that [95:49] you're doing this on eight requests, [95:50] then you're having more arithmetic for [95:53] the same weight read whereas we just use [95:55] one vector here. And this is allowing [95:57] you to amortize. It's the term that's [95:59] used this memory um transfer of the [96:02] weight onto memory to a larger extent [96:04] given that you're using this in a more [96:06] parallel and batched manner and in [96:09] particular when we're talking about [96:10] preill and um decode and this [96:13] disagregation [96:15] I wasn't just talking about this in [96:16] theory literally you have hardware [96:18] that's being introduced in this case in [96:21] Nvidia's case you have the Nvidia Gro 3 [96:24] LPX which is their product that is made [96:28] specifically for inference and what it [96:30] does is it has a Vera Rubin NVL 72. So [96:35] Vera Rubin is the generation after uh [96:37] Grace Blackwell and we have the same [96:39] Envy link 72 rack which processes the [96:42] prompt and does the prefill and then it [96:44] transfers the KV cache to these Gro 3 [96:47] LPXs which was a company that Nvidia [96:50] acquired and they made these special [96:52] chips that stream the answer really fast [96:54] because they have what's called SRAMM [96:56] that can have a extremely high bandwidth [97:00] in order to stream and process the [97:03] decode phase. in a in a quick manner. [97:07] Um, and then this has a really quick [97:08] time not time to not time to first token [97:11] but inter token latency. And this is [97:13] what allows you to have extremely fast [97:14] token generation both for the prefill [97:16] phase and the decode phase. Last time [97:19] I'll repeat myself. This is called [97:21] decode or rather prefilled decode uh [97:23] disagregation. [97:25] Okay. So now we're talk quickly about [97:28] batching and why do I have a picture of [97:30] a bus here? So um let me explain myself. [97:33] There's two different types of batching [97:35] or there are many types but I'll only [97:37] introduce two in this case. There's a [97:39] fixed batch which is the naive [97:43] implementation. So in in this case [97:46] essentially like let's say every 5 [97:48] seconds you will process some batches of [97:53] prompts that users come up with and you [97:57] only process new batches every 5 [97:59] seconds. So even though at slot three [98:02] you could process another batch um [98:04] because you have the space in memory, [98:06] you still wait until the next one or [98:08] rather you would wait until the last [98:10] batch is done. So you're kind of like [98:11] waiting for the slowest batch to finish [98:13] and this just wastes compute. And if we [98:16] have a bus here, the example here would [98:17] be like the bus can only pick up new [98:19] passengers once it's emptied all of the [98:21] passengers, which would make no sense, [98:23] right? You would essentially be passing [98:24] people even though the bus still has [98:26] space. Of course, if the bus is full, [98:28] then you can't let any more anybody else [98:30] on. But until that happens, you can [98:31] still welcome new passengers. And then [98:34] how a bus really works is what you would [98:37] call continuous batching. So as soon as [98:39] a is released, the C batch would uh fill [98:43] its place. And uh this is essentially [98:46] how most inference engines, so we talked [98:48] about tensor RT, we talked about VLM, we [98:51] talked about SG lang. This is how they [98:52] work in practice. They use continuous [98:54] batching. And in a bus this would just [98:56] be like every single stop if there are [98:58] passengers you will let them board. And [99:00] then every single stop if somebody wants [99:01] to leave you will let them come off. And [99:03] then the last edge case is like if the [99:05] bus is actually full if all the slots [99:07] are full then you do not let anybody on [99:09] the bus until somebody leaves. So that's [99:12] the example here about batching. And um [99:15] with respect to speed and like [99:17] processing tokens, we really emphasize [99:19] interactivity as one of the important [99:21] metrics. But once again power is [99:23] something that cannot be ignored. So [99:25] let's let's consider two runs run A and [99:27] run B which both have if we assume the [99:31] same completed tokens and quality on [99:33] whatever workload they're running. And [99:34] let's assume that run A uses 100 [99:38] kilowatts [99:40] um on average and that run B uses 80 [99:42] kow. So naively you would think, okay, [99:45] if they both do the same job, then run A [99:47] is worse because it uses more power at [99:49] any given moment. But let's assume that [99:52] run A takes 10 minutes and run B takes [99:53] 15 minutes. So if you just multiply this [99:55] to get kilowatt hours, it turns out that [99:58] run A is actually more efficient in [100:00] terms of power consumption or energy use [100:04] 16.7 kwatt hours as opposed to run B [100:06] that uses 20 kilowatt hours. So this is [100:08] because like run B took five more [100:10] minutes and this resulted in a 20% [100:12] energy increase. And all of this is to [100:14] say that [100:16] even if you have a system which uses [100:18] more power, if it can complete the job [100:20] faster, then it might actually have a [100:22] better energy consumption. [100:26] On that same note, we can also talk [100:28] about energy consumption in the sense of [100:32] the synchronization of GPU jobs in the [100:36] and to be more specific um when you're [100:38] training a model, the GPU falls or the [100:41] power consumption falls while you're [100:43] waiting for the exchange of the weights. [100:46] So essentially what happens right is [100:48] when you do a large model training run, [100:50] you have a forward pass where you pass [100:53] the the activate. So essentially you you [100:55] pass the prompt through the model to get [100:57] your activations and then use those [100:58] activations in a backward pass to [101:00] calculate your gradients and then once [101:02] you calculate the gradients [101:04] you will update your weights with the [101:05] optimizer and then you'll do what's [101:07] called an all reduce which will sync the [101:10] gradients across all of the models. [101:12] Sorry not the gradients but will sync [101:13] the weights across all of the models in [101:15] order to calculate the new model weights [101:17] for the next step. And what happens is [101:19] that during this all to all [101:21] communication which is the all reduce [101:23] you have less computation. [101:25] And what this means is that the low the [101:27] GPU power is much lower. And don't just [101:31] take my word for it. Let's look at this [101:32] graph which is really cool. So this [101:34] graph comes from a 2025 paper by [101:37] Microsoft OpenAI and NVIDIA which shows [101:40] the DGX H100 in a production training [101:43] run. And um the power on the on the y- [101:45] axis here is normalized just to be [101:47] between zero and one but it is actually [101:49] measured in watts. And um the GPU draw [101:53] in this time plot which is in seconds. [101:55] So we go from 0 to 200 seconds. So we're [101:57] on the order of 4ish minutes. You can [101:59] see how much this goes up and down. So [102:03] they don't actually label all right this [102:04] is the forward pass. This is the [102:05] backward pass. This is the optimizer [102:07] step. But I would assume that like this [102:09] would be the point in time when the GPUs [102:11] are communicating. And this is just an [102:13] extremely telling graph. Why is that the [102:15] case? It's because it already gives us [102:17] the intuition that when we're powering [102:20] these systems and we just naively give [102:23] them the same amount of power over time, [102:26] then there'll be a lot of problems where [102:28] at a at a given moment, if you're [102:30] serving like the maximum power, then [102:32] you'll essentially have a lot of excess [102:34] power that must go somewhere. So this [102:36] means that the the power supply that [102:38] we're feeding to these models needs to [102:39] be much more resilient. Of course, if [102:41] we're mixing up training with inference, [102:44] then that will also help because the [102:45] inference will kind of will kind of [102:47] serve to dampen these massive ups and [102:50] downs that occur during training time. [102:53] And we'll talk about these specific [102:56] techniques that Microsoft use for [102:57] example which are both on a software [102:59] level and a hardware level to make sure [103:01] that uh this synchronization that leads [103:05] to really low power uh consumption at [103:08] specific intervals does not affect the [103:10] downstream grid or the downstream [103:12] infrastructure and that will be in a [103:14] subsequent chapter. But for now just [103:16] understand that this is a problem which [103:18] must be addressed when we're training [103:20] models at a large scale. Okay. And we [103:23] already saw this graph in the previous [103:25] example, but let's actually determine uh [103:27] let's actually attribute numbers to it. [103:30] So the computing exchange phases creates [103:32] this time varying electrical load. Let's [103:35] assume in the simplified case that if we [103:37] have watts on or sorry kilowatts on the [103:40] y-axis and seconds on the x-axis that [103:43] the compute phase consumes 120 kow uh [103:46] the exchange is 40 kow. And then when we [103:49] are saving the model weights to the [103:52] model for the next compute phase, this [103:55] requires 60 uh kilowatts of energy. [103:58] Sorry, not energy but power. And that [104:00] the mean power is 85 kW. Well, once [104:04] again, if you now put shard this across [104:07] four GPUs, then the differences become [104:10] four times larger. And [104:15] the peak now is 480 kW instead of 120 kW [104:21] with a 340 kW mean. And per 60 cycle, we [104:26] are consuming 5.6 repeat kilowatt hours [104:29] of energy. And the one thing that I [104:32] wanted to say and the reason why I [104:34] brought up this entire example is that [104:37] this might not always be the case and [104:38] maybe it's actually difficult to make it [104:40] such but if we assume or if we can have [104:44] a instantiation where the jobs are [104:46] independent then in fact staggering them [104:49] by 15 seconds or starting them 15 [104:51] seconds apart to spread the power demand [104:53] would mean that the entire power band is [104:56] 340 watts sorry 340 kW and the energy [104:59] consumption still says the same because [105:01] we're doing the same amount of work over [105:03] this one minute of time, but we're just [105:05] staggering it such that the the the the [105:08] power spike doesn't occur like it does [105:10] here. So, this is kind of like an [105:12] idealistic uh framing of the problem. [105:15] But at a high level, this is one method [105:17] that we can use in order to prevent our [105:20] power requirements from peaking and [105:22] going to troughs over the cycle of our [105:25] workloads. [105:27] All right. So what did we discuss in [105:30] this chapter? Essentially what we said [105:32] is you need to size the supply from the [105:35] workload's measured power trace. So the [105:38] supply is like the actual energy. The [105:39] power trace is the power as a function [105:42] of time. And then the workload is what [105:44] are you actually like running? Are you [105:45] doing training? Are you doing inference? [105:47] And for each specific category there's [105:50] some key terms. So for the workload it's [105:52] like the model. So what is the size of [105:54] the model that you're training? the [105:56] context like how large is the KV cache, [105:58] how much memory do you need for it. When [106:00] we talk about token service is like what [106:02] are the requirements that you need to [106:03] serve to your users also are you doing [106:06] training are doing inference with [106:07] respect to power trace we need to know [106:09] the peak the energies the transition [106:11] speed so like how much is the energy [106:12] varying over as a function of time [106:14] essentially like what is the [106:15] acceleration of the of the power use um [106:18] and then the supply is just simply right [106:21] what is the grid what are what are we [106:24] dealing with in terms of power that we [106:26] can access how does the generation work [106:28] how fast can it adapt And also is there [106:30] like local storage essentially is there [106:32] like batteries which can supply a large [106:36] increase in energy surge and then also [106:39] turn off when the energy dampens. So [106:42] we'll we'll dive deeper into these uh [106:44] specific energy requirements and [106:48] accommodations in uh in subsequent [106:50] chapters but this is kind of a taste of [106:52] what we need to expect in terms of how [106:54] data centers need to adapt to the [106:56] workloads that um that AI comes with. [106:59] And these are very diverse. So now let's [107:03] talk about a bit more physical [107:06] infrastructure. So in this chapter we're [107:08] we'll really talk about like site [107:10] preference and how you like choose [107:12] sites. So if we literally look at this [107:15] on a three question basis. We'll talk [107:18] about when. So this is to say like at [107:22] what rate can the power actually get [107:23] online? How are we going to connect to [107:25] the grid? Are we going to connect to a [107:26] local supply? Are we going to use gas? [107:28] Are we going to use renewable energy [107:29] etc? And what so um what is like the [107:32] logistics of the power that we're [107:34] getting? How does it peak? What are the [107:36] backup supplies? And the bottom line is [107:38] we need to choose a site that can supply [107:39] each phase on its required deadline. And [107:42] in fact, if I just jump to the second [107:44] slide here in this chapter, if we have [107:46] multiple deadlines, what we can do is [107:50] stagger the construction such that we [107:53] incrementally put compute online. So [107:56] this isn't just theoretical. Literally [107:58] Cororeweave, which is one of the largest [108:00] hyperscalers, had a set of data centers [108:03] that they were building in October of [108:06] last year. And they received the first [108:08] first 50 megawws in October. So October [108:12] 27th is when the first phase was ready [108:14] for service. And then the second phase [108:16] was less than a month later on November [108:19] 24th. And at this point, they had 100 [108:21] megawatts of compute online. And you can [108:25] think of this as one way to cut your [108:28] oyster or whatever the idiom is. In the [108:30] second instantiation, you would just [108:33] build them all at the same time. And [108:35] maybe it would still take somewhere [108:37] around November to get the 100 megawatts [108:38] of comput online. But um you'll often [108:41] see that if you have like a massive 1 [108:43] gawatt project, it's probably faster and [108:46] better to develop them incrementally [108:48] based on the supply chain um constraints [108:52] and then at that point put the power put [108:54] the compute online as fast as possible [108:56] because right now there's an insane [108:58] premium for having compute that is ready [109:01] to go instead of just like building it [109:04] all at the same time and then um having [109:06] to wait longer for the entire thing to [109:08] come online. [109:10] And then we can see this same trend here [109:12] on Abalene which is the campus that is [109:16] being built in Texas by Crusoe [109:18] currently. So the entire target for this [109:21] target campus is 10 gawatt but the Abene [109:24] campus that's currently being built is [109:26] planned for 1.2 GW. And the reports from [109:30] Oracle as of um this month September [109:32] 2026 are that 75% of the capacity has [109:35] been delivered and is online. And they [109:37] didn't actually tell us the precise [109:39] amount of megawws that are actually [109:42] online. But if you just do 3/4 of 1.2, [109:46] then you get 900 megawatt as currently [109:49] online. And um this is just once again a [109:52] second case study of what happens when [109:55] you incrementally put compute online [109:57] versus wait until everything is done. It [109:59] just gives you the ability to make [110:01] profit much earlier than you would [110:03] otherwise. And um another cool thing [110:06] about Abalene is the fact that it uses [110:09] gas for on-site generation. So behind [110:11] the meter with these gas turbines, which [110:13] you can actually see right here, um [110:16] these these the power plants in Abalene. [110:18] And in order to service this, they they [110:22] actually made their own gas lateral. And [110:24] what's a gas lateral? You can think of [110:26] it as an offshoot of a main of a main [110:28] gas line. So we're in Texas where [110:30] there's abundance of gas. And um in [110:33] order to get this gas, they needed to [110:36] build a 14-mi offshoot, which they just [110:39] built themselves in order to supply uh [110:41] Abene with a source of natural gas to to [110:45] uh to supply their gas turbines. And um [110:48] yeah, this is just an example of if you [110:50] don't have the the resources, then [110:52] there's there's ways to get it. And in [110:54] this case, they literally built their [110:56] own gas pipeline to uh to supply these [111:00] gas turbines with fuel. And when one [111:03] talks about the feasibility and kind of [111:07] inspecting whether or not a data center [111:09] is ready and can actually access enough [111:13] power to be put online, you need to have [111:15] some joint studies to occur. Because [111:18] let's consider you're building two data [111:21] center campuses, campus A and campus B [111:23] right beside each other. If they both [111:25] depend on the same substation, then you [111:27] should ensure that this substation is [111:30] capable of supplying enough power to [111:33] both campuses and not naively assume [111:36] that okay um it meets the capacity for [111:39] campus A and like forget oh crap, we [111:41] need to double that because campus A and [111:42] campus B are both requiring the same [111:44] amount of power. Uh so this is just one [111:46] thing that we should keep in mind that [111:48] you need to do a lot of feasibility [111:49] studies before these data centers are [111:51] built. Um and this comes down to like [111:53] does the substation have enough [111:55] distribution capability and also does [111:58] the substation even receive enough power [111:59] from the grid or do we need to do our [112:01] own on-site generation in order to make [112:03] up for the deficit. And then um there's [112:06] four main types of power um consumption [112:10] or power generation techniques. So let's [112:13] distinguish between imports, local [112:15] supply, and just being completely island [112:17] off in a behind the meter um in the in a [112:19] behind the meter regime. So let's go [112:21] through each one one by one. Starting [112:24] off with grid supply. This is the case [112:26] where utility supply is the normal [112:28] demand and backup is a separate design [112:30] even though most data centers will have [112:33] some type of backup scheme. But in these [112:35] grid supplied um scenarios, the the [112:39] expectation is that the grid is what's [112:42] supplying the power at normal times. And [112:45] before this whole like data center crazy [112:47] GPU regime where everything is like out [112:50] of stock and there's these lead times of [112:52] multiple years, this is what the main [112:54] regime was because getting energy from [112:57] the grid is just much cheaper and much [112:59] more reliable than making generator [113:01] making energy yourself. And then if we [113:04] talk about grid parallel, this is [113:06] similar to just grid supplied except you [113:10] have local output which is to say like [113:14] you have your own on-site generators and [113:16] then you're also allowed to permit [113:17] imports. So you can think of like maybe [113:20] you know your you know your data center [113:22] is like 1 gawatt and there's always at [113:25] least like 500 megawws or 0.5 gawatts of [113:29] base load. So let's say that you bring [113:31] in a half a gigawatt or a 500 megawatt [113:35] turbine in order to supplement this base [113:38] load which is guaranteed and then you [113:40] will import from the grid in order to [113:43] essentially satisfy the spikes that [113:45] occur like maybe at noon there's a lot [113:48] of requests that happen. So you need to [113:49] import a lot of electricity to make up [113:52] for that um that spike that surge but [113:54] otherwise you're depending on this local [113:57] generator and then you will yeah just [114:00] reserve import capacity for whenever you [114:04] need it. Uh and this is essentially a [114:07] agreement that would be determined [114:09] between the utility provider and the [114:11] data center operator. [114:13] And then you have a export only setup [114:17] where essentially you supply all of the [114:20] energy that the data center needs to [114:23] data center uh yourself through your own [114:26] means of generation and then you export [114:29] or you have some agreement with the grid [114:31] to export energy that is created in [114:34] surplus. So in this case you would have [114:36] your IT capacity. Let's say that your IT [114:38] capacity is 1 me 1 gawatt and that your [114:42] um nonIT so your facility energy [114:46] consumption is 100 kilowatt so you have [114:49] 1.1 [114:50] um yeah you have 1.1 gawatt of energy [114:53] use in that case um sorry one let's say [114:58] that you have 1 gawatt of um energy [115:01] consumption by your IT capacity and one [115:04] megawatt of um energy use for the other [115:09] facility equipment. So that would be [115:11] 1,0001 megawatt of uh 1,0001 megawatts [115:14] of total consumption. So your your [115:17] generator would probably be larger than [115:18] that since you need to guarantee that [115:20] you can always support at least that [115:21] base load. So let's say it's like a 1.5 [115:23] gawatt generator. Then you'd be [115:25] supporting the data center through this [115:27] generator and then exporting access uh [115:29] to to the grid. And then the one thing [115:31] here is that there would be no imports [115:33] at all. So there would be no way to get [115:35] energy from the grid which means that um [115:37] if your generator goes down then you [115:39] should have like some redundancy in the [115:41] system and then completely off the grid [115:44] kind of behind the meter is saying that [115:46] you're just completely off the grid. So [115:48] you generate all of the energy uh [115:50] locally and you need some like type of [115:53] balancing some fast balancing to to make [115:56] up for [115:57] peaks and troughs in your energy demand. [116:00] So this might be in the form of a [116:02] battery system or other capacitors to [116:04] absorb these shocks. But in any case, [116:07] this is the fastest way to put a data [116:09] center online normally, but also likely [116:11] the most expensive. [116:14] And another thing that's interesting is [116:16] like Crusoe, which once again is this [116:18] data center AI factory serer or [116:21] contractor, have have had this hybrid [116:24] scheme. So what what have they done [116:25] here? Essentially they used gas turbines [116:27] as their initial power source [116:31] and then once the gas turbines or sorry [116:35] because why why did they do this? Let me [116:37] let me set this up for you guys. So [116:38] essentially there's these interconnect [116:41] cues which mean that you cannot [116:42] instantly get access to the grid and so [116:45] that they had this bridge where while [116:48] they were waiting to get grid access [116:50] energy they were using these gas [116:52] turbines to supply the data centers. So [116:54] you had this bridge supply, but then [116:56] once they finally got connected to the [116:58] grid, this gas turbine just became the [117:00] backup generators and then they instead [117:02] were importing energy from the grid. So [117:04] this is one one technique. I think also [117:06] SpaceX or XAI, whatever you want to call [117:08] them, have also had this same strategy [117:10] in one of their data centers. Um, and [117:12] it's kind of a way to get your data [117:14] center faster online so you can start to [117:16] generate tokens and make a lot of money. [117:18] And then once you finally get connected [117:20] to that grid, have a way to import [117:23] electricity from the grid and use this [117:26] latent capacity as your backup capacity. [117:30] All right. So when we talk about [117:32] generating behind the meter, the most [117:34] common form of generation is through gas [117:37] turbines. And you have two types. So you [117:41] have a simple cycle and a combined [117:45] cycle. The only difference between them [117:47] is a simple cycle has a gas turbine and [117:49] so does a combined cycle. But a combined [117:50] cycle has a steam cycle as well. Um [117:53] another thing is that you'll call these [117:54] you'll often hear these called CCGTs [117:57] which stands for combined cycle gas [117:59] turbine. And this would just be a simple [118:00] cycle gas turbine. [118:02] What you really need to know too is that [118:04] CCGTs are in immense scarcity. So like [118:08] you'll have to wait many years to uh to [118:10] get these if you want to order one. Um [118:12] the two biggest manufacturers or among [118:14] the two biggest manufacturers are GE [118:17] Vernova which is General Electric's Oh [118:20] there you go which is General Electric's [118:22] um [118:24] generator um department. So they make [118:26] the the generators and the steam [118:28] turbines these these massive uh gas [118:30] turbines for for the General Electric. [118:33] And then you have Seammen's Energy which [118:35] is the German company that also makes [118:37] these gas turbines. Um, the reason why I [118:39] said that they're in immense demand is [118:41] just because these are one of the most [118:42] efficient ways to turn gas into [118:44] electricity. In particular, CCGTs can [118:47] have up to 64% efficiency, which for [118:50] fossil fuels is almost unheard of. [118:52] That's like really efficient. And also, [118:53] they can just uh produce immense amounts [118:56] of electricity. So, you can have a CCGT [118:59] turbine in the order of gigawatts. [119:01] And um simple cycle is less efficient [119:04] and also less um less energy because it [119:08] only uses one of the cycles. And I'll [119:10] explain why that is the case. But [119:12] overall, these are really good ways to [119:14] turn gas into energy. And if you're [119:16] building a data center in Texas, which [119:18] is where a lot of data centers are being [119:19] built, or in Pennsylvania, which is the [119:21] second largest gas producing state in [119:23] the US, then you have a lot of access to [119:25] natural gas. And so this is the best way [119:27] to turn or to just get just get energy. [119:30] Um and the last thing I want to say is [119:32] an interesting dynamic which has [119:34] occurred is that there have been like [119:36] speculators who [119:39] foresaw the demand in data centers [119:41] before others and essentially had the [119:44] capital to place orders for these [119:47] combined cycle gas turbines or for these [119:49] simple cycle gas turbines with no [119:51] intention to actually buy them. and then [119:53] now they've essentially sold their spot [119:55] to others and made a pretty handsome [119:58] profit because of the fact that these [120:00] are in ma im immense demand. So that's [120:02] just an interesting anecdote that I [120:03] wanted to explain. But now I want to [120:05] actually explain how these work. So how [120:08] does a simple gas turbine drive its [120:11] generator. So at a at a high level these [120:14] simple cycle gas turbines kind of work [120:15] how like a a turbine works on a um on an [120:18] airplane. So you have a compressor [120:21] which compresses air into very high [120:23] pressure and then this high pressure air [120:27] is then combusted. So you inject the [120:29] fuel and then you burn it and then this [120:31] extremely high velocity um burned fuel [120:34] with high pressure just moving really [120:36] fast turns this turbine and then this [120:38] turning turbine turns the generator. Um [120:40] it's that simple. You can get into much [120:42] more detail if you want to but this is [120:44] at a high level how a simple cycle gas [120:46] turbine works. And I think you're [120:48] talking about like 30 to 40% efficiency [120:50] in terms of the amount of jewels in the [120:52] gas versus the amount of uh energy that [120:55] you produce from the generator. And [120:57] these are much simpler than a simple [120:59] than the combined cycle gas turbines. So [121:02] when I do the comparison, I'll first [121:04] describe how a combined cycle gas [121:06] turbine works and then compare the two. [121:08] So the gas turbine here in the in the [121:10] combined cycle is the exact same [121:12] mechanism as the simple cycle. So this [121:14] is the same. The only thing is now you [121:16] add a steam turbine also to the mix. So [121:18] what's happening here? Well, it turns [121:20] out right this combined cycle gas [121:23] turbine creates a lot of heat and in [121:25] fact like I think you have these [121:27] offshoots here which are offshooting the [121:28] excess heat and that's wasted in a [121:31] simple cycle gas turbine. But in instead [121:33] in a CCGT, you're recovering it and then [121:37] using a steam turbine in order to just [121:40] um turn a turn a turbine through steam [121:43] and generating excess power. Uh this is [121:45] also like how like a a nuclear power [121:47] plant works. It creates steam and then [121:50] turns a turbine. So essentially you have [121:52] this double cycle turbine which is not [121:55] wasting the heat but instead using it to [121:57] power a second generator. So I hope this [121:59] gives some intuition on how a combined [122:02] cycle gas turbine is more efficient than [122:05] a simple cycle gas turbine. This is a [122:08] diagram provided by Seammens. So then [122:11] the natural question is why the heck [122:13] wouldn't you just only use a CCGT? [122:17] There's multiple reasons. So first of [122:19] all they're more expensive and they're [122:21] more complicated. there's two parts like [122:23] there's two principal parts like [122:24] maintenance and just overall cost is [122:27] higher but nevertheless it's still like [122:29] on a very long basis um more cost [122:33] effective for the amount of power you [122:34] can produce there's also just the fact [122:35] that it's more complex so it takes [122:38] longer to build these and then lastly it [122:40] serves different roles so what do I mean [122:43] um you use CC CCGTs often for base load [122:46] so let's say that you know your data [122:48] center requires 500 megawatts all the [122:50] time then you would get a 500 megawatt [122:54] combined cycle gas turbine to serve that [122:56] load and always be on. And then let's [122:58] say that it spikes to 800 megawatts in [123:00] the daytime. So then what you might want [123:02] to do is buy a 800 sorry buy a um buy a [123:06] single cycle gas turbine that is 300 [123:08] megawws that will serve this peak. And [123:10] why would you do that? It's because [123:12] simple cycle gas turbines can actually [123:15] spin up faster since it's there's less [123:17] parts. It can reach its peak power [123:19] output faster. And then also there's a [123:22] specific trade-off point where because [123:25] the combined cycle gas turbines are more [123:28] expensive, [123:30] you want them to run all the time to [123:32] amortize their cost. But in fact, if [123:34] you're only using a generator for some [123:37] peak power since it's like probably [123:39] going to be off half of the time, the [123:41] fact that the simple cycle gas turbine [123:42] is cheaper means that in the long run, [123:44] it actually might be a better investment [123:46] just because it's only being used to [123:48] serve peaks. So you can see that these [123:49] are kind of complimentary turbines, but [123:51] both of these types are just in really [123:53] scarce quantity right now and have [123:55] multiple years of leadout time because [123:58] they're such a good way to turn um [123:59] energy into sorry to turn natural gas [124:02] into energy and you can into it the [124:06] extent to which these are in scarce [124:08] supply because literally Cummins for [124:11] example which makes truck engines is [124:13] turning its truck engines into [124:14] generators and then also you have like [124:16] airplane manufacturers airplane engine [124:18] manufacturers who are now pivoting into [124:20] data centers. You also have Nebus who [124:23] made a data center with Microsoft in New [124:25] Jersey using ship engines. Diesel ship [124:28] engines are being used to power data [124:30] center guys. So this is just describing [124:32] the extent to which they don't really [124:34] care how you make the the energy as long [124:36] as you can make it. Um it's it's fine. [124:39] And the reason why they're going to such [124:41] im immense um extents to procure energy [124:45] is because if you compare the bomb the [124:47] bill of materials of a data center the [124:50] energy is quite trivial compared to the [124:52] cost of the GPUs. So if you're like 2% [124:55] cost becomes 3% which would represent [124:58] like a 1.5x in the cost in the grand [125:01] scheme of things it doesn't really [125:02] matter that much because you're not even [125:04] paying that much in relation to the [125:05] GPUs. All this is to say that there's [125:08] many different ways to get make gener [125:09] make generate energy and um the the [125:12] CCGTs are the best but um there are [125:15] different ways and um since the bill of [125:17] materials is relatively small for these [125:21] uh generation schemes it's um it's not [125:24] that big of a deal if you have to pay a [125:25] bigger premium on making a diesel engine [125:29] use your gen create your energy as [125:31] opposed to some more efficient means. I [125:34] just want to quickly give you guys an [125:35] example of like a real CCGT in use. So [125:38] GE Vernova, once again, General Electric [125:40] has this complex in Danny Beach, [125:42] Florida, which um is a CCGT. So in fact, [125:45] there are two of them. These are the [125:47] General Electric 7HA.03. [125:50] What this means is, it doesn't really [125:51] make sense, but 7 stands for the 60 Hz [125:54] family. H just means high efficiency, A [125:56] is aircooled, and then 0.3 is the model [125:58] version. And so this can produce up to [126:00] 1.26 26 gawatt across these two CCGTs, [126:04] meaning that they're probably on average [126:06] 600 600 megawatts per unit. Um, and this [126:09] is just an example of what they look [126:11] like. You can see here the turbine is [126:13] probably somewhere in there. And then I [126:15] assume these are radiators, but uh do [126:17] not quote me on that. And then uh why [126:20] would you want to use a single cycle gas [126:22] turbine? Well, I already like motivated [126:24] this. On the left here you have a fossil [126:27] fuel based energy system where your base [126:29] load is served in black and you can [126:32] assume that this would be a combined [126:33] cycle gas turbine and then let's assume [126:35] that you have your peak load um that is [126:38] essentially peaking on um I guess like [126:40] probably the evenings on the weekdays [126:42] and this is increasing such that um you [126:46] would want to serve this with the the [126:48] the simple cycle gas turbines as we [126:50] explained previously. Um on the right, [126:53] this is kind of unrelated, but it's it's [126:55] explaining energy system with high per [126:58] uh penetration of intermittent renewable [127:00] energy. So for example, you'd have [127:02] solar, which you can see that these [127:04] blocks here are all the daytime. Um [127:06] they're producing they're producing this [127:08] power during the day and then the wind [127:10] in in contrast is is producing power [127:12] kind of constantly but is varying based [127:14] on the the strength of the wind. And [127:15] what this means is that if you have a [127:17] renewable system then the the necessary [127:19] base load is much smaller uh than it is [127:22] in a fossil fuel based system. But um [127:24] it's more interesting anecdote and it's [127:27] just saying that renewable energy is [127:28] cool. [127:30] So when we talk about like speed of [127:31] actually building these data centers [127:33] there are three meanings of fast. So [127:36] first of all there's like the delivery [127:38] which is like just like the EPC the [127:40] engineering procurement and [127:41] construction. how fast can you like get [127:43] this get the the parts and like [127:45] equipment on the site. And then there's [127:47] this there there's a startup. So this [127:49] means that like if there is a power [127:52] outage, how fast can you go from stopped [127:54] to generating? And then there's also [127:56] ramping. So once you have like actually [127:59] turned the data center on, how fast can [128:01] you go to peak output uh and produce [128:04] your tokens at the peak throughput that [128:08] the data center advertises? So we'll [128:10] talk about these in the later sections [128:14] as well as throughout this chapter. One [128:17] thing though is I talked about like this [128:19] break even point between using simple [128:22] cycle gas turbines as well as combined [128:24] cycle gas turbines. So if we assume that [128:26] a simple cycle gas turbine is 40% [128:28] efficient and a combined cycle is 60% [128:30] efficient. Then if we look at the uh the [128:33] fuel cost divided by time then the slope [128:36] of the combined cycle is is uh is lower [128:38] because it uses less fuel. And then [128:41] furthermore we'll assume that the fixed [128:45] cost of a simple cycle is 8 million per [128:48] year and the fixed cost of a combined [128:50] cycle is 60 million per year and that [128:52] fuel is $20 per megawatt hour. Let's [128:55] assume also that we're running 100 [128:58] megawatt generators on both scales. [129:01] Therefore, the break even point of using [129:04] a combined cycle gas turbine is 4,800 [129:07] hours in a year. And in fact, there are [129:10] around 8,000 hours in a year. So, if we [129:12] do 24 * 365, we get 8,760. [129:17] So, essentially, if you're running a [129:19] generator for more than half of the [129:20] year, then it's more cost efficient to [129:21] use a combined cycle. [129:24] All right. Now, let's talk about [129:25] Colossus 2. So, if you haven't heard of [129:27] Colossus 2, this is the co complex that [129:30] Space XAI is building in the Memphis, [129:33] Tennessee area, which is actually called [129:35] South Haven. And the interesting thing [129:38] here is that Colossus 2 and its power [129:40] plant sit on opposite sides of the state [129:41] line. So, Clauses 2 is in Tennessee, [129:44] which you probably know, but what you [129:46] might not have known is that they [129:48] actually built a power plant in [129:49] Mississippi, which is only like a few a [129:51] few uh miles south, and it's actually a [129:54] massive plant. So it's 41 turbines which [129:56] are 1.2 gawatt and I think now it's [129:58] actually been put online. So these [130:00] numbers are from January 2026 but 8 9 [130:03] months later it's actually in operation. [130:05] And if I do 1.2 divided by 41 you have I [130:09] guess 30 megawws per turbine. So these [130:12] are probably like less less complex [130:14] probably like diesel or small gas [130:16] turbines that are being used here. And [130:18] why why did they have it built in [130:19] Mississippi? So, first of all, it's a [130:22] really interesting and smart planning [130:23] choice by SpaceX AAI to have built this [130:26] on the state border with the future [130:28] thought of, okay, if Tennessee ever [130:32] determines or plans out to be turns out [130:34] to be kind of less flexible in [130:36] permitting, then we can always go to [130:37] Mississippi. And that's what they did [130:39] because the problem with behind the [130:41] meter power, which is what they did [130:43] here, they built their own turbines, is [130:44] you have to go through EPA, regulation, [130:47] the environment protective the [130:49] environmental protective agency in the [130:50] US. And why do you have to do this? It's [130:53] because you're actually producing a lot [130:55] of CO2 emissions when you when you're [130:57] burning gas, right? So, you can't just [130:59] like say, "Oh, yeah. I'm going to build [131:00] like 1.2 gawatt and put like 3,000 tons [131:03] of CO2 into the air per year." you have [131:05] to get a permission from or like a [131:07] permit from the EPA or you can get a [131:10] 12month temporary permit from [131:12] Mississippi which is what they actually [131:14] got um when they built this plant. So [131:16] that's that's why they went to [131:17] Mississippi is cuz I guess the county in [131:19] Mississippi gave them a 12-month [131:20] temporary permit which I think they've [131:22] now actually um converted into a [131:24] permanent permit to operate the the gas [131:27] plant. And so it's quite an interesting [131:29] uh setup here with how they managed to [131:32] get the site on one side of the border [131:34] and the power plant on another. Just a [131:36] really interesting um example here. So I [131:39] already said that twice. Let's now move [131:40] on to the let's move on to the next [131:42] slide. And so South Haven, right, which [131:45] is the which is the plant that they they [131:47] built in the Tennessee Mississippi area [131:50] has also planned um compute and battery [131:52] charging along with their local [131:54] generation. So if you look at these um [131:56] July 2025 permits, they have the 41 [131:59] turbines which we already talked about [132:01] that get natural gas from some um tap [132:04] which is somewhere in the in the region. [132:07] And then they also charge battery packs [132:10] in order to provide a sort of um buffer [132:15] whenever there is a lack or surplus of [132:18] energy to the data center. And this is [132:20] because they were not connected to the [132:21] grid at the time. They may or may not be [132:23] connected to the grid now. So you'd have [132:24] to search it up. But u in any case, when [132:26] you're on this behind the meter setup, [132:28] it is very important and essential to [132:31] have battery packs to serve as a sort of [132:33] buffer instead of having the grid which [132:36] usually serves that role. Another thing [132:38] that is really interesting and [132:40] underrated is how XAI overcame supply [132:42] chain constraints in the in the in the [132:44] clauses build out. So what do I mean? As [132:48] we mentioned in the primer or the [132:50] electricity section, I forget exactly [132:51] which one. Usually you will produce [132:55] energy at a gas plant or whatever plant [132:57] and then step up to high voltage do your [133:00] high voltage AC transmission and then [133:02] step down first to medium voltage and [133:04] then to the local low voltage supply at [133:06] the campus to distribute the energy. It [133:10] turned out that there was a shortage of [133:14] high voltage AC transmission equipment [133:17] and switch gear which XAI needed in [133:20] order to build Colossus. And this is [133:22] still the case today. You have some [133:23] insane like four-year leadouts for some [133:25] of these transformer equipments uh that [133:27] are being used for transforming and [133:29] distributing electricity. So what they [133:31] did instead is quite uh it's quite [133:34] impressive and creative. The XAI and [133:36] SpaceX engineers just said, "Okay, we're [133:40] going to skip high voltage and instead [133:42] we are going to only use medium voltage [133:45] for the transmission between the gas [133:47] plant and the local low voltage supply [133:49] to the campus." And this is from a [133:52] verified semi-analysis account uh uh [133:54] from last month in August 2026 of the [133:57] the way that they overcame this supply [133:59] chain obstacle. And what this means is [134:02] that they accepted a lack of efficiency [134:06] for a faster deployment time. And I [134:09] believe they actually got this medium [134:10] voltage feeder and switch gear. So a [134:12] feeder just to be clear is essentially [134:14] the infrastructure the conductors that [134:17] feed electricity from the source to the [134:19] destination. They they procured these [134:21] from China and u yeah just like really [134:24] went above and beyond. This is what Elon [134:25] Musk is known for saying okay the laws [134:28] of physics are such and therefore if it [134:32] does not defy the laws of physics then [134:34] we will find a way and that's what they [134:35] did but they did accept the trade-off [134:37] which is to say the efficiency of [134:40] transmission would definitely have [134:42] dropped and if we just talk about an [134:44] example right [134:46] let's assume that instead of [134:48] transmitting energy at 161 kilovolts [134:51] which would be considered high voltage [134:55] they instead had to do it at 345 sorry [134:58] 334.5 kts and if we assume that the [135:03] plant or actually we don't need to [135:04] assume like these are real numbers um [135:06] the the plant colossus has a power [135:09] requirement of 200 megawatt so that [135:11] means that they need to transfer 200 [135:13] megawatts from the plant to campus well [135:15] this means that the campus AC route [135:17] trades fewer transformer stages for more [135:20] current so if we assume that we have a [135:22] power factor of one which means that we [135:24] don't need to supply more power than is [135:26] necessary throughout the lines to reach [135:28] the destination. Then we need 3,347 [135:34] amps per line as opposed to 717 amps per [135:38] line to supply the same amount of power. [135:40] So this is 4.67 times the current at the [135:43] lower voltage. But more importantly, [135:46] it's the current squared to get the [135:49] increase in resistive heat loss. So this [135:52] would be somewhere around 20 times more [135:56] loss to heat. And this is of course [135:58] assuming that you use the same amount of [135:59] lines. Like one thing you could do is [136:01] just use like four four times the amount [136:04] of lines of conductors. And then in that [136:07] case you'll just have more metal to pass [136:09] the the the current through. So there [136:11] will be less resistive heat loss in that [136:13] regard. But in any case there is a [136:15] trade-off that must be made here. And [136:17] this this formula here is not that [136:19] important. Um if you guys remember P [136:22] equals ID is the formula for calculating [136:24] the current as a sorry for calculating [136:27] power as a function of current and [136:28] voltage but that's only for direct [136:29] current. If you're using alternating [136:31] current then you need to use this [136:32] formula which is a bit more [136:33] sophisticated and in particular uh this [136:36] is for three-phase AC. So we talked [136:38] about the different phases of AC based [136:41] on the phase of alternating current that [136:42] you're working with the the formula for [136:45] calculating the current given a specific [136:47] power will change. Okay. So, I think we [136:50] have a good idea of like you're trading [136:52] more current and more heat loss for [136:54] faster lead times and actually getting [136:56] the compute online faster. And it turns [136:59] out that that doesn't really matter in [137:00] terms of SpaceX making a a boatload of [137:03] money. So, let's just talk about their [137:05] disclosed compute fees because they show [137:07] that the stakes of delivering capacity [137:09] sooner were very profitable. So, we [137:12] actually know the exact amount of money [137:14] that SpaceX is making from their current [137:16] deals with Anthropic and Google. Uh so [137:18] you might have heard me names on X that [137:20] people are talking that people are [137:23] saying that SpaceX now that they've [137:24] merged with XAI is actually making more [137:26] money from their compute as opposed to [137:28] their space company. So it's more of [137:30] like a comput company that has like a [137:32] space side project. Um and this is [137:35] because they literally are generating [137:37] $1.25 billion per month from Anthropic. [137:41] And this is across Colossus one and [137:43] Colossus 2. These are full fees after [137:45] their May to June ramp up. So they I [137:47] think started with a specific compute [137:49] budget and then Anthropic even raised it [137:52] to a higher margin since they are [137:54] desperate for compute. And then for [137:57] Google they're currently making $920 [137:59] million per month off of them. And these [138:01] are from um I guess this this is at [138:03] least until October 2026 that they've [138:05] committed to this compute capacity. And [138:08] what we really need to know here um is [138:10] that first of all the the the contracts [138:13] are quite interesting. So both side can [138:15] can quit and say in the next 90 days [138:17] after that period of time I'm no longer [138:19] going to follow this contract but up to [138:21] now both sides have committed to the [138:23] contract. And these numbers are like [138:26] real numbers because in the SpaceX IPO [138:29] filings they literally reported these [138:31] numbers because they needed to report [138:32] their financials since they became [138:34] public. And if we sum these up we get [138:37] $2.17 billion per month. And according [138:41] to semi analysis which uh this is like [138:44] very reliable source at least on this [138:47] statement um not for all statements but [138:49] anyways they did the calculation and it [138:51] actually making this much money per [138:53] month pays back the capex for the entire [138:56] data center in less than a year. So by [139:00] essentially going through these hula [139:01] hoops and getting this inefficient [139:04] medium voltage transmission, they were [139:06] able to put this compute online faster. [139:08] And right when there was the compute [139:09] crunch, they could sell compute to [139:11] Anthropic and sell compute to Google at [139:13] a massive premium, which allowed them to [139:17] uh profit and pay back their capex much [139:20] faster than um than they normally would [139:23] if they had to wait to get the high [139:24] voltage AC installed. And we can [139:28] actually calculate this much more uh [139:30] precisely. So one month of earlier [139:33] service can outweigh a year of extra [139:34] extra fuel. Let's just put the numbers [139:36] here. Let's say that um you want 100 [139:38] megawws delivered 24/7 with 20 megawatt [139:42] $20 per megawatt hour of fuel. And [139:44] you're considering simple cycle versus [139:47] combined cycle where simple cycle costs [139:50] much more or rather costs $14.6 6 [139:53] million more per year in fuel because of [139:55] its efficiency that is less. But here's [139:59] the thing, you can literally open one [140:01] month earlier and use simple cycle [140:04] instead of uh combined cycle as long as [140:08] the earlier service contribution. So as [140:10] to say like fees less all other surface [140:13] costs from going online one month [140:15] earlier is greater than $14.6 million [140:18] plus the faster build costs. So, in the [140:22] case of these contracts which are in the [140:23] billions of dollars, if you're only [140:25] talking about like a $14.6 million [140:27] difference in like fuel consumption, and [140:30] then also don't forget to add the fact [140:31] that your electricity transmission is [140:34] less efficient, you're still not going [140:35] to be reaching a billion dollars, which [140:37] is just proving the point that you can [140:40] make a lot of money by being inefficient [140:42] in data centers right now. So here we're [140:45] going to do a quick interlude to talk [140:47] about what a place in a interconnect Q [140:50] actually buys. So I've mentioned this [140:52] already perhaps in the course, but the [140:54] interconnect Q is essentially the line [140:56] you need to wait in in order to get [140:58] access to utility power. And the two [141:01] systems I'm going to talk about are [141:04] URKT, the Energy Reliability Council of [141:08] Texas, and PJM, which is the largest [141:13] regional transmission organization in [141:15] the US and I think in North America. PGM [141:18] just stands for Pennsylvania, Jersey, [141:20] Maryland, if I'm not mistaken, since [141:21] those are three of the states which [141:23] originally constituted the interconnect, [141:26] but now it's actually 13 US states plus [141:28] Washington DC, whereas Urkott, as the [141:32] name suggests, is Texas primarily, but [141:34] also bleeds into a few other state [141:36] borders if I'm not mistaken. So, these [141:38] are the two main energy utility [141:41] consortiums that we want to be [141:43] interested in when we're talking about [141:44] the US. mainly because in Texas you just [141:46] have a lot of oil and data centers being [141:48] premium and then PGM it's just the [141:50] biggest one overall. So first of all [141:53] buying GPUs does not get them powered. I [141:54] think we can both agree on this. If you [141:56] have a bunch of racks that are not [141:58] plugged in then you're not going to get [142:00] anything done. In fact in a podcast last [142:03] year Satya Nandella is quoted as saying [142:05] that he had a bunch of chips sitting in [142:07] inventory that cannot be plugged in. So [142:10] we need to realize that power is [142:12] definitely a bottleneck in many [142:13] circumstances and therefore you need to [142:16] plan for power to get online when the [142:18] chips arrive. Let's look specifically at [142:22] a sec at Uricott's [142:25] data center que. So so if we look at the [142:27] queue, it's split up into three pieces, [142:29] but overall there are still 474.7 gawatt [142:33] that have no approved study. And what [142:36] this graph is actually is the large load [142:39] requests since 2022 by status as of June [142:45] 2026. Okay. And overall we can see first [142:49] of all that 5.2 g 5.9 gawatt are [142:52] actually operating and this is the very [142:54] small black margin here. But worth [142:56] noting here is that it's actually the [142:58] peak that they added. So since 2022 [143:01] there have been 5. 5.9 GW of data [143:04] center. or not rather data center but [143:06] just large load requests put online and [143:10] a large load request means typically 75 [143:12] megawatts or more of capacity. So of all [143:16] of the 474.7 [143:18] 5.9 have been online but this is only [143:20] counting the peak output. It's not [143:21] actually the average operating energy [143:25] draw. And then there's 284.3 GW which [143:29] have no study submitted to or [143:31] essentially how it works is that a study [143:33] needs to be submitted on behalf of the [143:35] utility in order to determine like [143:37] whether or not this demand is legitimate [143:40] and what new infrastructure needs to be [143:42] built to serve it. Okay, so that's the [143:44] that's the yellow. Then the blue is [143:46] there are 135.5 gawatt with the study [143:49] steady under review. And then five 55 [143:53] gawatt which is the green is the amount [143:55] that have a steady approved or further. [143:57] So you can see it's like almost a power [143:58] law where the the smaller and smaller [144:01] pieces is actually the further in the [144:03] process you are. And most people are [144:05] just in this gray zone or rather yellow [144:09] zone where there's no study submitted to [144:11] OT yet. And why is that the case? Let's [144:14] explain it with this simple example. So [144:17] let's assume that you want to build a 1 [144:19] gawatt data center and you have three [144:22] sites that you're considering for your [144:26] campus. Well, it turns out that you need [144:29] to have a separate study for each site [144:31] because of course if you want to build a [144:33] data center on one side of Texas versus [144:34] the other, that will determine to a [144:37] great extent whether you need more or [144:40] less infrastructure built to support [144:41] that gigawatt. And so what they do is [144:44] you submit these three requests for the [144:45] site since as we'll mention the cost to [144:49] join the queue is not immense and then [144:51] once that's done you can simply choose [144:54] your site and just forget about the [144:56] other two uh campus proposals just um [145:00] essentially remove those requests from [145:02] the queue. So the problem here though is [145:04] that the utility companies are not [145:07] perhaps aware of what's actually going [145:09] on with these requests. A new Texas law, [145:11] if I'm not mistaken, now makes it such [145:13] that they need to be transparent about [145:15] any redundant requests they have such [145:17] that the entire system knows like, okay, [145:20] this company is actually only going to [145:22] build 1 gawatt even though they have 3 [145:23] gawatt of requests. So, this is [145:25] something that is happening as we speak [145:27] in terms of a regulation change, but up [145:30] to now it was definitely a big problem. [145:32] And let's look at this semi analysis [145:34] graph to understand this better. This is [145:35] also from June of this year where they [145:37] say tracked data center projects back [145:40] only 13% of Urkott's 357 gawatt data [145:44] center cute. And so here what they're [145:46] saying is that there are 45.4 GW of real [145:50] compute. So this is essentially projects [145:52] that semi analysis can track because [145:54] there's a physical site. There's [145:55] actually boots on the ground. Whereas [145:58] the other 311 gawatt is just phantom [146:00] data centers. This green part here is uh [146:03] less interesting. It's crypto H2 I guess [146:05] hydrogen I'm not sure what H2 actually [146:06] stands for and industrial. So we can see [146:09] that we almost have a 1:10 ratio of real [146:13] to phantom uh data center demand and [146:16] this can be uh very concerning when you [146:18] think about people who are claiming this [146:21] is a indicator of a bubble situation. So [146:24] how can we resolve this? So there's some [146:26] new reforms that will be implemented [146:28] soon but not yet. So from October 8th, [146:30] 2026, these new laws will take form. And [146:34] what are we talking about? So a large [146:36] load request in UROT now has a security [146:38] for every megawatt. So before this came [146:40] in, there was no Urkotwide amount in [146:43] terms of putting its sort of deposit on [146:45] booking compute or uh booking energy and [146:48] each utility sets its own study charges [146:50] and security. But now there is going to [146:52] be these amounts that OT requires where [146:56] first of all there's a $100,000 deposit [146:58] for a study fee and then there's $50,000 [147:00] required for every single megawatt that [147:02] you request [147:04] if it's a 75 megawatt or more request. [147:07] So that would be a large load. And so if [147:09] you imagine a 1 gawatt request now [147:11] requires a $50 million security amount. [147:15] The one thing that is worth noting [147:16] though is that there is still relatively [147:20] lenient rules where as long as the [147:22] project isn't too far advanced. If you [147:25] pay this security deposit, you can still [147:28] get it refunded or at least get most of [147:29] it refunded. So don't think of this as [147:31] like a silver bullet that's going to [147:33] solve this phantom queue. It definitely [147:35] will help. I hope. But it's actually not [147:38] even implemented yet. So maybe what's [147:40] going to happen is that right before [147:42] October 8th, so right now it's September [147:45] and leading into October, you'll just [147:47] have a bunch of people making these [147:48] massive Q requests because they know [147:50] there's going to be no more uh because [147:54] they know that there's no deposit [147:55] required. And then after this date, [147:57] maybe the amount of requests in the [147:59] queue will massively shrink. I'm not [148:01] sure, but hopefully this can resolve [148:04] what we see in this chart. And let's [148:07] just shift our focus now quickly to PGM [148:09] interconnect. And the thing is since [148:11] it's such a wide zone, this is the zone [148:13] that is um encompassed by PGM. Let's [148:15] talk about Dominion, which is one of the [148:17] 21 utilities inside of PGM. So their [148:19] full name is Dominion Energy Virginia [148:21] and it's the main utility in this zone, [148:23] the purple one. And data centers signed [148:26] their utility energy procurement with [148:28] Dominion, not PJM. And since there's no [148:31] PGMwide data center aggregator, we'll [148:34] look specifically at Dominion's [148:36] contracts which far exceed what their [148:38] current data centers draw. So setting up [148:40] the stage right now at the 2025 peak, [148:42] there was 4 GW, sorry, drawn by [148:45] Dominion's data centers. And now looking [148:47] at the Q, first of all, if we just look [148:51] at July 2026, there are 32.4 GW of [148:55] engineering studies, which means that [148:57] it's paid for by the customer to [148:58] actually do the study. But we're talking [149:00] in the orders of like $100,000, right? [149:02] If you guys remember this slide, [149:04] probably it's in the range of uh six [149:06] figures, which for these hyperscalers is [149:08] not that much money. And then there's no [149:11] obligation to build as well. So you [149:13] don't actually have to build a data [149:14] center. You're just doing a study. Then [149:15] we have the construction authorized [149:17] where there's 9.4 GB of that gigawatt of [149:20] that. And in this case, the customer [149:22] repays spent costs if it leaves but [149:24] doesn't have to pay for the cost it [149:26] hasn't spent. And then there's 12 gawatt [149:28] of a service agreement. In this case, [149:30] it's a binding contract where the [149:32] customer pays whether or not it takes [149:33] the service. So we have these numbers [149:35] here compared to just 4 GW of data [149:38] center consumption. And you can see from [149:41] July 2023 over 3 years how this has [149:44] jumped from 16.5 gawatt to 53. So that's [149:48] a greater than 3x increase in this queue [149:52] of engineering studies, contract, [149:53] construction, authorization, and service [149:55] agreements. So the real thing that we're [149:58] really wondering here is how many of [150:00] these 53.8 GW will actually convert to [150:03] four because currently the ratio once [150:04] again is almost at a 1 to4 basis of [150:07] deployed compute versus cute compute. [150:10] And just to look at an OOTT forecast [150:13] what does this title say? It says the [150:14] 2025 OT adjusted load forecast breakdown [150:18] by type. And the only thing we're [150:20] actually interested in this in this [150:21] graph is the category which is gray the [150:24] data centers. And so here you can see [150:26] that after adjustments, data center load [150:28] remains the largest growth by type where [150:29] you can see gray becoming a much more [150:32] large piece of the pie. And here we're [150:35] going from 2025 to 2031. This is the OOT [150:38] forecast though and they're not really [150:39] AGI pill. So I would imagine that this [150:41] amount here is for 2031 pretty [150:45] conservative compared to what others [150:47] would be forecasting. So just keep that [150:49] in mind. Moving on. Now these [150:51] interconnect cues are of course very [150:54] long and because of that we have BTM [150:58] coming in instead to supply power behind [151:01] the meter and this means that a campus [151:04] can grow while its grid import stays [151:06] limited. So let's assume that you have a [151:10] pre-existing site that has 100 megawatts [151:12] of power that you can import and use [151:15] without any generators. This means that [151:17] you can only supply 100 megawatts worth [151:19] of compute. Well, instead of just going [151:23] out and trying to get more power from [151:25] the grid, which might be under extreme [151:28] load and these massive cues, you can [151:30] instead just bring in your own [151:32] generators. And let's let's say that you [151:34] found two generators each worth 500 [151:36] megawatt of power production. In this [151:39] case, you can still scale up to your [151:41] gigawatt scale and have actually [151:43] whatever your grid limit is as a sort of [151:45] backup. Um and this is what a lot of [151:47] hyperscalers are doing currently. In [151:49] fact, from 2028, semi analysis expects [151:52] most new data center capacity to need [151:55] its own power to be essentially powered [151:57] by behind the meter. So here you can see [152:01] blue is behind the meter, orange is [152:03] available grid capacity and green is new [152:06] grid supply. And so in this chart we [152:09] have the present state where most [152:11] everything is just available grid [152:13] capacity since that's the present status [152:15] quo. And just look at this grow [152:17] immensely according to their uh [152:19] projections from 14.8 in gigawatts. I [152:23] think here the title is mislabeled. It [152:26] should probably be gigawatt since even [152:28] here they mentioned that it will grow [152:30] beyond 50 gawatt by 2028. So we go from [152:33] 14.8 in 2027 to what is what they're [152:35] projecting to 65.7 which is a more than [152:39] quadrupling in the amount of behind the [152:41] meter infrastructure. And just once [152:43] again, if anybody has an investing [152:46] thesis regarding behind the meter and [152:48] its progression in the future, then you [152:51] can see that if you were to believe this [152:52] chart, there'll be a massive increase in [152:55] investment in behind the meter supply. [152:57] Okay, so that closes off our chapter. [153:00] Now, we're going to talk about carrying [153:02] the supply arrangement into the physical [153:04] site design. And um this will be [153:06] specifically focusing on like the [153:08] different phases of construction [153:11] connection routes and operating states. [153:13] All right, let's jump right into chapter [153:16] 5. At a high level, we're going to be [153:19] talking about how to turn a parcel of [153:20] land into a place you can build, [153:22] operate, and repair a data center. So [153:25] with the parcel, we're talking about the [153:27] land itself, connections, so fiber, [153:30] power, gas potentially. We already [153:32] talked about that a bit. uh the building [153:34] itself and then the equipment. So the [153:37] first concept that I want to introduce [153:39] here is green field versus brown field. [153:44] What am I talking about? If you haven't [153:45] heard this term before, it just means [153:47] green field being a development which is [153:50] on a green field. So like you're [153:51] building everything from scratch. And [153:53] then brown field means like it's some [153:55] existing infrastructure. So it could be [153:57] like the building is there or maybe like [153:59] the ground already has a foundation. [154:01] It's a pretty flexible term, but in the [154:05] in regards to this example here, so [154:07] you'll see if I move back up here, we [154:11] have Abalene, the Stargate campus in [154:13] Texas, which was a purpose-built layout [154:15] and was a green field ostensibly before [154:18] they started building. And then Colossus [154:20] 1, which was a former Electrolux [154:23] factory, and it already had an existing [154:25] shell as well as utility mains and some [154:29] power, only 8 megawatts, though. So they [154:31] had to build a lot of power. What is [154:33] Electrolux you might ask? It's actually [154:35] a it's actually a appliance manufacturer [154:37] from Sweden. So in this inside of this [154:40] data center they used to make like [154:42] washing machines which is kind of funny. [154:44] And um what the advantage here is that [154:47] Colossus 1, they already had the [154:48] envelope. They didn't need to build a [154:50] building and also the utilities were [154:52] there apart from needing to add more [154:53] electricity. So it just allows you to be [154:55] much faster but then also less flexible [154:58] because probably a washing machine [155:01] factory is not built tailor made for for [155:05] hosting H100s. But um there there's pros [155:07] and cons. So at a high level Abene is a [155:10] green field development and then [155:11] Colossus is a brown field and I I I [155:15] enumerated a few pros and cons but you [155:17] can also find many more. I encourage you [155:19] to comment any that are quite compelling [155:23] to you if you have any that you've came [155:24] up with. Okay. So when we jump our [155:28] attention right to Colossus one, it [155:30] right like I said had the utility mains [155:32] but needed more grid capacity. So they [155:33] needed 150 megawatts. So the substation [155:36] actually only could serve for 8 [155:38] megawatts initially. They had to add 842 [155:41] megaww. And what this meant is that um [155:44] whereas the existing water man and gas [155:46] man were kept, they had to add a new gas [155:49] tap was which was 8 in in diameter. I [155:52] suppose since the 16-in gas man was [155:54] insufficient for their electricity [155:57] generating needs and if you guys [155:58] remember they expanded their generation [156:02] to 41 turbines and 1.2 gawatt if I'm not [156:05] mistaken, something like that. Uh so [156:07] yeah, they needed to just bring in more [156:09] gas, but there was already some latent [156:11] infrastructure there. And an even more [156:13] crazy example here about actually a [156:15] green field development, a green field [156:17] development, which is just absolutely [156:20] mind-blowing is Meta who have this [156:23] Prometheus plant in New Albony, Ohio. [156:27] And a year ago, uh they published this [156:30] image where essentially they were [156:31] building a data center under a [156:32] waterproof tent. I'm not even joking. [156:34] It's like a lightweight frame and [156:36] fabric. So, the frame is aluminum and [156:39] then it's some puncturep proof fabric [156:41] that they're using. And it's worth [156:43] noting this is all temporary. So, this [156:45] is all being built while they have the [156:47] the main structure somewhere else. Uh [156:50] so, essentially what they would do is [156:51] have this as a temporary setup and then [156:53] whenever the the permanent structure is [156:56] built, probably just like destroy these [156:58] and then move the GPUs to the other one. [157:00] And like this is much more expensive [157:02] than just building it once and then [157:04] putting it in the original envelope, but [157:07] it's much faster to put up tents cuz [157:10] these tents can be put up on the order [157:12] of weeks or months if I'm not mistaken. [157:14] And if we remember, as soon as you can [157:16] put compute online, you're making [157:18] billions of dollars and this is probably [157:20] costing you millions of dollars to like [157:22] wreck these tents and stuff. So for [157:24] them, it makes sense uh because you get [157:26] the fast enclosure. Regarding the [157:28] Prometheus tents, these are publicly [157:30] stated as temporary. However, that may [157:33] just mean they'll be used temporarily, [157:35] but in the future when they make the new [157:37] data centers, they'll still keep those [157:39] tents. So, I may have misinterpreted the [157:42] meta news here. In any case, they [157:44] haven't reached the stage where they [157:45] would have to tear them down. And then [157:47] in terms of foundations, power and [157:48] cooling, it's probably much less [157:51] important and connecting and [157:53] commissioning it is much faster. So this [157:56] is an interesting and very like just out [157:59] of the box way of putting compute on as [158:01] fast as possible. And I'm emphasizing [158:03] this so many times, but it's really [158:06] important to realize like these [158:08] alternative methods are only occurring [158:11] and possible because of the premium cost [158:15] that occurs by adding incremental [158:17] compute to the supply since the demand [158:20] is so great. [158:21] Okay, let's just talk about utility [158:24] corridors quickly. So, [158:26] when you have a site that you're [158:28] evaluating for construction, you first [158:31] need to see, okay, is the building pad [158:32] like suitable for the building we want. [158:35] You don't want to build your data center [158:36] over a swamp in Louisiana. And that's [158:38] actually relevant because quite a few [158:40] data centers are being built in [158:41] Louisiana. So, you want to make sure [158:44] that's not the case. Then there's also [158:45] utility ement. So utility easement is [158:47] just like a part of private land which [158:51] utilities have some right to in order to [158:53] pass their utilities. So in the context [158:55] of a data center you would want a [158:56] utility easement nearby so that you can [158:59] procure power or water irrigation etc. [159:02] And then drainage. So why why am I [159:06] talking about drainage? is most of these [159:07] power plants are actually closed loop [159:09] liquid cooled which means that there's [159:11] no like net water that needs to be added [159:13] to the system once it's initially [159:16] filled. The reason for this is just like [159:18] if it rains a lot well you need to make [159:19] sure that the building envelope can be [159:21] secured that there's no like water [159:22] erosion or flooding occurring because as [159:25] we'll discuss when Hurricane Harvey [159:28] occurred in Houston there were quite a [159:30] few data centers which which had to deal [159:31] with a lot of water. So uh I'll save the [159:35] spoiler for a few slides. I won't spoil [159:36] it because we're going to be talking in [159:37] a few slides. But these are like at a [159:39] high level three concerns in terms of [159:42] the land parcel itself when a data [159:44] center is being built. And then when we [159:47] talk about Texas, which is where most [159:50] data centers in the US are being built, [159:52] there's an interesting law um and what [159:55] is it? So essentially buying Texas land [159:57] may leave mineral lights with another [159:59] owner. What I mean is that in Texas, [160:00] there's a split estate, which is to say [160:02] that there's a surface owner and there's [160:05] a mineral owner. And in fact, the [160:07] mineral owner retains rights to use the [160:09] surface. This is an insane property use [160:12] law that I had not known about until I [160:14] like researched it. But essentially, [160:16] given the background of Texas's [160:18] resourcerich history, they have [160:20] specifically established like leases to [160:22] land only for the mineral rights. And [160:25] this makes it such that you could very [160:27] well own the land on a property, but if [160:30] somebody else owns a mineral lease, then [160:32] they can just come to your land without [160:33] your permission at any time of the day, [160:35] any time of the year, and start drilling [160:37] for oil. Uh there are a few exceptions [160:39] here where like they can't like destroy [160:41] your building if it's already there. But [160:43] um it's still a very important [160:46] consideration to be made [160:48] and even more so because as I mentioned [160:52] most data centers in the US are being [160:54] built in Texas. So this great graph from [160:58] semi analysis says that Texas holds [161:00] about 17 gigabytes of booked on-site [161:02] generation. Also worth noting that [161:03] another 29 gigabytes of gaw of orders [161:06] have not been chosen yet in terms of [161:08] site. So there's like 29 gawatts that [161:11] are unattributed. It's probably the case [161:13] that a lot of these um sites will be [161:17] attributed to Texas, but who knows? And [161:20] then second place is Pennsylvania, as I [161:22] mentioned, because there's a lot of gas [161:23] there as well. But in any case, because [161:26] there's so many data centers being [161:27] attributed there, you need to make sure [161:29] that you have the mineral lease as well [161:31] in addition to the surface uh use rights [161:34] to the land that you're building your [161:35] data center on. And in fact, [161:38] um, there's this Odessa New Era data [161:41] center which is going to be on the order [161:42] of like a gigawatt that is on 493 acres [161:46] of land that's been secured, but as of [161:48] October 14, 2026 is still pending [161:51] because there's this waiver on the [161:53] mineral rights or some surface rights. [161:55] So this demonstrates that this is not [161:56] just like some trivial fringe law. Like [161:59] if you're building a data center center, [162:00] if you're building a data center in [162:02] Texas, you better be sure that you have [162:04] the mineral rights as well as the uh [162:05] surface rights or that the contract [162:08] specifies that there can be no gas or [162:12] oil excavation activities where the data [162:14] center is being built. In contrast, [162:17] there's also just parts of Texas where [162:18] there is no oil. So there's this main [162:20] like uh plane in Texas where most of the [162:23] natural resources are. So I would [162:26] imagine I haven't done research into [162:27] this. I would imagine that a lot of the [162:29] data centers in Texas are being built [162:31] outside of this oil region because it's [162:33] easier to secure the rights of the [162:35] mineral and property use. [162:38] Okay, let's talk about another [162:40] brownfield [162:41] uh development which is happening in [162:43] London in the UK as another case study. [162:46] So it's called the ADA or ADA is the [162:49] company that's building a new data [162:50] center in the Royal docks of East [162:51] London. And so this new campus has three [162:54] planned buildings and it's on old [162:56] industrial ground. So essentially what [162:58] they're doing is they need to support it [163:01] because this former industrial ground is [163:03] on soft river deposits and require up to [163:06] 6 m of like these fill piles to to [163:09] reinforce the foundation. So these deep [163:11] piles are cast in the ground to make [163:12] sure that the data center won't sink I [163:15] guess. And just with respect to the [163:17] external areas and utilities, there have [163:19] been around 6,000 ground improvement [163:21] columns installed. [163:23] And um it's just another example of [163:25] based on the land where you're building [163:27] the data center, there need to be case [163:29] specific accommodations made to ensure [163:32] the integrity of the site. And another [163:35] case where we had integrity in question [163:38] was the Houston case. So once again, [163:40] Hurricane Harvey was 2017 in particular [163:43] in August from the 25th to the 30th. Uh [163:46] there was massive flash flooding in [163:48] Houston and throughout the entire East [163:50] Coast, I believe. And what happened is [163:52] that Equinox, Equinex, which is a [163:55] massive data center provider, these [163:57] aren't particularly like GPU data [163:58] centers, just like normal data centers, [164:01] um they had their HO1 plant in Houston, [164:05] which stayed online despite actually the [164:08] flooding blocking access to the data [164:10] center itself. So what happened is that [164:11] the data center was staffed and operated [164:13] during the flood and there was no [164:15] interruptions even though the roads were [164:17] closed to the surrounding streets. And [164:19] how they did this is because these data [164:21] centers, funnily enough, are like made [164:24] or at least the ones with a sufficiently [164:26] high need to be always online are made [164:28] with like CS, so like dorms where you [164:31] can sleep with enough food and supplies [164:32] to last for some amount of time. And the [164:35] team literally stayed there for days, [164:37] like sleeping there, living there. And [164:39] according to the personal accounts, they [164:40] were literally pumping out water from [164:42] the data center to keep the the power [164:44] on. And I assume that they they used [164:47] on-site generation to power the data [164:49] center because most likely the power [164:51] infrastructure during hurricane Harvey [164:53] was also compromised, but I'm not [164:54] entirely sure. And then another [164:56] interesting example is QTS Swani, which [165:00] I believe is in like Georgia. It's [165:01] somewhere in the US. And what happened [165:04] here is that the campus plan here is on [165:07] the left. And they had to bring in the [165:11] fiber optic in order to like connect [165:13] their data center to the rest of the [165:16] internet. And what's done in this case [165:19] is that they had three uh interconnects [165:22] that or rather three fiber connections [165:24] to the data center. And just in order to [165:27] have a [165:30] design which is redundant, they made the [165:32] three entrances from approach from [165:34] different directions as opposed to [165:35] having a shared entry which could have [165:37] like one point of failure. And this is [165:39] just another example of ways that data [165:42] centers provide redundancy in the case [165:44] of like I don't know maybe somebody [165:46] drills a hole here and didn't realize [165:47] there's fiber. If that happens then [165:49] there's still two ways to communicate [165:51] with the rest of the internet. [165:54] Let's now talk about the cooling [165:56] considerations that you need when you're [165:58] talking about local climate and water [165:59] supply. So in Texas and in many places [166:03] in the US there are drought conditions. [166:05] So you cannot use what you call cooling [166:08] towers. So what are cooling towers? It's [166:11] a tower which essentially circulates [166:13] water around uh circularly in some [166:16] circumstances and essentially causes the [166:20] water some of the water to evaporate. [166:21] And what happens when water evaporates [166:23] is there's this phase change from liquid [166:25] to gas which [166:28] leads to a massive amount of energy [166:30] being released and essentially the the [166:32] remaining cooling the remaining water [166:34] will be cooled as opposed to the because [166:37] essentially you have energy exiting when [166:39] you're evaporating into uh liquid vapor. [166:42] And so this is a really efficient way to [166:44] cool water. But the problem is it's very [166:47] water intensive. It it leads to water [166:49] evaporation. So you need a constant [166:51] flow. You need a constant flow of water [166:53] in order to uh allow for this type of [166:55] cooling tower. So this is being done [166:58] with cloth one because I suppose in the [167:00] Tennessee Mississippi area there is a [167:03] relative abundance of water. But um you [167:05] should also realize that they also have [167:06] air cooled chillers. So what are air [167:08] cooled chillers? It's air cooled but [167:11] this doesn't mean that there's water [167:12] going to the air. It just means that you [167:14] have water going around passively uh and [167:16] essentially just convecting or [167:19] conducting. I forget which of the terms [167:21] is the correct one, but essentially it's [167:23] releasing heat into the atmosphere. Um, [167:24] but it's a closed loop. There's no [167:26] actually evaporation going on. So [167:28] whereas heat is just rejected to outdoor [167:30] air in Abalene, Texas where there [167:32] definitely is a shortage of water and [167:34] they cannot afford to use air cooled [167:35] chillers in um the Colossus one, you [167:38] need to have some sort of makeup water. [167:40] And that's why in the previous slide we [167:42] saw they had like a 16in water man. So [167:44] they can probably have access to quite a [167:46] lot of water in order to replenish their [167:48] supply. [167:49] And this is also why you hear a lot of [167:51] rhetoric of people saying, "Yeah, man. [167:53] The data centers, they're using all the [167:54] water. They're they're they're using all [167:56] the water." That's what you hear people [167:57] saying. And if you actually use air [168:00] cooled chillers, that's like patently [168:01] false. The only thing that is required [168:03] is to fill up the initial water supply, [168:06] which is like still a lot of water. [168:07] We're talking like I think millions of [168:09] gallons. But if you amortize if you [168:12] amortize that across the entire [168:13] lifespan, then it's like extremely [168:16] negligent for the data center's water [168:18] use. And another interesting fact is [168:20] that 80% of American water use is [168:23] agriculture. So the data centers are not [168:26] using all the water even though there [168:28] are other valid arguments against data [168:30] centers like water is like relatively [168:33] weak. The one caveat to that if I may [168:36] add is that power generation can use a [168:38] lot of water because if you're using [168:40] like a steam turbine you have it you're [168:41] essentially evaporating water into the [168:43] air. Um but so if you're using a lot of [168:45] like evaporative water generation then [168:47] you can have like a down or sorry an [168:50] upstream water use effect but still if [168:52] we compare it to agriculture use it's [168:54] very minimal. [168:56] Okay. Another interesting uh case study [168:59] here that I want to make is Rogers which [169:01] is a Canadian telecommunication company [169:04] which have a data center um I think it [169:07] it's at their offices and that it uses [169:10] these chillers to cool the the units and [169:13] these are actually opposite residencies. [169:14] So I think these are residency like [169:16] these are residential buildings and or [169:19] rather I I think like the residential [169:21] buildings are facing us cuz the walls [169:22] are here. So to to to put the short to [169:25] get to the point, they have 16 sections [169:27] which are essentially walls that um se [169:31] separate the chillers from the [169:33] residential area and this interrupts the [169:35] direct sound path while the open top [169:37] still allows air to flow upwards and [169:39] cool the cool the data center. So this [169:42] is another consideration like if you're [169:44] building a data center nearby [169:45] residential areas then the nimbies which [169:48] is like completely valid like you don't [169:49] want these massive fans like disturbing [169:51] you all the time they will tell you that [169:52] yeah you need to you need to block and [169:55] enclose your coolers and your chillers [169:57] so that um they don't always like [170:00] propagate this massive noise just like [170:01] how you have highways blocked off with [170:04] these sound insulation panels [170:07] right if we now go inside the data [170:09] center I also mentioned at the very into [170:11] the course that a GB300 NDL72 rack is on [170:14] the order of 1.5 tons, metric tons to be [170:18] more precise. And this is a Lenovo rack. [170:21] So, one thing about um these Blackwell [170:25] uh racks, these NVL72 racks, is that [170:26] there's multiple different OEMs or [170:29] rather manu manufacturers which make [170:32] them. So, Dell makes a lot of these [170:34] racks. Lenovo does too. I suppose Nvidia [170:36] has their own like proprietary version. [170:38] But um in any case, since these are [170:41] actually quite heavy, like moving the [170:42] rack and replacing a tray requires [170:44] separate handling plans and you need to [170:46] like plan for this. You can't just like [170:48] consider this an afterthought. And the [170:49] computer trays themselves even are 29 [170:51] kg. And what this means is that you need [170:54] to serve the service them with a [170:55] material lift. So if I just quickly [170:57] select quick quickly skip one side, [170:59] that's a teaser of what these look like. [171:01] Um but also just want to mention so [171:03] these racks are quite interesting. You [171:05] can see that on the outside you have the [171:07] water input and output and then um the [171:09] bus bar clip which is where the energy [171:11] comes through and then I suppose that [171:13] these four um these four pins here or [171:16] input clicks or whatever you want to [171:18] call them that's where you actually have [171:20] the data flowing in and up. Uh now if we [171:22] just clip skip ahead you need to reserve [171:24] space for the tray lift and the [171:26] technician. So let's say that one of the [171:29] GPUs in a rack goes offline. Well you [171:31] need to service it. you need to take out [171:32] the rack and put in a new one while you [171:34] fix or do an RMA. So, what is RMA? It [171:38] stands for return merchandise [171:40] authorization. This is just to say that [171:42] like Nvidia has these contracts with the [171:45] people who buy their GPUs where if a [171:48] specific quantity are like fail or [171:52] something happens to them like with [171:53] within some period of time, they will [171:55] replace them for you. So, if you're [171:56] doing an RMA on a GPU, you need to make [171:58] sure that you have the lift and the the [172:00] methods to uh to actually get it out [172:02] because you don't want to like hand [172:05] manually uh insert and and take out a 29 [172:09] kg rack, which probably costs like a lot [172:12] of money. Uh I I don't know what the the [172:15] rack cost is, but I know that the the [172:17] racks themselves are on the order of [172:18] hundreds of thousands of dollars. So, I [172:20] mean, I don't know what the tray cost [172:21] is, but the as I said, the rack cost is [172:23] in the hundreds of thousands. Okay. So [172:25] that was talking about the the trays [172:28] with the compute but there's also the [172:30] power trays. So as I talked about [172:32] previously uh we have these PSUs. So in [172:36] one tray you have six 5.5 kW hot swap [172:40] PSU models. So let's just uh decompose [172:43] this. So 5.5 kW is the power rating of [172:46] each power supply unit which converts AC [172:48] to to DC. And then hot swap just means [172:50] that you can take one out and put one [172:54] back in without powering down or [172:56] deenergizing or isolating that specific [173:00] uh that specific rack. And this is [173:02] really useful because time is money and [173:05] tokens per watt is how you make money. [173:07] So if you have to turn off the freaking [173:09] rack every time you want to replace a [173:11] PSU, which probably go down every now [173:15] and then, then uh that will be costly. [173:17] But that's not the case. So essentially [173:19] what's done is you replace one while the [173:21] remaining qualified supplies carry the [173:22] load. But then if you're replacing a [173:24] compute tray the if you're and you're [173:27] dealing with the Lenovo rack then Lenovo [173:29] requires the tray to be powered off [173:30] before removal which makes quite some [173:32] sense right if you're lying taking off [173:34] the rack physically then you probably [173:35] want to power it off or else uh bad [173:38] things could probably happen. Another [173:40] like kind of trivial consideration but [173:42] so important is like you need to make [173:44] sure the replacement route can fit the [173:45] equipment through every turn. like some [173:47] of this equipment is heavy and can be [173:50] wide. So when you're building a data [173:52] center, just make sure that it can fit. [173:54] And then also if you're dealing with 1.5 [173:57] ton data centers or rather racks, sorry, [173:59] you need to make sure that the floor [174:02] supports the load wherever it travels. [174:05] So have supports where it's actually [174:07] installed. And then the moving [174:08] equipment, make sure that it uh doesn't [174:10] cause cracks in the ground when it's [174:11] moving. [174:12] Right now, let's do like one final [174:16] knowledge check or just like make sure [174:18] that we're on the same page before we [174:19] end this chapter. Let's say that you [174:22] have one data hall A which is live and [174:25] hall B is being constructed but uh the [174:29] engineers didn't really think think it [174:30] through and the fiber actually goes [174:33] through both the road and sorry the [174:37] fiber goes through where they want to [174:38] build the hall be and the road does too. [174:41] So this is kind of a simple question, [174:44] but what must move before digging [174:45] starts? And I think you know the answer. [174:48] So the road needs to move and so does [174:50] the fiber. So what you would do in this [174:52] case is build the new road and the [174:54] fiber, test and switch them and then [174:56] start the excavation as opposed to [174:58] excavating and then realizing, oh crap, [175:00] we just cut off the fiber from hall A [175:02] and now we're losing like a million [175:03] dollars a day from uh opportunity cost [175:06] of loss revenue. So um just once again [175:08] another consideration to be made. This [175:10] may seem trivial, but like if you [175:12] consider the previous example of the [175:14] meta Prometheus tents that are being [175:16] built, you have a lot of like build [175:18] something temporarily and then destroy [175:20] it and then put it in the permanent one [175:22] and a lot of like this reconfiguration. [175:24] In fact, another crazy example is Meta [175:27] was building these data centers [175:29] somewhere in the US with a specific [175:32] design and then during that design [175:35] process, [175:36] they had switched to like a more [175:38] advanced design and once again I forget [175:40] the details of this but the short of the [175:42] short of the story is that while the [175:44] data center was being built, they [175:45] switched their main design and because [175:47] of that they demolished like a [175:48] half-built data center and started it [175:50] again from scratch to uh to build it [175:53] with like the new spec. effects because [175:54] for them it was worth it. The article on [175:56] this is on semi- analysis. So if you [175:58] just ask CHBT roughly what I said, [176:00] you'll definitely find more information [176:01] if you're interested. But once again, [176:04] like these considerations, even though [176:05] they sound trivial, they definitely come [176:08] up in uh in real in real life. All [176:10] right, guys. So in editing, I found the [176:12] article and here's what it says. For [176:14] example, Meta demolished an entire [176:17] building under construction because it [176:19] was their old data center design with [176:21] low power density which they have used [176:23] for many years. Instead, they replace it [176:25] with their brand new AI ready design. [176:28] Most data center designs are not ready [176:30] for GB200 and Meta in particular has by [176:33] far the lowest power density in their [176:34] data centers compared to the other [176:36] hyperscalers. And here are the [176:39] images which are the evidence [176:40] themselves. So in November they were [176:42] building it into January, demolished it [176:44] in May and said, "Yeah, we're [176:47] restarting." And that's the new data [176:48] center. So really incredible the the [176:51] econom uh the economies of scale here [176:52] that we're talking about, right? So this [176:56] is one of my favorite chapters of the [176:57] course where we're going to test we're [176:59] we're going to discuss power and [177:00] specifically follow power from the [177:02] campus connection to a working rack. Uh [177:05] so essentially you have the high voltage [177:07] DC that comes in the substation goes to [177:09] the transformer goes to these busways [177:11] which is once again like electron [177:13] highway and they're just called the row [177:15] bus way cuz it's in rows and then uh you [177:17] have the rack here which actually [177:18] accepts the energy in order to power the [177:21] GPUs. So that's a very uh primitive and [177:25] like crude explanation. So we'll go into [177:28] much more detail throughout the course [177:29] or throughout this chapter rather. But [177:32] let's still like look at a high level [177:34] image just a bit more with a bit more [177:35] granularity here. So let's trace a load [177:38] from high voltage AC to the actual rack. [177:43] So in Abalene at the Stargate campus [177:46] their second phase of expansion is [177:49] actually importing high voltage AC at [177:52] 345 kovts or 340,000 volts in [177:55] alternating current. And then um first [177:59] of all at the substation you're going to [178:01] have a step down transformer which goes [178:04] from 345 kV to 34.5 KV. So a 10:1 [178:08] voltage step down. And then you'll have [178:11] medium voltage switch gear which does [178:13] essentially switching. What does [178:14] switching mean? First of all, you're [178:15] going to protect the electricity. So [178:17] you'll have like a bunch of circuits and [178:18] breakers to handle like any surge events [178:21] or any current/voltage irregularities. [178:24] And then also you're going to divert the [178:26] current to multiple feeders such that [178:29] each hall gets its own independent [178:31] electricity feeder. So once you've [178:34] reached the data hall, you'll transform [178:36] once again from medium voltage to low [178:38] voltage. In this example, we're going [178:40] from 345 kovts to 480 volt AC. And then [178:44] once it reaches the building bus, we [178:47] will feed each of these uh 480 volt AC [178:52] lines to the rows which will then go to [178:54] the racks or the trays. And this will in [178:57] turn make the GPUs go burr. Uh so there [179:00] you go. And then don't forget the [179:02] cooling pumps. The cooling pumps also [179:03] run on the same 400 volt AC uh network. [179:06] So those also need to be powered in [179:08] order to make sure that the GPUs don't [179:11] go burr and burn. Uh and then also if [179:14] you're having an expansion which happens [179:16] right because as we mentioned a lot of [179:18] these data centers are being built in [179:19] phase. So you'll have the first phase 50 [179:21] megawatts go online and then you have [179:22] the second phase to get 100 megawatts in [179:24] total. If that's the case then you'll [179:27] have like these future feeders which [179:29] just remain open which is to say the [179:31] current does not flow. Um but it will [179:33] eventually be connected once the future [179:36] hall is complete. Right? So now let's [179:41] zoom into one specific part. Let's zoom [179:42] into the medium voltage switch gear. [179:45] Let's do it. So, let's look at a [179:48] Seaman's NX Airs switch gear, which is a [179:52] medium voltage switch gear. In fact, if [179:54] you look here, it's up to 12 kilovolts. [179:57] And if you guys have seen a pattern [179:59] here, Semens actually builds a lot of [180:00] the infra for these data centers, which [180:02] is interesting to note. And just at a [180:05] high level, we have a low voltage [180:07] control compartment, which is just for [180:08] like controlling the switch gear. This [180:10] is what like the electricians and the [180:11] technicians will use. Then we have the [180:13] bus bars which is for the medium voltage [180:16] AC to actually pass electricity. [180:18] And then we have a vacuum circuit [180:20] breaker. So this is for breaking the [180:22] current if there is some current event [180:25] which some surge event which requires [180:27] the current to stop. Then you also have [180:28] a connection to earth. And then you have [180:30] the feeder connections to actually split [180:32] off to the different streams. So this is [180:34] what it looks like instead of the [180:35] cross-section of one type of switch [180:37] gear. And this is for example the type [180:39] of stuff which is really in high demand [180:42] and in scarcity because you need this in [180:44] order to build the data centers. For [180:46] example, Cruso they're actually building [180:49] a factory or this factory in Tesla Tulsa [180:53] Oklahoma just went online this month [180:55] like a few days ago where they're [180:57] building switch gear themselves. And [180:59] this is probably because they can't find [181:01] it from other sources or rather the [181:03] other sources are having really large [181:07] lead times. So they're trying to build [181:08] it themselves in order to build data [181:10] centers faster because that's probably [181:12] one of the biggest bottlenecks for them. [181:14] Right now let's follow the three-phase [181:17] connection. So let's quickly zoom in on [181:19] that one. So just jumping back really [181:22] quickly. Right. In this engineering [181:24] diagram, you just have a single line [181:26] which kind of hides what's going on at a [181:28] at a low level. So the single line [181:30] diagram is demonstrating this [181:33] three-phase 480 volt AC current as just [181:37] one circuit and it's sending it to the [181:39] power power distribution unit which is [181:42] one of the ways that we split [181:44] electricity into and send it to the [181:46] different racks. Whereas like if we look [181:48] at this on the physical conductor level, [181:51] sorry, you have three lines which have [181:54] the three different phases of AC which [181:56] if we recall from the primer are each [181:58] 120° [182:00] uh apart. And actually if you forget the [182:03] primer, don't worry, we're going to talk [182:04] about that once again in the next slide. [182:06] And then you have neutral which is also [182:08] used to complete the circuit. And then [182:10] you have protective earth which is just [182:12] used to ground. And just as a reminder, [182:15] right, when you have a when you have a [182:16] circuit, you need to pass it through two [182:18] conductors to have the electricity flow. [182:20] So if you pass it through two of the [182:24] energized phases, you'll get a 480 volt [182:27] potential difference across those two [182:28] phases. So between any of these three [182:30] lines and then if you have the current [182:33] connected between any of these energized [182:36] AC lines and the neutral, then the phase [182:38] to neutral voltage is 277 volts. And so [182:41] this is just one of the properties of [182:42] three-phase AC. [182:45] Then let's talk about once again the [182:49] fact that three balanced phases fill the [182:51] power dips of a single phase. So let's [182:54] say that we have a 100 kilowatt average [182:56] at the load [182:58] with three-phase AC we can serve 100 [183:02] kilowatt total 100 kilowatts total at [183:04] every instant. And let's like let's go [183:06] through this and see what happens. So [183:07] first of all, let's assume that we just [183:09] have 60 Hz AC and that um we're feeding [183:14] this power over time where on this plot [183:17] you have current in each phase. And then [183:19] on this plot you have the three phases [183:22] power. So this would be the P uh equals [183:24] IV. But I just realized it's not [183:27] technically P equals IV because that [183:28] would be for direct direct current. you [183:29] see it's using the more sophisticated [183:31] formula which um includes this the the [183:34] square root the square not the yeah the [183:36] square root three term as well as the [183:38] RMS uh which stands for the root mean [183:40] squared which is used in order to [183:41] calculate the power for AC once again we [183:45] don't really need to understand the [183:47] fundamental equation here we just need [183:49] to know that with 350C you have this [183:51] constant power band um which is [183:53] supplying varying levels of current per [183:56] conductor [183:57] um but altogether is supplying at any [184:01] given point 100 kW. Interestingly enough [184:04] though, if you add the currents because [184:06] um they're alternating like the [184:09] instantaneous current across all three [184:10] phases is is zero. But because of the [184:14] fact that we have these three phases [184:15] that are shifted by 120°, we're able to [184:19] get this 100 kowatt um constant power [184:22] band. And this is assuming the power [184:25] factor is equal to one just once again [184:27] and that each load is is essentially [184:29] providing the same amount of energy. So [184:31] we have a balanced load here on 480 volt [184:33] 3phase and this allows us to have a good [184:37] amount of energy output in particular [184:41] 100 kow total at every instant from the [184:43] AC source to the load. Okay. And here [184:45] let's quickly just discuss a [184:47] misconception that some may have [184:49] including me. So that's why I included [184:50] it that's why I include this slide. So [184:52] 480 volts is the voltage between two [184:54] live phase conductors. Um one thing that [184:56] I was thinking is like okay you have 480 [184:58] volts here and 480 volts here. So when [184:59] you connect them shouldn't you get 960 [185:01] volts? Well first of all that's kind of [185:03] mal formulated because voltage at a [185:05] single point is not even defined. [185:07] Voltage is essentially this potential [185:08] difference between two different [185:10] conductors. And [185:13] let's just go into this section where I [185:15] say why not 960 volts? Well 48 volt is [185:19] an RMS magnitude. It's not actually a [185:20] fixed 400 vol 48 480 volt um amount. And [185:25] this is because once again, if you like [185:27] actually sum up the voltages of [185:29] three-phase AC at any given time, you [185:31] get zero volts because they're purposely [185:35] like shifted by 120 120° each to have [185:40] like this net 0 volt uh average. But [185:44] then if you take the RMS magnitude [185:47] then you will get a positive uh average [185:49] voltage and this should give you 480. [185:52] And just once again if you're interested [185:54] in the specifics of this then you should [185:56] probably do a course on electrical [185:58] engineering because that's not my field [186:00] of expertise and also there's just a lot [186:02] of depth and richness into how AC [186:05] current works in the different phases. [186:06] But we just need to know that off if [186:09] you're on a lineto- line uh voltage [186:11] reading on 480 volt AC which is [186:15] three-phase then you'll get a reading of [186:17] 480 volts but if you're on the phase to [186:20] neutral you get 277 volts. [186:23] Okay. Now let's talk a bit about like [186:26] how we manage the electricity [186:27] infrastructure. So we have a protection [186:31] relay which detects any faults and then [186:34] tells the breaker to interrupt the [186:35] current if that happens. So this blue [186:38] line is our is our conductor our supply [186:41] which is giving electricity from the [186:43] supply to the load and we usually have a [186:46] current transformer in series on the [186:48] supply. What does it do? It scales the [186:51] current and then um essentially like [186:53] samples it. So you can think of it as [186:54] like a sample of the current reading to [186:56] see if there's ever a surge in current. [186:58] And then the protection relay if it sees [187:00] that will go ahead and trip the circuit [187:02] breaker. And then once the circuit [187:06] breaker receives the trip it will fault [187:08] that feeder and then open the circuit [187:11] such that uh there's no like [187:15] energy being fed to that downstream [187:17] appliance which would be outside of its [187:20] current uh input range. because you're [187:23] usually dealing with really expensive [187:24] equipment and you don't want to have [187:26] anything bad happening in terms of like [187:28] shorting or just like burning the [187:30] internals of your equipment due like a a [187:32] current surge. And then conversely for [187:35] voltage what you have is usually called [187:37] a current transformer or a capac sorry a [187:40] voltage transformer or a capacitor [187:42] voltage trans sorry a capacitor voltage [187:45] transformer. Both of these just sample [187:47] the voltage and scale it in order to get [187:49] a proper reading. And these are usually [187:52] in parallel or they're yeah they're [187:54] always in parallel the the voltage [187:56] transformers or the capacitor voltage [187:57] transformers which read once again the [187:59] voltage and then the protection relay [188:01] will detect if there is need to [188:05] to trip. For example, if a lightning uh [188:08] strike hits the the the current, the [188:10] conductor, sorry. That would be a case [188:12] where the voltage would spike and you [188:14] would perhaps want to open the circuit [188:16] in order to prevent any bad things from [188:19] happening with your downstream [188:20] equipment. And then we talked about the [188:23] circuit breaker which can open and close [188:25] automatically, but then there's also a [188:26] disconnect which literally causes an air [188:28] gap. So the difference between a circuit [188:30] circuit breaker and and a and a [188:32] disconnect is that a disconnect is [188:33] literally like usually like motor [188:35] powered and just like separates the [188:37] current with air so that there is like [188:40] no way for the current to pass unless [188:41] there's an arc but you usually control [188:43] for that and then circuit breaker is [188:45] usually some like m like some mechanism [188:47] where perhaps the I actually don't know [188:51] how most circuit breakers work but you [188:53] can just imagine as like a disconnect is [188:55] a more robust way to isolate a section [188:59] of the u the circuit from its supply. So [189:03] in this case we're isolating the [189:04] transformer from its supply as well as [189:07] the downstream data hall. And another [189:10] thing that's interesting is a surge [189:11] artor. So what is a serger artor? Well, [189:13] it protects transformer insulation from [189:15] over voltage. What does what does this [189:17] mean? It's literally a path to earth [189:20] which normally [189:23] conducts extremely little. So you can [189:25] think of it as in in normal operating [189:27] conditions this serger artor is [189:30] conducting on the order of like 100 [189:32] watts of electricity to the ground. And [189:34] in the grand scheme of things that's [189:36] like really nothing. And however if [189:40] there is a massive voltage increase on [189:42] the line all of a sudden this sergetor [189:44] becomes an amazing conductor. These are [189:46] usually like zinc oxide um zinc oxide [189:49] serger arresttors if I'm not mistaken. [189:51] and it will cause a massive current to [189:55] go down into the earth in order to [189:57] protect the transformer. But [189:59] importantly, it will not like send all [190:01] of the current down. It will only send [190:02] enough current down into the earth such [190:05] that the amount that's on the conductor [190:07] remains in standard uh operating [190:11] conditions. And this means that even if [190:13] a voltage h sorry a lightning bolt hits [190:15] the line, thanks to the serger [190:17] arresttor, you should still be able to [190:19] run the the the distribution system [190:23] without problems. And so once again, you [190:25] have this really strange step-wise [190:27] function where normally there's like no [190:29] conductance [190:31] um high resistance and then as soon as [190:33] voltage spikes a lot, the serger [190:35] arresttor will route current to the [190:38] ground. And the last thing that I want [190:40] to mention here is that usually put them [190:41] right before the transformer and not [190:43] like right before the supply or right [190:45] before the circuit breaker because you [190:46] want this to be as close to the [190:48] transformer as possible. Like let's say [190:49] that the serger artor was here before [190:51] the circuit breaker. Well, in the off [190:53] chance that like a lightning bolt strike [190:55] right here, then it would reach the [190:57] transformer before the serger artor and [190:59] could still like just ruin the [191:00] transformer due to the high voltage [191:03] spike. Right now let's jump jump jump to [191:07] a different topic here and talk about a [191:09] bit of code design. So we'll talk about [191:13] Compass which is a data center uh [191:15] manufacturer or like they procured and [191:18] construct data centers and Seammens [191:20] which we've already talked about who [191:21] designed the equipment and they made a [191:23] switch gear and transformer into one [191:25] package. So what am I talking about? On [191:27] one factory-built skid, they [191:29] incorporated medium voltage switch gear [191:31] which does switching and protection of [191:33] the conductor and the current with a [191:36] transformer which which steps down or up [191:39] the voltage. In this case, probably [191:41] steps down from medium voltage to low [191:43] voltage. And here's a picture of it. So, [191:46] Seammens built the switch gear and [191:48] transformer skids for compass data [191:49] centers. This type of code design was [191:51] used in order to modularize their [191:54] construction. And what you have at the [191:56] end of the day is a custom factory built [191:58] skid with MB switch gear plus [191:59] transformer and it was co-developed. So [192:01] there's an interesting case study of how [192:03] you can combine two parts into one in [192:05] order to modularize and make your [192:08] factories more uh modular essentially. [192:11] Sure. So [192:13] we've already discussed kind of the the [192:14] pros and cons of having high current in [192:17] terms of heat loss. And in general, the [192:21] lesson that we've take away from this is [192:23] that you want to put the high current [192:24] route close to the hall. So let's [192:26] consider these two options where you [192:28] step down at the campus entrance and you [192:31] go immediately from medium voltage to [192:35] low voltage and then you have to [192:37] transfer 2 megawatt at 2,46 amps, which [192:42] is insane for heat loss. That's why the [192:44] lines are blue. And instead if you step [192:46] down beside the hull then there's much [192:48] more there's much less sorry loss to [192:50] heat and much more efficiency. In this [192:52] case we're assuming 2 megawatts with a [192:54] power factor of one and threephase uh [192:57] three-phase AC. So just the high level [193:00] overview is like you don't want to have [193:03] too much transmission happening at low [193:05] voltage if you can help it. Of course [193:08] this bottom picture is kind of what [193:10] SpaceX did, right? They did quite a bit [193:12] of medium voltage transmission, but that [193:15] was worth it for them since they were [193:17] able to put their compute online much [193:18] sooner. First of all, I misspoke earlier [193:22] calling the top line blue instead of [193:24] red. And second of all, the SpaceX [193:27] example is somewhat contrived in this [193:30] sense because we're talking about [193:31] different distribution schemes. The [193:33] lesson here is that medium voltage [193:35] inside of the campus is better than [193:36] lower voltage. So high voltage is the [193:39] best for power efficiency and then once [193:42] you're in the the campus then you switch [193:44] down to medium voltage and you want to [193:46] keep the low voltage in order sorry [193:49] within the data hall as much as [193:51] possible. Right now let's just uh [193:54] quickly talk about something that is [193:56] important for transformers. So [193:57] transformers work essentially by having [194:00] these coils which tune the ratio of [194:05] how much you will step down or up in [194:07] voltage. So let's say that you have a [194:10] transformer with 80 turns of coil on the [194:12] input and 20 turns on the output. Then [194:14] you have this like 4:1 ratio such that [194:17] 480 volts coming in would would step [194:20] down to 120 volts. And then this means [194:22] that even if you supply 504 volts, it [194:26] will step still step down by a quarter. [194:28] It won't it won't go to 120 because the [194:30] transformer doesn't really care what the [194:32] input voltage is. As long as it's in the [194:34] range and uh that's it. It will just [194:36] divide by four or whatever the the [194:39] amount of taps the taps are. But what [194:41] you can do instead is you can modify [194:46] where on the coil you connect the condu [194:49] the conductor on a transformer. And this [194:51] means that you can just modify them such [194:54] that instead of having 80 turns, you [194:56] change the input to 84 turns and the [194:58] output to 20 turns on the on the AC [195:00] output. Keep that the same. And this [195:02] more primary turns restores the output [195:05] voltage to 120 volts. So this is an [195:07] interesting observation or reality of [195:10] how transformers can be tuned to their [195:14] specific output voltage. And if you have [195:16] a dry core transformer, the taps are [195:19] physical winding connections. So this is [195:21] a dry type transformer and you literally [195:24] change where these conductors step up [195:27] and down on the uh on the the course of [195:31] the transformer here. And there's [195:34] different labels here. Usually these are [195:36] two 2.5 to 5 point to 5% difference in [195:40] terms of the ratio of the the voltage [195:42] when you switch from one to the next. [195:44] And yeah, we'll leave this for the [195:46] electricians cuz I assume if you mess [195:48] this up then there are some pretty bad [195:49] consequences. But this is actually how [195:52] this diagram works if you at least have [195:54] a dry type transformer. Worth noting [195:56] though that a lot of transformers are [195:58] not of the dry type. Instead, they're [196:00] actually immersed in this mineral oil in [196:02] order to cool the the the windings. But [196:04] um this is a specific case where it's a [196:06] it's a dry type. Okay. And you already [196:10] have intuited this or understood this I [196:12] imagine by now. But the building bus [196:14] feeds much more than just it. So let's [196:16] assume that you have this transformer [196:18] that's stepping down to 480 volts of [196:20] alternating current. Um there's a [196:21] switchboard which takes the incoming [196:23] supply and then splits it onto these [196:25] feeders which um supply current to the [196:28] separate parts of the data center. And [196:32] for example, let's say you have the IT [196:34] feeder which just which just feeds [196:37] energy to the busway that is then routed [196:40] to the rack PSUs the power supply units [196:43] that will take this AC energy and [196:45] convert it to DC energy. Then you have [196:47] the cooling feeder and then the house [196:49] feeder and just term and this is just [196:50] doing like lights and controls and other [196:52] building services. So this is a general [196:54] example of what we're doing in terms of [196:56] switching and feeding to the different [196:59] uh end sources. And so like at at each [197:01] of these junctions, you'd probably have [197:02] a a PDU, which is the power distribution [197:05] unit. And we'll see an example of that [197:07] in the next slide. Another thing I want [197:09] to mention is that [197:11] data centers are actually transitioning [197:14] to having more direct current in the [197:18] data hall. And this is for a variety of [197:20] reasons which we'll discuss for [197:21] efficiency. But what you can already [197:23] guess is that if they're converting some [197:25] of the energy supply and distribution to [197:28] DC, I'll actually spoil it for you right [197:30] now. What's happening is that the rows [197:33] and rather the GPUs in a few years time [197:36] and already it's happening now. They're [197:37] developing the standards will be fed [197:39] with 800 volt DC current. Well, this [197:43] means that you still need to actually [197:45] have a source of 400vt AC because the [197:48] cooling and the regular uh equipment [197:50] within the data center still require [197:52] alternating current to be powered. So, [197:54] this is like one one thing worth [197:57] mentioning and considering with the [197:59] overall transition to direct current for [198:03] GPU power. Uh then we'll discuss this in [198:06] much more detail. So, I just wanted to [198:07] give a quick teaser and then I discussed [198:09] PDUs. So we have power distribution [198:11] units and there's two types. So we have [198:13] a floor PDU and a rack PDU. So the floor [198:16] the floor PDU distributes the feeders [198:18] and this is somewhere on the data hall. [198:20] And then the rack PDU is for the [198:23] specific racks of the GPUs. It [198:25] distributes AC to the outlets. So this [198:28] distributes AC to the to the different [198:30] PSUs. And then the server PSUs are the [198:33] individual um power supply units which [198:36] deliver AC to DC for the specific for [198:39] specific compute units since compute and [198:41] the motherboard they take current and [198:44] direct current. So what we can consider [198:46] is like PDU is in the AC uh AC regime or [198:49] domain and then the PSU is what converts [198:52] AC to DC finally. [198:55] All right. And then I've I've talked [198:56] about the busways where we have like [198:58] essentially copper conductors passing [199:00] electricity through the data hall and [199:04] these are usually done now like above [199:07] the rack and you have these tap offs [199:09] which then feed to the individual uh [199:12] compute units and that's where you'll [199:14] supply the PSUs with power. And just to [199:19] repeat myself here the tap off connects [199:21] one rack to the overhead busway as we [199:23] saw in this diagram. So you have the end [199:25] feed which powers the entire busway. The [199:27] tap off box which takes a protected [199:30] branch from the busway and then the [199:32] cable to the rack PDU which then [199:35] supplies power to the separate PSU. So I [199:37] said that we go from tap off directly to [199:40] the uh PSU in the previous slide. That's [199:42] not correct. So the tap off first [199:44] connects to the rack power distribution [199:47] unit. uh and then those power shelves [199:50] will then convert in the PSUs the [199:54] alternating current to the direct [199:55] current. [199:58] As you saw in the diagram, we're usually [200:00] using overhead busways. This wasn't [200:02] always the case. And in fact, what this [200:04] means is that the metering could be done [200:07] at each tap off. New branches could be [200:09] added as the rack loads changed over [200:12] time. And then also uh this freed up the [200:15] underfloor for the for cooling. Um so [200:18] assembly essentially simplify the design [200:20] here. And I think nowadays you'll see [200:22] the overhead bus way as the most common [200:24] way of supplementing your racks with [200:26] power. If you guys remember all the way [200:28] back to the TPU slide, we also saw that [200:31] the power was being supplied to the [200:33] racks from above. [200:35] Right? And when we talk about the tap [200:39] offs, we need to make sure that we are [200:41] staying within the voltage uh [200:43] requirements or respecting how much the [200:46] busway can support. So to to put it [200:49] concisely, a new top off still uses the [200:50] same upstream conductors. So let's [200:52] assume that this feed has a 250 amp [200:56] budget and we currently are using 56 [201:00] amps per rack in series. So that at this [201:04] point there's 111 amps remaining or [201:06] rather there's 111 amps at this point of [201:09] the conductor and then once it spits [201:10] down to rack 2 then we have 56 amps and [201:13] we arrive at this number because we're [201:16] using 415 volt lineto line. If you use [201:19] the ACP power uh power formula as a [201:23] function of current and voltage you'll [201:25] get these amp numbers. And then let's [201:27] just go to a fourth uh go to a fourth [201:29] rack. So since we have 56 amps per rack, [201:34] 56* 4 is still much below 250. But then [201:37] as soon as we graduate to a five rack [201:41] system on one uh busway, then you'll be [201:46] 28 amps over your usable budget and um [201:50] that's probably going to cause issues [201:52] down downstream with your electricity [201:54] equipment since you've essentially [201:55] exceeded the n the end feed a lotment. [201:59] So, we need to be conscious of the the [202:02] amperage and the other values on the [202:04] feed in order to ensure that we're not [202:07] exceeding them since we're still using [202:09] the same upstream conductors as as we [202:11] add racks onto the feed and the busway. [202:15] All right, so this is the first meme of [202:17] the uh of the course. So, let's see what [202:21] it says. How is apparent power like a [202:23] pint of beer? And then it shows this [202:27] this this funny this funny image which [202:29] says apparent power is equal to active [202:32] power as well as reactive power. It's in [202:35] fact the um the Pythag so it's [202:38] essentially the hypotenuse. So just to [202:41] to say it concisely you calculate [202:43] apparent power as active power squar [202:45] plus reactive power squar square root. [202:48] And what does this mean? Essentially, [202:50] active power is the power that actually [202:52] goes to your equipment. And then [202:53] reactive power is the power that's [202:57] stored and returned inside of your [202:59] alternating current. So essentially, [203:01] since you have like this alternating [203:03] current, which is constantly uh [203:06] energizing and deenergizing, there's [203:07] some energy that's just stored inside of [203:09] the conductor and is then subsequently [203:12] released, but is not actually used as [203:13] active power. And that's called reactive [203:16] power. If you add reactive power with [203:18] the the power that actually does go down [203:19] to the end load, you get apparent power. [203:22] So this is this is just a quick [203:24] introduction. Now let's actually talk [203:25] talk about this in in an example. So [203:30] let's assume that you have a transformer [203:35] with the real power in your system equal [203:38] to 900 kts and the apparent power s [203:42] equal to 900 kovt amps. So this is the [203:44] this is the unit of um apparent power [203:47] and the line current here is 1,083 amps. [203:51] You can calculate this once again and [203:53] this is only for AC using the same [203:55] formula that we saw in a previous [203:56] section. And what this means is that if [203:59] your transformer if your transformer can [204:02] carry 1,00 kovt amps then you've only [204:04] loaded at 90% since your power factor is [204:07] one. But in real life your power factor [204:09] won't be one. So if we assume that the [204:10] power factor is 0.8 eight. That means [204:13] that the apparent power or rather 80% of [204:16] the air apparent power is real power, [204:18] but then 20% is this foam at the top of [204:20] the beer that doesn't do anything called [204:21] reactive power. And this means that you [204:24] essentially need to make up for it by [204:26] supplying more current to get to this [204:28] same 900 kW reading. And therefore, you [204:33] would actually be surpassing the [204:36] transformer load by 12.5%. [204:39] So in summary we can say that real power [204:41] is consumed but reactive power [204:44] represents just energy stored and [204:45] returned inside the conductor. And then [204:47] apparent power measures the loading the [204:48] transformer must carry since it's the [204:52] the combination of this real plus [204:54] reactive power. And another like [204:58] paragraph here that really gives that [205:00] really drives the idea home is that in [205:02] some current in sorry in AC so once [205:04] again this is not DC this is AC some [205:06] current can flow to temporally store [205:07] energy in magnetic or electrical fields [205:09] then return it rather than delivering [205:11] that energy to the load. This is called [205:12] reactive power. The current still passes [205:14] through and heats the conductors. So the [205:16] transformer must handle more current [205:17] than the real power alone would suggest. [205:19] A power factor of 80% does not mean that [205:22] 20% of the energy is wasted. It simply [205:24] means that more current carrying [205:25] capacity is needed for the same real [205:27] power. That additional current just [205:29] incre in increases resistive losses. So [205:31] the in the additional current from this [205:33] reactive power increases reactive losses [205:36] but itself is just like once again [205:38] temporarily stored in magnetic or [205:40] electrical fields within the alternating [205:42] current. This is another reason why [205:44] direct current is advantageous. you [205:45] don't have this uh concept of power [205:48] factor because the polarity of direct [205:51] current does not change over time and [205:52] therefore the magnetic and electrical [205:54] fields also stay constant. [205:56] So um we're just talking about the [205:58] example previously if we consider [206:01] different levels of power factor for a [206:03] transformer with a 1,00 kilovolt amp [206:05] rating then we'll be fine with a power [206:08] factor of one. We'd actually still be [206:10] fine with a power factor of 0.9. We'd [206:12] actually be right at the limit. So I [206:14] don't know if that's fine or not. And [206:15] then we would have a transformer loading [206:17] of 1,125 [206:19] k amps if we only had an 80% power [206:22] factor. So we've been talking a lot [206:24] about three-phase alternating current in [206:26] this course. And so let's consider in [206:30] this slide a three-phase rack power [206:33] distribution unit that feeds singlephase [206:37] power supply units. So essentially this [206:40] power distribution unit takes in the [206:42] three-phase AC and then each power [206:45] supply unit takes in one [206:49] active phase. So let's say line one [206:54] connects to one active phase in group A. [206:56] So this is a PSU and then also it [206:58] connects to the common neutral in order [207:00] to get this 277 volt potential [207:02] difference. Um, and then this gives 20 [207:05] amps per power supply group. Meaning [207:08] that across A and B, we add up the [207:10] voltages. Sorry, not the voltages, but [207:12] rather the currents to get 40 amps. And [207:14] we do the same for every single group. [207:16] Uh, and then let's assume that each [207:18] busway has a 60 amp limit. Well, if we [207:21] split them equally across each line, [207:24] across each phase, then we have no [207:25] problems. But then let's for some reason [207:29] put four of the power supply units onto [207:33] the first line. What happens here is [207:36] that essentially we have now 80 amps on [207:39] the first phase of the three-phase AC [207:42] and only 20 amps on the other ones and [207:44] we've simply um over we've we've gone [207:48] over the the voltage sorry the amperage [207:50] that we can supply on a single phase and [207:53] this is why you want to balance the [207:54] loads across the phases in three-phase [207:56] AC. [207:59] Now if we talk about like breaking and [208:02] the downstream effects of something that [208:05] happens in terms of a a feeder failing [208:08] or some like fault happening on a feeder [208:11] an upstream backup opens when the feeder [208:14] breaks sorry when the feeder breaker [208:15] fails to open. So what I mean here is [208:16] that like all right let's say that [208:18] there's a fault in hall A and for some [208:20] reason hall the hall A feeder breaker [208:22] fails to open [208:24] even though the trip is issued the [208:26] breaker remained closed. Maybe it's like [208:27] some mechanical issue. This mean that [208:30] now the shared bus between hall A and [208:32] hall B is compromised and even though [208:35] hall B has no fault because it is [208:38] connected to whatever happened in hall [208:40] A, the upstream backup now gets tripped [208:43] and that and it opens. So essentially [208:45] you what I'm saying here is that you [208:46] have this cascading reaction where as [208:48] soon as there's a fault, it will [208:50] propagate to every single breaker until [208:52] one opens. And this is just a [208:55] illustration of the fact that whenever [208:58] there's a fault, the breaker that opens [209:00] will the first breaker that opens will [209:02] isolate it. But until that happens, you [209:04] will have to remove supply from every [209:06] single hall. So in most operating [209:08] conditions, this breaker should open and [209:10] then you'd only have to isolate hall A. [209:12] But assuming this one fails to open for [209:13] whatever reason, then you would also [209:15] have to break hall B and isolate that as [209:19] well, which isn't good for the bottom [209:20] line, of course. [209:22] Okay, so that's a close to this chapter. [209:26] Next, we're discussing how to keep racks [209:29] and cooling running through an [209:31] interruption such that if this happens, [209:33] we can still supply backup power. All [209:36] right. So, in order to talk about how we [209:39] can keep racks online in interruptions, [209:42] we need to discuss continuity, storage, [209:45] and protection. So, let's not waste any [209:47] more time. As we talked about [209:49] uninterruptible power supplies already, [209:51] let's let's look at a specific example [209:53] of a product here. So here you have a [209:55] Schneider Electric Easy UPS3 phase [209:58] modular. So these are the same unit. One [210:00] is just black and one is white. And [210:02] Schneider is another company which makes [210:05] a lot of different components and [210:08] electrical infrastructure for data [210:10] centers. Specifically, [210:12] these racks can support up to 50 or [210:15] rather up to 250 kW and as little as 50 [210:18] kW. And in terms of the dimensions, it's [210:21] uh almost 2 m tall by half a meter wide [210:25] and 1 0.85 m in depth. [210:29] So, what's inside the actual [210:31] uninterruptible power supply? You have [210:33] the rectifier and then the DC link which [210:35] connects the rectifier to the inverter [210:37] to the inverters. Sorry. Then you have a [210:39] battery interface. So not the actual [210:40] battery itself. The battery is somewhere [210:41] else. But the interface to connect to [210:43] the battery is in the UPS. You also have [210:45] a static bypass which is a conductor [210:50] that skips this entire rectifier DC link [210:53] inverter path. And we'll explain why [210:56] that's that's a good idea. Essentially, [210:59] it allows the UPS to work even if the [211:02] rectifier DC link or inverter are down. [211:05] And uh we'll show we'll show that in a [211:07] diagram so that it makes more sense. And [211:09] then you have the electrical controls [211:10] and cooling as well. So once again the [211:12] batteries the batteries are external [211:14] cabinets and connect to the battery [211:16] interface of the UPS. And these are you [211:19] usually rack battery backup units. So [211:22] BBUS that uh yeah bring direct current [211:26] back up at the rack. And this type of [211:28] facility uninterruptible power supply is [211:31] usually in an electrical support room. [211:32] So in the gray space not the white [211:34] space. And in particular, these specific [211:37] UPS's are made so that they can be put [211:40] into a configuration of four, which [211:43] gives you 1 megawatt. And then either [211:45] that can be run as like four, which [211:46] supplies 1 megawatt, or 750 kW across [211:51] the four with one level of redundancy. [211:53] So that' be a N plus1 system. And then [211:55] in addition to the batteries that are in [211:56] external cabinets, you also have the [211:58] rack battery backup units which are [212:00] direct current backups at the actual [212:02] rack itself for the uh for the racks. [212:06] All right. Oh, I just skipped a slide [212:07] there. So if we look at once again here [212:10] at uh a flow state of what happens with [212:13] an uninterruptible power supply, you can [212:16] see how normally you have the utility AC [212:18] going to the rectifier DC link inverter [212:19] and then the projected load which in [212:21] this case is 100 kilowatts. And whenever [212:25] the utility AC goes down or if something [212:28] happens in this side of the battery [212:31] supply chain then the UPS so the [212:34] unintentional power supply battery will [212:36] discharge through the DC link and supply [212:38] power to load and um as soon as it's [212:40] restored we will remove the uh power [212:44] supply sorry we'll remove the battery [212:46] from the from the DC link and actually [212:49] if the utility is providing sufficient [212:52] power we' also So not only supply power [212:54] to the inverter but also put current [212:56] through this birectional battery [212:58] interface to charge back the battery to [213:01] its normal voltage. [213:04] Okay. So now that we're talking about [213:06] batteries and we'll also jump into a bit [213:08] of capacitors. We need to discuss the [213:10] concept of capacitance. [213:12] So what is capacitance? It relates [213:15] stored charge to voltage. So first of [213:20] all we know voltage is this electrical [213:22] potential in the in the units of jewels [213:25] per kum and then also we have charge [213:29] which is in units of kum. So in fact um [213:33] this unit uh of charge is just like [213:36] saying how many electrons do you have. [213:39] So if you have a charge of 160 kum [213:42] across some dialectric. So I didn't [213:46] explain what a dialectric is, but a [213:48] dialectric is just like an insulator. So [213:51] an example of a dialectric is literally [213:53] air. So air is not a good conductor. So [213:55] it separates electrons from passing and [213:58] like forming a current. So let's say [214:00] that you have some type of conductor on [214:03] this side of the plate. So you have like [214:06] two conducting conductive plates that [214:08] make an electric field and you have a a [214:11] dialectric such that there is no way for [214:14] the electrons to pass through and there [214:16] you therefore you have this electrical [214:18] potential of 800 volts and in particular [214:21] you have a difference of charge between [214:23] the two sides of 160 kum. So in this [214:27] case you would say that the charge is [214:28] 160 kum and then what happens is that [214:33] when you have a charge across a [214:35] dialectric that's by definition a [214:38] capacitor. So a capacitor is literally [214:40] like the definition of a capacitor is [214:42] some charge across two conductors um [214:45] that creates this electrical field and [214:49] each capacitor has a capacitance. Okay. [214:53] So what is capacitance? [214:55] You can think of it as this fundamental [214:57] property of every capacitor that says [215:01] for every so capacitance first of all is [215:03] in is this C here. So the formula is [215:06] charge equals capacitance time voltage [215:08] or capacitance equals charge time [215:10] voltage. So what capacitance is telling [215:12] us is for a physical conductor this is a [215:15] fixed value. It's telling us how much [215:17] charge can we store for every unit [215:19] voltage. So that's how you should think [215:22] of capacitance. It's like okay let's say [215:24] that you have a specific like a a [215:26] ceramic capacitor. So for that specific [215:29] capacitor it'll have a number of its [215:31] capacitance which is in units of farrod. [215:34] So that's the unit it's it's just f um [215:36] and it stands for 0.2 kums per volt or [215:41] you could have a capacitance of one [215:42] which would say for every one kulum uh [215:46] it's essentially one volt is across the [215:50] across this dialectric. Okay. So that's [215:52] just the unit of capacitance. And once [215:55] again, since capacitance is equal to [215:58] charge per unit volt, if you divide out [216:01] uh the V on both sides, we can also flip [216:04] the formula to get charge is equal to [216:08] capacitance time voltage. So in this [216:10] case, um the whole exercise is okay, we [216:12] have a capacitor with a known [216:14] capacitance of 0.2 and the voltage [216:17] across the two conductors in the [216:19] capacitor is 800 volts. Therefore we can [216:22] calculate the number of kulum is 160 and [216:24] then if one kum is like 10^ the 30 [216:27] electrons which is just a random number [216:29] I forget what actually a kum is equal to [216:31] then you would say that um the charge [216:33] difference so the difference in number [216:35] of electrons between the two sides of [216:36] the plates would be 160 * 10 ^ 30 or 160 [216:40] * however many electrons is in a koolum [216:43] and [216:45] it may seem contrived to really [216:47] understand what this means at this low [216:49] of a level but it's important important [216:51] to know because capacitors are really [216:53] important in the context of like these [216:55] uninterable power supplies and these [216:57] battery systems. So let's jump to this [216:59] next slide where we talk about how a [217:02] capacitor actually is is charged. So [217:08] what are we what are we plotting on this [217:09] on this graph? So let's assume that a [217:13] capacitor [217:15] is being charged. So what that means is [217:17] that you would be moving electrons onto [217:20] this side of the plate and you'd be [217:22] increasing this uh this charge [217:24] difference and then you'd also be [217:26] increasing the voltage because [217:28] capacitance is a fixed value. So if [217:31] [snorts] you want to increase Q equals [217:32] charge then V must also increase in [217:35] order to maintain the ratio. Okay. So as [217:38] you increase the charge so does the [217:40] voltage increase. [217:42] And in this graph we increase the charge [217:46] from zero. So that that would be a case [217:48] where like uh there is actually no [217:49] voltage or charge between the two [217:51] conductors to 160 kum. And then this [217:56] also gives us a voltage across the the [217:59] gap across the dialectric of 800 volts. [218:03] And so we have this formula which says [218:07] that the charge multiplied by the [218:11] average voltage applied across the [218:14] across the gap gives us the amount of [218:16] energy that's stored in the capacitor. [218:18] But I just like skipped ahead. So let's [218:21] first describe like how did we actually [218:23] get get there. So first of all we have [218:26] energy which is in units of jewels. And [218:29] so by the formula of uh by the units [218:34] right of capacitance which is just sorry [218:37] of charge which is just in kulum and [218:39] volt which is in jewels per kum. What we [218:42] can do is multiply these two terms and [218:44] then just get jewels. But we need to [218:47] divide by two because essentially the [218:50] voltage is making this triangle. So in [218:54] order to get the area of this triangle, [218:56] we need to do um base * height / 2. So [218:59] in this case, it's literally 160 * 400 / [219:03] 2. So this gives us the amount of energy [219:05] required to charge a capacitor uh up to [219:10] a up to a specific voltage essentially. [219:12] And we can also alternatively [219:16] express the energy of charging a [219:18] capacitor with respect to only voltage [219:21] and capacitance because if you guys [219:23] remember Q is equal to CV. So uh we can [219:27] substitute Q here and get energy equals [219:29] CV ^ 2 / 2 as the amount of energy [219:32] required uh to charge a capacitor or [219:35] equivalently the amount of energy that a [219:38] capacitor can discharge. So this formula [219:40] can be used both to express how much [219:42] energy is required to charge a capacitor [219:44] and how much energy a capacitor can [219:46] release. And once again this is going to [219:48] be really important because the [219:49] capacitor is essentially what provides [219:51] the buffer while the battery and the [219:53] generator are slowly winding up to their [219:57] operating voltage whenever there is a [219:59] power shortage. Okay. So let's assume [220:04] that [220:06] we have a 1 megawatt load which has been [220:09] cut off due to some issue with the [220:13] with the supply chain or there's just a [220:16] power outage and we want to know how [220:19] long can our capacitor support this 1 [220:22] megawatt load. So we already know that [220:24] the capacitance which is once again like [220:26] the kums per unit volt is 0.2 2 far odds [220:30] and the load is 1 megawatt and the [220:33] capacitor is currently sitting at an [220:36] electrical potential of 800 volts which [220:38] if we come back here means that there's [220:40] 160 kum of charge across this uh [220:42] dialectric and so it's pretty [220:45] straightforward. We have this energy [220:48] formula which we can just change to be [220:51] delta energy = 12 C of V ^2 at the [220:55] beginning minus V at the end squared. So [220:59] literally uh that's just this formula [221:01] here except we take the difference in [221:03] voltages to get the difference in [221:05] energy. And essentially what happens is [221:07] we're assuming that on this bus voltage [221:12] at 700 volts the load will cut off. So [221:15] this this assumption just comes to the [221:16] fact that we assume that whatever this [221:19] bus which has electricity is supplying [221:23] has a permitted output or rather it can [221:26] input voltage up to 7 down to 700 volts [221:30] but as soon as the voltage decreases [221:31] below 700 volts then it's no longer uh [221:34] the the bus can no longer actually power [221:36] the load. So that's why we calculate our [221:40] V start as a 800 cuz that's the start. [221:43] But more importantly that's why our V [221:44] min or our V end is 700 volts because [221:47] after that there's no point. In fact we [221:50] already see here that once the capacitor [221:52] has discharged to 700 volts it still has [221:55] 49 kJ of energy but it just does not [221:57] expend it because it's pointless to [222:00] bring the bus below 700 volts. Uh we [222:02] would just like shut off the power to [222:03] the downstream components at that point. [222:06] Okay, so enough with the talk. If you [222:08] actually calculate this formula, you [222:10] would get uh 15,000 jewels as the amount [222:14] of energy [222:16] that the capacitance the sorry the [222:19] capacitor can discharge between 800^2 [222:22] minus 700 volts. And then if it has to [222:25] supply 1 megawatt, [222:28] then if we just do time equals energy [222:32] divided by power. So that's the way that [222:34] you can get the amount of energy that is [222:37] required or sorry that's how you get the [222:40] time required to expend a specific [222:41] amount of energy then it would only be [222:44] able to support the 1 megawatt load for [222:46] 15 milliseconds. So it's not that much [222:48] time but also the capacitor doesn't have [222:50] that much energy and you require 1 [222:52] megawatt to support the load. So now we [222:54] know that up to down to 700 volts the [222:57] capacitor can support the the load or [223:00] rather the bus for 15 milliseconds at [223:02] which point it would cut off power to [223:04] the downstream components and they would [223:05] no longer have a power supply. [223:08] But that's actually not what happens in [223:09] reality. So in reality what happens is [223:11] that as the capacitor is uh ramping up [223:15] it's supplying essentially the deficit [223:17] sorry the capacitor is like actually [223:19] instantly uh activated because the [223:22] electrons will just start to move from [223:25] the negative terminal to the positive [223:26] terminal instantly once the voltage [223:28] drops in order to dampen this uh voltage [223:31] drop. Um and so the capacitor is [223:35] supplying a deficit during the battery [223:36] ramp up is what I meant to say. And so [223:38] instead of having just like only one [223:40] thing happening, we have the capacitor [223:42] which is acting while the battery is [223:45] ramping. So instead of calculating the [223:49] capacitor supplying the the voltage um [223:52] like on its own alone, instead we should [223:55] um calculate this while the battery is [223:57] ramping up. And we can assume that the [223:59] battery essentially is providing power [224:03] in an increasing manner linearly. [224:06] And therefore the area that the [224:08] capacitor needs to supply is this [224:10] deficit which is this triangle. So if we [224:13] just calculate it it's just um 1/2 [224:15] because there's a triangle of 1 megawatt [224:19] because that's the that's the load times [224:21] 10 millconds because that's how long it [224:23] takes for the battery power to equal the [224:25] load power. Okay. And this means the [224:28] capacitor needs to support 5 kg of [224:29] energy in this time. And so if we have [224:32] the same formula as last time except we [224:35] solve for the V end then um and we know [224:38] that jewels is 5,000 we can rearrange [224:40] for V ^ 2 and then we would get that [224:43] after 10 milliseconds [224:45] the deficit is now 768.1 volt or more [224:51] precisely the bus voltage drops down to [224:54] 768.1 volt but then stabilizes because [224:57] now the battery is supplying the 1 [224:59] megawatt load and the capacitor now is [225:02] no longer discharging because its [225:04] voltage is equal to the bus voltage. [225:07] Okay, so now let's uh let's quickly jump [225:11] back because those are pretty scary [225:12] formulas. So now we're at 768.1 volt and [225:16] if we just supply the 1 megatt load on [225:18] the battery, we'll never get back to 800 [225:20] volts because we need to essentially [225:23] charge the capacitor back, which means [225:25] we need to supply a surplus over 1 [225:27] megawatt in order to get back the [225:29] energy. And so that's what the formulas [225:30] on the on the next page are going to [225:31] show us. So I promise you this this [225:33] isn't as scary as it seems. So let's see [225:36] what it says. First of all, we just [225:38] using once again the time equals energy [225:40] divided by power. And just to give some [225:42] intuition again, we have time in [225:44] seconds. And then we're dividing jewels [225:46] divided by jewels divided by seconds. So [225:48] the jewels cancel out and the second [225:50] comes into into the numerator. And we [225:53] already know that energy with respect to [225:55] capacitance and voltage is equal to [225:59] CV 12 CV^ squ. So we have the energy in [226:02] the numerator which is 1/2 CV^ squ and [226:04] then the denominator is saying [226:06] essentially [226:08] we have the time and we want to [226:10] calculate the energy to replace it and [226:13] then at the very bottom we want to know [226:16] how much extra power do we need to [226:18] provide uh which is just P extra equals [226:21] the source minus the load. So in this [226:23] case the load would be 1 megawatt and [226:26] then we want to know how much more power [226:28] above 1 megawatt do we need to supply [226:32] and that's the question. So if we if we [226:36] for example supplied 50 megawatts above [226:38] the extra power sorry above the yeah the [226:41] extra power of the load then we would do [226:43] 5 kg divid by uh 50 megawatt and then we [226:46] would get the recovery time. So this [226:48] would be the constant positive surplus [226:50] at the data at the DC bus that we um [226:54] that we supply in order to recharge the [226:56] capacitor. So we get this 5 kg just as a [226:59] quick intuition by doing all right our [227:03] target voltage is 800. So 800 squared [227:06] and then initial voltage is 59 which is [227:11] um just 768.1 squared. And so this means [227:14] that we want to go uh we want to get to [227:17] 5 kg essentially that's what the [227:18] numerator is saying and therefore we [227:21] need to add this 5 kg with some extra [227:23] power right so let's look at what can [227:27] happen here so the more surplus power [227:29] restores the bus sooner and once again [227:33] we're assuming that we have a 1 megawatt [227:34] load that needs to be powered and we [227:36] also need to supply some surplus to get [227:38] to the 5 kg and restore that. So let's [227:41] say that if we supply one megawatt to [227:44] the bus, right? There's no surplus. So [227:46] the the voltage stays at 768.1. This is [227:48] this dotted line. Instead, let's suppose [227:51] that we now supply 50,000 kW extra power [227:55] to the bus. At that point the recovery [227:58] will happen in 100 milliseconds because [228:03] if we simply divide 5,000 divided by uh [228:07] 50,000 we will get 100 milliseconds so a [228:10] tenth of a second if I'm not mistaken. [228:13] And then conversely if you supply more [228:16] power 1.1 megawatt which is 100 kow then [228:19] since you're supplying double the power [228:20] the recovery time will be half. And this [228:23] means um we're doing 50 sorry 5,000 [228:26] divided by 100,000 and that will give us [228:30] 50 milliseconds in recovery time. Okay. [228:33] And also once we get to 800 volts we [228:36] assume that we reduce the source power [228:38] at the bus to 1 megawatt because if you [228:41] increase above that then you'll just [228:43] start to charge the capacitor above 800 [228:45] volts but you also don't want to do [228:46] that. So um that's what happens when [228:49] we're using a battery. Also, um let's [228:53] assume that we're not using a battery to [228:55] recharge the capacitor. Let's assume [228:56] that we're using a generator and a [228:58] rectifi rectifier. So, once again, the [229:00] rectifier is going from um is going from [229:03] AC to DC. So, in this case, the uh the [229:07] generator would also perform the exact [229:08] same job of of recharging the capacitor [229:11] in order to reach the the the operating [229:14] bus voltage. [229:16] All right, so that was pretty long. I [229:17] hope you guys survived through those [229:19] equations. It's pretty pretty [229:20] complicated sometimes. But now let's uh [229:23] get back to a bit more fun diagrams. [229:26] So if we look at here, right, the UPS [229:30] battery bridges the weight for accept [229:31] the generator power. So let's assume [229:34] just to re resume or give a resume of [229:37] what happens in every step of this [229:40] cycle when we go from um utility to [229:43] failing power to generator. So first of [229:46] all the utility is supplying power and [229:48] then let's assume that the utility is [229:50] unavailable and now the generator is [229:52] starting but disconnected since it needs [229:53] to take a few minutes to ramp up. At [229:55] this point the capacitor has taken up [229:57] the first 10 milliseconds and then after [229:59] that the battery makes up for the um the [230:03] the load which is 100 kow in this [230:05] example and the battery is discharging. [230:07] then we would say that the generator has [230:10] now reached its uh operating output of [230:13] power and therefore it goes through the [230:15] same UPS as normal as the utility would. [230:18] And then finally once the battery or [230:21] once the util the generator is ready to [230:23] supply power not only to the to the load [230:26] but also to the battery then we will um [230:29] essentially supply more than 100 kow to [230:33] supply power and recharge our battery [230:36] such that in in a future uh situation [230:39] where the generator or the utility come [230:41] down then the battery is at its charge [230:44] state in order to supply the DC link. [230:48] All right. And let's just quickly talk [230:50] about battery output. So this is pretty [230:52] simple [230:54] math and physics here. So assuming a [230:57] battery has a 1 mill sorry a 1 megatt [231:00] hour uh energy storage. So this would [231:03] mean that it can supply 1 megawatt of [231:04] power for 1 hour. And also if we [231:08] consider the fact that you probably will [231:10] not discharge the battery past 20% so [231:12] that only the 80% is actually useful. [231:14] And then finally that actually the [231:17] battery cannot perfectly convert its [231:19] energy into electricity. There would be [231:22] a 95% conversion ratio. Then you [231:24] actually only get 76 megatt hours to [231:26] load from an initial 1 megatth battery. [231:30] And assuming you have a 8 megawatt [231:34] output limit on that battery through the [231:36] DC link here, then if you have a 6 [231:39] megawatt demand, you could you could [231:41] supply that demand for 7.6 minutes. And [231:43] this is simply just getting our energy [231:46] dividing this by the power and then [231:48] multiplying by 60 to get minutes. And [231:51] then assuming that we have a 4 megawatt [231:56] output limit, then we actually could not [231:59] even meet our 6 megawatt demand because [232:01] the full load cannot be supplied. There [232:03] is essentially 2 megawatts of demand [232:05] that's unsupported. So your ensemble of [232:07] batteries should at least be able to [232:09] supply the load. So in this case, you'd [232:11] want batteries that at least can supply [232:14] 100 kilowatts after you consider both [232:16] the conversion um and the usable window [232:20] and also like what is the output of the [232:22] battery. Right? So up to now we've [232:25] actually been only talking about fossil [232:27] fuel uh systems from my understanding of [232:31] supplying systems when there is a power [232:34] outage. But actually Cruso has this [232:36] spark model which is like their modular [232:38] data center which uses solar and battery [232:40] storage in order to supply their data [232:44] center. And what happens here is that in [232:46] their pilot project which was from last [232:48] year they had 12 megawatts of solar and [232:51] 63 megatt hours of battery to supply a 1 [232:54] megawatt data center which also had a [232:57] grid backup. So should like there be [232:58] like no sun for like many days and they [233:00] couldn't u feed it. But in fact, if you [233:04] just do the calculations here, right, [233:05] since they have 63 megatt hours of [233:07] batteries, of course, in this example, [233:10] we we um don't really consider the fact [233:13] that you can't just get pure efficiency [233:15] out of your batteries. But if we ignore [233:17] that, you get approximately 63 hours of [233:19] run time without having any solar [233:21] charge. So that's already a pretty [233:24] robust system in terms of the gross [233:26] energy to load ratio. So, this is an [233:30] example of a completely renewable [233:32] system. Completely renewable in the [233:33] sense that hopefully you would never [233:35] have to dip into the grid. Maybe from [233:36] time to time you would. Um, but on net [233:39] it would be running on renewable energy. [233:41] So, I think that's a pretty cool concept [233:42] as well in terms of autonomy from the [233:45] grid and from fossil fuels. [233:47] All right. I mentioned the static bypass [233:49] before, but this is the first time we're [233:50] talking about it in a diagram. So, the [233:54] static bypass supplies the load when the [233:55] inverter is unavailable. So essentially [233:57] what's happening here is that since the [233:59] source is an AC we can technically like [234:01] skip the entire rectifier DC link [234:03] inverter stage uh completely if anything [234:07] on this line in the UPS ever is faulted [234:10] or gets disconnected by this uh [234:12] disconnector or breaker. So this just [234:15] allows us to still power the load if the [234:17] inverter is unavailable for example. But [234:19] what the problem is is if you lose the [234:22] access to your static bypass then you [234:24] can no longer actually supply the [234:27] battery because as we said the [234:29] assumption here is that the inverter is [234:30] uh faulty. So the problem is that when [234:32] you use the static bypass you're kind of [234:34] living on the edge because if ever the [234:36] static bypass goes down then you cannot [234:39] power your load. There's actually a [234:41] another type of bypass which is called [234:44] the maintenance bypass. So the [234:47] maintenance bypass carries the load [234:48] around the completely isolated UPS. You [234:50] can see here we have these disconnects [234:52] on the entire load which make it such [234:54] that the UPS is not at all connected to [234:56] any source of energy. And the purpose [234:58] for this is that if you ever need to [235:00] service the UPS then you don't want it [235:02] to be connected to anything which is [235:03] energized. So essentially this external [235:07] maintenance bypass allows you to still [235:09] power the load while the UPS is [235:11] completely shut off. And what happens [235:14] here is that once again if you lose the [235:17] bypass source if for some reason you're [235:19] doing maintenance on this UPS and the [235:21] maintenance bypass decides to fail then [235:23] this load also gets no supply. So one [235:26] thing worth noting is that whenever [235:27] you're using a bypass then you have a [235:28] single point of failure as opposed to [235:30] the UPS where if one of the input [235:33] sources fails you have usually a second [235:35] option ahead of you. [235:37] I also have been talking about once [235:39] again AC and DC in comparison. And one [235:42] thing that's interesting is that even if [235:46] you [235:48] isolate the AC from the rest of the grid [235:51] here, the DC can still be alive because [235:54] of the battery and because of [235:56] capacitors. So in this example here, [235:59] it's really important to disconnect the [236:01] battery as well and make sure that this [236:02] DC link is completely deenergized before [236:05] there is any maintenance attempted on [236:08] the machine. And furthermore, because DC [236:12] is at a constant current and voltage, [236:15] even if you disconnect it, it could [236:17] still remain energized. And this is [236:19] opposed to AC where essentially AC [236:22] passes through zero twice period, [236:25] meaning that it's much easier to [236:26] deenergize alternating current. Um so [236:29] what this means is when you're using [236:30] direct current you need some load side [236:33] um method to disconnect and dis [236:35] discharge the direct current before any [236:38] maintenance is uh conducted on this [236:41] component. So once again the the message [236:43] here is that even if the AC is off the [236:45] DC the DC can still be live. So that's [236:48] more of a safety precaution. [236:50] And in fact um what can happen as well [236:56] with DC is occur let's let's let's talk [236:59] about um the the base case before we go [237:02] into into arcs. So when current is when [237:06] the circuit is closed current flows [237:08] because the touching contact completes [237:09] the circuit but an open gap can carry [237:12] current even if there is a gap uh if [237:16] there is sufficient voltage. literally [237:17] hot ionized gas which is the gap and [237:19] current still flows. What happens to [237:22] destroy this or to remove this arc is [237:24] that well in alternating current since [237:27] the [237:29] uh polarity switches so often an arc can [237:33] actually be self-extinguished because in [237:35] the intervals when the current reverses [237:38] and there's no voltage [237:40] the air between the gap can actually [237:43] become sufficiently cool that it just [237:45] self-extinguishes the arc. The problem [237:47] is though with direct current is there [237:49] is no such uh change in polar polarity. [237:52] So if you just keep a circuit closed or [237:56] rather not closed but uh constant with [237:59] an arc in direct current it will [238:02] continue like indefinitely unless the [238:04] source is deenergized. And so what this [238:06] means is that either one must deenergize [238:10] the source to stop a direct current from [238:11] arcing if it's occurring or you can move [238:13] the move the two uh nodes of this [238:17] current that are open farther away [238:18] because at a certain distance then there [238:20] will be no longer enough voltage to [238:21] actually cross the the gap. So or rather [238:24] the distance will be too long too large [238:26] for that given voltage to cross the gap. [238:28] So this is just something also worth [238:29] noting um where direct current has a [238:34] downside or a disadvantage here as [238:36] opposed to alternating current when it [238:38] comes to these like fringe fringe cases [238:41] of [238:42] electric faults or arcing in specific uh [238:45] in particular. So just retain that in an [238:48] arc with direct current unless something [238:50] is changed in the system the arc will [238:52] persist. Whereas in alternating current [238:54] since the polarity is changing there is [238:56] the possibility that in the intervals [238:59] the gap will become sufficiently cool in [239:02] order to stop the ionized plasma [239:04] essentially from passing this current [239:06] across the air. All right so we have [239:09] more memes. Um it's because we're [239:11] talking about redundancy now. So this [239:13] first one says or second one says I see [239:14] you don't have enough redundancy in your [239:15] data center I also like to live [239:17] dangerously and then this first meme [239:20] says take it easy buddy think of the [239:21] data centers because it's a really hot [239:22] day and turns out that data centers [239:25] don't do too well in the heat. So let's [239:27] use these two memes to segue into the [239:29] redundancy section of the chapter. So [239:33] let's talk about a 100 kowatt source or [239:37] rather load which is fed by two 50 kow [239:41] modules on a shared output bus. So in [239:43] this case you have 100 kow available and [239:45] 100 kow also required. So in this case [239:48] we can supply the it and in our case of [239:54] two 50 kow units we have n um n [239:57] redundancy or essentially there's no [239:59] redundancy cuz the same amount of power [240:02] that's available is what's required. So [240:04] what happens is that if you fail one [240:05] module then you can no longer um supply [240:08] this it [240:10] happens when you don't have any [240:10] redundancy. How about let's add a third [240:13] module. So in this case we have N plus1 [240:17] redundancy for 100 kilowatt load and [240:19] let's let's do some maintenance. So if [240:21] we do maintenance on this module would [240:23] essentially disconnect it from the from [240:25] the bus and isolate it and then uh we [240:29] would still be able to serve our end [240:32] load. But now let's say that we're [240:34] isolating and we failed one. Well, in [240:36] that case, we only have 50 kilowatt [240:38] available and therefore we cannot supply [240:40] enough power through the UPS to supply [240:43] our um our end demand. And then if we [240:48] fail one module and keep one um keep [240:51] both of them alive, then also we can um [240:54] keep them we can keep this IT demand um [240:57] satiated because we have M1 and M3 these [241:01] two modules supplying power to the to [241:03] the load. Okay, now we're going to jump [241:05] to N plus2 redundancy. I I think you [241:07] guys can get the get the picture here. [241:09] Let's say that we are isolating one [241:10] module and one of them is failed. In [241:12] this case, we can still feed the load at [241:14] the end of the bus because we have this [241:15] N plus2 redundancy and essentially two [241:19] of the modules can fail and we can still [241:21] run. Uh if we actually return both of [241:23] them to be online, we have 200 kW [241:25] available across the shared output bus [241:27] even though 100 is required and we're [241:29] also still in business with 100 kW to [241:32] spare. Now let's go to a different [241:35] example. Okay, so we have n plus1 [241:37] redundancy but the output bus is shared. [241:40] And so this means that the paths are not [241:42] independent. [241:43] And suppose there's a bus fault [241:45] somewhere along the shared output bus. [241:48] Well, that means that despite n plus1 [241:50] redundancy, we cannot do anything and [241:52] there's insufficient [241:54] uninterruptible power supply capacity [241:57] due to this bus fault. If we restore the [241:59] shared bus, that will no longer be the [242:00] case. We can now supply power to the IT [242:02] demand. But know that even if we have N [242:05] plus1 redundancy, there's still a single [242:07] point of failure failure if your shared [242:09] output bus is [242:11] shared. And then now we'll use this as [242:14] an example to go into a 2N redundancy [242:17] configuration. So essentially you have a [242:19] A and a B path. That's what you call [242:20] them usually or you would call this a [242:22] dual corded uh system. Essentially [242:24] there's there's two chords. And so what [242:26] happens is that if you isolate path A, [242:29] let's say that you're doing some [242:30] maintenance here, then you're still able [242:32] to carry electricity because half of the [242:35] paths can carry the whole load. Well, if [242:37] we fail um one of the modules here on B, [242:40] then unfortunately even two end [242:42] redundancy is not enough. But then if we [242:44] return uh the 50 kows on path A and even [242:48] if B1 is failed then we can still supply [242:52] the 100 kows because we have these two [242:54] independent power supply units such that [242:57] um yeah in in this case unless both of [242:59] them fail to this extent we're able to [243:02] supply power to our end load. All right. [243:05] And we can even go further and talk [243:07] about 2 n plus one redundancy which you [243:10] can understand through this diagram. And [243:13] in this case it's the same thing as 2n [243:15] but now we add one to each of the each [243:17] of the different lines. So let's say [243:19] that we isolate this entire path. Now we [243:21] have 150 kow available to this 100 kow [243:24] uh end load. And even if we fail one [243:27] module we can still supply the 100 kow. [243:30] So this is a much more resilient and [243:32] redundant manner to uh supply our end [243:35] demand as we have in the title here. A [243:37] spare on each path adds margin during [243:39] maintenance. So this is essentially as [243:42] as good as you can get. If you have more [243:45] redundancy than this, then you're [243:46] starting to get into dimishing returns [243:48] cuz you need some pretty strange fringe [243:51] events to occur for that type of [243:53] redundancy to fail. And moving on, let's [243:57] talk about N plus1 and a situation where [244:01] we then increase the IT demand to 150 [244:04] KOW. So essentially what's happened here [244:06] is that we've gone from N plus1 to N [244:09] redundancy. So the good thing about [244:11] having more redundancy is if you do [244:13] upgrade your equipment later on then you [244:15] can serve more with the same power [244:18] units, but of course the problem is you [244:20] lose your redundancy. So, it's a it's a [244:23] trade-off there for sure, but it's one [244:25] thing to know is that having redundancy [244:26] also allows further upgrades to happen [244:29] without changing the power supply [244:31] infrastructure. [244:33] All right. And just to give a quick [244:36] example here too with having faults on [244:40] these different branches on feeders. So, [244:43] let's say that we have a fault on group [244:45] B and it's isolated. Well, as long as we [244:49] have the disconnect open here before it [244:54] reaches the shared bus, then the fault [244:57] stops the load before um the breaker [244:59] opens upstream. And we can keep group A [245:02] and group B, sorry, group A and group C [245:04] supplied. But now let's consider [245:06] something else where you have a branch [245:08] fault and the upstream trips because for [245:11] some reason the group B uh breaker did [245:14] not open. In that case, you have to fail [245:17] all three of these buses or all three of [245:20] these groups. Since they are dependent [245:22] on this single point of failure from the [245:24] upstream supply and then additionally [245:27] even if there's no failure within any of [245:30] the groups, let's see all the groups are [245:31] fine but there's some problem on the bus [245:34] and the bus voltage collapse. Well then [245:37] this entire upstream supply will [245:39] probably be closed somewhere because you [245:42] have this propagation of the fault to [245:44] the first breaker that opens and [245:46] therefore these three groups also [245:48] receive no power. So I think the lesson [245:51] here is in these hierarchical grid [245:54] systems you need to see where is the [245:57] single point of failure. You know, in a [245:59] really redundant system, you actually [246:00] have completely independent grids or [246:02] rather paths for energy to flow such [246:05] that one entire line can fail and the [246:07] power can still be supplied to the load. [246:09] But that isn't always possible in terms [246:11] of just like from a fiscal standpoint. [246:14] So you need to balance um resiliency [246:17] with cost. [246:19] A different a different type of fault [246:20] that we could talk about is um the racks [246:23] themselves. So let's assume that you [246:27] have some type of live conductor that [246:30] has an insulation fault. So essentially [246:32] let's assume that like the wire casing [246:34] has shredded and like the wire is [246:36] exposed to the metal cabinet of the rack [246:37] literally like the metal cabinet and it [246:40] would if if that conductor came into [246:43] contact with the cabinet it would become [246:44] energized [246:46] and that could lead to problems like if [246:48] somebody touches the energized rack then [246:51] bad things could happen and therefore [246:54] interestingly enough like to prevent [246:57] this from happening you would connect a [246:59] return path from the metal metal cabinet [247:02] to the source such that it completes the [247:05] fault loop. And why would you want to do [247:06] this? So once again, a protective [247:08] conductor connects the metal cabinet [247:10] itself back to the source of the circuit [247:12] such that in a fault scenario, the fault [247:15] current makes the protection disconnect [247:17] the circuit. So, what is meant here is [247:19] that if ever there's an insulation fault [247:22] and the metal cabinet becomes energized, [247:24] you'd actually want to connect the metal [247:26] cabinet back to the source to complete [247:28] the circuit and allow the breaker to [247:30] trip and close the circuit properly as [247:33] opposed to having the metal cabinet [247:35] remain energized and have nowhere to go [247:37] if it wasn't connected back to the [247:38] source. That's an interesting thing [247:40] where like you're purposely connecting [247:42] the metal cabinet back to the source [247:43] source. If ever there's some insulation [247:45] fault that occurs which causes that [247:47] metal cabinet to be energized, you want [247:49] that current to flow back to the source [247:50] as opposed to elsewhere. [247:53] All right. So when we talk about [247:54] redundancy, there's actually a ranking [247:56] and it's called the uptime institute [247:58] tiers with one being the lowest and four [248:00] being the highest. So if we if you start [248:02] from the bottom, number one, it says [248:05] your data center just provides basic [248:07] power and cooling. So pretty standard. [248:10] And then number two, obviously these are [248:12] like slightly generalized. There's [248:13] there's much more stringent uh [248:14] requirements for these systems, but [248:16] we're just giving a a high level [248:17] overview. Number two says you add spare [248:19] power and cooling equipment. The number [248:21] three states that you maintain equipment [248:23] with it running. So this means is that [248:26] as in this example here, if you isolate [248:29] path A to perform maintenance because [248:32] you have path B, you can still uh keep [248:35] your load running and that would qualify [248:37] for the third tier. And then the fourth [248:40] tier is saying that if there's an [248:41] infrastructure fault, which is to say [248:43] this is planned maintenance. If this is [248:46] essentially you can think of it as [248:47] unplanned maintenance, if that happens, [248:49] then in tier four, the system is still [248:51] able to survive. And tier four data [248:53] centers are less common because they're [248:56] much more expensive to build in all the [248:58] redundancy. You you'll mainly see this [249:00] with for example uh government data [249:02] centers where well the government [249:03] doesn't have that much sensitivity to [249:05] the cost. So they'll give the they'll [249:08] give the the money for the highest tier [249:10] redundancy and also like government data [249:12] usually is pretty pretty high um high [249:15] worth. So they don't want to have the [249:16] data center go down or have some vital [249:19] government uh service lose power. Okay. [249:24] And just just an example here, right? We [249:26] have tier three and tier four are [249:27] essentially saying in plan maintenance [249:29] plan path A is disconnected and isolated [249:32] and path B still supplies the it. And [249:35] then tier four is saying okay if there's [249:37] a sudden failure and path A goes down [249:40] then path B can still be used to supply [249:43] other it. Whereas in tier three, we're [249:46] saying we can only guarantee that in a [249:49] planned maintenance event, we can keep [249:51] the it online. It's not saying that if [249:53] path A goes down to some fault, then [249:55] there could be some um issue where the [249:57] path B also becomes compromised, for [249:59] example. [250:01] All right. One thing also worth noting [250:03] is that on-site generation is actually a [250:07] part of all four uptime tiers because it [250:10] feeds into both it power and cooling. [250:12] But when you go from tier one to four in [250:16] um in redundancy, you add more spare [250:19] equipment and that's the second tier. [250:22] Then in the third tier, you add the [250:24] ability to do maintenance without [250:25] turning off the data center. And the [250:27] fourth tier, you're saying that even in [250:28] a fault tolerant situation. So you can [250:31] think of like Hurricane Harvey with that [250:33] Equinox uh Equinex data center, that [250:35] would be a tier 4 data center. [250:38] So there's tiers and then there's the [250:40] nines which are separate concepts. So [250:43] three, four and five nines of [250:45] availability simply say how much of time [250:48] in the year is your data center up. So [250:50] if you have three nines of of [250:52] availability, which is for example the [250:54] Cruso Spark data center in Nevada, which [250:57] is the one that uses solar panels, this [251:00] means that for 8 hours and 46 minutes in [251:02] the year, you will have that much [251:04] downtime. And literally, this is just by [251:06] doing um 0.01% [251:10] of the year. And this gives you 8.46%. [251:12] And then if you have 49s of [251:14] availability, then this is an example of [251:17] Microsoft Fairwater in Atlanta, Georgia, [251:19] which we'll talk about in the next [251:20] slide. This would be the equivalent of [251:22] 52.6 minutes of downtime in a year, [251:24] which is already like less than an hour. [251:26] And there are like 8,000 hours in a year [251:28] if you guys remember, 8,760, I think. [251:31] And then 59s of availability is pretty [251:33] insane. So there's this uh data center [251:35] in Vienna which must have a lot of [251:37] redundancy and the SLA which is the [251:40] service level agreement I guess [251:42] specifies that you can expect at maximum [251:45] 5.26 26 minutes of downtime per year, [251:47] which is quite impressive if you think [251:49] about it. Right? And let's just talk [251:52] about an example of a data center with [251:54] 49 of reliability. This is the Microsoft [251:56] Fairwater data center. And in fact, it's [251:59] quite interesting. So this data center [252:01] achieves 49s of availability at 39's [252:04] cost. How is this possible? It's because [252:06] they actually don't have on-site [252:08] generators. They don't have UPS systems [252:10] and they don't have dual corded [252:11] distribution. So just as a reminder, we [252:13] know what generators are. We know that [252:14] the uninterruptible power supply is to [252:17] supply power through a battery and then [252:19] we know that dual corded systems is to [252:21] have essentially an independent A and B [252:23] path for every single power supply. So [252:25] how do they achieve four 9s of [252:27] availability without any of this [252:30] infrastructure? [252:32] Let's uh let's let Scott Gutry who made [252:35] this quote tell us how he does it. So [252:38] what they say is we have also worked [252:40] with our industry partners to co-develop [252:41] power management solutions to mitigate [252:43] power oscillations created by large [252:45] scale jobs a growing challenge in [252:47] maintaining grid stability as AI demand [252:48] scales. This includes a software-driven [252:50] solution that introduces supplementary [252:52] workloads during periods of reduced [252:53] activity a hardware driven solution [252:55] where the GPUs enforce their own power [252:56] thresholds and on-site energy storage [252:58] solution to further mask power [252:59] fluctuations without utilizing excess [253:01] power. So I think I just realized this [253:03] quote actually talks more about how they [253:05] dampen the effects of the power which is [253:10] caused uh by AI to fluctuate. Um to talk [253:14] about how they actually achieve this [253:17] power redundancy without having any of [253:19] the on-site generators themselves. It [253:22] turns out that the grid that they're [253:23] connected to already has a lot of [253:25] built-in redundancy. So for whatever [253:28] reason uh their grid connect in [253:30] Fairwater Atlanta has a lot of latent uh [253:33] redundancy already within the grid. So [253:36] they don't need to build that secondary [253:38] power on themselves cuz I suppose they [253:41] already have this guarantee that the [253:42] grid itself is going to be 56 uh is [253:46] going to be up for most of the year and [253:49] will be down for at most 52.6 minutes [253:51] per year. So essentially what's [253:54] happening is that instead of having to [253:56] rely on their own generators to [253:58] guarantee 99.9% 99.99% availability uh [254:02] they got this agreement or they got this [254:04] deal from the grid to do that. And then [254:05] this paragraph my mistake is just [254:08] talking about how they are able to use [254:10] both software and hardware as I [254:12] mentioned uh I guess in a previous [254:14] chapter to dampen the load changes and [254:19] fluctuations between these different [254:21] jobs. [254:22] Okay, so this is the last uh second last [254:25] chapter, second last slide in our [254:27] chapter. It's going to quiz us on what [254:29] happens in a fault situation. So let's [254:32] say the racks have power, but can the [254:35] workload keep running? So what what's [254:37] the situation here? We're saying that [254:38] utility fails and the generator starts [254:40] and supplies a cooling plant where the [254:43] compute racks have a uninterruptible [254:45] power supply and a generator, which is [254:46] good. That means they have the rectifier [254:47] and the inverter. And then the cooling [254:49] controls though is only powered by the [254:51] utility and the pump and heat rejection [254:53] is powered by a generator. So the [254:56] question is what still stops the [254:58] workload. So I'll let you guys think [254:59] about this for a second and I'm going to [255:01] reveal the answer. Essentially the [255:04] cooling control has no more power [255:06] because it's only supplied by utility. [255:09] If you put the cooling on a protected [255:11] supply, then at that point you could [255:13] have a redundancy which would still run [255:17] the data center even if utility goes [255:19] down. But until the utility is not the [255:23] only source of energy for the cooling [255:25] controls then um even in a utility [255:27] outage you could not supply enough uh or [255:30] you cannot keep the data center running [255:32] because of the lack of cooling control [255:34] simply. [255:36] Okay. So the second question and our [255:40] last slide in this entire chapter is [255:41] saying the following. So it's saying [255:43] that the UPS is on a bypass. So let's [255:45] say that um the inverter has failed. [255:47] Actually that's what it says here. The [255:49] inverter has failed. Static bypass [255:50] carries a load. Now utility power has [255:52] failed. The question is will the load [255:54] stay powered without interruption? So do [255:56] you guys think yes or no? I don't know. [255:59] I think the load will not stay powered. [256:02] Okay. Correct. And what's the reasoning [256:03] here? The failed inverter blocks the [256:05] battery path and a generator can restore [256:08] bypass power after startup and transfer [256:10] but it does not bridge this [256:11] interruption. So essentially what [256:13] happens is that [256:16] technically you can supply power. So [256:17] this is kind of a nuance question I must [256:19] admit. You can supply power with the [256:21] utility after a few minutes but you [256:24] can't supply power with the battery. So [256:25] what I'm trying to say here is okay we [256:27] initially had this inverter fail and [256:30] then what happened is that the utility [256:31] also failed. So therefore, the battery [256:35] cannot supply power to the load because [256:36] the inverter is down. But the one thing [256:38] that still can happen is that if you [256:40] have a generator here, it can now take [256:42] the static bypass and supply the load. [256:45] But the problem is that that would take [256:47] like a few minutes to ramp up. So for [256:48] some time, you would have no supply and [256:50] then after that you could reboot the [256:52] system and get your rack running again. [256:54] So that was the second and last question [256:56] to verify the knowledge that we learned [256:57] in this chapter. Now let's move on to [256:59] chapter eight. [257:02] Okay. So here in chapter 8, we'll be [257:04] discussing how we feed the rack while [257:07] keeping chip voltage steady. And this is [257:10] going to be quite a compelling chapter [257:11] because you'll see how starting off at [257:14] let's say 345 kilovolts at the high [257:18] voltage AC transmission when electricity [257:21] actually comes into the data center [257:23] substation, we now will get like in the [257:26] range of 1 volt. And if we remember the [257:29] formula P= IV if you just naively did [257:32] this entire conversion then you would [257:34] have technically a 345,000 [257:38] increase in current. Of course this is [257:41] massively simplified and also assumes DC [257:43] current even though a lot of this [257:44] current is AC but the highle idea that I [257:48] want to get across here is that we are [257:50] now finally going to get to the current [257:52] which is actually accepted at the chip [257:53] level and this is like in the range of 1 [257:55] volt usually. So to get to this final [257:58] stage, we first need to go from the rack [258:00] bus, which we've already discussed [258:02] previously. So this vertical rack behind [258:05] our NVL72 [258:07] uh rack, for example, which supplies [258:11] electricity to all of the constituent [258:13] trays, which will then go to the board [258:16] itself. So if we're looking at once [258:18] again that same example, we have on [258:20] every single tray two boards in the [258:22] NVL72B300 Ultra. And then on the boards [258:25] we have voltage regulator modules these [258:27] VRMs which convert energy to the chip [258:31] itself. And um in this example these [258:35] these are some parts of the VRM. Um but [258:37] this is once again just an illustration. [258:39] So don't trust it as like the truth of [258:42] how every single VRM looks like. And [258:44] then we also have capacitors which are [258:47] on the chip level. These capacitors [258:49] essentially serve to buffer voltage or [258:51] rather to buffer current. So the voltage [258:54] remains constant so that the chip can [258:56] always have a relatively uh acceptable [259:00] input range of voltage. [259:02] Okay. So as a motivation for this slide, [259:06] I want to discuss a research piece that [259:09] was published by Bank of America Global [259:12] Research on the following statement. So [259:15] by the end of the decade, we think [259:17] Nvidia's road map could lead to greater [259:20] than 1.5 megawatt racks for the Fineman [259:23] era nearly 100x higher than a standard [259:27] CV server P CPU rack at 10 to 15 kow. So [259:31] the 10 to 15 kilowatt we're talking [259:32] about here is the traditional server [259:35] rack. And in fact, I spoke with somebody [259:37] who is a Berkeley professor in high [259:39] performance computing and has been like [259:41] an industry veteran for many years. And [259:43] he was telling me how at a certain time [259:46] the conventional wisdom that was that 60 [259:48] kW was the maximum rack power density [259:52] that one could achieve. And already [259:54] today we're at this phase currently [259:57] where there are N as said there are Vera [259:59] Rubin NVL 144s or NVL72s. Um, in fact, [260:03] this naming scheme is slightly [260:04] confusing. I'm going to jump ahead one [260:06] slide to con to explain why this is a [260:09] slightly confusing naming scheme. It's [260:10] because [260:12] on these GPUs, you have two dyes which [260:16] compose one GPU. So, this GPU has one [260:20] die here on the top and one die here on [260:22] the bottom. And just to explain why this [260:25] term is 72 and this term is 144. This [260:29] naming convention says that we count [260:32] these two dyes separately. And then the [260:35] convention where you say NVL72 [260:38] says that we count this GPU as one uh [260:42] functional GPU and we don't count the [260:44] individual dieseS. So all that is to say [260:47] that the NVL72 and the NVL 144 actually [260:52] have the same number of dyes which is [260:54] what is com which is what composes the [260:57] GPUs. Um the only thing that changes [261:00] between these two different uh between [261:01] these two generations is that the dies [261:03] probably become bigger and that's why it [261:05] has a higher power demand despite the [261:08] fact that actually from generation to [261:10] generation the actual dies themselves [261:13] become more efficient. But just to [261:15] recap, because the die is becoming [261:17] larger and it's just doing more [261:19] computations despite increases in [261:22] efficiency from generation to [261:23] generation, the overall power [261:25] consumption is increasing. So the entire [261:28] reason I went on this digression is to [261:29] say that currently we are in 2026 with [261:32] the Ver Ruben NVL 144/ NVL72. Both [261:35] naming conventions work and it's around [261:38] 130 to 140 kW. It hasn't actually [261:40] changed that much since Grace Blackwell [261:42] with notwithstanding in 2028. So this [261:46] this graph or rather this piece of [261:48] research whenever it's published is out [261:50] of date because what happened is that [261:52] the Vera Rubin Ultra Kyber NVL NVL 576 [261:56] so it's really a mouthful is this new [261:59] massive rack scale solution by Nvidia [262:02] which will be 500 sorry not 576 kW but [262:05] on the order of 600 kW and because of [262:08] its extremely sophisticated spine back [262:11] plane which is essentially the [262:14] interconnect on the rack level, the [262:16] scaleup domain that is behind the rack. [262:19] Um, it's essentially like a 7v2 or it's [262:21] like a 70s something layer PCB, a [262:23] printed uh circuit board, which allows [262:26] all of the GPUs to comm communicate with [262:28] each other. um because of leaked and [262:31] like this is industry confirmed [262:33] difficulties with like actually getting [262:35] this printed circuit board to work like [262:38] all of these issues with respect to the [262:39] different layers like warping and [262:41] ruining the board in manufacturing and [262:43] also the overall complexity of routing [262:45] and interconnecting all the GPUs. This [262:49] specific rack has been delayed to 2028. [262:51] So that's the whole short of it which [262:52] means that probably Rosa Fineman which [262:54] is the generation after uh the Ver Rubin [262:57] Ultra Kyber which is this one will [262:59] probably come in 2029. Uh we never know [263:01] maybe Nvidia will really get their game [263:04] on for this version but in any case [263:07] there are projections that the Rosa [263:09] Fineman era will be in the 1.5 megawatt [263:12] class. Um this is just like mindboggling [263:15] in terms of increase in power density. [263:17] One other thing though that you're [263:18] seeing is that whereas previously or up [263:21] to now you have these standard rack uh [263:24] configurations that are like rectangular [263:26] prisms [263:27] and not Nvidia but AMD has actually [263:30] already released a new type of rack [263:32] which is called the Helios. So, as you [263:35] can see here, the AMD Helios is almost [263:37] like a fat rack because it's two racks [263:40] wide. And this is in order to just have [263:42] a larger amount of compute inside of one [263:46] single system. And you may also see [263:48] Nvidia doing this at some point. You may [263:51] uh is actually probably a 2 week [263:54] qualifying term. You will definitely see [263:56] this happening on Nvidia's side as well [263:58] at some point in the future and um [264:00] likely in like the Rose of Fineman [264:01] because in this vertical configuration [264:04] like even after like 2 to 3 years of [264:07] insane progress it's probably not [264:09] possible to fit 1.5 megawatts of uh [264:11] compute inside of this small of a [264:13] enclosure. So that's just um essentially [264:16] giving some uh foresight of how these [264:19] racks will also be morphing in over over [264:22] time. And currently as I said AMD Helios [264:25] is the only rack which has this double [264:28] wide um size of the hyperscaler. So AMD, [264:32] Nvidia and other ASIC providers. So what [264:36] is actually this slide saying? Um we're [264:38] going once again from a rack to a board [264:42] when we want to convert down and [264:45] transfer power all the way from this [264:47] vertical bus bar to the voltage [264:49] regulator modules. I really like this [264:51] diagram. It's a Lenovo GB300. So, [264:54] interesting to know is that like there's [264:56] a lot of different manufacturers which [264:58] take the trays and the GPUs from Nvidia [265:00] and then put their own envelope around [265:02] it. So, Lenovo is one example. Also, [265:05] Dell is a really big uh provider of [265:08] these GB300 and other Nvidia racks. In [265:11] this example, we can really see every [265:13] single tray in in fine detail. So on the [265:15] top if you guys remember we have four [265:18] shelves which have six PSUs each six [265:22] power supply units which means that [265:24] there are 24 power supply units in these [265:26] four shelves. And similarly at the [265:29] bottom so that means 48 power shelves [265:32] not power shelves but power supply units [265:34] and eight power shelves. And so if you [265:37] have 48 power shelves and each one is [265:38] 5.5 kW, then we can actually just do the [265:41] math here and get the total power output [265:44] of a GB300 rack which is 264 kW. And [265:48] this is nice because it allows a lot of [265:49] redundancy since the peak demand of a [265:52] such a rack would only be 142 KOW. So [265:56] it's not perfectly n plus2 redundancy. [265:59] Sorry, 2N redundancy because that would [266:01] have to have I suppose 284 kW of power. [266:05] but it is still quite redundant, which [266:07] means you can hot swap the PSUs without [266:10] having to power down the GPUs. And where [266:14] are the GPUs? They're here. So, there [266:16] should be 10 racks here. Uh, sorry, 10 [266:18] trays here and eight trays here. Each [266:21] tray has two boards, which means that [266:23] there are 36 boards since we have uh 18 [266:27] trays and two boards per tray. It's [266:31] really hard to resolve these things [266:32] sometime, but I think we got it right [266:34] there. And then we also have the NVL [266:36] link switch trays which have nine trays. [266:39] And each tray has two NVLink switches [266:42] which means that there are 18 NVL link [266:43] switches in this rack. Okay. And so the [266:46] compute tray which is what we're really [266:47] interested in once again has the two [266:49] boards. And this is what a single board [266:50] looks like. And each board has two GPUs [266:53] and one CPU. That's why it's called the [266:56] NVL72. Because if we do two GPUs per [267:01] board times two [267:05] uh trays, sorry, two boards per tray, we [267:08] get four GPUs per tray. And if there are [267:12] 18 trays, we get 72 GPUs per rack. So [267:16] there is the entire endto-end [267:18] calculation of how you can get 72 GPUs [267:21] from an NVL 72 rack. And then uh one [267:23] once again there's only 36 grace CPUs in [267:26] this configuration. [267:27] I want to quickly zoom in and first of [267:29] all say that if you watch my course [267:31] called chips which describes how [267:33] semiconductors are made from silicon to [267:36] this final board uh you'll you'll learn [267:38] that process. So that's what we're [267:39] learning today. But I want to talk about [267:40] a few things here uh before we switch to [267:42] the next slide. So first of all I talked [267:44] about these voltage regulator modules. [267:46] And one component of these VRMs are [267:50] chokes, which are like inductors. And [267:52] all you need to know about an inductor [267:54] is it's a type of component you add to a [267:57] circuit to resist changes in current. [268:00] And it makes it such that you can have a [268:01] more constant current. So you don't need [268:03] to know anything about a choke more than [268:06] that. But what I do want to point out is [268:07] that the chokes themselves are here [268:10] around the around the GPU. And it makes [268:12] sense because this is what's actually [268:13] supplying uh the current to to the GPU. [268:15] So you want them to be as colloccated as [268:17] possible. Why? Because of resistive heat [268:20] loss. Since you're at 1 volt, the [268:22] current will be massive because of P= [268:24] IV. And you want the the the VRM, which [268:27] is actually supplying current to the [268:29] chip to be as close as possible. Um, [268:31] it's not the closest thing to the chip. [268:33] In fact, these gold like uh squares is [268:37] the high bandwidth memory, which you [268:39] want to be closer because memory latency [268:42] is king. So that's way more important [268:44] than having your resistive heat losses [268:47] minimized as much as possible. But you [268:49] can see the hierarchy here. So like you [268:50] have memory latency as the most [268:51] important and then also after that the [268:53] the voltage regulator metals voltage [268:55] regulator modules which are composed of [268:58] these chokes which are inductors are [269:00] also close by. These circular things as [269:03] well here might be might may or not be [269:04] capacitors. I'm not sure because Nvidia [269:07] doesn't actually release the detailed [269:08] spec of their boards. But in any case, I [269:11] wanted to highlight that you have a lot [269:12] of chokes here which compose the VRM [269:15] around the GPUs and similarly actually [269:17] with the CPU here below the gray CPU, [269:20] but there are less because the CPU is [269:22] much less power intensive than the GPUs [269:24] as we all know. And then the last thing [269:26] here is that the GPU has high bandwidth [269:29] memory whereas the CPU has a different [269:32] type of memory. It's still it's still a [269:35] DRAM but it's LP DDR5X. What does that [269:39] mean? It stands for low power double [269:41] data rate 5x. Um, and this is just a [269:44] type of DRAM. So, low power means it's [269:47] um low power as opposed to GDDR, which [269:50] is the graphics version of uh double [269:52] data rate power. Not double data double [269:54] data rate memory. And then double data [269:55] rate memory just means that in one clock [269:58] cycle, you can transfer both on the up [270:00] edge and the down edge of the clock [270:02] cycle, which just means that you have [270:03] better uh transfer and memory latency [270:06] and throughput. And then like five is [270:07] just a version. So that's low power [270:09] double data rate 5x memory. You might [270:11] have heard of LPDDR or GDDDR. Now you [270:14] know what that term actually means. And [270:15] then that's everything we need to know. [270:17] The last thing that is also just uh [270:19] quickly quickly worth noting is the [270:21] connect x8 supernix which forms the [270:23] scale out domain of these chips. So [270:25] we've already discussed the scale up [270:26] network. Um and then now we have the [270:29] scale out network which is mediated [270:31] through these super nicks which stands [270:33] for network interface card. So that was [270:35] that's what the NIC there means. Okay, [270:38] so that's a really high level overview. [270:40] Now, let's get into a bit more details [270:42] from the PSU to the CPU cores. So, the [270:45] power supply unit once again looks like [270:47] this. So, there's six of them in a in a [270:49] shelf. And this in our example takes the [270:53] alternating current that comes in [270:55] through the busway to 51 volts DC. So it [270:59] both converts from AC to DC which is [271:04] called rectification and then also will [271:06] step down the voltage to the voltage [271:09] range. You'll also see like these PSUs [271:12] will supply 51 volts and then the [271:16] motherboard or tray component that [271:18] accepts this voltage will like for [271:20] example have a input rating between 52 [271:24] and 46 volts. So that's just an example [271:27] of a meta specification. So this would [271:30] be the specific Nvidia trade that was [271:32] made for meta that had this input [271:33] voltage range uh which we'll discuss in [271:36] a bit uh in a few slides. But uh just [271:38] realize once again that you do have this [271:40] leniency in voltage ratings because you [271:43] can't always have like an absolutely [271:44] perfect voltage. Okay, so that's what [271:46] the PSU does. And then the voltage [271:48] regulator module makes and regulates the [271:50] core voltage. So there could be an [271:52] intermediate voltage step down from like [271:55] 51 to 12 and then to one or you could [271:57] just have a 50 to1 conversion. Once [272:01] again depends on the implementation. All [272:02] we need to know is that the voltage [272:03] regulator module is what actually is [272:05] what does one of the steps down in this [272:07] pro one of the step downs from this [272:08] process. And then we have the Vcore rail [272:10] which is like a abbreviation for voltage [272:12] core rail. It's just a copper path or [272:15] rail that carries the onevolt current to [272:18] the CPU or GPU cores. And then you want [272:22] to have these local capacitors that [272:24] connect to the rail beside the chip in [272:26] order to buffer if there is increase in [272:28] current absorb current. And then if [272:30] there's a decrease or like there's low [272:32] current on the bus to release some [272:34] current uh in order to buffer the [272:36] voltage. Okay. And so what's actually [272:41] supplying the the VRM at the downstream [272:44] level is the power supply unit. In this [272:47] case, we're looking at advanced energy [272:50] OV3 PSUs, which are 3 kW and go anywhere [272:55] from 200 to 277 volt AC to 50 volt DC [272:59] with six per shelf as we discussed in [273:02] the uh the Nvidia diagram there. But [273:04] these would not be used in the GB300 [273:06] because the GB300 uses 5.5 kilowatt [273:09] PSUs. In any case, every single rack [273:11] would then give you 18 kilowatt [273:13] installed capacity in this specific [273:15] example. [273:16] And what's interesting and what's worth [273:18] noting is that we have the incoming [273:20] three-phase alternating current. And in [273:23] fact, in order to load balance, what you [273:26] would have is like on a rack, sorry, on [273:30] a shelf in the rack with six PSUs, you [273:35] would likely have each of the three [273:37] phases of AC going to two going to two [273:40] power supply units in order to balance [273:42] the load. Or in any case, you would want [273:44] in of a of a group of uh six to have 1/3 [273:48] going to uh sorry, 1/3 of the lines [273:51] going to each of the power supply units. [273:53] And so in this case, we would have like [273:55] line one connecting to one here, line [273:57] two to another one, and then so on. And [274:00] then this would take this uh in this [274:02] case 277 volt AC and then step it down [274:05] to 50 volt DC. What you might be [274:07] wondering is okay if it's 480 volt AC [274:09] current why are we saying it's 277 volt [274:12] alternating current here it's because [274:13] once again 480 volt since voltage is [274:16] defined as a difference between two [274:19] electrical potentials this is the lineto [274:22] line voltage whereas 277 volt is the [274:26] line which would be any of these to [274:27] neutral difference in electrical [274:29] potential. So there's there's the reason [274:31] why we're talking about 277 volts here. [274:33] And then this would be fed as the arrow [274:36] is pointing onto this vertical bus bar [274:38] of the rack. And once again, this is [274:40] showing a DGXB300, but these PSUs would [274:43] actually not be there because we need [274:44] the 5.5 kW PSUs because of how power [274:47] hungry these GB300 chips are. [274:51] Okay, I'm I'm now going to do a a [274:54] calculation with you guys so that we can [274:56] really walk through like what does it [274:58] look like going from a input load. So [275:02] like our load requires some piece of [275:04] power. How do we determine the amount of [275:07] power we actually need to feed the rack [275:09] given like the GP the CPU or the GPU's [275:12] um power requirement? So let's assume [275:14] that we have a 72 requirement 72 kW [275:17] processor power load input and let's say [275:21] that the local voltage regulator modules [275:24] are 92% efficient. So essentially what [275:27] we're going to do here is work backwards [275:28] from the lack of efficiency and the heat [275:31] that we lose to get the required input [275:34] voltage at the alternating current [275:37] level. So let's just go through this in [275:39] our calculator. So 72 / 0.92 gives us [275:45] the amount of watts we need to input to [275:48] this regulator to get 72 kW which turns [275:50] out to be 72.26 just as we get here. The [275:54] other thing too is that the power from [275:56] this vertical bus right here isn't just [275:59] going exclusively to the chips. It's [276:01] also going to go to memory, networking, [276:03] fans, and auxiliaries. So we'll also [276:05] assume that we have 12 kow that need to [276:07] be added to this process as well. So [276:09] we'll just add 12 to get 90.26. As you [276:12] see here, this is the shelf DC bus. And [276:15] then once we actually want to get the [276:17] power into the bus, let's assume that [276:19] the rack power shelves, which is what [276:22] the PSUs do, which is what the PSUs are [276:24] rather, are 97% efficient. So what we're [276:27] going to do now is put this around [276:29] brackets and divide it by 0.97, [276:32] giving us 93.05, [276:35] just as stated in the diagram here. So [276:37] this would be the watts entering the [276:39] rack and the the calculation also is on [276:41] the bottom here where we add in the heat [276:43] in order to get our final amount. So you [276:46] can imagine that at a crude level this [276:48] is what data center engineers are doing [276:51] when they're determining how much energy [276:53] to feed to every single rack when they [276:55] already know the efficiency of each of [276:57] the uh the conversion steps. So that's [277:00] actually a walkthrough example that once [277:02] again a data center engineer would do in [277:04] real life. Okay, let's now get back to [277:07] the theme of the fact that a VRM is [277:11] doing this massive voltage switch which [277:14] causes potentially a really high current [277:18] and therefore heat losses. So at 1 volt [277:22] a small path resistance can consume a [277:24] large share of the voltage and create a [277:26] lot of resistive heat loss. So, let's [277:28] assume that at the board level, we [277:31] require 1,000 watts of energy to power [277:34] our GPUs and the rest of the components [277:36] on the board. What this means is that at [277:38] 1 volt, we need 1,000 amps since P= IV. [277:42] However, this does not account for [277:44] resistance. So if we assume that in this [277:48] regulator output to chip path there is [277:51] 100 micro ohms of resistance then due to [277:55] Ohm's law which states that the change [277:57] in voltage is equal to the current times [277:59] the resistance we'll have a voltage drop [278:02] of 0.1 vol which means that actually we [278:08] need to supply 1.1 volt at the regulator [278:11] to get one volt at the chip and then the [278:14] Other thing that I want to add in [278:16] addition is that since we have both [278:19] values of resistance and current, we can [278:21] also calculate the heat loss in this [278:23] path which is 100 W. And this is once [278:26] again the reason why we want to make [278:27] this as short as possible from the [278:28] regular output to the chip. If we [278:31] decrease the resistance by an order of [278:33] magnitude, you can see how much that [278:35] helps. So in this case, we only need to [278:36] supply 1.01 volts as opposed to 1.1 [278:39] volts at the regulator to get to one [278:41] volt at the chip. And then also the heat [278:43] in the path that is lost with resistive [278:46] with resistive heating is only 10 watts. [278:48] So really the essence of the slide is [278:53] small resistance changes can make large [278:56] downstream uh effects both in terms of [278:59] voltage drop and heating which it makes [279:01] it such a important space of efficiency [279:05] in the cycle of converting high voltage [279:08] AC to like one voltage DC to power the [279:11] chip. [279:13] This slide is really cool because it's [279:15] titled all roads lead to Rome. And the [279:18] the reason I titled it this is that [279:20] there are many ways to get to one volt. [279:22] And here are two methods. There could be [279:24] others. The data center industry is very [279:27] diverse. And once again, these are only [279:30] two possibilities. Let's just quickly [279:31] walk through them. So, one possibility [279:33] of going from the rack to the compute [279:35] die is to have the 51vt rack bus that [279:38] we've already talked about the vertical [279:39] bus and having a 51 to 1 approximately [279:43] ratio DC toDC converter meaning that we [279:46] go from 51 volts and approximately 20 [279:48] amps to 1 volt and 1,000 amps with a 1 [279:51] kow load. This would make us have to [279:54] have the final version beside the die or [279:56] else there'll be way too much resistive [279:58] heat loss and that's why the high [280:00] current stays local. Then you can have [280:02] an intermediate rail though and this is [280:03] actually what's done with the meta chip [280:05] which we'll see in the next slide where [280:06] you have 51 to 12 DC toDC converter. [280:09] This gives you approximately 83 amps in [280:11] this intermediate phase and then we [280:14] convert back to or to the 1V desire [280:18] voltage at the compute die and this [280:20] gives a higher voltage board [280:21] distribution. This also means that the [280:23] regulator choice and layout flexibility [280:26] may be better simply because you can [280:29] split up this conversion into two [280:31] phases. Okay. So, as I promised, we're [280:34] we're going to discuss the Metag GB300 [280:36] tray which actually has a complete [280:38] specification. So, if any of you guys [280:40] are interested, the open compute project [280:43] talks about this tray and this was made [280:45] uh last year in September in fact and [280:47] you can see all of the people here are [280:49] just meta engineers. So really good [280:52] reading if you would like to have some [280:54] extra uh homework after this course. And [280:56] what they do on their board at a high [280:58] level is what we just explained here in [281:00] the intermediate rail phase. So they go [281:02] from 51 to 12 to approximately 1. And [281:07] this furthermore has a allowance for [281:10] between 46 and 52 volts at the power [281:15] distribution board. So the power [281:16] distribution board is somewhere on the [281:18] board level. So essentially this is what [281:20] would happen when you get from the rack [281:22] to the board and there you go. So that's [281:25] a real life example of how local [281:27] regulation occurs to get to one volt at [281:29] the process level. Okay. So now let's [281:33] talk about the VRM not just as a [281:36] blackbox but like actually see what are [281:38] they doing in practice. So where can I [281:40] put myself to not block this slide? [281:43] Okay, there you go. So sharing current [281:44] inside a VRM as I mentioned these VRMs [281:49] they have these chokes and they have [281:51] many of them based on the power [281:55] hunginess of the compute that you're [281:57] serving and these chokes roughly [282:00] correspond to phases of current and in [282:06] the industry you have like multiple [282:08] phases per per VRM or rather you have [282:12] multiple parallel phases per voltage [282:13] regulator module. So if we consider the [282:16] extreme case which you would really [282:17] never use where you only have one phase [282:19] of of current in a in the VRM then what [282:23] happens is that although let's say you [282:26] have a 1,00 amp average since it's 1,000 [282:28] kil sorry 1,000 watts the the amperage [282:31] itself would be very volatile and this [282:33] is not good for the chip which expects a [282:35] constant voltage and if you if you [282:37] modify the amperage then also your [282:38] voltage will change because of P= IV and [282:41] then instead what's done is that you [282:43] have multiple paralle phases in a VRM. [282:46] So in this case we have 250 amp phases [282:48] which of of which there are four which [282:50] add constructively to 1,00 amps. And if [282:54] you interle these then what you can do [282:56] is stabilize the current and then [282:59] furthermore have a steady voltage on [283:02] your downstream compute dot [283:05] at at a at an even lower level. How this [283:07] works is okay, you have a voltage [283:10] regulator model which is actually just [283:12] called a buck converter which if you'd [283:13] like to search keep uh just just come to [283:16] this link here and you'll get a really [283:18] good rundown of how that works. In [283:20] essence, you have two uh transistors [283:24] here as well as inductors and these [283:26] switch at really high rates in order to [283:29] get this voltage differential. So in [283:31] this case we would be going from 12 to 3 [283:33] volts. That would be a 4:1 ratio. Um and [283:37] if we look on the right here we have the [283:39] switch node voltage which just says what [283:42] is the voltage applied at every switch [283:44] um as a function of time. And then [283:46] furthermore we get the inductor current [283:50] here. So the inductors are what resists [283:53] current. So I misspoke. I said the [283:55] inductor resists current but it's the [283:57] opposite. The inductor resists changes [283:59] in current. How does it do this? Well, a [284:02] an inductor stores a magnetic field and [284:05] when the current changes, the inductor [284:08] flips its voltage polarity to force the [284:11] current to keep moving by generating a [284:13] back voltage. So all of that is to say [284:16] that the inductor resists changes in [284:18] current and this is what's used to aid [284:20] in the voltage conversion even when the [284:23] switch is open and this allows current [284:26] to flow in the circuit. And what this [284:28] means is that when we interle these two [284:31] currents, we get the uh the the the [284:34] normalized or rather the smoothed [284:36] current where there's less spikes. And [284:39] once again, this allows us to buffer the [284:42] power that the load requires, which is [284:45] the chip itself. And the more the phases [284:49] you add, the more the current and the [284:53] voltage are essentially stabilized. So [284:55] if we look here uh we have a bunch of [284:58] illustrations more phases share the load [285:00] and interle them reduces ripple. Um it [285:03] really depends on multiple factors how [285:05] many phases you can have. But these are [285:06] just examples. In general though like [285:09] the more power hungry and sophisticated [285:10] your workload you'll likely have more [285:12] phases which is once again why the GPU [285:14] had more of these chokes which are the [285:16] inductors specifically uh than the CPU [285:19] in the GB300 board. Okay. So we've been [285:22] talking like pretty low level. Let's [285:24] let's jump back up to a higher level [285:26] overview of how we power the system and [285:30] in particular let's talk about how [285:32] backup power plays a role in these data [285:35] centers. So at a facility level we have [285:38] the grid which can be supplemented both [285:41] by a generator and a bees which stands [285:45] for a battery energy storage system. So [285:47] if we think back to the Cruso [285:50] Sparks center which had a 12 megawatt [285:53] solar panel grid with 63 megatt hours of [285:56] batteries that would be the battery [285:58] energy battery energy uh storage system [286:01] and these both feed onto the main AC bus [286:04] which would probably be in a medium [286:06] voltage uh setting so that you can [286:08] balance essentially resistive heat loss [286:10] with power density. And then you have [286:13] your uninterruptible power supply which [286:15] runs through the the AC bus in order to [286:18] allow a battery on the UPS to take the [286:22] load if ever there is a supply [286:24] disruption. So that we have the [286:26] rectifier inverter and then now we go [286:29] from the broader campus to the building [286:32] to the rack. So once at the rack level [286:34] we have the rack level PSU which we've [286:36] discussed in detail that goes now from [286:38] AC to DC for good. And then what you [286:40] might have are some BBUS. So these [286:43] stands for battery backup units. And [286:46] these would be like coll-located beside [286:48] the rack. So you might have one rack [286:49] here and then a second rack with some [286:52] battery backup units. They also could be [286:54] on the rack itself. Once again depends [286:56] on the design choice, but these feed [286:58] onto the rack DC bus. And then this is [287:01] an even more local way to buffer power [287:05] shortages or power uh overages. So you [287:08] can think of it as like just a local [287:10] version of what the UPS batteries are [287:12] doing and a faster latency version [287:14] because the farther you wait, the [287:16] farther away you are from the compute, [287:18] the longer the delay is for some type of [287:20] system to kick in. Like the most extreme [287:22] example would be a generator which takes [287:23] multiple minutes to kick in after a [287:25] supply of power disruption. [287:28] And then if we zoom in all the way to [287:30] the chip, which which is what we were [287:31] just talking about, right? We have the [287:33] capacitors which are at the same level [287:35] of the VRM that through this Vcore rail, [287:38] this voltage core rail supply power all [287:40] the way to the chip and make the GPUs go [287:42] burr. Okay, so let's talk about the BBUS [287:46] for a sec and see where they come into [287:49] play. In particular, let's say that [287:52] we're doing a compute load and maybe [287:54] somebody just like sent in a new request [287:57] on a really heavy ML workload that makes [288:00] the load rise by 40 kow. So the thing is [288:04] the power supply units although fast are [288:07] non instant. So let's assume that the [288:09] power supply units take a linear amount [288:12] of time to increase by 40 kW in 0.2 [288:16] seconds. So what that means is that for [288:19] this triangle of time there actually is [288:22] a deficit in power which needs to be s [288:26] uh which needs to be provided by. So [288:28] this is what the battery backup unit [288:30] does. We can assume that the battery [288:31] backup unit is much faster than the PSU [288:33] and can instantly supply this load. And [288:35] if we just simply calculate the amount [288:37] of energy this would require since it's [288:39] a triangle we first multiply by half and [288:41] then we multiply by 0.2 seconds and then [288:44] the y- axis here would just be 40 kow. [288:46] So if you multiply these out, you get 4 [288:47] kg required of energy from the battery [288:50] backup unit to make up for this load [288:52] rise. And if we look at the top here, [288:55] both the power supply unit and the [288:57] battery backup unit are putting power [289:00] onto the vertical rack DC bar. And if we [289:05] consider another example where the PSU [289:09] is actually twice as slow, then we have [289:12] twice the energy need. So 0.4 instead of [289:14] 0.2 2 seconds being multiplied here [289:16] gives us 8 8 kg required from the [289:19] battery backup unit. And then as you can [289:20] imagine once we have the power supply [289:23] unit giving the energy it requires for [289:26] the load then the battery backup unit [289:28] would taper off and stop powering the [289:30] bus bar. Okay. So just as we can have a [289:35] load deficit, we can also have a load [289:37] surplus. So let's say for example you're [289:39] doing like a heavy matrix multiplication [289:41] and then now the GPUs stop to all reduce [289:45] or essentially share their weights and [289:47] update their parameters such that they [289:49] are in sync. In that case you'll [289:51] definitely get a decrease in load. So if [289:53] we consider the exact same 40 kowatt [289:56] drop this time um if you had the uh the [290:01] PSUs taking 0.2 2 seconds to stop giving [290:04] that 40 kW access. Then now instead of [290:08] uh like discharging 4 kg, the battery [290:10] accepts 4 kg to prevent the bus from [290:14] increasing in voltage because [290:17] essentially if you have excess power on [290:18] this DC bus. What that means is that [290:20] your voltage will go above the 50 50 [290:23] volt rating or whatever the rating is of [290:25] the of the uh uh the the bus just [290:28] because of P equals IV our favorite [290:30] formula. And if the PSU takes twice as [290:32] long to slow down, then we get twice as [290:35] much energy required to absorb. With [290:37] that being said, if the battery is full [290:40] or is for some reason charge limited, [290:42] then the bus voltage will rise. There's [290:44] no other choice. The energy has to go [290:46] somewhere and it essentially just goes [290:48] into increasing the the potential [290:50] difference of the voltage on this [290:52] vertical bus. And in practice, that [290:54] could mean going into some capacitor. [290:56] Okay, so we've discussed the battery [290:58] backup units in the abstract. Let's now [291:00] look at what they are. So, they look [291:02] quite similar to the power supply units. [291:04] And I would assume this is not by [291:06] accident. You want to have these [291:07] components standardized and fitting into [291:10] the same trays so that you can not have [291:12] to make a custom tray for every single [291:14] part. And in this specific example, we [291:16] have delta BBUS which are 3 kW and [291:19] output 48 volt DC. Interestingly, this [291:23] is like a onetoone correspondence with [291:25] the power supply units that we discussed [291:26] previously, which outputed 3 kW uh from [291:30] AC to DC. These ones just directly [291:31] output DC since that's how batteries [291:33] work. The one thing worth noting here, [291:35] though, is that this system is only [291:36] rated for 15 kW, which likely means that [291:38] you have N plus1 redundancy. So, at any [291:40] given moment, maybe only five of the [291:42] batteries are on, or maybe the six [291:44] batteries are on, but they're not [291:45] running at peak uh power. That might be [291:47] the the the best example here. And what [291:49] this means is that yeah, you just have [291:51] one level of redundancy as opposed to [291:53] outputting 18 kW where as soon as one of [291:56] the battery units uh has a error then [291:58] you are below your output uh rating. So [292:02] the last thing here to note is that when [292:04] our batteries are recharging between [292:06] different cycles if we don't give it [292:08] enough time then over time the battery [292:10] could completely discharge. So if we [292:12] first look at the right side here we [292:14] have a discharge of 40 kW in 0.2 2 [292:16] seconds leaving a 8 kg deficit. And [292:19] let's assume that the battery only has [292:22] five sorry half a second to charge back [292:25] 10 kow. And in this assumption we assume [292:27] that the discharge speed or rather power [292:29] rate is four times the charge rate. In [292:32] this case if this occurred repetitively [292:34] then you might be in a situation where [292:35] your battery is not charged which is not [292:38] desirable when you actually need the the [292:39] battery to support some excess load. And [292:43] then on the left uh we have a situation [292:46] where you do have enough time. So if you [292:48] have a 4:1 [292:50] uh ratio between discharge and charge, [292:53] you need at least four times as much [292:55] time in order to recharge the battery [292:58] just because of energy equals power [293:00] times time. Okay, so hopefully by now [293:03] you have some intuitions on the [293:05] trade-offs between having a higher [293:07] voltage and less current which gives [293:10] better power density and the contrary [293:12] which in most cases is not practical but [293:17] also helps in terms of having lower [293:20] voltage which is safer overall. And with [293:24] this trade-off in mind, in this chapter, [293:26] we're going to discuss how 800 volt [293:28] direct current can reduce copper and [293:30] free compute rack space with increases [293:32] in density of these racks. And what I [293:36] really want you to have in your mind [293:37] right now is that graph we saw from the [293:39] previous chapter which demonstrated [293:41] racks reaching the 1 megawatt level as [293:44] soon as a few years from now. In [293:46] particular, in this case, we're going to [293:48] discuss throughout this uh chapter 100 [293:50] kilowatts of energy or rather of power [293:53] and how this differs in supplying it in [293:57] alternating current versus direct [293:59] current. So, let's jump right into it. [294:02] First of all, we can already say that [294:05] 1/3 less copper is required when [294:09] converting or rather supplying power and [294:12] current through direct current. Why is [294:14] this? It's because if you're using 400 [294:16] volt three-phase alternating current, [294:18] you need three phases which requires [294:20] three equal copper legs. And then if [294:22] you're using direct current, direct [294:24] current just has a positive and a [294:26] negative conductor. And so there's just [294:28] like one less conductor that you require [294:30] in order to deliver the same 100 kW. [294:33] You might be asking, okay, but maybe the [294:36] current is different. So actually this [294:37] is a poor representation or a poor [294:40] comparison. But in fact the same 100 [294:43] kilo the same 100 kilowatts needs nearly [294:45] the same current in each conductor for [294:48] 400 volt 3-phase AC versus 800 volt DC [294:51] assuming for example the power phase is [294:54] one for simplicity. So let's just reveal [294:56] the currents for 480 volt 3-phase AC you [294:59] need 120.3 amps and then in 800 volt DC [295:02] you need 125 amps. Um we can also [295:05] modulate this. So if you want to get [295:06] exactly the same amount of amperage, you [295:08] need to increase the the voltage [295:09] slightly to like 829 or something. But [295:12] uh without loss of generality, 800 volt [295:14] DC for the same current uses a third [295:18] less of the conductors. And then because [295:20] it uses a third less of the conductors, [295:23] it also has almost a third less heat [295:27] loss in in this comparison. So let's [295:29] assume 10 micro ohms or 10 milliohms [295:33] sorry per conductor of resistance and [295:35] the same power efficiency of one [295:36] assumption. If we calculate this for the [295:40] three-phase AC using the uh the [295:42] resistive heat loss for alternating [295:43] current formula we get 434 watts of heat [295:46] in resistive heat loss. And then in [295:49] contrary we get 312.5 watts of heat loss [295:52] 800 volt DC. So already you can see that [295:55] we need less copper and we lose less [295:57] heat because there's less copper and [295:59] it's just allowing us to supply more [296:02] power in an efficient manner at a high [296:04] level. There's also some cons and some [296:07] pros. So that's what we're going to [296:08] discuss in more and more detail. In [296:10] fact, this road map of converting data [296:13] centers to direct current is going to be [296:17] done over the next few years in [296:19] approximately three phases. So this is [296:21] uh the three phases in one slide but [296:23] we'll talk about each one in separation. [296:25] So first of all what happens is that in [296:28] this current setup and the regime we [296:31] have alternating current entering the [296:33] compute rack and then being converted to [296:36] DC inside it. So this is what's [296:37] happening today. You have the facility [296:38] alternating current from upstream power [296:40] room. Then inside of the data hall [296:42] itself you move you move power in [296:44] alternating current. And then within the [296:45] compute rack you have the power supply [296:47] units that go from AC to DC. And then um [296:49] on the device itself now you're in the [296:52] direct current regime. And so in this [296:54] example we are just talking about what [296:57] is happening today. And so this AC toDC [297:00] equipment space which is being occupied [297:02] by the power supply units is taking up [297:04] space. Uh and you can you can see in the [297:07] next slide that will actually be freed. [297:09] So before we jump to that let's see what [297:11] like what are we actually talking about [297:12] in this slide. So now in phase 1/2 which [297:16] will be I guess starting this year but [297:19] extending to 2028 we have a sidec car [297:22] which converts AC to DC beside the [297:25] compute rack. So literally what a sidec [297:27] car is is just like a a rack beside it. [297:29] Uh I don't know why they call it a car. [297:31] I think a better name would be a side [297:32] rack but you can think of it as a rack [297:35] on the side of the main uh compute unit [297:38] which converts converts AC to 800 volt [297:41] DC. So in this case we go from facility [297:44] AC and then it goes into the side car [297:48] which converts to 800 volt DC and then [297:51] now within the rack itself we have [297:53] nothing that's alternating current. This [297:56] may be a this may be a exaggerated [297:57] statement but much less alternating [297:59] current than what was done previously. [298:01] So you only have this DC toDC regulation [298:04] and then we still need to obviously go [298:06] down to the low device voltage on the [298:08] compute rack itself uh so that we can [298:10] supply the chips at their desired input [298:12] range. But now the power supply units [298:14] which took up in the example of the GB [298:17] 300 eight shelves are free. So we can [298:21] either put more compute trays there, put [298:22] more Envy link switches there. The [298:24] compute density is increasing which is [298:26] what you need to take away from this [298:28] chapter at a high level. And then [298:30] finally in this third and final phase [298:33] which is projected to be occurring in [298:35] 2028 to 2029 upstream conversion lets [298:39] 800 VC 800 volt direct current run [298:42] through the data hall. So what do we [298:43] mean here? in some upstream power room, [298:47] you probably go like directly from [298:49] medium voltage alternating current to [298:51] 800 volt direct direct current and then [298:54] within the hall within the data hall [298:56] most of the direct current will be [298:58] distributed in 800 volts or sorry most [298:59] of the current will be distributed in 8 [299:01] 800 volts at least the current that's [299:02] for the compute and then this side [299:05] actually does not change. So if you guys [299:06] remember this uh this final box it is [299:09] not changing at all. In fact, the reason [299:11] why you're adding these side cars is so [299:13] that you can develop like the final [299:15] output of what this transition is trying [299:18] to reach, which is to say a compute [299:19] wrap, a compute rack that is purely in [299:22] direct current. But before we can get to [299:25] this 800 volt DC hollow distribution, [299:27] which you can imagine will require a lot [299:29] of new parts, a lot of new changes, [299:31] we'll just have this temporary phase [299:34] where the side cars are kind of acting [299:36] as this quasi data hall distribution [299:38] before that's actually built up. But the [299:41] thing to retain from this is that in the [299:44] second and third phases, the compute [299:45] rack is actually constant. Okay, so one [299:49] thing that we did not raise uh in the [299:51] last three phases is like when are we [299:54] actually stepping down from 800 volt DC [299:57] to the lower voltages. So like for [300:00] example maybe we'll go from 800 to 50 in [300:02] order to be uh able to convert down to a [300:06] lesser voltage. So once again we talked [300:08] about all roads lead to Rome with [300:10] respect to voltage conversion. We'll con [300:13] we'll consider two two examples here in [300:15] in this uh mini case study. So let's say [300:18] that you step down at the rack entrance. [300:21] So what do I mean? Let's say that you're [300:23] inputting 800 volts direct current into [300:25] the rack and that once you get on the [300:27] vertical bus bar before actually [300:30] entering this bus bar, you do a DCTDC [300:33] converter such that the voltage of this [300:36] vertical bus bar is 50 volts and then [300:38] afterwards you just have the device [300:40] conversion at the tray/board and then [300:41] the rails uh for the chip voltage itself [300:44] would be in the one voltage range. [300:46] Conversely, what you could do is step [300:49] down at the trays, which is to say that [300:51] make the vertical bus bar 800 volts and [300:54] now at the tray convert with some [300:57] voltage ratio there and then once again [300:59] at the voltage regulator modules which [301:01] gives you the same voltage at the end. [301:03] So these are two options of how you can [301:07] step down 800 volts with the main [301:09] difference being in one option you have [301:11] a 50volt vertical bus bar and another [301:13] option you have a 800volt vertical bus [301:14] bar. In this case, the good thing is [301:16] that you are going to have a better [301:20] power density or rather less resistive [301:22] heat losses because you're keeping the [301:23] voltage higher for longer. But in the [301:26] top example, you have more compatibility [301:27] because there's already much more [301:29] support for racks which have a vertical [301:32] bus bar in 50 volts for the power [301:34] distribution. [301:36] And then another thing which is a [301:38] bifurcation is saying when we go to 800 [301:42] volts do we step down first or rectify [301:45] first because there are two ways to [301:47] reach 800 volt uh DC from medium voltage [301:50] alternating current. What I mean is like [301:52] okay let's say that you have MVAC let's [301:55] say that you have 10,000 volts [301:57] alternating current. Either you could [301:59] rectify that to direct current and then [302:02] step down to 800 volt uh direct current [302:05] or you could first step down the 10,000 [302:08] volts AC to like 800 volt AC and then [302:12] rectify to um direct current. So the one [302:17] that I just explained is the top where [302:20] you go from medium voltage AC you [302:22] transform to low voltage AC then you go [302:25] through AC to DC converter to rectify [302:27] and regulate to 800 volt AC. This is [302:29] noted as the conventional path. Why is [302:31] that why is that the case? It's because [302:33] it uses established transformers and [302:35] lower voltage power techn power [302:37] electronics which are much more [302:38] developed and established. [302:41] But in the second in the second example [302:43] here, we rectify first or we can [302:45] actually do this in one shot. So we can [302:47] both rectify and perform this um uh [302:52] voltage step down if we use what's [302:54] called a solid state transformer. So [302:56] conventional transformers uses like [302:58] massive iron core and magnetic coils to [303:01] step down voltage or step up voltage. [303:03] And solid state transformers are [303:04] different because they use actually like [303:06] a bunch of electronics and [303:07] semiconductors. It's much more [303:09] sophisticated but actually is smaller [303:10] than a traditional transformer. And [303:12] because of this, it is like more tunable [303:16] and able to be uh integrated in [303:19] different electronic paths. So to go [303:21] through this stage here inside of the [303:23] solid state transformer, we go to medium [303:26] voltage AC to rectify it and then have [303:28] this internal DC link. And then once [303:30] we're there, what you would do is do [303:33] this high frequency transforming with [303:35] switching to convert to 800 volt DC. [303:38] This is not a solid state transformer [303:40] course. These are quite uh cool and [303:42] sophisticated systems. There is a really [303:44] good resource on this by professors at [303:46] the ETHZurich in Switzerland. So I would [303:49] just recommend searching up solid state [303:51] transformers data centers ETHZurich if [303:54] you're interested in learning more about [303:56] solid state transformers. But the simple [303:58] thing to note here is that it integrates [303:59] the functions in a medium voltage [304:01] electronic system such that both the [304:04] rectification and the voltage step down [304:08] happen in one system. One thing worth [304:10] noting is that there are these new high [304:14] voltage silicon carbide devices that [304:17] could make solid state transformers [304:19] simpler and more competitive than they [304:21] are today. However, their widespread [304:23] adoption in data centers still has to be [304:24] demonstrated. And another thing worth [304:26] noting is that like the first ever solid [304:29] state transformers that were put into [304:31] widescale use were in Switzerland as [304:33] well and that was I think around 2012. [304:35] So the problem here is that there's only [304:37] like 14 years of data of how these [304:40] things perform and when you're doing [304:42] like a cost uh cost trade-off analysis [304:45] for a data center that's like not a lot [304:47] of data for informing choices and also [304:50] the solid state transformer made in 2012 [304:52] is not the same one as today. Whereas [304:54] with conventional transformers, they're [304:56] tried and tested and everybody knows how [304:58] they work. So once again, the bottom [305:00] path has more variance on like will it [305:02] work, will it not work, but could [305:04] potentially make them much more simpler [305:07] and competitive compared to the [305:09] conventional process. [305:11] Okay. And to talk about a very early [305:15] data center that moved to DC, this is [305:18] perhaps one of the earliest of them all. [305:21] So Zurich West, which is a data center [305:23] in Zurich in Switzerland, moved [305:26] rectification upstream in 2012 by having [305:29] essentially a 1 megawatt DC system that [305:32] went from 16 kovt AC input to transform [305:36] that into 300 volt DC distribution at [305:40] this central conversion unit which first [305:42] step down the voltage of the alternating [305:45] current and then rectified it from AC to [305:47] DC which is the conventional path and [305:50] then would pass through this 300VT DC [305:52] distribution to the servers and storage. [305:54] You can see here though that this is [305:56] like a non-compatible unit since this [305:58] was like probably a bespoke system. They [306:01] chose their own direct current [306:02] distribution voltage. In any future [306:05] system, the open compute project which [306:08] is what what essentially uh sets the [306:10] standards for these voltages and power [306:13] requirements. they have settled on 800 [306:15] volts as the direct current voltage [306:18] which will be used in future data hall [306:20] distribution systems. [306:23] Okay. So one thing that is new when you [306:27] change much of the feeding of the [306:30] electricity to direct current is it now [306:32] needs DC rated protection. So what am I [306:36] talking about here? Here we just have [306:38] some [306:40] sample circuit and first of all let's [306:44] establish that both AC and DC currents [306:46] need fault protection. This is not [306:48] something that only DC needs. But the [306:50] thing is with alternating current it [306:53] naturally crosses zero twice period [306:56] helping a breaker which is right here [306:58] distinguish an arc after the contacts [307:01] open. We mentioned this briefly, but [307:03] since DC has no periodic natural zero [307:07] polarity, the protection must instead [307:09] force the current to stop. So, so in [307:12] this case, both the rectifier, the [307:14] capacitor and this inductor can feed a [307:18] short circuit. So, the breaker must [307:20] interrupt that DC fault and withstand [307:23] the voltage afterward. In fact, zero [307:25] current does not mean zero voltage, [307:27] right? Because if you have a pipe system [307:29] which is closed off, you can still have [307:31] pressure from the mentioned the example [307:33] we mentioned in the primer. So in short [307:35] when it comes to direct current [307:38] disconnecting the supply alone does not [307:40] remove the stored energy from the [307:41] circuit. There are some similarities and [307:43] differences with this as well as [307:45] alternating current. But overall the [307:47] challenges posed with direct current are [307:49] larger and require adaptation from the [307:52] current set of standards for redundancy [307:56] and current breaking that are being used [307:59] in alternating current systems. [308:01] All right. So let's quickly discuss a [308:05] example of migrating from alternating [308:08] current to direct current through these [308:11] side car or these side racks. So let's [308:14] assume that we have a existing [308:16] alternating current allocation of 240 kW [308:20] and that this side car that we add is [308:22] 96% [308:24] efficient [308:26] with 3 kW of upstream auxiliaries which [308:30] just means that we also need to supply 3 [308:32] kW to some other like fans or whatever [308:34] they may be and that we have two [308:38] 120 kW racks that are in direct current [308:42] or essentially accept power and direct [308:44] current just like most racks. So if we [308:48] just run through the calculation here, [308:50] what we do is first of all multiply 2 by [308:53] 120 to get 240 and then divide this by [308:56] 0.96 because that's how much heat we [308:58] lose or rather we need to multiply this [309:01] value slightly to account for the amount [309:04] of heat we lost and then also add 3 kW. [309:07] So in this calculation we get a required [309:11] AC allocation of 253 kW which is 13 kW [309:15] more than allocated meaning that in this [309:17] system the existing AC allocation is [309:19] insufficient. So either you would [309:21] increase the AC allocation or another [309:24] option is like somehow reduce the power [309:27] of the racks. So maybe this just means [309:29] like remove one GPU from the racks each. [309:31] Of course this is kind of contrived but [309:32] in any case if you actually did do this [309:34] you would act you would get 7.83 83 kW [309:37] uh of capacity remaining since the [309:39] required AC input would now be 232.17 [309:43] kW. So there's a cool example which is [309:45] probably a similar scenario to what data [309:50] centers will have to face when they're [309:51] calculating what they need to do in [309:53] terms of integrating these side cars in [309:55] their data centers. And now let's [309:57] discuss one final time the three phases [310:00] of upstream rectification that will be [310:03] used to replace AC distribution in the [310:05] hall. So in fact the current phase is [310:08] just what's happening today. So I should [310:10] have rephrased as let's discuss today [310:12] and what's happening in the future. Once [310:14] you get the DC side car you'll go from [310:17] medium voltage AC to step down [310:19] transformers to step down to lower [310:22] voltage AC. And then you have this AC [310:24] switchboard which essentially will [310:26] distribute the alternating current to [310:27] the various feeders and the various [310:28] buildings and the various uh racks what [310:31] whatever they may be with these power [310:33] distribution units. And then where you [310:35] distribute where you're sorry where [310:37] you're becoming uh diverging from this [310:40] first uh diagram is that now the power [310:42] distribution units are going to these [310:45] side cars which convert to 800 volt DC [310:48] and then the 800volt DC is going to the [310:50] racks themselves as opposed to the power [310:53] distribution units going to AC going to [310:56] the power supply units [310:59] as opposed to the power distribution [311:01] units giving AC to the racks giving that [311:03] AC to the power supply units which are [311:05] then finally going to direct current and [311:08] then in the data hall level direct [311:11] current or direct medium voltage DC [311:14] scenario we go from medium voltage [311:16] alternating current and then we can use [311:19] a medium voltage rectifier or a solid [311:22] state transformer to get to 800 voltage [311:24] DC in our distribution hall of the data [311:27] center and then this will go on a on a [311:29] busway above the racks that's also 800 [311:31] volt DC and then it will also go at 800 [311:34] volts within the racks. So when you keep [311:38] all of this transmission at 800 volts, [311:41] you can imagine you just need much less [311:43] like rectifiers, much less uh [311:45] transformers. It just vastly simplifies [311:48] the system and then furthermore allows [311:50] us to pass energy uh in a more efficient [311:54] manner as opposed to 400 volt alternate [311:57] current. given what we established at [312:00] the beginning of the chapter where we [312:03] talked about the conductors being like a [312:06] third less required as well as the heat [312:07] losses also improving. [312:10] Okay, so in the last two slides, we've [312:14] essentially gone over the AI data center [312:16] power stack. And this is just a figure [312:18] which kind of goes from layer 1 to the [312:21] endpoint, which is the Nvidia GPU die, [312:23] and talks about some of the different [312:25] companies that are involved in each of [312:27] these sectors. So if you're had if [312:30] you've had like a specific interest to [312:32] any of the specific uh sections of these [312:34] chapters then you may want to check out [312:36] companies like Hitachi Energy, Genosa, [312:38] Genova etc. So just quickly going [312:41] through these layers at the grid and [312:43] substation infrastructure level we have [312:45] high voltage transformers we have switch [312:46] gear which once again kind of [312:48] distributes the voltage and also uses [312:50] circuits and breakers to secure the [312:52] power. Then we have the substations. [312:54] Then you have high voltage DC [312:55] interconnects which may be used if [312:57] you're actually transmitting energy in [312:59] direct current. That is the case in some [313:02] specific scenarios but mostly it's [313:04] alternating current. And then if we look [313:06] at the lead times here, we're talking [313:08] about 4year transformer lead times for [313:10] specific high voltage transformer and [313:12] switch gear which uh is just a evidence [313:15] to the immense scarcity of this type of [313:19] energy infrastructure. [313:21] Then in the second layer, we're at the [313:22] data center and power distribution [313:25] level. So we have these power [313:26] distribution units. We have [313:27] uninterruptible power supplies. We have [313:29] busways. We have high voltage direct [313:31] current distribution which will be more [313:34] and more important as we move on in the [313:36] future cuz right now HBDC is not really [313:39] used in that many data centers. And then [313:41] of course you have cooling distribution. [313:43] And then you have critical power which [313:44] is to say like battery backup systems [313:46] and other ways to ensure that the power [313:48] is sustained whenever there is a utility [313:51] or grid issue. Okay. Now going one level [313:55] deeper at the rack level/ power level. [313:57] So here we're talking about the power [313:59] supply units, the shelves themselves, [314:01] the high voltage DC rack conversion. So [314:03] this would be the side cars in the 800vt [314:06] transition. And then furthermore, you [314:08] have these new PSUs that advertise [314:11] between 18.3 and 30 kW per unit, which [314:16] if you remember is like 10 times more [314:19] power than the 3 kW units that we talked [314:21] about previously. And just to be [314:23] specific here, this is Delta [314:25] Electronics, which is advertising these [314:28] levels of power per unit with the 800vt [314:32] DC transition in the future. Moving on, [314:36] we have power semiconductors, which is [314:38] not really a layer in it itself, but is [314:40] more so the different materials that you [314:43] will be will be used to allow 800 volt [314:46] DC to be just even more efficient. So [314:49] for example, you have these silicon [314:52] carbide metal oxide semiconductor field [314:55] effect transistors which are essentially [314:58] just these transistors which are used in [315:01] the solid state transformers. And then [315:03] similarly you have these gallium nitride [315:05] field effect transformers which can be [315:08] used to replace silicon sorry not [315:10] transformers but transistors um which [315:12] can be used to replace silicon [315:14] transistors at apparently 25 to 40% [315:17] fewer losses but you should definitely [315:19] caveat this because these specific [315:22] technologies are much less tested than [315:24] the conventional transformer techniques [315:27] and this is in the context of once again [315:29] 800vt rectification. So some companies [315:32] here that are worth noting are Infinion, [315:34] Wolf Speed and SC Micro in particular. [315:37] And then at the final layer we have the [315:39] point of load which is to say the actual [315:40] GPU die and the chip level voltage [315:42] regulator module. So that's what we just [315:45] discussed uh in this course or in this [315:47] chapter. But then you also have the [315:50] integrated voltage regulators which are [315:52] essentially a more intelligent voltage [315:55] regulator module which can be used to [315:57] replace that component in the future. [316:00] And then you also have the power [316:02] management integrated circuit if I'm not [316:04] mistaken which is just another way of [316:06] managing managing the power that you [316:08] feed at the circuit level. And here [316:11] there is also a few companies such as [316:14] Vicer Texas Instruments monolithic power [316:17] systems which you can check out if [316:19] you're interested. Okay. So in chapter [316:22] 10 we'll discuss networking and [316:24] interconnects which is one of my [316:25] favorite topics here. In fact, it's kind [316:28] of along the same hierarchy scheme of [316:31] rack to compute to GPU as power, except [316:34] this time instead of electrons, we're [316:38] talking about like data moving through [316:40] optical fiber and copper in order to [316:43] communicate between these different [316:44] devices. So, let's start off this [316:47] chapter with a little meme to uh lighten [316:50] up the mood. So, I mean networking and [316:53] interconnect of course requires lots of [316:55] memory. So, as you guys can see here, [316:57] the AI [317:00] Grim Reaper has already actually got rid [317:02] of all the RAM, GPU, and SSD and made it [317:05] hard for consumers to buy it. Next is [317:07] actually CPUs because of reinforcement [317:09] learning, but that's something [317:10] completely unrelated to this. Uh, but [317:12] there you go. We'll start it off with a [317:13] nice little meme. Hope you enjoyed that. [317:15] And now, let's actually get on topic. [317:17] So, a distributed model uses different [317:19] networks at different scales as we [317:21] looked at like the the first image [317:23] there. So the the first thing we'll talk [317:26] about here is a scale up which we [317:27] already know about. Uh let's talk about [317:29] a specific example. So let's say that [317:31] you have a GB200 NBL72. In that case you [317:34] can run a Kimmy K3 instance which is a [317:38] few par a few trillion parameters. I [317:39] think it's 2 point something trillion [317:40] parameters Kimmy K3 is. And these GPUs [317:44] 72 will communicate with Envy link which [317:47] is an alltoall communication platform or [317:50] paradigm which is to say that every GPU [317:52] has a connection that is zero steps away [317:54] so direct connection to one another. And [317:57] then when it's on a scaleout domain this [317:59] is more relevant in terms of uh training [318:01] and pre-training in particular. So, [318:03] we're talking about going between [318:05] different racks. And an early example of [318:08] this was Llama 3.145B, [318:11] which trained on more than 16,000 H100s [318:14] in 2024. And this was within a data [318:17] center. And then outside of a data [318:19] center, we're talking about across [318:21] campuses and regions. And I really want [318:24] to highlight for a second Fairwater, [318:26] which is Microsoft's platform for [318:29] connecting different data centers. In [318:30] fact, um the two examples here that [318:32] we'll highlight in the next slide is [318:34] their major data centers in Atlanta and [318:37] in Wisconsin. So, let's just go straight [318:39] ahead and I'm going to zoom in first. [318:41] So, these are like their their flagship [318:43] data centers. So, you can see that it's [318:45] going from uh Georgia to Wisconsin and I [318:47] think it's in Atlanta. I don't know [318:48] where the specific data center in [318:49] Wisconsin is. And it uses something [318:51] called AI wide access network, AI WAN. [318:55] And what this is is essentially [318:58] Microsoft's own [319:01] fiber network which allows them to [319:04] communicate between these two data [319:06] centers in milliseconds which is also [319:08] still possible on the regular internet. [319:10] But the improvement here is that they [319:12] can save a bit of latency by not having [319:14] to have any cues or like essentially [319:16] compete with other traffic on the [319:18] internet. It's literally just these two [319:20] data centers which are communicating [319:22] with Microsoft owned or rented um [319:26] interconnect that is just laid in the [319:28] ground. And another thing I want to note [319:30] is the fact that this these two data [319:33] centers are actually two floors. And [319:35] this is quite unheard of like most data [319:36] centers are one floor in length uh just [319:39] or in height rather to simplify things. [319:41] And the reason why they did this is just [319:43] to improve uh density of compute because [319:45] you can like have twice as much [319:46] essentially if you have two floors. But [319:48] if you go and like read about the fair [319:50] water buildout, they have some [319:52] interesting observations and notes on [319:54] like well what are the bottlenecks and [319:56] obstacles they had to overcome. For [319:58] example, with just like supporting the [319:59] weight of all these 1.5 ton racks on a [320:02] second floor to make sure that they [320:03] don't fall on the first floor. So I'd [320:05] recommend um if you're interested to go [320:07] search up the Microsoft um fairwater [320:10] architecture to see how they actually [320:12] set this up themselves. So that's the [320:15] data center to data center level of [320:18] communication. And two quotations that I [320:20] do want to highlight from their blog [320:22] post is according to them it's it can [320:24] support in the future training models [320:26] with hundreds of trillions of [320:27] parameters. So we're not there yet. And [320:29] um I think by the time we are at a [320:31] hundred of trillion of parameter model [320:34] uh regime that like this data center [320:36] will probably be like obsolete. So I'm [320:38] not sure to what extent this statement [320:40] is actually like reliable. But the other [320:43] statement which is more pertinent here [320:45] is the dedicated fiber between the sites [320:47] which is to say that uh according to [320:49] them it allows the data to travel [320:50] congestion free which is the important [320:51] thing at nearly the speed of light. So [320:53] of course the speed of light is at its [320:55] maximum inside of a vacuum otherwise [320:57] it's slower than the the the actual like [321:00] value for C which you see in all the [321:02] formulas. But in any case the [321:04] congestionfree aspect of this network is [321:06] quite compelling and you also see this [321:08] with like Google TPU data centers as [321:10] well. So within the GPU rack and across [321:14] rack hierarchy, what's the actual like [321:17] memory layout? So on a GPU, it's very [321:20] simple. You just go from GPU HPM to the [321:23] compute unit. So if you're doing uh [321:25] compute on a Nvidia GPU, you'll be using [321:28] the tensor cores to actually do the math [321:30] and then loading in the weights and the [321:31] activations etc. as well as the KV cache [321:33] from the HPM. And then with respect to [321:36] within the rack, uh we already actually [321:38] discussed this. You're going between the [321:39] NVLink switches and then across racks, [321:42] you'll probably be going from the GPU to [321:44] some adapter or network interface card. [321:47] So that was the connect x connect x uh 8 [321:50] super nick in the case of the NDL72. [321:53] And then uh you go from this adapter to [321:56] these fabric switches which are optical [321:58] links between racks. And then you go [322:00] from one adapter to the next. go to the [322:02] other GPU's network interface card and [322:04] then you have essentially achieved a [322:07] connection between two different racks [322:09] and you can now pass data between them [322:10] or do whatever you please in terms of [322:12] communication. So [322:15] you might be asking yourself right now [322:16] like why are we talking about data [322:19] center interconnect or rather um model [322:22] and GPU interconnect in a data center [322:25] course and the reason for this is [322:28] hardware software codeesign. So just in [322:31] this example here we have two GPUs which [322:34] are in some sense combining results to [322:37] uh perform some calculation across NVL [322:40] link switches and in this case maybe [322:41] it's a form of parallelism like tensor [322:43] parallelism and let's say that you were [322:46] like naively building a data center and [322:48] we're like completely unaware of the [322:51] levels of interconnect and communication [322:54] that your GPUs would need to [322:56] orchestrate. So maybe you're building a [322:59] data center that's meant for training [323:00] models, pre-training, and you don't [323:03] invest in scale out bandwidth, which is [323:06] to say that within the racks, the chips [323:09] can communicate really fast. Maybe it's [323:10] NDLink, but let's say that you don't [323:12] invest in like the Infiniband, which is [323:15] the Nvidia scale out network such that [323:18] now when it comes to training the [323:20] models, you have like insane memory [323:22] latency and you can't even train them [323:23] fast. and then uh like what's the point [323:25] of having the data center anyways. So [323:28] just as just to like motivate why we [323:30] actually want to focus on the networking [323:33] and like the topologies it's because if [323:35] you don't use this to inform the [323:37] construction of a data center then you [323:39] might regret the final product once it's [323:41] built. So with that, let's actually jump [323:43] into Infinity Band or Infinity Band, [323:46] sorry, which is Nvidia's scale out [323:49] network technology. And these connect X7 [323:54] adapters are what is essentially used to [323:57] communicate between the chips on [323:59] different racks. So this specific [324:02] network interface card which is what you [324:04] can also call this adapter runs at 400 [324:06] gigabytes per second and uses PCIe5 [324:09] which is just to say principal uh [324:11] peripheral component interconnect [324:13] express 5.0. This is what you use just [324:14] to connect a lot of different um devices [324:16] and it's a universal um standard with a [324:19] 16wide bus and then you put these in [324:23] these Quantum to QM9700s. [324:25] The name isn't important, but once [324:27] again, these are what holds the [324:30] individual adapters. And this is the [324:31] Infinity Band, Infinity Band itself. So, [324:34] each one of these leaves, which is what [324:37] we'll call them in the next slide, has [324:38] 32 physical cages. And each cage has two [324:41] of these adapters. So, you have 64 ports [324:44] per Quantum 2, and each one is 400 GB [324:48] per second. And like I called it a leaf, [324:50] right? This is this is because like the [324:52] way that they're actually organized in a [324:55] data center topology is through this [324:58] spine leaf uh sort of arrangement. So [325:02] essentially if you have two DJX H100's [325:05] uh which is a older version Nvidia rack [325:08] then they might be interconnected via [325:12] two leaf uh infinity band nodes and then [325:14] one spine infinity band node such that [325:16] to communicate between the two you would [325:18] jump to the leaf and then go through the [325:20] spine and then get to the other H100 [325:23] through its leaf and this is in a super [325:26] pod specifically. So this is the name of [325:27] this specific H100 architecture. And [325:30] then you might be asking like is this [325:33] the only technology or rather topology [325:35] that we use for communicating between [325:38] different racks? Well, no. So it's [325:41] common but not universal. So here you [325:43] just have a copy of the architecture [325:46] that we saw previously where traffic [325:49] between different leaves crosses the [325:50] spine and these are all called switches [325:53] in general. So like the actual [325:55] infiniband node which performs the [325:58] communication between the servers you'd [326:00] call it a switch but we have other [326:02] systems as well. So Taurus and Dragonfly [326:05] are two types of topologies that are [326:07] used specifically for tensor processing [326:09] units the Google TPUs. So when you talk [326:13] about a Taurus is essentially where you [326:15] have each um each TPU connected to all [326:20] of its neighbors in like this six-way [326:22] configuration. And then Dragonfly is [326:24] also this sophisticated thing where you [326:25] have like these pods which are connected [326:27] to other pods with these really fast uh [326:29] data center interconnect networks. In [326:31] any case, these topologies are like less [326:34] important to know viscerally and just to [326:35] be aware of like them existing. And then [326:38] um you also have different custom [326:40] fabrics which I imagine like if you look [326:41] at the AWS trrenium which is their chip [326:44] perhaps they're using their own [326:45] topology. I'm not sure, but I'm I'm [326:47] certain that this information is [326:49] publicly and readily available if you're [326:51] interested to find out for yourself. [326:53] Okay. So, when it comes to the actual [326:57] substrate, like where or like what is [326:59] the information flowing through, there's [327:01] usually two prime candidates. So, you [327:03] have copper and then you have optical [327:05] fiber, which is like to essentially say [327:07] um you're moving through light or you're [327:09] moving through like electricity, which [327:10] is um copper is electric and then [327:12] optical fiber is based on light. Um here [327:16] E to O just means like you're going from [327:17] the electric domain to the optic domain [327:19] because essentially what I'm trying to [327:21] de uh develop in this slide is to say [327:23] that when you use copper it's usually [327:25] over short shorter distances and then [327:28] light is over longer ones. So why is [327:31] this the case? Well copper loses and [327:35] distorts electrical signals. So the [327:39] reason that we actually have up to 2 m [327:42] long coffer in this specific 400G linkx [327:45] cable which is Nvidia's uh cabling for [327:49] connecting these uh highspeed scale out [327:51] domains is because beyond 2 meters you [327:53] get into these like distortion and [327:55] signal losses. So what you do is you [327:57] have these short passive copper links [327:59] that are simple and use a little little [328:01] power but then you switch to the fiber [328:04] which uses these modules at both ends to [328:06] convert electricity to light and back. [328:09] And we've talked in this slide [328:11] specifically we're talking about a 2 m [328:13] passive copper cable as one option for [328:15] moving data across two different ports. [328:18] And then in optical fiber there's [328:19] actually two different uh options here. [328:21] So we have up to 50 m for this multi [328:24] mode optics which means that there's [328:26] like multiple light signals to pass [328:28] information and then we have up to 500 m [328:32] for single mode DR for optics and in [328:36] this case it's just like one beam of [328:38] light passes information which you can [328:40] imagine reduces the latency but also [328:44] reduces the bandwidth but allows you to [328:47] have a like stronger signal uh and make [328:49] the signal loss lower. So these are the [328:52] two different ways that we can move [328:55] information and they're both essentially [328:56] just pragmatically developed for either [328:59] balancing like simplicity versus like [329:02] distortion over over long longer [329:04] distances. And when it comes to optics [329:08] on the chip itself, there's a new [329:10] concept or not really that new but an [329:13] important concept which you can which [329:15] you should know. It's called co-ackaged [329:17] optics or CPO as opposed to pluggable [329:21] optics. So on the left you have [329:22] pluggable optics which puts electrical [329:26] to optical conversion at the front panel [329:28] right here. So it's a longer electrical [329:30] path whereas co-ackage optics moves it [329:33] beside the switch chip itself. So the [329:35] switch silicon here the optic the [329:37] optical engine is here shortening the [329:39] electrical path. What you need to know [329:41] is that both approaches remain in use. [329:42] Like in traditional servers, you'll [329:44] probably have like just standard [329:45] pluggable optics, whereas in like high [329:47] performance AI accelerators, it'll [329:49] likely be co-acked optics. And I would [329:52] like to liken this to the [329:55] lesson we learned about voltage [329:57] regulator modules, which want to make [329:59] the path of high resistance, high [330:01] current electricity as short as [330:03] possible. Similarly, here you want to [330:06] make the electrical path really short so [330:08] you can get into the optical domain [330:10] faster and just send the information. um [330:12] at light speed, right? So, this is kind [330:15] of a trade-off here where the copackage [330:19] optics definitely does introduce some [330:20] more packaging complexity, but then [330:23] allows you to get an optical domain much [330:25] faster than in the left diagram with [330:28] pluggable optics. So, I realized in [330:30] editing that I forgot to mention two of [330:32] the biggest advantages of co-ackaged [330:35] optics. Number one being much lower [330:38] power consumption since the electrical [330:40] path is much shorter. And number two, [330:41] the bandwidth you can serve is much [330:43] greater than with electrical conductors. [330:47] Okay. Now, let's do a quick case study [330:48] to make sure that we're actually [330:49] learning what we're talking about. But [330:51] before we discuss the example, I [330:53] realized there's a few things that I [330:54] still want to mention about these [330:56] switches. So, with the Quantum 2 [330:59] Infinity Band platform, as I mentioned, [331:02] there are 32 cages with each one having [331:05] a pair of two 400 GB gigabit per second [331:09] ports. So I think I've been so far mis [331:12] misspeaking here. So the capital GB is [331:15] gigabit and then capital G, capital B is [331:18] gigabyte. So that would be eight times [331:20] faster, but we're doing gigabits here. [331:22] So I apologize if I've been using this [331:25] term incorrectly up to now. It's also [331:27] funny that Nvidia will publish their [331:29] numbers in gigabits because it makes it [331:32] sound faster. If you report in [331:34] gigabytes, it would be eight times [331:35] slower. This is a classic example of [331:37] Jensen math. Or another example would be [331:40] like if they have some network adapter [331:42] which is 200 Gbits at one way and 200 [331:45] Gbits another way. Even though the [331:48] simultaneous speed of any given moment [331:51] is 200 Gbits, if you add them together, [331:53] you get 400 Gbits. And sometimes this is [331:55] also uh presented by Mr. Jensen himself. [331:58] So just a funny thing to note there. But [332:00] the reason I mention this is because [332:02] when you have a network that is in a [332:05] leave spine kicker configuration with [332:08] this quantum 2 infinite band platform, [332:10] you have these up link and these down [332:12] link arrows. So I think if I show you [332:15] here, it's the easiest to to see. What [332:17] this essentially demonstrates is that [332:20] you have 32 GPUs connected below at the [332:24] DJX1 H100 level. And in fact, there are [332:27] eight GPUs per rack in the H100 DGX [332:32] system. So, if you do the math, you get [332:34] four H100s connected to one leaf because [332:37] you have 32 connections down um sorry, [332:40] four H100 racks, which gives you 32 GPUs [332:43] connected below. And then at the spine, [332:46] you have 32 up links which go into the [332:49] spine. So, 32 down, 32 up. This is how [332:52] you connect the leaf spine network. It [332:54] is pretty logical when you think about [332:55] it with these ports mapping one to one [332:58] in this type of relationship. And then [333:01] coming back now to our case study, we [333:05] talking about cross leaf bandwidth. What [333:08] do I mean? So when I mentioned four [333:11] racks being collected connected to one [333:13] leaf, what this means is that if you're [333:16] doing any type of communication between [333:17] these four racks, you don't need to go [333:19] to the spine to get to there. You can [333:22] simply go just to the leaf and then do [333:25] the communication there and then come [333:26] back down and this is what is done in [333:28] practice and let's just go through an [333:30] example here. So let's say that you want [333:32] to communicate between four ports on a [333:36] leaf without going to the spine and you [333:38] would call this cross leaf bandwidth and [333:41] then going through this leaf would just [333:43] be called like the cross leaf cut. And [333:45] so if we have four uh four separate [333:48] ports, that gives us 1,600 gabits per [333:51] second from those four server ports. But [333:54] let's assume that we only have two [333:56] uplinks between them. So this would give [333:58] us a 800 GB per second uh uh transfer [334:04] speed across this boundary. So that's a [334:06] 2:1 over subscription and the 2:1 down [334:10] link to uplink ratio. So if we actually [334:12] go in gigabits and say let's let's [334:13] transfer 256 Gbits, well, we can only do [334:16] that in 0.32 seconds. And this is the [334:19] minimum transfer time. So this is the [334:20] lower bound with some overhead and other [334:23] things going on. You'll probably be [334:24] slower than this. And this also assumes [334:26] bounds traffic in one direction. So [334:29] these are the assumptions under this [334:30] calculation. But if we now add another [334:34] two uh links that can give the one one [334:38] down link to uplink ratio, well then we [334:40] can half the time at this 1600 Gbit per [334:43] second uh transfer speed to 0.16 [334:46] seconds. So you can see that there are [334:47] some considerations to be made to be [334:49] made in the cross leaf cut bandwidth um [334:52] in terms of how fast we can transfer [334:54] storage or weights or whatever it is [334:57] between the leaf. And then you can [334:58] imagine if you go up to the spine, well [335:00] now you're going to have to add even [335:02] more latency because you're adding [335:03] actually one hop to the spine and one [335:05] hop to the uh the leaf that you want to [335:08] do. You can actually think about this in [335:10] the context as well of just like a RAM [335:11] hierarchy. So if you've heard like a [335:13] cache hit or a cache miss, this is [335:15] essentially when the CPU is trying to [335:17] query from its RAM and its registers. [335:19] And the higher you have to go in the RAM [335:21] stack, the longer you have to wait to [335:23] get the data back. Um, so you can just [335:25] think of this leaf and spine type of [335:28] topology as a blown up version of CPU [335:32] memory in this context. And the whole [335:35] like topic of like multiple senders [335:38] overwhelming one receiver, which was [335:39] kind of what we were seeing here when we [335:41] only had two uplinks available at each [335:42] leaf is called incast. So you when you [335:47] when you essentially have one receiver [335:48] receiving from many different senders at [335:50] the same bandwidth as its um uplink, [335:52] then you just overwhelm that receiver. [335:53] So that's just a piece of terminology [335:55] and as I mentioned here, so traffic [335:57] staying within a leaf avoids its shared [335:59] uplinks. So if you're going across these [336:01] two leaves, let's say you're going [336:03] between leaf one and leaf four, then you [336:05] might pass through these two spines, [336:07] split the split the workload into two [336:09] different uh essentially spines and go [336:12] twice as fast to go between these two [336:14] leaves. But if you're only going between [336:15] leaf one, then it the exchange stays [336:17] stays in leaf one. There's no need to uh [336:19] go outside of the leaf that you're [336:21] calculating without within. if you don't [336:24] need to. Okay, now let's talk about a [336:27] real implementation of this type of [336:30] infiniband scale out. Interestingly, [336:33] Meta has both built large clusters with [336:36] in Ethernet and infiniband. So, [336:38] infiniband being the [336:42] proprietary technology that is used by [336:44] uh Nvidia or rather that is created by [336:46] Nvidia and then Ethernet being the other [336:48] standard. So in particular they used [336:51] Ethernet plus R OCE which stands for [336:54] RDMA over converge Ethernet and RDMA [336:57] stands for remote direct memory access. [337:01] What does remote direct memory access [337:03] mean? Well, it essentially just means [337:05] that the adapters, which are the network [337:07] interface cards, can transfer data into [337:09] remote memory without going through the [337:11] CPU, which usually manages the copying [337:14] of memory into inside and outside of uh [337:18] the local chip. So, this just allows you [337:22] to move memory at a much faster rate [337:24] with less overhead. But in any case, [337:27] whether using this Ethernet plus ROCE or [337:30] the Infinity Band standard developed by [337:33] Nvidia, both fabrics need to coordinate [337:35] routing, congestion control, collective [337:37] software, and job placement. So these [337:41] specific criteria determine how well the [337:43] workload will use the installed network. [337:46] And as you can determine from this uh [337:49] set of criteria, it's like a [337:50] four-dimensional optimization space. And [337:52] maybe there's even more criteria that I [337:54] haven't mentioned here. In any case, um, [337:56] at a low level, what you have is these [337:58] adapters communicating to these switches [338:00] in either protocol in some type of [338:02] topology and when you reach a 24,000 GPU [338:07] level orchestration, which is actually [338:09] small given the level of uh GPU runs [338:11] that you have today where you have like [338:12] a million scale uh GPU and TPU rollouts, [338:15] then the networking gets just as much [338:18] complex as you scale just gets even more [338:21] complex is what I mean as you scale the [338:22] number of devices. [338:24] So I mentioned um optical switches these [338:27] OCS so optical circuit switches which [338:29] can change which fiber endpoints connect [338:32] and we mentioned this like in a previous [338:35] chapter when we talked about TPUs. I [338:37] just wanted to talk about like a really [338:40] low uh not low level but actually a high [338:43] level of how these work. So let's say [338:45] that you have an optical circuit switch [338:47] with these mirrors that is pointing from [338:50] port A to C and B to D. Well, since the [338:53] mirrors can see the light path, let's [338:55] say that you actually want to go from B [338:56] to C and A to D. Well, you just augment [338:58] the mirrors and you can in real time [339:01] change where the data is going. And this [339:04] diagram here actually shows how it works [339:06] for TPUv4 with these MEMS mirrors. In [339:09] any case, you can see that the [339:11] complexity is astounding. you have 136 [339:14] mirrors in this array and it's just [339:16] passing this light uh through 855 [339:20] nanometer wavelength and it's the one [339:23] once again that can like actively [339:24] reconfigure if there is some like path [339:26] which is no longer available. Uh so I [339:28] think this is just quite a compelling [339:30] graph and diagram to look at to [339:32] determine like how does optical circuit [339:34] switch work at a at a high level. And uh [339:37] once again go to the link here on the [339:40] bottom left screen if you want to know a [339:42] bit more about that. And then when it [339:45] comes to like actually going outside of [339:47] the data center we talked about [339:49] Microsoft Fairwater which had their own [339:53] AI wide access network is what they [339:55] called it I believe. So that's one [339:57] option and then also you can just go [339:59] through the classic internet. So let's [340:03] talk about the campus fiber handoff [340:05] which connects the AI cluster to [340:07] external services. So this could [340:08] actually be its own users or maybe to [340:10] another data center. So you first go [340:12] from the AI cluster to the router which [340:14] directs traffic and then once at the [340:17] patch panel this is what actually [340:19] organizes the the physical connections [340:22] and sends the data to the meet me room. [340:25] What's the meet room? has a funny name. [340:27] It's essentially the data center's fiber [340:29] connector to the carrier. So I guess [340:31] when you think of carrier, you can think [340:32] of like AT&T or other like internet [340:35] providers which actually connect [340:37] different customers. So this may be yeah [340:40] a a customer or it could also be the uh [340:43] case with the fiber that's used [340:45] exclusively by Microsoft in their um own [340:47] data centers to communicate with each [340:49] other. And then once you're outside the [340:51] network, it's just either on the private [340:53] network or the open internet. So it's [340:56] worth noting just in terms of like a [340:58] spacing problem that the fail facility [341:00] needs space for this handoff and the [341:02] planned fiber fiber entrances and ducks [341:05] such that um it doesn't interfere with [341:07] like power and cooling or other aspects [341:09] of the data center. And then furthermore [341:11] if you remember we talked about how [341:13] there's this one data center where it [341:15] had three separate approaches for the [341:17] fiber. Well, once you have the three [341:19] different fiber approaches coming in, [341:21] the question becomes, are you gonna have [341:22] three different handoff points or [341:24] consolidate them all into one room? The [341:26] problem is there you have like a single [341:27] point of failure, but it'll be less [341:29] redundant. So, a lot of questions here [341:31] that come into play when you're talking [341:33] about how you're actually getting the [341:34] data in and out of the data center. And [341:36] I think it's quite provoking. So, now [341:39] just to give you an idea of like the [341:41] distances that we're talking about, [341:42] let's say that you want to go 100 km or [341:44] 60 mi. And this is what's the distance [341:47] that's separating two different uh [341:49] facilities. [341:50] Since uh signal's speed in this [341:53] calculation is not the speed of light, [341:55] we'll assume it's instead 200,000 km/s. [341:58] Well, to go between these two different [341:59] facilities, it would take around 1 [342:01] millisecond. And in fact, there's this [342:05] set of [342:07] really interesting like fermy [342:08] calculations that Jeff Dean has posted [342:11] where essentially it demonstrates, okay, [342:13] if you're going between Europe and North [342:14] America, this is the millisecond [342:16] bandwidth or latency you should uh you [342:18] should expect. If you're going between [342:20] two states in the US, this is the [342:22] latency you should uh you should expect. [342:24] And I think it's quite interesting [342:25] because when you're really optimizing at [342:29] a Google level your bandwidth and your [342:31] network, then these are actually like [342:33] numbers that you need to consider when [342:36] you're serving your end and your and [342:38] your users, your customers. And um this [342:40] is just one example, but having these [342:42] numbers in the back of one's head as [342:44] just a sort of a first order [342:46] approximation can be quite helpful to [342:48] reason about the different latency times [342:52] that you should expect when [342:54] communicating between different data [342:56] centers. [342:58] As we know in like everything it's like [343:01] the slowest link that sets the pace. So [343:03] even if you're dealing with servers, [343:05] let's say that uh you're waiting on the [343:08] fourth server to do some collective [343:09] operation. Maybe you're trying to [343:10] synchronize synchronize sorry the [343:12] weights and do some all reduce operation [343:14] to compute the gradients of your of your [343:17] forward pass or sorry of your backward [343:19] pass in a in a training run. Well, if [343:22] one server is taking much longer than [343:23] others, then you're set up at uh being [343:26] 50 milliseconds for all four, even [343:29] though three servers at 20 milliseconds. [343:30] So we've already kind of talked about [343:32] this already in terms of well if you're [343:35] doing a collective communication then [343:36] you're as slow as your slowest [343:38] constituent. So this is also applying to [343:41] like even the topology of the switch and [343:45] spine leaf hierarchy. [343:48] Okay. And now before we close off this [343:51] chapter I want to talk about a quick [343:54] example once again with meta. So in [343:58] their RSC which is their research [344:00] supercluster here if I zoom in then you [344:03] guys will see the name the meta research [344:04] supercluster which is from January 2022 [344:06] so quite some time they built storage in [344:08] in tiers to keep the GPU supplied. So in [344:11] particular they had 175 pabytes of bulk [344:13] storage 46 pabytes of cache and 10 [344:16] pabytes for shared file storage. So what [344:18] was this what was this actually for? In [344:20] essence, they prepared the data sets uh [344:23] that would be used for the training runs [344:25] in uh like in advance to reduce repeated [344:29] preparations and transfers. And you can [344:32] think of like at the scale of internet [344:34] data when you're training these massive [344:35] models. It's pabyte scale. So it's not [344:39] just like loading some like USB file to [344:41] send the data and get the model trained. [344:44] you're really requiring storage [344:46] equipment and network capacity to make [344:48] sure that um the models have access to [344:50] whatever data they're using to train. [344:53] And this storage and networking capacity [344:56] also needs its own rack space, power, [344:58] cooling, and cabling. So, we don't just [345:01] need to think about the GPUs, but also [345:04] how we're networking in together. [345:06] especially in the context of like [345:07] massive pre-training runs. When you're [345:09] just serving users at inference scale, [345:11] this is like less of a concern. But if [345:14] you're specifically making a cluster for [345:16] pre-training, then this definitely needs [345:17] to be considered. All right, now we're [345:21] on chapter 11, which is about chip and [345:24] rack heat chap uh heat capture. And I've [345:26] started off with quite a provocative [345:28] title, namely, why is air cooling dead? [345:31] And if you remember once again this was [345:33] a way back in the beginning of the [345:35] course I mentioned that I would show [345:36] some statistic which demonstrates the [345:40] comparative advantage of using air uh [345:42] liquid cooling over air cooling. So [345:45] let's set up the the the standards here [345:47] of wanting to have a 100 kowatt system [345:52] being cooled with some 10° [345:56] uh Celsius temperature differential. [345:59] Okay. So if this is the system that you [346:01] want to want to cool essentially you [346:04] would be cooling the the the system by [346:07] 10° then in order to do this by using [346:10] the fundamental heat transfer equation [346:12] which is essentially stating that uh [346:14] this dot here by the way is just like [346:15] the delta time that's what it's [346:17] referring to because it's referred it's [346:18] referring to some change in time. The [346:21] way that you calculate this is by [346:22] multiplying the mass flow rate, which is [346:24] this mass flow rate here, by the [346:27] specific heat capacity and the delta [346:30] temperature that you want to get the the [346:32] heat transfer equation. And if we [346:35] actually solve for the uh the mass flow [346:38] rate or equivalently the volume times [346:41] the density then you'd get that for this [346:43] cooling situation 8,292 L of second of [346:49] air would be required and 2.39 L/ second [346:53] of water would be required for the [346:54] equivalent cooling. And there's two [346:55] reasons for this. First of all, it's the [346:57] density of water which is 1,00 kg per [346:59] meter cubed versus 1.2 2 kg per meter [347:03] cube of air. And then also the specific [347:05] heat capacity, which is kind of like how [347:07] much heat can water like absorb and [347:10] store in its bonds is four times larger [347:13] than air. So air has a specific heat [347:15] capacity of around 1 kg per kilogram [347:18] centigrade and water has around 4 kg per [347:21] kilogram centigrade. So you don't really [347:23] need to know like the background theory. [347:24] You just need to know like a higher [347:25] specific heat capacity means it can [347:27] store more it can store more energy [347:29] essentially. So this is just like a a [347:31] more theoretical way to start off the [347:33] chapter. But don't worry, we're going to [347:34] talk about these numbers in a much more [347:36] tangible way. But let's first talk about [347:39] what we're doing now with air cooling. [347:41] So out with the old, in with the new. [347:43] Just for the sake of completion, I'm [347:45] going to tell you guys how air cooling [347:46] works in general. So an aircooled rack [347:49] is just like fans, just like a just like [347:50] a CP uh sorry, not a CP, a PC that you'd [347:53] have at your house where you have warm [347:55] air coming out and cold air coming in. [347:57] And the heat exchanger is called a CR, a [348:01] computer room air handler, where you [348:04] exchange the air with either water or I [348:08] think usually it's water since water is [348:10] just better or perhaps you could have a [348:12] airto-air handler. I'm not sure if those [348:13] exist, but in any case, you're [348:15] essentially exchanging the air from the [348:16] racks with some type of exchanger here [348:19] which is converting the heat from our [348:22] computers to go outside. And then right [348:25] now with the current systems, we use a [348:28] coolant distribution unit which is [348:30] essentially coolant on both sides of the [348:32] exchange where you have the technology [348:33] coolant that goes to the cold plates and [348:36] the facility water that goes outside. [348:38] The CDU brings the heat from the left to [348:40] the right. And the interesting thing [348:43] here is you may have some remaining air [348:46] heat. So like let's think about it. If [348:48] you put a cold plate on top of the GPU [348:51] and circulate water to get the heat and [348:53] absorb it, there'll still be some that [348:55] escapes outside of the sides and you [348:57] need to do something with with this uh [348:59] so that it doesn't just go into the air [349:00] and like heat up the data center that [349:01] way. And the way that you actually can [349:03] resolve this is called is either via [349:06] using your own CR in addition to the CDU [349:10] or more interestingly using a RDXH which [349:14] stands for a rear door heat exchanger. [349:17] Oh, so it' be a RDHX. Sorry. Uh, so I'll [349:20] talk about the rear door heat exchanger [349:21] in detail in a future slide, but just [349:23] know that even if you're using a cold [349:25] plate liquid cooling closed loop, you [349:28] may still have some remaining air [349:29] escaping. And that needs to be addressed [349:31] because although over a short amount of [349:34] time, it would like likely not lead to a [349:36] massive increase in temperature, over [349:38] many days, you would definitely get a [349:40] noticeable increase in the temperature [349:41] of the data center if this is not [349:43] addressed. since you don't have like [349:44] this perfect energy capture via via the [349:47] chip cold plates. And what are actually [349:49] the chip cold plates? So these are two [349:52] pictures of GB300 cold plates. They're [349:54] made of copper as well just because [349:55] copper is a really good conductor of [349:56] heat. And essentially you have this in [349:59] and out flow for the liquid because you [350:01] have the hot coming out and the cold [350:04] coming in with the which is probably the [350:06] blue node there. And then in this [350:08] diagram they're on the left and the [350:09] right as opposed to in the center. And [350:11] then how this connects to the CDU is [350:14] well you have the cold plate with its [350:17] own quick disconnects which is just [350:19] there so that if there's a leak or if [350:21] you need to replace something you can [350:22] actually just remove only one cold plate [350:24] instead of the entire system. Uh and [350:26] then this is going into the manifold [350:28] where the technology coolant circulates [350:31] and then this is how you get from the [350:32] cold plate to the coolant distribution [350:35] sorry the coolant distribution unit. how [350:38] this actually looks like in practice at [350:39] least with respect to the uh overarching [350:42] plumbing is such. So this is a look at [350:45] the liquid cooling inside of Abalene [350:47] where you can see that you have these [350:49] headers that are going to the top of the [350:50] data centers. Sorry, not the top of the [350:51] data center but the top of the racks and [350:53] this is how you exchange the heat. Once [350:56] again likely the blue and the red [350:58] demarcate whether it's hot or cold uh in [351:02] terms of which way the the liquid is [351:03] circulating inside of the uh inside of [351:06] the racks. And then I talked about the [351:08] rear door heat exchanger. So now I'm [351:10] actually going to properly uh introduce [351:12] what it does. So let's say that uh yeah, [351:15] you do have this extra compute that [351:18] escapes the cold plate and goes into the [351:21] surrounding air. Well, now you have some [351:23] warm air in the rack. And then what you [351:25] do to get rid of this warm air is you [351:27] have some fans at the front of the rack [351:29] which blow air to the back of the rack. [351:32] And then you have a rear door heat [351:35] exchanger which literally is just a [351:38] bunch of liquid cooling manifolds. Uh so [351:41] essentially think of like liquid pipes [351:43] that are flowing through the back of the [351:45] rack and capturing the air so that it [351:48] doesn't escape the rack. So I think this [351:50] diagram on the right here really shows [351:51] it best. Let's say that you have the [351:53] coolant coming in from the bottom and [351:54] then filtering through like a radiator [351:57] just like a car radiator actually such [351:59] that the fans blowing the hot air to the [352:02] back of the rack get the air captured in [352:05] this air sorry in this uh water and then [352:07] the warm coolant goes back to the same [352:09] coolant device unit. So that it's [352:11] actually going through the same uh loop [352:13] as the the liquid cooled chips but just [352:15] through a separate uh through a separate [352:17] circulation. And then now your air is [352:19] cooled such that um you don't have this [352:22] problem of like excess air going and [352:23] heating in the data center. So that's [352:25] another way that you can kind of use [352:27] liquid cooling to heat sorry to cool the [352:30] air by essentially pushing the air to [352:33] the coolant and then uh having it [352:36] exchange heat that way. So that's how a [352:37] rear door heat exchanger works. Now I [352:40] want to talk about something really [352:41] funky which is called immersion cooling. [352:44] And I would say that generally this is [352:45] less applicable in the context of AI [352:48] accelerators in like these massive AI [352:49] data centers, but is nevertheless still [352:51] used in in data centers. And it's [352:53] exactly what the name suggests. [352:54] Literally, you put the racks inside of [353:00] fluid. So I was very careful not to say [353:02] water because water can conduct [353:04] electricity and that will just like [353:06] shortcircuit everything. So instead you [353:08] use dialectric liquid. And dialectric [353:11] liquid means it's a liquid which acts as [353:13] an insulator. So essentially you have [353:15] this data center that has racks filled [353:17] with dialectric fluid. So literally I'll [353:20] just skip one side quickly. The [353:22] computers are in fluid. And I don't want [353:24] to say water once again cuz it's not [353:25] water but it's some dialectric fluid. It [353:27] might be some like type of mineral oil. [353:29] And essentially what happens here is [353:31] that you have coolant that pumps the [353:33] water through this rack and then it has [353:37] its own heat exchanger which then will [353:40] chill the water in its loop. This can be [353:42] in any type of method. So it can be a [353:43] chilled water loop, an evaporative [353:44] cooling tower or a dry cooler. We'll [353:46] discuss each of these in detail in a few [353:49] uh slides. But in any case, what you [353:51] should know is that you have two types [353:53] of immersion cooling. So you have [353:55] singlephase liquid cooling which is [353:57] shown here where the liquid inside of [353:59] the rack does not evaporate and then [354:02] secondly you have two-phase. So in this [354:05] setup what happens is that once the [354:08] liquid that's dipped uh sorry once the [354:10] the liquid that's with the compute units [354:13] gets heated sufficiently it will boil [354:16] and then condense and then once it [354:18] condenses it will cool back down and [354:19] then you would recirculate it into into [354:21] the loop. So those are two ways that you [354:23] do immersion cooling. Either purely [354:25] through liquid or liquid than gas. And [354:30] um yeah, this is just an image of what [354:32] it looks like inside an immersion tank. [354:34] I learned this literally while I was [354:36] making this course. So maybe this is not [354:37] new news for anybody watching this. But [354:39] in any case, it still shocked me that [354:41] you have computers that were dipped in [354:44] liquid. I thought this was a fake image [354:45] initially. So now that we've gone [354:48] through immersion cooling, let's let's [354:51] switch subjects to define two terms that [354:54] are important. Number one is heat flux [354:56] and number two is thermal resistance. So [354:59] the tagline here is that the same heat [355:01] can be harder to remove. What I mean [355:03] here is that um with heat flux, it's [355:06] essentially heat transfer rate divided [355:08] by area. So on the left you have a 4 [355:13] [snorts] cm cubed or not cubed but [355:15] rather squared uh compute die. Let's say [355:17] that it's like some chip and you want to [355:20] remove 400 watts of heat in a second or [355:22] 400 jewels in a second. So that would be [355:24] 400 watts. And in this case you would [355:27] have a heat flux of 100 watts per [355:30] centime squared. That's like the heat [355:32] transfer that you need to do per square [355:33] centimeter. But if you had an even [355:35] smaller compute die, so let's say it's a [355:38] quarter of the size. So instead of being [355:39] 4 cm squared, it's 1 cm squared. Then [355:42] the heat flux would be four times [355:45] larger. If you consider it has the same [355:48] heat transfer rate and you can think of [355:50] it as in the right, it's just harder to [355:53] remove that same heat because it's more [355:55] concentrated as opposed to the left. All [355:57] right? And actually we'll talk about [355:59] thermal resistance in the next slide. So [356:00] for now just recall that heat flux is [356:03] talking about the heat transfer rate per [356:06] unit area of some type of object. In [356:09] this case the denser it is you can [356:11] imagine the more difficult it will be to [356:14] cool. And now let's get into thermal [356:16] resistance. So the formula if you recall [356:19] is delta T equals P R theta with P [356:24] equaling the heat rate and R theta being [356:28] the thermal resistance such that when [356:30] you divide these terms out you get units [356:32] of Celsius which makes sense because um [356:34] the temperature is on the left so it [356:35] could be Celsius could be Fahrenheit, [356:36] Kelvin, Rankine etc. And now let's [356:40] develop some intuition on what this this [356:41] this formula is saying. So let's first [356:43] start off with what does thermal [356:46] resistance mean? It means that [356:49] the GPU in this example of a GPU being [356:52] cooled by water must be 0.2° [356:58] C hotter than water per watt of heat [357:03] flow. So if we think of this at a at a [357:06] really actually low level, if we want to [357:10] cool the the GPU or want to remove one [357:13] watt from the GPU, then the temperature [357:16] difference between the GPU and the water [357:18] needs to be 0.2. So in that case, the [357:21] GPU would be 30.2° [357:24] and the water would be 30° C. But now [357:28] let's say that well we want to remove [357:31] 100 watts from the GPU and the [357:34] temperature of the water is 30°. Well, [357:38] if we know these two uh these two [357:40] numbers, then we just plug them in to [357:43] the the formula here of well the heat [357:45] rate is the amount of heat and the [357:48] thermal resistance is a specific [357:50] property of this GPU and we get 20° is [357:53] the difference required between the [357:55] water and the GPU to to remove 100 watts [357:59] um from from the GPU to the water. So [358:02] thermal resistance you can kind of think [358:04] of as like for a specific material how [358:07] much does it resist getting cooled or [358:11] getting heated depending on which way [358:13] you look at the equation and which way [358:15] you have the delta t. But in any case [358:18] this is a useful formula because it [358:19] tells us for a specific material what is [358:22] the cooling capacity that we will need. [358:25] In this slide, let's talk about why 35 [358:29] degree coolant can cool two chips [358:31] differently. So in this case, we're [358:34] going to really apply this formula for [358:36] thermal resistance in practice. So let's [358:39] consider the the strange situation where [358:42] you have two chips, one at 67° and one [358:47] at 83° despite the coolant being the [358:50] same temperature and despite having the [358:54] same amount of heat transfer. [358:56] How what could explain this? Well, it [358:58] would be this thermal resistance. So if [359:00] we assume that chip one has a thermal [359:03] resistance of 0.08 08 Celsius per watt [359:05] and that chip 2 has a thermal resistance [359:07] of 0.12 [359:09] per uh Celsius per watt. What that means [359:10] is chip 2 needs a higher temperature [359:12] differential to achieve the same amount [359:14] of cooling. And if you just plug these [359:16] numbers in, then assuming a chip limit [359:20] of 80°, well, to get 400 W of heat out [359:24] of this chip, then you would be [359:25] exceeding the temperature by 3° cuz your [359:27] temperature difference would be 48°C. [359:29] Whereas in this time in this [359:31] consideration because it has lower [359:33] resistance it can essentially have a [359:37] smaller temperature differential to cool [359:39] the same amount of watts on that coal [359:42] plate. So you would imagine that when [359:45] you are building these cooling systems [359:48] you want to have the interface between [359:49] the chip and the cold plate to have low [359:52] thermal resistance. Okay, let's look [359:54] back at the fundamental heat transfer [359:57] equation that we saw on the first slide, [360:00] which once again says that the mass flow [360:03] rate times the specific heat capacity [360:06] multiplied by the temperature difference [360:08] is [360:10] your heat rate. And what we want to [360:14] develop from this is to realize that [360:17] more flow means less temperature rise. [360:21] So let's consider two situations here. [360:24] First of all, we have water coming in at [360:27] 35° C and then passing through a heat [360:32] pickup on a 100 kowatt system and then [360:36] now it's heated up to 44.57° [360:41] given a giving a delta t of 9.57. [360:45] And what you need to notice is that we [360:47] plugged in 2.5 kg/s as the mass flow [360:51] rate in this formula. But let's say that [360:54] now we double the flow to 5 kg/s [361:01] because we've doubled the flow uh here [361:04] the m now the delta t becomes half which [361:07] means that there's less temperature [361:09] rise. So essentially what we're saying [361:10] is that if you can move more water [361:12] through the system then the water will [361:15] get heat get heated less essentially and [361:17] I think this somewhat makes intuitive [361:18] sense like imagine like you just double [361:21] the amount of water well double the [361:22] water can absorb uh the same amount of [361:25] heat but get heated half as much because [361:28] there's twice as much water to absorb [361:30] the heat. And uh this is one way that [361:32] you can kind of intuitit this [361:34] fundamental heat transfer equation. [361:36] Alternatively, if we half the heat rate, [361:40] then the temperature rate sorry, if we [361:42] half the mass that's flowing through the [361:44] the system, then the temperature will [361:47] double in terms of the amount of [361:48] increase. And of course, the specific [361:50] heat capacity CP is a number that is [361:54] specific to that type of u water or [361:57] whatever type of coolant you're using. [361:59] So glycol is also really common in terms [362:01] of the liquid that you use for cooling. [362:03] But this is not something that we can [362:04] change unless we actually change what is [362:07] in the side of the system. Okay. So on [362:09] the previous slide we saw why coolant [362:12] flow matters. Here we'll talk about what [362:15] determines how much flow we actually [362:16] get. So the horizontal axis is flow in [362:20] liters per second and the vertical axis [362:22] there is the pressure difference the [362:24] pump adds to drive coolant around the [362:27] loop. And in this case the pump speed is [362:31] fixed and the the the line is downward [362:34] sloping because at this speed it can [362:37] produce a larger pre pressure difference [362:38] at low flow and a smaller one at high [362:41] flow. That's just how the nature of a [362:42] fixed speed pump is. And now if we look [362:45] at the two upward sloping lines, they [362:48] show what the cooling circuit requires [362:51] for a given flow rate. What would the [362:52] what would the corresponding pressure [362:54] be? And essentially pushing more coolant [362:57] through the pipes, valves and coal plate [363:00] makes a larger pressure difference. In [363:02] fact, these two quantities are related [363:05] quadratically. So as you increase flow [363:07] rate, the corresponding pressure is uh [363:10] quadratically scaling. So a flow rate [363:12] that would double would cause a [363:14] corresponding pressure difference that [363:16] would quadruple. And when I say prefer [363:18] pressure difference, I mean like the [363:19] pressure of the water that is coming out [363:21] of the pump versus the pressure of the [363:23] water that's coming inside of the pump. [363:25] Because essentially as a pump goes [363:27] through a circuit or as the pressure of [363:29] the water goes through the circuit, it [363:30] will lose pressure uh as it goes [363:33] throughout and then it will regain [363:35] pressure as it uh passes the pump. So in [363:38] fact where these two lines meet where [363:40] the pump line and the two different um [363:44] flow rate lines meet is where the system [363:46] settles. That's what you could call the [363:48] operating point. And why are there two [363:50] different points? It's because I'm [363:52] considering two different circuits. So [363:53] at point A, the pump provides the [363:56] pressure the circuit needs to sustain 2 [363:58] L/s, which is 120 kilopascals. But now [364:02] let's imagine that the valve is partly [364:04] closed or a filter is clogging. Maybe [364:07] there's like a lot of minerals in the [364:08] water and it starts to scale and make a [364:10] clog. Well, then getting the same flow [364:13] that restriction requires more pressure. [364:16] So, that gives us this steeper dashed [364:18] red line. And even though the pump speed [364:20] hasn't changed, the flow falls to about [364:22] 1.4 L/s while the pressure difference [364:25] rises to 140 kilopascals. So, in [364:29] summary, the pump is still running and [364:30] it's producing a larger pressure [364:31] difference, but less coolant is [364:33] circulating. And that's why we need to [364:35] check flow as well as pressure since the [364:38] equipment still needs enough coolant to [364:40] carry away its heat. All right. Now, [364:43] let's talk about what's called the [364:45] approach for the coolant device unit. [364:48] And this is essentially the gap between [364:50] the two supply temperatures. So, it's a [364:52] really simple calculation. The [364:54] calculation asks what is the temperature [364:57] of the water that is going to the chips [365:00] subtracted by the water that is coming [365:02] from the facility. So these are both of [365:04] the cold sides of the loop. So if you [365:05] look at the returns that is when it's [365:07] heated up in instead the the the [365:10] approach is asking for the chip cold [365:14] length and the facility cold length. [365:15] What is the difference? And essentially [365:17] a lower approach gives more coolant [365:20] temperature margin but may require more [365:24] exchanger area flow pumping or cost. So [365:29] in this example we have a five five [365:32] degree difference in temperature. The [365:34] extreme case would be like the [365:37] temperature approach could like approach [365:39] zero if you had like almost a perfect um [365:43] heat exchange between the facility loop [365:44] and the chip loop. Of course in practice [365:46] that isn't the case and you usually have [365:49] a approach in like the five to three [365:53] degree range but that's just a estimate. [365:56] In any case, this is one of the [365:58] important metrics that we use in order [365:59] to determine the heat exchange at our [366:02] coolant device unit. And like what is a [366:04] coolant device unit in terms of like [366:06] what a picture would look like. So [366:08] here's an example of a cool IT CHX2000 [366:12] which is a heat exchanger that has a [366:14] rating of 2 megawatts thermal at 5° [366:17] approach. So you can now kind of reason [366:20] why approach is important. It's [366:21] essentially the number that we use to [366:24] report the statistics of this heat [366:26] exchanger, which is to say that at a [366:28] five different five degree difference [366:30] between the water that's going to the [366:32] chips and the water that's coming from [366:33] the facility, we can exchange 2 [366:35] megawatts thermal. And this essentially [366:37] means like you can exchange 2 million [366:40] jewels per second of heat. And the pumps [366:44] themselves are running at 2.125 [366:48] or sorry 2,125 L per minute uh at 30 and [366:52] this gives you 35 PSI. And then what [366:54] else you're controlling? Well, you can [366:56] you can modify the temperature, the [366:58] flow, and the pressure based on the [367:00] system that you're working with. But [367:01] these are just a few of the standard [367:03] rates that would be used in the in the [367:05] in the C in the in the CDU. And this is [367:08] what it looks like with the cabinet. So [367:10] you can see the difference between [367:12] inside the CDU and uh just the outside [367:15] of the cabinet. I want to quickly [367:17] mention to you that Cool It is a cool [367:19] company because they started off making [367:21] gaming C uh gaming PC cooling equipment [367:24] and then they pivoted into data centers [367:26] and now we're worth like billions of [367:27] dollars. So it's quite a interesting [367:30] story and turn of events for this [367:31] company. And we talked about redundancy [367:34] in the context of uninterruptible power [367:36] supplies and feeder lines and these AB [367:40] corded dual paths. Now let's talk about [367:42] it in the in the in the context of [367:45] coolant distribution. So if you have [367:48] 1,00 kW of heat that you produce in your [367:51] computation, then n redundancy or [367:54] approximately n redundancy would be if [367:56] you have two 600 kow CDUs because you [367:59] can you can provide 1200 kW of cooling. [368:02] So you can meet the cooling requirements [368:04] and then um N plus1 would be okay can [368:07] you now provide 1.5,000 [368:10] watts of heating and in this case sorry [368:14] 1,00 kW that is actually 1 megawatt so [368:17] 1.5 megawatt of cool of heating and uh [368:21] in this case you would have three of [368:22] these CDUs and then if we jump to a [368:25] two-end system this would be two [368:27] separate loops so a a loop and a b loop [368:30] of two CDUs which are completely [368:32] independent because if one facility loop [368:35] fails, we need the other one to pick up [368:37] the slack. And if you want to get a [368:39] better intuition of what this looks [368:41] like, it essentially means that either [368:42] cooling train can carry the load. So at [368:45] normal operation, [368:47] you have 2400 kows of capacity, but [368:50] you're probably only using around 1,000 [368:52] since that's what your load is running. [368:54] And you may either have one on and one [368:57] off or perhaps they split the load at [368:59] like 500 kW each. And then once there is [369:04] a path loss or maybe you're just doing [369:08] maintenance, uh you can have the second [369:10] path serve that cooling requirement and [369:14] uh allow your computational workloads to [369:16] continue to persist. So you can imagine [369:19] that when you're dealing with a data [369:21] center that needs like a 59's runtime, [369:24] which I if I'm not mistaken was only a [369:26] few minutes of downtime per year, then [369:28] most likely you'll have some form of 2N [369:31] redundancy or maybe even a 2N plus1 [369:33] redundancy on your cooling. And believe [369:34] me, there are many different [369:35] configurations which you can have in [369:38] order to cool your equipment. It simply [369:40] is a question of the the requirements [369:43] that the project has and the budget as [369:45] well since it's more expensive the more [369:47] redundant your system is. And now let's [369:50] consider that you have this n redundant [369:54] system. So you only have two CDUs and [369:58] you can cool it normally but then let's [370:02] say that one of them fails. So [370:05] essentially what happened is that you [370:07] had one CDU at 600 kW and another at 600 [370:12] kW which was fine but now that there's [370:15] only one remaining your original load is [370:18] in excess of the heat uh that you can [370:20] cool or that the heat that you can [370:21] extract. So what you might do with some [370:24] controlling um is simply reduce the load [370:27] so that you can at least reduce and [370:29] produce some computation maybe delay [370:32] some work which is not like time [370:36] sensitive and continue to run the load. [370:38] So you'll definitely see in data centers [370:41] some form of controlling that uh would [370:44] do this [370:46] reduction in load whenever there is in [370:50] fact some cooling issue and you have [370:53] reduced cooling capacity. So what this [370:55] would look like in a graph is you're [370:57] running at 1,000 kW you lose power to [371:00] one of your CDUs. Maybe now you can only [371:03] capac you now have a cooling capacity of [371:04] 600 kW. You reduce the power and then [371:06] you measure to make sure that the [371:08] coolant capacity is above your current [371:12] load. Okay. I also mentioned at the very [371:17] uh beginning how we have this rear door [371:19] heat exchanger and let's just do like a [371:22] branching uh problem to [371:26] exercise our new found capacities in how [371:30] data center cooling works. So let's [371:33] assume that we have a rack that's 100 [371:34] kow and we know that 85% of the heat can [371:38] be removed by the coal plates. This [371:40] means that 15% is residual air or 15 [371:42] kow. So what are you going to do in this [371:45] case? What you can do is either use a CR [371:50] a computer room air handler to get rid [371:52] of that 15 kows and let's assume that [371:55] you have a 20 kow capacity. So in this [371:58] case we can just simply um use the CR [372:02] and then have actually 5 kW to spare if [372:06] in some case the the weather's really [372:09] hot or there's some issue where we need [372:11] to have a bit more air cooling capacity. [372:14] But once again, the other option which I [372:16] think is more interesting is the rear [372:18] door heat exchanger where not only can [372:20] you use the liquid cooled plates to [372:22] remove heat from the the chips, but also [372:24] push the air to the back of the rack [372:27] where the heat is collected at the rear [372:29] door and the liquid then brings that [372:32] heat to the outside of the building. So [372:34] this is two ways of how you can think of [372:36] removing heat from air and that closes [372:40] this chapter. So now we're going to move [372:43] on to the next one where we're [372:45] discussing similarly heat but instead [372:49] rejection on a more global level and [372:52] also water usage. So we've mainly been [372:54] talking about like heat within the rack [372:56] and like a local lens, but the reality [372:59] is the heat still has to leave the site. [373:02] So, like if we just look at the [373:03] three-step flowchart here, we collect [373:05] the chips heat in some warm coolant. [373:08] Then we carry it via the coolant [373:11] distribution unit to a separate facility [373:13] loop with this heat exchanger. And then [373:15] we release it with some uh heat transfer [373:17] outside. That's kind of how it works. [373:19] It's ominously like pretty much like a [373:21] heat HVAC system uh just like in a home, [373:24] but uh now it's like liquid cooled [373:27] instead of just air cooled or depend [373:29] depends on actually what your HVAC [373:30] system is. Anyways, at Avalene, the heat [373:34] goes to outdoor air. So, that's actually [373:35] how their system is implemented. How it [373:37] works is that you have the facility [373:39] water that gets the heat from the uh the [373:43] the CDU and then an air cooled chiller [373:46] is used to move heat outside, which you [373:49] can think of as essentially like an air [373:51] conditioner. So you have this [373:53] refrigerant in the chiller and the [373:56] electricity [373:58] compresses that refrigerant to have like [374:00] essentially a larger ratio of heat [374:03] transfer with the environment and you [374:06] use this compressor to remove heat from [374:08] the building. Another term for this type [374:10] of heating is non- evaporative [374:12] rejection. But one thing you do need to [374:14] note is that the fill and maintenance [374:16] still use water. So the initial fill of [374:18] the system requires a lot of water and [374:20] also maintaining the system might mean [374:22] that you have to dump all the water out [374:23] or replace some percentage at some point [374:25] in time. [374:27] Okay. So when it comes to like [374:29] interacting with the environment, [374:30] there's two different uh classes of [374:32] heating. Once again, evaporative and [374:33] non-eaporative. So you can kind of [374:35] generally classify the evap the [374:37] evaporative cooling into like wet [374:40] cooling and then dry cooling for the [374:42] non-eaporative. It kind of makes sense. [374:43] So start off with dry coolers. What you [374:45] do is essentially you collect heat from [374:48] your building and then you move it [374:50] across a radiator and then you blow air [374:51] through it. It's that simple. You can [374:53] also add a refrigerant here with a [374:55] compressor if you want to have a bigger [374:57] ratio of heat transfer, but once again [374:59] that depends on the environment in the [375:00] system itself. And then if you're in a [375:03] wet cooling regime, what's going on? So [375:06] you have heat collected from the [375:07] building and then you move this heat [375:11] across what's called a fill with [375:14] nozzles. So a fill is essentially this [375:18] material. It's usually like a polymer [375:20] that maximizes surface area given a [375:23] volume. So it's like some type of [375:24] lattice shape where essentially you put [375:26] the water along this fill so that it's [375:28] exposed to as much air as possible such [375:31] that it evaporates at least some of it [375:33] will evaporate into water vapor and take [375:35] away the heat. And what this means is [375:37] that the resulting water that comes back [375:39] into the system is much cooler since a [375:41] lot of heat was lost through [375:42] evaporation. Of course, now you need [375:45] what's called makeup water to make up [375:48] for the heat that sorry, the water [375:49] that's lost. And then secondly, you also [375:52] have what's called blowdown. What the [375:53] heck is blowdown? So blowdown is [375:56] essentially water you have to remove [375:58] from the system because it has too many [376:01] minerals in it. Why is this the case? [376:04] When you lose water with water vapor, [376:06] you essentially increase the [376:07] concentration of minerals in that water. [376:09] And this is really bad because then [376:11] you'll have like scaling and like [376:12] calcification of your pipes which will [376:14] reduce the flow rate and the pressure as [376:16] we saw in a slide of the previous [376:18] chapter. So in order to prevent this [376:20] there is some sort of periodic blowdown [376:22] where you just evacuate water from the [376:24] system that has too high of a mineral [376:26] concentration in order to avoid this. So [376:28] not only do you have to add makeup water [376:30] but you also have to remove blowdown. So [376:32] it's like a double whammy of like [376:34] requiring water which just demonstrates [376:36] the intensiveness of uh wet cooling [376:39] towers. So once again, do you want your [376:41] cooling wet or dry? That's the that's [376:43] the question here. It's very much a [376:44] environment question and if you're in a [376:46] desert, then the answer is clear. Uh [376:49] just to give a another way of kind of [376:52] intuiting what what was going on with [376:54] the evaporation. So water evaporates, [376:56] you get a high concentration of [376:59] minerals, then you discharge it which is [377:01] called blow down and then you refill it [377:03] with the makeup water. So that's the [377:04] simple cycle in three steps. Okay, I [377:08] talked about dry cooling and wet [377:10] cooling. Now there's another set of [377:12] concepts that are somewhat related but [377:15] not really. It's called dry bulb [377:17] temperature and wet bulb temperature. So [377:20] first of all, dry bulb temperature is [377:22] ordinary temperature. It's simply the [377:24] air temperature measured with a dry [377:26] shaded thermometer. In contrast, what [377:30] bulb temperature is the temperature [377:31] measured with a thermometer covered in a [377:34] water soaked cloth and exposed to air [377:37] flow? And what happens is that [377:39] evaporation cools this water and makes [377:43] the difference in temperature larger. [377:45] And I like to think of it as like [377:46] sweating. So what I mean is if you make [377:50] your skin wet and you measure the [377:53] temperature of your skin, it will [377:55] actually be cold due to evaporation. And [377:58] guess what? That's what sweating is. [378:00] Essentially, your body lets water [378:03] accumulate at the surface of your skin [378:04] through sweat. And then as soon as that [378:06] water leaves your skin, it evaporates [378:08] and removes heat from your skin. And [378:10] that's why if you touch like a sweaty [378:12] arm that's like not directly in the [378:14] sunlight, it will be cold to the touch [378:15] because the evaporative effect has [378:17] released heat from your body. That is [378:19] how sweating works. And interestingly, [378:23] there's only one time when dry bulb [378:26] temperature and wet bulb temperature are [378:28] the same. Maybe you can guess. It's when [378:30] the relative humidity is 100%. And what [378:32] do I mean? If the relatively if the [378:34] relative humidity is 100%. Then there is [378:37] no water to become the water is [378:40] saturated in the atmosphere. There's no [378:41] more water that can go enter the [378:42] atmosphere. So if you sweat your water [378:45] will just stay on your skin and actually [378:46] it won't work. Uh and similarly here if [378:49] you have a temperature a a thermometer [378:51] that's covered in water but the entire [378:53] uh atmosphere and like the air around [378:56] that temperature that thermometer is [378:58] also is also saturated with water then [379:00] there is no evaporative effect and the [379:02] temperature reading will be the same as [379:03] the dry bulb temperature. So this is [379:06] obviously rare unless you're like living [379:08] in a tropical rainforest. So what we can [379:10] do is actually manipulate or take [379:13] advantage of this uh temperature [379:14] differential when we are using wet [379:17] cooling. [379:19] So once again humid air leaves less room [379:21] for evaporative cooling. But if we do [379:23] have dry air then you can have a massive [379:26] difference where the ordinary air [379:27] temperature might be 35% on the dry bulb [379:29] but the temperature of a wet weted [379:31] sensor may be 22° because of the [379:34] evaporative effect into dry air. [379:37] However, if you increase the humidity, [379:39] then less water evaporates and the weted [379:41] sensor becomes warmer. But still, unless [379:44] you reach reach 100% relative humidity, [379:46] then you will still have a colder wet [379:49] bulb than your dry bulb. And this will [379:51] be really important when we discuss wet [379:54] cooling in more detail. Okay, but let's [379:57] first start off with dry cooling. So, [380:00] what is the set of uh set of [380:03] requirements here? [380:05] First of all, we have a rack which is [380:08] cooling which is being cooled via this [380:10] rack circuit here. That is just a closed [380:13] loop liquid circuit that's going to the [380:14] CDU and this is the collection phase. [380:18] Okay. Then in the transfer phase we have [380:22] 55 degree uh coolant that is heated up [380:26] that is then transferring its heat and [380:28] now heating up the facility circuit and [380:32] it now is going to the sealed coil which [380:37] essentially passes air across the water [380:40] and you can think of it as like a [380:42] radiator essentially but the water is [380:43] inside and sealed and what happens is [380:46] that the air passes outside the coil and [380:48] Water stays inside but heat still [380:50] transfers outside to the air. And then [380:52] you have this return where [380:55] now the cooled facility circuit would [380:58] come in at 40° and it will cool down the [381:02] 55° uh rack circuit to 45°. One thing [381:06] that I want you to calculate here is [381:08] what is the approach? So hopefully you [381:11] got 5 degrees because it's once again 45 [381:13] divided minus 40°C. [381:16] Okay. But let's say that the coolant [381:20] limit is 35° C [381:24] in the chips. So in this case, we cannot [381:28] actually we cannot actually cool the [381:30] chips to a sufficient uh temperature in [381:33] order for them to run properly if 35° is [381:35] their operating temperature. So dry [381:37] cooling would not help us run this [381:40] workload at peak throughput. And you can [381:43] even kind of reason about this by [381:45] looking at the dry bulb temperature. So [381:47] if the dry bulb temperature is 35° then [381:49] you can't get drier or rather you can't [381:53] get cooler than the dry bulb temperature [381:55] in a passive system like this because of [381:58] the fact that uh there's no like [381:59] compression there's no compression [382:01] there's no refrigeration there's no [382:02] evaporation. So the best that you could [382:04] get is 35° and that would involve like a [382:07] perfect temperature sorry a perfect heat [382:09] transfer which we cannot achieve in [382:11] reality. So in this case the the system [382:14] is not able to cool the racks to the [382:16] different to to the allotted [382:18] temperature. But now let's hit the [382:20] nitrous oxide. Let's go need for speed [382:22] mode. And what can we do to improve the [382:24] system? What we can do is called [382:27] adiabatic cooling. So what does [382:29] adiabatic cooling mean? It's actually a [382:32] meteorology term what's used in like [382:34] weather forecasting. And all it means is [382:37] that even though the water loop is [382:40] closed, what we can do is actually wet [382:45] the co the coils of the water. And you [382:49] can see here you'd make the coils wet. [382:51] And what this does is that as the warm [382:54] coolant comes in, this cool air that um [382:59] is humid will deposit water vapor on the [383:02] coils and then cause that water to [383:05] evaporate uh and then become warm. [383:08] Sorry, it will cause that water to [383:10] evaporate. Exactly. uh it become warm [383:12] air and then it will remove much more [383:16] heat from the warm coolant than through [383:18] just like passive air cooling with dry [383:20] air which would be this step here. So [383:23] essentially what you do is you have this [383:24] wetted pad which you pass air through [383:26] and they get this cool humid air that [383:28] that would probably condense on the [383:30] coils and then as you pass the hot fluid [383:33] across the coils that water will [383:35] evaporate and then reject more heat than [383:38] normal. So this would be one way we can [383:40] overcome this heat limit if the air [383:44] outside is 35° C using what's called [383:47] adiabatic cooling. And worth noting here [383:49] is that the coolant loop is still closed [383:52] but now we have to add water on the [383:54] outside. [383:56] Okay. And now let's just discuss a [383:58] completely wet tower which is to say um [384:01] there is a evaporative effect going on [384:03] within the circuit within the coolant [384:06] loop. So what you'll notice first of all [384:09] is that the rack circuit is the same. So [384:13] here we're collecting water from the [384:14] racks and we're transferring it to the [384:16] coolant device unit and then the coolant [384:18] device unit is transferring it to [384:20] another heat exchanger which has the [384:22] tower circuit. So essentially the water [384:24] tower is what passes water along a some [384:27] type of tower which has a large uh [384:29] structure in order to pass water through [384:32] like this type of fill and allow as much [384:36] of it to evaporate as possible in order [384:38] to reject heat through evaporation. So [384:40] if we go to the release phase, [384:44] what we what happens? So water falls [384:46] over fill, air takes heat, and some [384:49] water evaporates. And even though it's [384:51] 35° outside, the wet bulb temperature is [384:53] 28° [384:55] on the actual fill. And therefore, we [384:59] have this heat and water vapor that's [385:01] escaping through the water uh through [385:03] the water uh tower. And then also any [385:06] remaining water will fall into the basin [385:08] where we'll also pump in some makeup [385:09] water and at some point also do the [385:12] water rejection for any water that has a [385:14] high mineral content. And then if we do [385:16] the return we see that in this case we [385:20] still get a rack coolant that exceeds [385:24] 35° C even though we have a 28° wet bulb [385:28] temperature. And this is simply because [385:30] we have to add another heat exchanger in [385:32] this wet tower configuration which means [385:36] that um there's just another place where [385:38] we have this temperature difference [385:40] which um is not perfect. And then this [385:43] is also considering the high humidity uh [385:46] environment. So if we come back right [385:48] here right we said in high humidity it's [385:50] 28° wet bulb but in low humidity we have [385:54] 22° wet bulb temperature. So if we [385:58] switch to low humidity now with 22° C, [386:03] we actually can reach our desired rack [386:07] coolant supply of 35° C and run our GPUs [386:12] at the desired throughput and the [386:15] desired load. So that was a kind of a [386:17] nice worked example of going from [386:19] collect where we warm the coolant from [386:22] the chips. Then we also cross the two [386:24] exchangers and interestingly we have [386:26] three water circuits here. Once again we [386:29] release water into the atmosphere [386:30] through evaporation and if we have [386:33] sufficiently low humidity we can achieve [386:36] our desired rack circuit coolant [386:39] temperature. [386:41] Okay. One thing that I want to note is [386:43] once again that you can still have a [386:46] closed rack loop but a evaporative [386:49] cooling tower because you have these [386:51] separate loops. So you really need to [386:53] reason about a system and say is all of [386:56] the coolant uh closed loop or is only [386:58] some of it because this could be a [387:01] differencemaker between whether or not [387:02] the system requires a high amount of [387:04] water in order to run. Whereas with a [387:06] dry cooler, right, um you have a closed [387:08] rack loop, you have a closed facility [387:10] loop that is either water or glycol. [387:11] Those are two common uh materials that [387:13] you use or liquids rather that you use [387:15] for the coolant and then the dry cooler [387:17] which is just exposing the pipes to [387:19] outdoor air without any evaporation [387:22] within the the facility loop itself. So [387:24] those are the two different options [387:26] here. We have the dry cooler and the wet [387:28] tower. And there you have it. So moving [387:31] on, let's talk about the compressor a [387:33] bit. So if you're using a compressor as [387:37] is the case in Abalene, let's say that [387:39] you have a 10 megawatt heat transfer [387:43] from the racks to outside. But now also [387:46] the compressor electricity becomes heat [387:49] in this cycle. So you need to add that. [387:52] And it's worth noting that you'll get [387:54] more heat in the output because of the [387:56] electricity that the compressor used. In [387:58] this simple example, we have 12 [388:00] megawatts of computing heat plus [388:01] compressor heat that will go through the [388:04] chiller and um lower the temperature [388:06] through refrigeration. [388:08] Without a compressor, outdoor cooling [388:10] can still remove all 10 megawatt, but [388:12] that's only if temperature and [388:13] capacities allow. So, most likely you [388:16] cannot always guarantee that through [388:18] passive cooling, you can remove a [388:21] certain amount of heat throughout your [388:22] data center, especially if it's really [388:24] hot outside. That's why having a [388:25] compressor is really handy. So, as I [388:29] mentioned, cold weather can essentially [388:31] let the compressor rest where [388:33] essentially you have the warm water from [388:35] the racks going to outdoor cooler, which [388:36] is cooling with the air outside and the [388:40] cool water comes back to the racks and [388:41] you'll have like a sort of like economy [388:43] mode or eco mode where the compressor is [388:46] off and you only have the fans and pumps [388:47] on. So you still have some electricity [388:50] with the fans and the pumps, but the [388:53] compressor, which is definitely [388:54] consuming the most of the electricity, [388:55] is not running in that circumstance. [388:58] Okay, so now we're going to do just the [389:01] big boss um comparison here of all of [389:04] the different types of cooling that we [389:05] just described. So first of all the [389:08] front end is consistent which is to say [389:12] you have the IT coolant loop then the [389:14] coolant device unit distribution unit [389:17] and the facility water loop. But the [389:19] question is what connects after the [389:20] facility loop. So we can split up into [389:22] three categories of what is happening [389:24] after the purple rectangle here. First [389:27] of all, we have the no chiller example [389:31] where there is no type of like [389:33] refrigeration or condenser with a [389:34] compressor. In this type of example, we [389:37] can use a dry cooler which just has this [389:41] passive heating through a fan with the [389:45] coils outside with the optional adabatic [389:48] assist which we talked about was like [389:49] the nitrous oxide from Need for Speed. [389:51] Or you can actually use a wet tower in [389:55] addition to having no chiller. So what [389:56] what would this look like? It's [389:57] essentially the evaporative coolant that [389:59] we discussed where you have this [390:01] separate third exchanger loop which has [390:04] the coolant distribution that then [390:06] evaporates as water and then needs to be [390:09] recycled with the makeup water through [390:10] the wet tower. And these are the two [390:13] types of no chiller options. Okay, now [390:17] let's talk about the chiller options. So [390:19] there's two of these as well. There's a [390:21] air cooled chiller and a water cooled [390:23] chiller. So starting off with the air [390:24] cooled chiller, you have this sealed [390:26] refrigeration loop where essentially you [390:28] have the refrigerant that's looping [390:30] through with a compressor in order to [390:33] cool down. And then once the compressor [390:35] is heated up, you cool it down in an air [390:38] cooled condenser such that uh you have [390:40] the the vapor refrigerant that is [390:43] condensed down to liquid again and [390:45] recycled into the loop. And then we have [390:48] opt optionally as a third way the water [390:51] cooled chiller. So the seal refrigerant [390:53] loop stays the same except here you have [390:57] a water cooled condenser. What does that [391:00] do? It's essentially a heat exchanger [391:02] between the refrigerant loop and the [391:06] other loop which is the water loop. So [391:08] you have the seal refrigerant loop that [391:10] brings in this refrigerant that captures [391:13] as much of the heat as possible from the [391:15] facility water loop. And then once the [391:18] refrigerant loop is heated up, you have [391:20] another heat exchanger called the water [391:23] cooled condenser that will either [391:25] interact with a dry cooler or a wet [391:27] tower. And if you're paying attention [391:29] here, that's this or this. So the water [391:32] cooled chiller is like almost like the [391:34] big boss because it really combines like [391:36] a lot of different components together. [391:39] And if we look at the diagram way over [391:43] here, what that would look like is you [391:46] would add another loop here called a [391:48] sealed refrigerant loop where before [391:51] this heat exchanger, you would have the [391:53] refrigerant going through the compressor [391:54] that would be interacting with the [391:56] facility circuit. So you'd have the [391:57] facility circuit, the refrigerant loop, [391:59] and then the tower circuit. So four [392:01] circuits in total that perform the [392:03] heating obligation. Um, I think we [392:05] should like not get too lost in the [392:07] weeds here, but just realize that there [392:09] are many different ways that you can [392:10] cool these data centers and these are [392:13] some of the most common configurations. [392:15] Okay, now let's talk about a really [392:17] important uh metric here called the [392:19] coefficient of performance, the coop. So [392:22] the coefficient of per performance tells [392:24] us how much heat each unit of [392:26] electricity moves. So what you should [392:27] know is that um when you have a [392:29] compressor you can kind of have like a [392:31] almost infinite infinity uh efficiency [392:34] ratio where like for one unit of [392:36] electricity you can use you can remove [392:39] four units of heat. And how is this [392:40] possible? It's because you're taking [392:42] advantage of the temperature difference [392:44] with the building and the environment. [392:46] So if you used one unit of electricity [392:48] to remove four units of heat, you would [392:50] have a coefficient of performance of [392:52] four. Okay. So this is a really common [392:54] way that we use to determine to what [392:57] extent our cooling is efficient in [393:00] particular when we're using compressors. [393:03] And this also is a very good way to [393:06] determine how hot weather affects [393:09] cooling electricity. [393:11] So let's assume that we have a 8 [393:13] megawatt computing heat aotment that we [393:16] need to remove from the facility and [393:21] assuming that our cooling weather [393:24] coefficient of performance. So sorry [393:26] cool weather let's just say like outside [393:28] it's cold the coefficient of performance [393:30] is eight then we have 1 megawatt of [393:34] cooling electricity required to get rid [393:36] of that heat and if we have 10 megawatts [393:39] of electricity we can do it um in fact [393:41] even if we reduce the computing power [393:43] and we consider the auxiliary power as [393:45] well we are completely within our limit [393:48] but if it's too hot outside and let's [393:51] say that the hot weather coefficient of [393:54] performance right here is half. So we [393:57] can only remove four units of heat for [393:59] every one unit of electricity. Then now [394:01] we need 2 megawatts of electricity to [394:03] remove 8 megawatts of heat. And at a [394:06] full load, we would actually require [394:08] more electricity precisely 10.4. [394:12] So 400 kW more than what we can actually [394:14] afford. So in that case, we'd have to [394:16] fit to the available power, which is to [394:18] say we would have to slightly reduce our [394:20] load to 7.68 megawatt from 8 megawatt. [394:24] And this is another way that you would [394:26] have to have to buffer and essentially [394:28] to reduce the threat of overheating your [394:32] data center. And now you probably [394:34] understand the meme that I showed you [394:36] guys a long time ago where we talked [394:37] about take it easy buddy. Think of the [394:40] data centers when it's really hot [394:41] outside. So you need to realize that the [394:43] quotient of performance is heavily [394:45] affected by the temperature that's [394:47] outside. Okay. Now let's also talk about [394:51] an interesting question which is these [394:53] data centers they generate a lot of [394:54] heat. So sometimes when people are smart [394:57] they will essentially use this heat for [395:00] district heating and what what is [395:01] district heating is essentially like the [395:04] heat from a data center will be used and [395:05] recycled to heat up other buildings [395:07] which is a really nice way to not waste [395:09] this heat. Uh so let's assume that the [395:12] data center runs obviously 24/7 and from [395:16] midnight to 6:00 a.m. maybe the data [395:18] center has a contract with a hospital to [395:21] supply 2 megawatts of heat. So that [395:24] would be their aotment for the entire [395:27] day whereas the data center produces 4 [395:31] megawatts of heat throughout the day. So [395:33] if you had such a such a coordination [395:35] scheme then for 6 hours of the day you'd [395:38] have to direct this heat to the hospital [395:40] and then for the rest of the day uh you [395:42] would have to release it outdoors. So [395:44] this is just to simply highlight the [395:46] fact that there are ways to re reuse [395:48] this heat and recycle it and it may not [395:52] be the case that you can always recycle [395:53] it. So this would require the cooling [395:56] processes to adapt in real time or at [395:58] least have a schedule for diverting heat [396:01] outside versus to some district heating [396:03] system. Okay, to close off this chapter, [396:06] I want to discuss this interesting case [396:08] study which was the 2013 flood in [396:12] Toronto, Canada. So let's first set up [396:16] the uh the context of what's going on [396:18] here, the background. So, interestingly, [396:20] Toronto uses Lake Ontario to cool [396:22] downtown buildings. So, Lake Ontario is [396:25] a lake that is along the border of [396:26] Toronto and at the bottom of the lake, [396:28] it's always 5° C. So, not freezing, but [396:32] very cold. So, they use this deep intake [396:35] and then pass it through their water [396:36] treatment plant and use this as both a [396:39] water supply for the city, but also a [396:42] heat exchanger to essentially cool down [396:45] buildings, but also cool down data [396:47] centers as the heat exchanger. So, it's [396:49] really cool how this works and uh I [396:51] think you can understand the mechanism. [396:52] It's like a free air conditioner [396:54] essentially except you have to power the [396:56] pumps of course. What's worth noting [396:58] here is that there was some issues [397:00] during the flood. And all I'll say is [397:03] that this building 151 Front Street West [397:06] is Toronto's telecom hub and stores like [397:09] all of the different telecom [397:10] connections. So, you might have already [397:14] estimated or predicted what would [397:15] happen. There was some issues with the [397:18] power supply for the uh telecom hub. In [397:21] fact, the generator started which meant [397:24] that you could still heat up all of the [397:26] build all of not heat up but rather [397:27] power all of the racks. But [397:30] unfortunately, this district cooling [397:32] system was disrupted and and essentially [397:36] they had a redundancy on the generators [397:38] here but not on the N-wave cooling and [397:40] that meant that the heat started to [397:41] build up in the system even though the [397:43] racks were running. So what they did as [397:45] a sort of a mitigation member uh [397:47] mitigation measure is to shut down [397:49] equipment and move services to like only [397:52] the essential ones and essentially shut [397:54] down non-essential systems and they [397:56] furthermore um move the compute [397:59] workloads from a suite that was hoting [398:00] that was heating sorry above 43° C to [398:04] another like portion of data center [398:06] which was recovering sooner um and [398:09] transferring the services there. So they [398:11] had to do some like real time uh [398:14] transfer of their workloads which was [398:16] quite interesting. In fact, there's this [398:17] uh case study where the CTO of this [398:20] company called Uberflip reported that [398:23] they worked from like midday all the way [398:25] to 1:00 a.m. to make sure that these [398:27] jobs were running and also that they [398:29] were removing the non-essential uh [398:31] workloads in order to reduce the heat as [398:34] much as possible. And there you have it. [398:36] So interesting case study. The entire [398:39] account here on what happened is in the [398:41] link. So I recommend it. This was once [398:43] again the 2019 2013 uh July 9th flood in [398:47] Toronto. Okay. As a summary in this [398:50] slide, what does each cooling path [398:52] depend on? Well, if we're doing dry non- [398:53] evaporative cooling, then it's only [398:55] based on electricity apart from the [398:57] initial fill as well as operating uh [399:00] dependencies. But then if you're using a [399:02] wet cooling tower or just wet cooling in [399:04] general, you're both you're based both [399:06] off of water and electricity. So this is [399:09] the set of factors that you need to take [399:12] into account when you're considering [399:14] different cooling technologies. And [399:16] within each one, as we saw in that like [399:18] wacky chart, there are a lot of [399:20] different options. All right, chapter [399:24] 13, engineering, procurement, and [399:27] construction, aka EPC. Let's get the [399:30] boots on the ground and the hammer is [399:31] flying. So to give a one sentence resume [399:36] or rather summary of what each of these [399:38] three terms could mean in relation to a [399:39] data center, let's say that engineering [399:41] is sizing the switch gear, transformer [399:44] and connections. So switch gear being [399:46] the electric electronic components which [399:49] route the electricity to the different [399:52] data halls and the different buildings [399:53] and also make sure that the voltage and [399:55] the current are in a safe and operable [399:57] range. the transformers for changing the [399:59] voltage and then connections for [400:00] actually connecting the circuits and [400:02] then procurement. Order the coordinated [400:05] package and reserve factory capacity. [400:07] Finally, construction, build the [400:08] foundations, bring the skids in which [400:11] have all the equipment and connect the [400:12] plant together. So, let's do a little [400:15] case study here. Let's say that you're [400:18] building a data center which has 20 [400:20] megawatts of IT capacity or rather load [400:24] that needs to be supplied and you [400:27] originally had a rack design that was [400:30] 100 kW but now it's been changed to 200 [400:33] kW. So essentially what we're saying [400:35] here is that the rack density has [400:37] doubled. So if we just make this clear [400:40] through this diagram, the same 20 [400:42] megawatt is concentr concentrated now [400:44] into half as many racks where before in [400:47] a 2 megawatt zone of this data center of [400:50] which there would be 10 since we're at [400:51] 20 megawatt you have half as many racks [400:54] as before. So in this uh 2 megawatt [400:57] zone, we originally intended to have 20 [400:59] 100 kow racks, but now it's 10 200 kow [401:03] racks in the zone. And then you'd have [401:05] 10 of these zones to get your 20 [401:07] megawatts of it power. And so let's just [401:11] talk about some hypotheticals here. [401:14] First of all, the new rack exceeds its [401:16] existing electrical branch rating. So [401:18] let's assume that we're using the same [401:20] shared bus to power these data centers, [401:23] these racks, sorry. In that case, each [401:26] branch, if we assume had a 160 amp [401:28] limit, then we're approximately doubling [401:30] the amperage if we double the power. and [401:33] therefore we will exceed that branch [401:36] limit. So already the branch current [401:40] needs to be modified or to be more [401:43] specific the equipment that actually [401:46] restricts the branch amperage needs to [401:47] be upgraded. And then let's also talk [401:50] about the cooling. So as we remember [401:54] since the pressure is proportional to [401:57] the mass flow rate squared let's talk [402:00] about some math. So assuming that [402:02] originally we needed 2.4 kg/s of mass [402:07] flow rate to cool a 100 kow rack without [402:11] loss of generality. Let's assume we need [402:14] to double that mass flow rate in order [402:16] to maintain the same 10° temperature [402:19] rise. [402:20] So therefore we're at 4.8 kg/s of um [402:25] flow rate. And and once again if we [402:28] assume that the branch limit in the [402:30] cooling circuit is 3 kg/ second 3 kg/s [402:34] then now you need to rethink what's [402:36] going on there and then also let's [402:38] propose or hypothesize that the pressure [402:41] difference across the connection was [402:43] limited to 60 kilop 60 kilopascals on [402:46] the branch then now you went from 20 to [402:49] 80 kilopascals because of this um [402:52] relationship with um the pressure and [402:54] the mass flow rate. So the cooling [402:57] system would also need to be rethunk or [402:59] rethought rather if you had such a [403:02] increase in power density and then let's [403:06] not talk about the weight right so the [403:08] rack change [403:10] concentrates the floor load at few [403:11] positions where originally you had a one [403:15] unit load per foot because there was 20 [403:18] racks in one zone now you've doubled the [403:20] load in the same 4ft area or whatever [403:22] the area is of the rack that you're [403:24] building and So, this is kind of like a [403:27] a contrived example of like all of a [403:30] sudden your racks are twice as dense. [403:32] However, I would argue that once again [403:36] by referring to that Bank of America [403:39] research analysis [403:41] graph where they showed almost the [403:43] exponential increase in rack density [403:45] over time. It's almost like this exact [403:47] situation is playing out in real life [403:50] just over a longer time period. So every [403:53] few years the rack density doubles and [403:56] then therefore you need to make sure [403:58] that your pipe branches your valve [404:00] fittings can handle the increase in [404:02] pressure the increase in mass flow rate [404:04] that your branches are able to high have [404:07] a higher amperage or you convert them to [404:10] 800 volt DC as we discussed in chapter [404:12] 9. So many different ways to skin the [404:16] cat in this case. In any case, that's [404:18] some food for thought for what changes [404:21] would be required as density of compute [404:24] increases as a function of time. Okay. [404:27] So, we also want to talk about a bit of [404:29] prefab in the EPC chapter because it's [404:32] becoming an integral process of how data [404:35] centers are built. So, prefab is almost [404:37] a self-explanatory term. It means [404:39] shifting assembly into the factory. So [404:42] in the factory we can probably handle [404:44] power distribution make it more [404:47] integrated. The coolant manifolds can be [404:49] assembled. The controls and testing can [404:51] come pre-shipped. The internal wing can [404:53] be done within the factory. And it can [404:55] even be tested as well before reaching [404:57] the site. And once it's on the site, [404:59] well you still need to do the [405:00] foundations and the access. Uh you need [405:02] to make sure that the utility and the [405:03] cooling plant is actually connected and [405:05] uh distributed. Then you need to place [405:09] the different components of the [405:11] foundation and the field joints and then [405:13] also bring in the IT racks and the [405:14] integrated tests. Essentially what we're [405:16] trying to do here is bring as much of [405:17] the work that can be parallelized into [405:19] the factory and then just only do the [405:22] parts where we have to accept this join [405:24] system on site. So let's talk about a [405:27] reall life example. It's called project [405:29] Houdini which is an AWS project that uh [405:32] is doing precisely that. So, they've [405:34] partnered with Certino Electric and this [405:36] is their factory. So, you can see a lot [405:38] of switch gear that's being assembled in [405:41] the factory as opposed to on site. And [405:44] these data hall skids are both power and [405:48] cooling infrastructure. And once again, [405:50] they do the factory assembly test on [405:52] site and then they connect it once it is [405:54] arrived. [405:56] If we talk about a specific product, [405:58] Seammens, which we all know as the [406:00] company that makes a lot of different [406:02] data center infrastructure, partnered [406:05] with Compass, which is a data center [406:08] company that makes data centers to make [406:10] a single skid switch gear transformer. [406:13] So on one factory-uilt package we have a [406:16] seaman's switch gear which is [406:18] distributing the power to its separate [406:20] feeders and also has it circuits and [406:21] other breakers to ensure that in the [406:24] case of a voltage surge or some current [406:27] irregularity the power is faulted and [406:31] then we have the transformer which is [406:33] actually going to be doing some voltage [406:34] conversion into one units and internal [406:37] connections and assembly therefore for [406:39] this specific process cow into the [406:42] factory. [406:44] In fact, one thing that you might have [406:45] thought is, okay, if you're doing all [406:47] this like factory prefab work, do you [406:50] get like this insane like modeled Lego [406:52] situation where you have a bunch of [406:54] pieces that are being fit together, but [406:55] it's all custom and hard to reason about [406:58] what will fit into where. So, luckily, [407:01] the open compute project, which I've [407:03] referenced once so far in this course, [407:05] has essentially a guide which defines [407:07] the interface between suppliers. So for [407:09] example, a UQD which stands for [407:12] universal disconnect might be specified [407:14] in a open compute project guide such [407:17] that supplier A can make its piece and [407:20] just add a UQD plug plug and then you [407:24] supplier B can have the UKD socket which [407:27] will accept that plug and then now they [407:30] don't have to worry about the pet the [407:32] piece parts not fitting in. So [407:34] essentially it's kind of [snorts] [407:36] setting the contract for the output and [407:37] the input and allowing the companies to [407:39] still make the function inside. So let's [407:42] say that night you've now connected [407:43] these universal quick disconnects. And [407:46] unfortunately though if we remember [407:48] since you're still using those 200 kow [407:50] racks uh these sockets are not that [407:53] useful to you because uh it they only [407:56] support 3 kgs per second of coolant and [407:58] you need 4.8 kgs per second. So even [408:01] though the connectors made the branch [408:02] still needs redesign. [408:04] Another example of a open compute [408:06] project initiative is [408:10] essentially standardizing the open [408:12] source rack. So whereas conventional [408:14] racks with server level PSUs would look [408:16] like this where on every single shelf [408:18] you have a PSU in the open compute [408:22] project standard you have the rack power [408:24] distribution unit on the top and it's de [408:27] delivering three-phase power just like [408:29] the the conventional form but instead [408:31] you have these open source racks with [408:33] consolidated PSUs and this gives you [408:36] just a lot of benefits because now the [408:38] PSU providers can focus on making a tray [408:41] level solution [408:42] and the comput providers can focus on [408:45] making their own compute level solution [408:47] without having to worry about [408:48] integrating the power supply units. So [408:50] there's one great benefit of the open [408:53] compute project and in fact it's the [408:55] open compute project that is [408:57] spearheading the 800vt DC effort along [409:00] with its industry partners. So the open [409:02] compute project is kind of like [409:04] sponsored and supported by Nvidia, [409:06] Microsoft, all of the usual suspects but [409:08] it is technically still independent from [409:10] my understanding. [409:11] All right. So just to give a high level [409:14] overview of what we've been discussing [409:15] in terms of pre-fabrication, [409:17] the one line we need to remember is [409:19] moving acceptance testing into the [409:21] factory. So what's now done in the [409:24] factory is internal wiring and [409:25] protection, internal piping and leak [409:26] test, controller logic and simulated [409:29] signaling. And what's now checked at the [409:31] site is field cables and protection [409:33] coordination, the actual joints that [409:36] attach what has been prefabed and then [409:38] measuring the flow once you connect the [409:40] different pipes of for example a cooling [409:41] manifold and then real sensors and [409:44] response equipment. So even though you [409:47] can simulate the signals in the factory, [409:48] it's probably still smart to verify that [409:51] the signals and the sensors work on site [409:54] as well so that you don't have any bad [409:55] surprises. And before we end off this [409:59] example of the 20 megawatt power [410:03] concentration shift, let's say that now [410:08] we've modified our equipment and in [410:11] these zones which each have 20 racks, we [410:16] have electrical, cooling and network [410:19] done to varying degrees. So what we can [410:23] see pretty clearly is that in order to [410:25] satisfy the rack's operating capacity, [410:29] you need to have all three of these. So [410:31] at this current point in time, let's [410:34] propose that only zones A21 or rather [410:37] racks A21 to A60. So the first two zones [410:40] or rather the second and third zone [410:42] since this is the first zone have now [410:44] met all of the homologation for [410:46] electrical cooling and network. Um let's [410:48] also actually add cooling to this zone. [410:52] And now we have 60 more racks which are [410:55] powered. And um this is just a process [410:58] which you'll do over and over in this [411:01] contrived example in order to eventually [411:03] get at your 20 megawatt target and power [411:07] everything after this relatively large [411:10] change in rack density from 100 to 200 [411:13] kW. So at this point we would have 60 [411:15] racks of the 100 and that would give us [411:17] 12 megawatts of it. So there would still [411:19] be these two zones which need to be at [411:23] one in one instanceorked and then [411:25] another instance um electrically [411:28] connected. So I didn't even talk about [411:30] networking but you can imagine that when [411:32] I had the infinib quantum 2 switches [411:35] which had 32 down links and 32 up links. [411:39] Of course, here we're talking about some [411:42] GPUs that are only um imaginary. We [411:45] didn't actually specify what GPUs we're [411:46] talking about. But if you can think of [411:48] the original switches having double the [411:50] links, now that we've made the density [411:54] twice as dense, we'll probably no longer [411:56] need half of those links because of the [411:58] increase in GPU density. So that's [412:00] another thing that we didn't discuss, [412:02] but in any case would probably be [412:03] something worth considering. Okay, so [412:06] this is the last slide of this chapter [412:08] on EPC. It's just saying that as we [412:10] mentioned before, [412:13] Core Weeave had a deal with Applied [412:16] Digital to make them a set of data [412:18] centers last year in the fall. And one [412:21] important thing here is the time is [412:23] money argument where we need to connect [412:26] the next 50 megawatts while keeping the [412:27] first megawatt online. So essentially [412:30] the moral of the story here is that when [412:32] you're actually building the site in the [412:33] construction phase of the EPC and you're [412:36] able to put a data center online before [412:39] the entire envelope is done before the [412:41] entire site is complete then you need to [412:43] make sure that um you can keep the first [412:46] one online while building a second one [412:48] which would definitely require a lot of [412:50] contingency planning and allowing for [412:54] different variables. planning for [412:56] perhaps like some construction incident [412:58] that could happen while building the [412:59] second cluster and making sure that that [413:00] doesn't put the first one offline [413:02] because these data centers are bu [413:05] printing money and the the downstream [413:07] customers core we've want it to be [413:08] online. Okay. So now chapter 14 of 16 is [413:13] on the topic of controls operations and [413:16] reliability. So keeping a data center [413:18] running and troubleshooting when the S [413:23] word goes to the fan to speak politely. [413:26] So let's consider a hypothetical case [413:30] study here as we start off the chapter [413:32] where we have the plant dashboard that [413:34] demonstrates that the plant supply of [413:36] coolant is at 30° which is normal. [413:39] However, the row B rack has a chip [413:43] exceeding its 80° operating threshold. [413:46] So, it's at 85° despite the limit being [413:49] 80°. And this definitely signals a [413:52] problem. And it also signals the fact [413:54] that the adequate measure is not just [413:58] looking at the coolant temperature [414:00] alone. You need to make sure that you're [414:01] measuring like the chip level [414:02] temperatures as well. And we'll we'll [414:04] demonstrate why this is the case in this [414:06] chapter. So, essentially the short of it [414:09] here is that row B is still getting [414:11] water at 30° but only half the flow. So [414:14] if we look at the look at the column the [414:16] rows here the rack power is the same the [414:20] water into row B is the same but now the [414:22] flow rate has gone from 100 kg/s to 50 [414:25] kg/s and consequently the water leaving [414:29] row B has changed from 35° to 40° a 5° [414:34] increase and not only that the hottest [414:37] chip is now operating at 85° C which is [414:41] a 50° jump from its initial ial [414:44] temperature and 5° over the limit. So, [414:49] we can like postulate that perhaps [414:52] there's been some leak or that would be [414:56] really bad or probably a more a more [414:59] feasible example or a more real [415:01] realistic example would be there's [415:03] something in the circuit which is [415:05] causing a uh a reduction in flow. So, [415:08] let's look at how this chip incident [415:10] could unfold as a function of time. And [415:13] the title here is that lower flow can [415:15] leave chips too hot after temperature [415:17] stabilized. So what does that mean? [415:18] First of all, let's talk about the top [415:22] graph which plots the heat generated [415:25] versus heat removed as a function of [415:28] time. So the heat generated is constant [415:30] function. It's just this dotted line [415:32] here that's 2.09 megawatt. But then we [415:36] can see that there's a stepwise function [415:38] here when the flow fails when it halves [415:40] where the heat removed drops um [415:45] aggressively. In fact, since we know the [415:49] fundamental heat transfer equation, this [415:51] would mean that the heat removed would [415:53] have halfed. But then the interesting [415:56] thing worth noting here is as the heat [415:58] accumulates which is this golden uh [416:01] shape in between these two lines the [416:04] heat remove actually normalizes back to [416:07] 2.09 megawatt and reaches equilibrium. [416:11] So that's quite interesting to note. [416:12] It's like saying that essentially as the [416:14] chips are absorbing this extra heat [416:16] which is not being removed properly then [416:18] the difference in temperature is [416:21] increasing which allows for a bigger [416:23] heat transfer to occur because of this [416:26] uh equation the thermal heat transfer [416:28] equation and therefore it will reach [416:31] back this energy transfer of 2.09 09 [416:34] megawatt. But at this point, we've [416:36] already accumulated a lot of extra heat [416:38] within the rack. And then if you just [416:40] look at what would be the the graph of [416:42] the hottest chip over time, essentially [416:45] it would have this relatively sharp [416:48] increase at the beginning since all of a [416:49] sudden the heat removal has halfed and [416:52] then slowly taper off as the heating [416:57] equilibrium is regained. And once again [417:00] at each chip you have heat flow equals [417:02] temperature difference divided by [417:03] thermal resistance. So at the very [417:06] beginning the temperature difference is [417:08] smaller but now because there's less [417:11] fluid flowing over the uh chip it [417:14] increases in temperature with respect to [417:17] the fluid and then now once the [417:19] temperature increase temperature [417:20] difference gets big enough we'll reach [417:22] back this um equilibrium state but at [417:24] that point the chips are already at [417:25] 85°C. So in this example we can see that [417:29] the moral of the story is even if the [417:32] coolant loop is normally and still at [417:36] 30° Celsius that is not an indicator [417:39] that in the other parts of the system [417:41] everything is going okay. [417:43] Okay. So that was all for our example in [417:46] this chapter. Now let's talk about some [417:48] real case studies. So Google developed [417:53] with de mind a autonomous control [417:56] mechanism for cooling their data centers [418:00] and essentially in in 2016 the extent of [418:03] this algorithm was the AI would advise [418:06] some human to act. So it was still very [418:08] much human in a loop but then only after [418:10] 2 years the AI would control the cooling [418:12] at 5minut intervals with the operator [418:16] still setting limits and retaining the [418:17] override. But let's look at how this [418:19] system works in a set of four steps. So [418:23] I'm going to zoom in on each one. And [418:25] just as a reminder, this is every 5 [418:27] minutes. So what happens is that in step [418:29] one, our cloud-based AI pulls a snapshot [418:32] of the data center cooling system as [418:34] represented by a thousand thousands of [418:36] physical sensors. So essentially, it's [418:38] reading the sensors. Then in step two, [418:40] which is to the right, it will predict [418:42] the outcomes based on a neural network. [418:44] And then on step three, it will choose [418:46] within the limits defined by the safety [418:50] settings and with the goal of minimizing [418:53] future energy consumption. And then step [418:55] four, the optimal actions are sent back [418:57] to the data center since it's in another [418:59] data center that you're actually running [419:00] this neural network where the local [419:02] system verifies them against its own [419:03] safety constraints before [419:05] implementation. So you see that there's [419:06] a verify locally step and also there is [419:10] a step where the AI selects its actions [419:12] within the safety constraints. So [419:14] there's like a two-step safety uh [419:16] protocol here just to make sure there [419:18] isn't any uh instances where I suppose [419:21] the AI does some freak prediction and [419:24] makes the chips overheat. But that's not [419:26] really worth worrying about because if [419:28] we actually look at the plot of what [419:32] happened over 9 months, cooling energy [419:35] per unit improved from about 12% to 30% [419:39] below the preAI baseline. And you can [419:43] also see that as training data [419:45] increased, this is one of the market [419:47] indicators of the performance in the [419:49] cooling system. And I guess this is just [419:51] a demonstration of the power of scaling [419:54] data. So this actually worked in [419:56] practice and this was implemented in [419:58] 2018. So I imagine Google has [420:01] generalized this to me many more of [420:02] their data centers today and these [420:04] algorithms are like working as I'm [420:06] speaking and as you're watching this [420:08] video which I think is quite [420:09] fascinating. Okay. So when it comes to [420:13] um operations in a data center there's [420:15] three control layers and each one solves [420:17] a different problem. Local pump controls [420:20] just tells the pumps what flows to have [420:24] and what pressures to maintain. This is [420:26] not something that a human would ever [420:29] manually tune, but it's something that [420:31] you could watch in an instant analysis [420:33] for example. Then you have the plant [420:35] controls which is to say like when do [420:37] you run the cooling units when do you [420:40] decrease the max energy that the cooling [420:42] units are moving and then you also have [420:44] the compute level which is to say what [420:47] are the workloads that you're running [420:48] and should you move or delay compute [420:51] based on for example cooling [420:53] restrictions. So this is the lowest [420:56] level on the left and then these two are [420:57] at a similar level abstraction except [420:59] this one is on a facility level versus [421:01] the workload schedule workload schedule [421:03] being on the compute level. Okay. So [421:06] let's first focus on the plant controls [421:08] and see how this plays a role in how we [421:12] operate a data center with the following [421:14] scenario. So the title here is a new job [421:18] needs cooling that takes 3 minutes to [421:20] start. And let's assume that this new [421:23] job is 2 megawatt which is to say that [421:26] its compute budget or the amount of GPUs [421:28] it requires is equivalent to 2 megawatt [421:30] of heat and that currently there's [421:33] already 4 megawatts of base load running [421:35] on this data center. those like [421:36] inference workers or maybe there's [421:38] already like a pre-training job that is [421:40] happening simultaneously. [421:42] Given these uh circumstances and the [421:46] fact that at the given moment in time [421:48] when this new job comes online we have [421:51] only 5 megawatt of cooling available. [421:55] If we started the job instantly [421:58] then we can assume that the cooling [422:01] would not come online in um at the same [422:03] time and so there would be one megawatt [422:04] excess heat for 3 minutes while the [422:07] plant controls bring cooling units [422:09] online and that's not good. 1 megawatt [422:11] for 3 minutes may be detrimental and you [422:14] could lose millions of dollars of [422:15] hardware. So instead what the plantin [422:18] control does is it will first of all [422:20] bring the cooling online by uh or [422:24] increase the cooling online by 2 [422:25] megawatt equivalent to the amount of the [422:27] new job and then once the standby is [422:31] ready after 3 minutes we can start the 6 [422:34] megawatt job and avoid any issues from [422:38] having excess amount of heat build up in [422:41] our data center. And then you can also [422:43] notice in this in this example that [422:45] generally we have 1 megawatt extra [422:48] cooling versus the amount of heat. This [422:51] is just one way of having redundancy in [422:54] the cooling regime. Let's talk about a [422:56] few other ways. So Johnson Controls [422:59] which publishes information on cooling [423:03] standards for data centers has the [423:06] following guideline. So there's two [423:08] different types of chillers here. I [423:10] don't even know what they do and you [423:11] don't need to know either. We just need [423:12] to know that for the positive [423:14] displacement chiller, they recommend an [423:16] 80% uh cooling or load to cooling ratio, [423:19] which would say that if you have 8 [423:22] megawatts of heat, you should have 10 [423:24] megawatts of cooling. And then this [423:26] constant speed centrifugal chiller can [423:29] handle up to a 9 megawatt heat load [423:31] given 10 megawatt of cooling. And then [423:36] whenever there's a sustained load above [423:38] the thresh threshold, you would start [423:40] the next stage of of coolant such that [423:43] you're always rounding up uh to have [423:46] like if you were doing now um 9.5 [423:50] megawws of of heat, you wouldn't want to [423:53] only have 10 megawatts of cooling. You [423:55] would probably go to 11 or 12 depending [423:57] on what the ratio says. In any case, we [424:00] can imagine that based on the chiller [424:02] specification, you would give the real [424:04] level of redundancy for the cooling [424:07] cushion or the cooling buffer in your [424:09] data center. And what are the criteria [424:10] that you would consider overall? Well, [424:12] first of all, look at demand. So, if [424:14] you're using machine learning models [424:15] like Google, they'll probably see that [424:17] historically at 2 p.m. the amount of [424:20] compute spikes. And so, this would be an [424:23] expected load. you could ramp up the [424:25] cooling to preempt that and not have any [424:28] uh throttling on your jobs. Also, usable [424:31] capacity is just a question of like how [424:33] much flow do you have, how many cooling [424:35] units do you have, what's the weather, [424:37] and what is the required failure case? [424:38] And then lastly, what's the response [424:40] time? So there's this always this [424:41] question of startup delay versus thermal [424:43] buffer duration which is to say this [424:45] thermal buffer duration is this this [424:49] area here where maybe your data center [424:51] can withstand a bit of it but too much [424:53] and you're in a bad territory. So it's a [424:56] trade-off between living on the edge and [424:59] having a startup delay which costs more [425:02] electricity of course. Okay, this slide [425:04] is also cool because it talks about how [425:07] Intel stored water to outlast the server [425:11] uninterruptible power supplies, which is [425:13] to say that [425:15] in the case where in Toronto we ran out [425:18] of cold water because of the system [425:19] failing due to the flood, Intel has a [425:22] system in one of their data centers in [425:24] 2007 actually this was published where [425:26] it could still withstand or rather [425:29] support 12 minutes of cooling. um which [425:32] is to say it's designed to cover five [425:33] minutes of uninterruptible power supply [425:35] plus 7 minutes if there is a uh issue [425:38] with the cooling system itself. And in [425:41] fact during a 2006 outage it was able to [425:44] withstand 15 minutes of um autonomous [425:47] load with a lighter IT uh amount. But [425:51] nevertheless, this is one way of like [425:53] having essentially cold water as your [425:56] collateral to your cooling system [425:58] failing. [426:00] Okay, another cool thing that Google did [426:02] is partnering with the utilities in [426:05] Oregon to reduce demand during busy grid [426:07] hours. So essentially what happened is [426:09] that they had coordinated to have this [426:11] day ahead demand response where with the [426:13] utility pro partner northern Wasco [426:16] County uh PUD they scheduled less [426:18] compute during a forecasted g grid [426:20] constraint. So essentially what happened [426:23] is that Google would still serve their [426:25] search, their map, their video playback, [426:27] the stuff which needs to be served in [426:29] real time but as you know YouTube videos [426:32] are processed um asynchronously. So [426:36] likely they could do the translation for [426:38] the YouTube videos and video processing [426:40] on a deferred basis. So this is [426:42] precisely what they did. And in fact, [426:44] this is not actually what the load was, [426:47] but this is a representation of let's [426:49] say in the most intense hours of the [426:52] day, let's say 2 p.m. to 4 p.m. or [426:54] probably 2 p.m. to 6 p.m. would be more [426:56] accurate. Anyways, it's usually evening [426:57] actually. um [427:00] Google would pause the flexible job [427:02] during the grid and then just like shift [427:04] it to afterwards when there is less [427:06] pressure from other consumers like [427:09] houses or other people that are taking [427:11] advantage of the grid. So I think this [427:12] is a really interesting example of [427:14] essentially deferring work that is not [427:16] urgent for grid reliability and grid [427:21] resiliency. Okay, now let's talk about [427:24] Cloudflare, which everybody knows is [427:25] like the spine of security in the [427:28] internet. And what happened with [427:30] Cloudflare was a single point of failure [427:32] disrupted their three site cluster in [427:36] Portland, so also in Oregon. And what [427:38] happened is that their core services [427:41] including their dashboard API and [427:42] analytics all went down for some large [427:45] amount of time. And why did this happen? [427:47] It's because their site PDX04 [427:50] which stored their click house database [427:53] and their CFKA [427:55] went down and sites B and C which still [428:00] were up were no longer operational [428:02] because it turns up it turns out that [428:04] they depended both on this data center. [428:06] So the real outage removed the entire [428:08] PDX04 facility but sites B and C were [428:11] also nonoperational. [428:14] And so what happened after this incident [428:17] was that there was the code orange [428:19] declared and they had this whole uh [428:22] follow-up work to make sure this doesn't [428:24] happen again where they essentially [428:25] expanded the capacity to have more [428:27] servers and networks. They also fixed [428:29] the failover to remove single-sight [428:30] service dependencies and they tested the [428:32] facility lost to find and repair [428:35] remaining gaps. So obviously with [428:37] hindsight you'd say, "Oh, this is so [428:40] simple. Why did they not have the data [428:41] set or their database on all the other [428:43] sites? But given the complexity of data [428:46] centers, you can imagine that there are [428:47] like a thousand other potential single [428:49] points of failure. So maybe Cloudflare [428:52] had addressed 999 and this was the last [428:55] one and all it takes is one collateral [428:58] single point of failure to make a large [429:01] outage occur. But in any case, they did [429:05] some type of testing to find and repair [429:08] the remaining gaps. And I'm happy to [429:10] report or I guess they were happy to [429:12] report that the next power failure saw [429:14] APIs and dashboards recover in 7 [429:16] minutes. So the next year I think this [429:19] this was in November 2023. Yep. In March [429:22] 2024, the same facility loses power [429:24] again. So I mean I guess already this is [429:26] a sign that maybe this facility has some [429:28] power distribution inefficiencies or in [429:31] inadequacies. In any case, it lost power [429:33] at 1458 or 258 and at 3:05, 7 minutes [429:38] later, everything was back up and no [429:40] human intervention was required for the [429:42] API/ dashboard recovery. And um yeah, [429:44] analy analytics recovered later that [429:46] day. So clearly they did something right [429:48] here and um they went from a multi-hour [429:51] or multi-day outage, I actually don't [429:53] remember how long it was to a 7-minute [429:55] outage. In fact, now let's talk about [429:58] another type of outage which was in [430:00] Google's case due to hot temperature. So [430:03] this was in Europe West 2A which I [430:06] believe is in the UK. And what the quote [430:09] here says is that one of the data [430:10] centers that hosts zone Europe West 2A [430:12] could not maintain a safe operating [430:14] temperature due to simultaneous failure [430:15] of multiple redundant cooling systems [430:18] combined with the extraordinary high [430:20] outdoor temperature. So the first thing [430:23] is Google being the massive company it [430:26] is and serving billions of people [430:29] already has multiple redundant cooling [430:30] systems. But despite this still had this [430:34] outage which was probably in part due to [430:36] these extraordinarily high outside [430:38] temperatures in July of 2022. And what [430:41] they did in response was quite um quite [430:45] quite fascinating. So there was the [430:46] extreme heat which failed the redundant [430:49] cooling and then they did a protective [430:50] shutdown which is to say that Europe [430:52] West 2A was partially shut down and they [430:57] stopped some virtual machines such that [431:00] the other services could not get this [431:03] collateral damage from from the rest of [431:05] the system failing. And so this is like [431:07] some type of preemptive maintenance in [431:09] this um really bad scenario to prevent [431:12] it from propagating to the rest of the [431:14] system. And the actual timeline was that [431:17] restoring curling, restoring cooling [431:19] began a further 14 hours after the [431:21] service recovery. So let's look what [431:23] happened. Um 4 hours and 8 minutes was [431:25] the cooling repair. And then cooling was [431:27] restored but it still took another 14 [431:30] hours and 15 minutes to recover the [431:31] service which was the next day which [431:33] really involves restarting servers, [431:35] restoring dependencies and reconciling [431:37] state despite restart sequencing and [431:40] service state reconciliation continuing [431:42] right after cooling was restored. So, it [431:44] took them 14 hours, which makes sense [431:45] given the scope of these data centers [431:47] and how how much data they process. But [431:50] the the summary in this scenario is you [431:54] can purposely turn off your system when [431:58] it's starting to go south in order to [432:01] prevent the situation from becoming [432:03] worse than what it is currently. Okay, [432:05] let's now talk about Meta and Llama 3 in [432:08] particular where for the Llama 3 [432:09] pre-training which lasted for 54 days, [432:11] they were able to make progress despite [432:14] interruptions. So, in fact, in the [432:16] training of Llama 3, there were 466 [432:19] training interruptions with 47 being [432:21] planned and 419 being unexpected. And [432:25] all of these interruptions were [432:27] recovered from, meaning that there was a [432:29] greater than 90% effective training time [432:32] with only three incidents needing [432:33] significant manual intervention. And you [432:35] can imagine that when you're training [432:36] models on like the tens to hundreds of [432:39] million dollars in terms of cost, that [432:42] an interruption, which does reduce [432:44] effective training time substantially, [432:46] is costing millions and tens of millions [432:49] of dollars. So having a data center [432:51] which is able to recover from some type [432:53] of systemwide outage or lack of [432:55] synchronization is extremely critical. [432:58] So to talk about what Meta describes in [433:01] their redundancy scheme, first of all [433:04] they only update one maintenance group [433:05] at a time. So they call them trains and [433:08] these are groups of clusters where [433:11] they'll put one offline and purposely [433:14] make these groups such that one being [433:17] offline allows the entire workload to [433:18] persist. Um this is how they do their [433:20] maintenance and you can check this [433:22] article here to see what the domain [433:24] sizes are specifically. I just like this [433:27] graph in their article where they talk [433:28] about how the maintenance cost is such [433:30] it's like a U curve or like a V-curve [433:32] where the domain size as you increase it [433:36] well you have less interruptions because [433:40] you don't have to go and like turn off [433:42] all the individual units to maintain [433:43] them. But then there's a sweet spot [433:46] where in a specific domain size you have [433:49] just enough that you can maintain a [433:51] large amount and like efficiently [433:53] maintain your units but also the rest of [433:55] the units are still a good enough amount [433:57] to have the capacity where if you keep [434:00] on going to the right you get this [434:02] higher capacity loss from the domain. So [434:05] you can see that the maintenance is like [434:07] already this optimization problem of how [434:09] much should we turn off and how much [434:11] should we put online when we're [434:12] maintaining our systems. And in this [434:14] case, Meta has their answer and read [434:17] this article if you want to learn more. [434:18] Okay, this is the last slide in slide [434:21] 14. It's a knowledge check. So what is [434:24] it saying? Can row C take another 2.5 [434:27] megawatt under the given uh restraints. [434:30] So first of all in in normal operations [434:34] row C does 2.09 megawatt of power [434:38] consumption and this is done through [434:40] water cooling and we have 100 kg uh per [434:44] second of mass flow rate to to give us [434:47] 30 3 to 30° of Celsius input and 35° of [434:51] Celsius output with a maximum return of [434:54] 40°C. And now the question is if we [434:57] increase the workload by 2.5 megawws and [435:01] we assume that we have enough electrical [435:02] headroom in terms of like the amperage [435:04] on the buses as well as cooling headroom [435:07] in terms of the pressure increase that [435:10] we must achieve. Can we still maintain [435:14] this maximum return below 40° C? So if [435:19] you want to use the fundamental heat [435:21] transfer equation to see whether or not [435:24] we can uh get this answer, please be my [435:27] guess. Already the heat capacity is [435:30] known for water. So we can plug that in [435:32] here. And also the mass flow rate is [435:35] also constant. So we don't need to [435:36] change that. In fact the only thing [435:38] changing is delta t. In fact we just add [435:42] 2 we just put 2.5 megawatt on this side [435:44] of the equation and see what we get. And [435:46] in fact, it turns out the way that we [435:48] solved it here is simply by saying, [435:49] okay, we're already at 35° C. And if we [435:54] want to go to 40, how much energy will [435:57] we need to transfer? And it turns out [435:59] it's 2.09 megawatt. So, what you could [436:02] do here is get up to 4.18 megawws of of [436:07] compute, which would then turn into [436:09] heat, but you can't get into what would [436:11] be 4.59 megawws in on the right side. So [436:16] there's two ways of solving this. Either [436:17] you can go 40 minus 35 or equivalently [436:20] what I would do is say okay we want to [436:23] calculate Q. In fact we know Q should be [436:26] 2.5 megawatt and then we'll divide this [436:29] by the specific heat capacity of water [436:31] and then we'll divide this by the mass [436:33] flow rate which is 100. And of course [436:34] the units here will be wrong but what we [436:37] can see is that we would have a [436:39] temperature increase of 5.9° if we fix [436:42] the the the decimal there. And that [436:44] would give us what would that give us? [436:45] That would give us 40.9° C. So it's [436:48] above our limit. And the last thing here [436:50] is that what you could do instead is [436:53] increase the mass flow rate. So if you [436:55] instead substitute 100 into this [436:57] equation, you would get 2.5 [436:59] approximately. So that's another uh [437:01] parameter that we can tune is to say the [437:03] added load needs approximately 100 kg/s [437:06] of total throw. um if we also want to [437:09] modify that in order to take another 2.5 [437:12] megawatt. Okay. All right. So, chapter [437:15] 15, GPU cloud economics. And the Grim [437:19] Reaper Grim Reaper has made its return. [437:21] In fact, here the Grim Reaper is going [437:24] after PC gaming as a hobby as opposed to [437:26] CPUs, which was the last time. And it's [437:28] already taken consumer VRAMm, mid-range [437:30] availability, and low MSRPs, [437:32] unfortunately. But we'll just say too [437:36] bad for the consumers because we need to [437:39] make money for the data centers. And [437:41] let's talk about GPU cloud e economics. [437:43] So if we talk about coreview which is [437:45] one of the biggest hyperscalers [437:47] 98% of their revenue came from committed [437:49] contracts in 2025. So the real question [437:52] here is what is sold? Who carries the [437:54] risk and how is the cluster financed? [437:56] And that's what we want to dive right [437:57] into when it comes into when when it [438:00] comes to like the profit model for a [438:02] data center. You have of course on one [438:04] side selling compute through the model [438:07] API which is selling tokens tokconomics [438:10] and then on the left you have GPU hours [438:14] and cluster commitment where you [438:15] probably selling to somebody that's [438:17] selling to somebody else through their [438:19] model API. So this would be customers [438:21] running training or inference whereas [438:23] the right side is the provider operates [438:25] the model service and the end user is [438:27] just buying tokens. So these are the two [438:29] different examples here and you can [438:31] think of it as you run the inference on [438:33] the left. So you have your model your [438:35] application and you have to handle [438:37] serving scaling updates but you're still [438:40] using the core GPU infrastructure. So [438:42] this was here where you just buy GPU [438:44] cloud capacity from core and there's [438:46] another option where core runs the [438:48] inference for you. It's still your model [438:50] and your application but core does the [438:53] serving the scaling and the updates. So [438:55] the common denominator the common [438:57] denominator here is core of GPU [438:58] infrastructure but the how it's run is [439:01] core on the right where they handle it [439:03] and you on the left where you do. So [439:05] what are we talking about when we say [439:07] there are two ways to skin the cat? Um [439:10] let's look. So Anthropic for example is [439:13] a is a core customer and what they do is [439:18] use core's compute capacity but [439:22] self-management in order to run their [439:24] own workloads. So this would be the left [439:26] side where they're running running the [439:27] serving scaling and the updates. Whereas [439:29] another example of cursor who's a [439:31] fireworks customer or at least was until [439:34] the XAI acquisition acquisition. I'm not [439:36] sure if they still are. They have a [439:38] managed inference deal with fireworks [439:40] which is a um inference compute [439:42] provider. So what happens is that they [439:45] have the fast apply model which is like [439:47] their autocorrect tab model and [439:49] fireworks runs it and deploys it for [439:52] cursor on their behalf. So of course in [439:55] the managed inference uh paradigm you [439:58] have less worries about scaling and [440:01] serving massive increases in traffic. [440:03] That's more of the problem for fireworks [440:05] to deal with. Whereas in the anthropic [440:08] case you just have much more flexibility [440:09] in how you can run your workloads. And [440:11] also you don't have to worry about [440:13] coreweave looking at your model weights [440:15] or having to do some fancy encryption to [440:18] ensure that they don't see what you're [440:20] working on in terms of like the actual [440:22] architecture. So these are the two ways [440:24] that you sell GPU cloud capacity either [440:26] just the raw capacity which is sometimes [440:28] called bare metal and then the managed [440:30] inference which is not bare metal. Okay. [440:33] So when it comes to renting out compute [440:37] this market is becoming more and more [440:38] interesting given the volume and a lot [440:41] of people are comparing it to just like [440:43] regular financial markets and there's [440:45] three categories which are relevant to [440:48] discuss here. First of all, there's [440:50] ondemand, which is essentially paying [440:52] for use capacity without having a [440:55] multi-year term. So, there's no [440:56] contract. And you can think of it as the [440:58] enterprise claw code plan or the [441:00] enterprise um codeex plan is tailored [441:05] where you pay for your tokens that you [441:06] use. And so, the more you use, the more [441:09] you pay, but at the end of the month, [441:10] you can say goodbye. And this is on [441:13] demand. So what is worth noting with on [441:16] demand is it's almost reliable but it's [441:19] still not guaranteed. If you want [441:21] guaranteed capacity you need to sign a [441:23] contract but it's still much more [441:25] reliable than spot. What is spot? So [441:27] spot as the title suggests suggests is [441:30] renting spare capacity at the current [441:32] price where the prices are really [441:35] determined by market dynamics and can [441:37] change and also the provider can reclaim [441:41] the compute without notice or [441:44] essentially you're not guaranteed to [441:46] have this instance for however long [441:48] you're willing to pay. Instead it can be [441:50] taken away from you to uh essentially [441:53] adapt to some other capacity term which [441:56] is much more committed. So for example [441:58] to give a customer who has a contract [442:00] access to the compute. And so with [442:01] respect to which with with respect to [442:03] spot what you want to do is run only [442:06] workloads using redundancy. And what I [442:10] mean by that is if you're training a [442:11] model on a spot instance, you better be [442:14] checkpointing it, which is to say saving [442:16] the weight of the model intermittently [442:18] such that if the instance is reclaimed, [442:20] you don't have to restart the training [442:21] from zero. Instead, you can restart at [442:24] the latest checkpoint once you get the [442:26] spot instance back. Okay, so that's how [442:28] spot works. And it's usually the most [442:30] expensive of all of the options here. [442:32] And then sorry, it's usually less [442:34] expensive than on demand. So on demand [442:36] is more expensive because it's less [442:37] guaranteed. And then depending on the [442:39] committed capacity, it can be more or [442:42] less than spot. So what committed [442:43] capacity is is just a res. It's just a [442:45] defined term where you pay for like one [442:47] year of having access to this compute [442:49] and it's a contract. So it's an [442:51] obligation. Um and that's pretty much [442:53] it. So you have a GPU per hour [442:55] definition on the contract and you pay [442:57] that much over the over the term of the [442:59] contract. Okay. So when it comes to [443:03] having a contracted compute deal, a [443:06] longer contract trades repricing [443:08] opportunities for committed revenue. And [443:10] this is from the standpoint of the [443:12] contractor. So the person that is [443:14] selling this contract. If they sign a [443:16] one-year deal, then this means that [443:18] afterwards at year 2, they can renew or [443:21] sell to a new customer. And if it's the [443:24] case that now compute is even more [443:25] constrained, this is the most attractive [443:27] deal because you can sign a one-year [443:29] deal, which is usually more expensive [443:31] than a 5-year deal, and then renew it. [443:33] But then the risk is that if compute is [443:35] now worth less than it was uh that that [443:38] when you sign the contract, then just [443:40] like any other contract or like just [443:42] signing a fixed mortgage, then you'll [443:44] have a worse rate. And the same thing [443:46] applies to three years and fiveyear [443:48] deals. So, these are like the main deals [443:50] that you'll see in terms of compute. And [443:54] it's quite interesting because a lot of [443:56] the deals that are signed as a five-year [443:58] deal are currently like insane bargains [443:59] because, as you know, the compute [444:02] environment currently is insanely [444:03] constrained. And what this means is that [444:05] if you signed a 5-year deal like last [444:07] year, the price that you're paying is [444:09] much less than the market rate, and you [444:11] still have that that that uh that [444:13] pricing power for four more years. So if [444:16] you have like a hyperscaler or just even [444:17] like a Frontier Lab that has one of [444:19] these contracts, they'll be making a lot [444:21] of profit off of it for the next 3 four [444:23] years, but then once they have to [444:24] resign, they'll have to pay a nice [444:26] premium on what they had before as their [444:28] contract price. Let's just look at what [444:31] the H100 prices were over the past year. [444:34] So, what specifically we're looking at [444:36] here is the semi-announces public price [444:38] history of the 25 to 75 percentile uh [444:42] price range for H100 one-year contracts [444:45] on a $1 per GPU hour basis with the [444:49] caveat or with the condition that [444:51] usually these contracts have a one [444:53] quarter prepayment and then the rest is [444:54] paid for uh as time progresses. So, in [444:57] October of 2025, you had a $1.45 45 to [445:01] $1.95 [445:03] uh 50% quartile in the middle. This went [445:06] then to 1.5 to $2 and then in April when [445:08] you had just the coding agent frenzy [445:10] going on it went to 2.1 to $2.7. So a [445:14] compute asset which should depreciate [445:17] actually doubled in its contract value [445:20] over how many months like sevenish [445:22] months from October to April which is [445:24] just quite quite amazing to see. [445:27] Okay. And now when it comes to like the [445:31] rental rate that you charge your [445:33] customers if you're like a Neocloud or [445:35] hyper hyperscaler, let's look at an [445:37] example where you're doing a one-year [445:39] deal on 1,024 GPUs across the fleet. and [445:44] assuming that you could get a better [445:46] price by renting less to like a more [445:50] uncommitted rental rental pool, but with [445:53] the caveat that you can only guarantee [445:55] 50% rented, then it would be better to [445:59] get a $2.5 GPU per hour rate, which is [446:02] pretty cheap, but guaranteed because all [446:05] hours are paid versus getting $4 per [446:07] hour and having 50% rented. So in this [446:12] case, we're talking $22.43 million [446:14] versus $17.94 million ARR, annual [446:17] recurring revenue given these prices on [446:21] 1,024 GPUs. But we can calculate the [446:23] break even revenue point, which is [446:26] simply dividing 2.5 by 4. And it turns [446:29] out that at 62.5 rented hours, you make [446:32] just as much as m just as much as the [446:34] full fleet commitment by charging $4. [446:37] And you still get to keep what? 37.5% of [446:40] your fleet. And then if you can see 80% [446:44] as uh rented even though it's an [446:47] uncommitted rental pool, then you can [446:49] make even more money. You can make like [446:51] 6.3. What is that? You can make $6.27 [446:54] million more versus what you would get [446:56] if you had a full fleet commitment at [446:58] $2.5 per GPU hour if you have an [446:59] uncommitted rental pool with 80% rented [447:02] over that year at a $4 per GPU price per [447:06] hour. Of course, this this price would [447:09] fluctuate over time. We're assuming that [447:10] this is the average price that you could [447:12] get over the entire year. And it's just [447:14] a really interesting trade-off between [447:16] guaranteed money versus a more risky [447:20] bet. And this is kind of just like what [447:21] a variable versus fixed mortgage is at [447:24] the end of the day. Okay. What is worth [447:27] noting though is that if you if you opt [447:29] for this option here, that's what's [447:30] called committed revenue because it's a [447:31] contract, right? And the benefit of [447:33] committed revenue is that it can help [447:34] finance GPUs. And how this works is, [447:37] okay, I'm the GPU cloud. I buy hardware [447:40] and deliver a service and I get a [447:43] contract from my customer to pay for [447:46] that hardware in full for like the next [447:49] year. And as a lender, it's much more [447:51] attractive to finance that hardware if I [447:54] have the guaranteed contract which acts [447:57] as like the debt service um to the [447:59] lender in order to backs stop or to [448:01] guarantee that this capital the lend [448:04] lender is lending to is going to be [448:06] repaid through this customer contract. [448:08] So you can see that the the asset class [448:12] here is interesting because it's a GPU [448:14] and it's being act it's being used as [448:16] like this form of collateral capital and [448:18] then the capacity contract is acting as [448:22] a way to help convince these lenders to [448:24] finance the GPUs. [448:27] Okay. And all right now let's talk about [448:29] Nvidia. So this is a very interesting [448:32] graph which may many have seen. It's the [448:34] whole circular graph of how Nvidia and [448:36] OpenAI fueled the AI money machine [448:37] apparently. And this is from Bloomberg [448:39] News and [448:42] I'll use this as like a sort of [448:43] introduction to what is Nvidia doing in [448:46] this whole environment given that [448:47] they're the big bosses and let's talk [448:50] about an example that they made a deal [448:53] with Cororeweave. What did they do? They [448:55] made a deal to buy Core Weave's unsold [448:57] capacity at a $6.3 billion contract [449:01] value. So, what does this mean? It means [449:03] that Nvidia sold Coreweee a bunch of [449:07] GPUs and guaranteed them that if you [449:09] cannot rent these GPUs, we will pay you [449:12] $2 per hour for the GPUs that you cannot [449:16] rent. I'm just giving an example here. I [449:18] don't know what actually the the term is [449:20] that they had agreed upon where the [449:21] Nvidia would buy the unsold remainder. [449:23] In any case, they agreed on a price and [449:27] what you should know is that this price [449:28] is below the market rate. And what this [449:30] means is that Corore has the incentive [449:32] to find customers to rent the capacity [449:34] because actually the Nvidia uh price [449:37] that they agreed upon is really just [449:40] like break even or maybe even slightly [449:42] below break even. I'm not entirely sure. [449:44] So this actually gives a commitment of [449:47] capacity for up to April 2032 that Corey [449:52] can use when it goes to its lenders and [449:54] says, "Hey, look, this $6.3 billion [449:57] cluster that I have, it's actually [449:59] committed until 6 years from now. So you [450:03] should give me a loan so that I can buy [450:05] this and make this GPU cluster." And so [450:08] this is kind of how it works where [450:10] Nvidia is trusting that there will be [450:13] enough demand for compute such that [450:15] coreweave will never have to sell it [450:17] back to Nvidia which is probably not a [450:20] good thing for Nvidia and that coreweave [450:23] can use this financing uh this backs [450:26] stop as they call it in order to get [450:28] financing from other lenders. And if we [450:31] just like look at this in terms of uh a [450:33] timeline, what happens is that a [450:35] six-year backs stop supports shorter [450:37] customer rentals too. So what it means [450:39] is that instead of coreweave being [450:42] forced to like sign a 5-year deal, which [450:45] probably wouldn't be the best interest [450:47] for them because of the the price [450:49] locking that they have to do. They can [450:51] instead do like for example three [450:54] two-year contracts and every single time [450:57] have the assurance that if there is like [450:59] some buffer where they don't have any uh [451:02] person to rent their compute to, they [451:04] can still take advantage of this [451:06] six-year capacity commitment from [451:07] Nvidia, but in reality what they want to [451:09] do is either renew their contract or [451:11] replace it with an even more lucrative [451:13] one depending on the market conditions. [451:15] And so to summarize this contracted [451:17] fallback revenue is what is allowing [451:19] them to search this financing for the [451:21] cluster. [451:23] Then if we look at Nvidia's perspective [451:26] they earn a share of the upside and take [451:27] some risk. So how so how does it work [451:29] for Nvidia? When the customer rents the [451:31] GPU from coreweave it's not just [451:33] coreweave making all of the money. In [451:35] fact they get some revenue but Nvidia [451:38] receives an agreed upon share of the [451:40] money that they get in excess of some [451:42] margin. So that's where the Nvidia [451:45] that's where the incentive is coming in [451:46] for Nvidia. But once again, if the [451:49] capacity goes unsold, then Nvidia buys [451:50] that capacity and they may or may not be [451:53] able to use it or resell it. It's kind [451:55] of like hard to think about that in [451:56] today's market where like you say, man, [451:59] if I had some extra capacity, I could do [452:01] so many things with that. But this is [452:04] still a risk, right? You have like some [452:06] fringe possibility that maybe there is [452:08] like just a lot of extra compute. [452:10] Actually, not even a fringe. Maybe this [452:11] is actually a very real possibility that [452:13] in the future there's a lot of excess [452:14] comput on the market and that therefore [452:17] um this is a risk at the end of the day [452:18] for Nvidia and in fact as of July 2026 [452:22] the Nvidia AI cloud commitments in terms [452:24] of back stops according to semi analysis [452:26] were $36 billion [452:28] that's a lot of money but also Nvidia is [452:30] worth like $5 something trillion they [452:32] have a massive balance sheet so if you [452:34] compare this to companies like Oracle [452:35] it's actually relatively conservative in [452:38] any case you do have this sort of backs [452:40] shop circular financing that some people [452:42] are criticizing that Nvidia is also [452:44] participating in and this is a specific [452:45] example that they did with Cororeweave. [452:47] Okay, so we're talking about like the [452:48] Neoclouds how they reason about selling [452:52] their compute at like a certain cost per [452:55] hour and let's not forget that [452:57] electricity is also a cost that they [452:58] have to pay on a GPU per hour basis. So [453:01] what's what's this example here? Let's [453:03] say that over 100 hours you are able to [453:06] rent your GPU at [453:09] 80%. So essentially 80 of the 100 hours [453:12] it was rented and it ran at 1 kow. So [453:16] that was 80 kowatt hours of energy that [453:18] you had to pay for and that for the [453:19] other 20 hours it was not rented out. [453:22] But luckily you can idle the GPUs when [453:24] they're not rented out to any customers. [453:26] So the energy usage is much less. And [453:29] what we can read here from this chart is [453:30] that for 80 build hours, the Neocloud, [453:34] the hypothetical Neocloud had to pay for [453:37] 84 kwatt hours of energy. And so the the [453:40] price here is 1.05 kW per build GPU [453:45] hour. And if we consider the following [453:47] prices of $80 per megawatt hour, that [453:50] would mean the following price of 0.084 [453:53] per build GPU hour. But if we double the [453:56] price, we get now 16 cents uh or 16.8 [454:00] cents per uh hour of build GPU time. So [454:05] these numbers are obviously much smaller [454:07] than the numbers that we see in terms of [454:10] the rents. So those are in the dollars [454:12] whereas these are in the cents. Of [454:14] course, let's not be naive and think [454:16] that the only cost for running a GPU is [454:18] the electricity. The the GPU itself is [454:21] the biggest cost. It costs a lot of [454:23] money to get these GPUs because of [454:25] Nvidia's 75% profit margins. So, this is [454:28] only one cost. Of course, Neocloud up to [454:31] now have been making positive profit [454:33] margins from everything that I've seen. [454:35] But it is worth worth remembering that [454:38] if we consider this future where there [454:39] could be an excess of supply of compute, [454:41] then this would become a very important [454:44] calculus for these new clouds in [454:46] determining whether or not they'll be [454:48] profitable. And when it comes to who [454:50] pays when electricity prices rise, so [454:54] this graph right here, well, there's two [454:56] ways you can run this. First of all, you [454:58] can have a fixed all-in price where the [455:00] provider absorbs both the increase and [455:02] the decrease, which is to say like, [455:04] well, if I'm Cororeweave and I'm selling [455:06] compute to Anthropic and let's say that [455:09] the electricity price drops drastically [455:12] or I can find a more reliable and [455:14] renewable energy source which is [455:16] cheaper, then now I'm making a bigger [455:18] profit margin. And then conversely, if [455:21] there is the war going on and oil prices [455:24] are going crazy, then now I have to [455:26] absorb the increase in price. So that's [455:28] one way you can do it. And this is [455:30] essentially hiding the risk behind the [455:32] provider. And then the second way that [455:34] you can structure this contract is to [455:35] say that it will be an electricity pass [455:37] through which is to say the customer [455:39] pays the increase or there's some agreed [455:41] upon scaling so that with the increase [455:45] in electricity there is some percentage [455:46] increase in the build GPU per hour cost. [455:50] So these are just two ways that you can [455:51] structure the electricity pricing and [455:54] when we talk about uh Abene itself the [455:57] plant like you might have heard of Cruso [456:01] oracle OpenAI in terms of how this is [456:05] actually like being financed like who [456:06] builds and operates Abalene. Well the [456:08] actual structure here is Cruso builds [456:10] this builds builds the facility and then [456:14] Oracle is the hyperscaler that operates [456:16] it with their Oracle cloud [456:18] infrastructure. I believe that's what [456:19] OCI stands for. And then OpenAI runs the [456:22] AI workloads on that facility. So, it's [456:25] kind of funny how you have this [456:27] three-way deal. It would it would it [456:30] would it would be sensical for Caruso [456:32] just to directly sell to OpenAI, but [456:34] given the way these deals are structured [456:36] and uh Oracle's interesting tactic to um [456:39] have this massive compute obligation to [456:41] OpenAI, this is the exact operation [456:44] that's going on at Abalene. And so one [456:47] question that is coming up a lot in data [456:50] center land is people who talk about [456:54] half of the US data center supply being [456:57] cancelled or half of the US data center [456:59] supply being uh delayed. And so before [457:02] we move on to the 16th and final [457:04] chapter, let's look at the case study [457:06] for Abene of what's actually going on. [457:09] So this is actually f is making [457:11] reference to a semi-analysis article [457:13] from June of this year where they [457:14] actually go into the statistics of what [457:18] percentage of data centers are being [457:20] delayed. So please read this if you're [457:21] interested. But getting to the short of [457:24] it for Abene specifically, let's talk [457:26] about the plan versus the reported [457:27] delivery. So on this column we talk [457:30] about the announced target and then now [457:31] on the right column we talk about the [457:33] later report which is cool because these [457:35] numbers were announced in 2025 and now [457:37] after a year we can see what has [457:38] actually been accomplished. So the first [457:42] two buildings is was announced to be [457:44] energized in the first half of 2025 and [457:47] they were energized within a year of [457:49] June 2024. So the precise date was [457:52] somewhere in September 2025. So it's not [457:55] in the first half of 2025, but it still [457:57] was in 2025. And then those are the [458:00] first two buildings. Now when it comes [458:02] to the six other ones, the construction [458:04] was announced to be complete mid 2026. [458:07] And as of September, only 75% of total [458:11] campus capacity was delivered and the [458:13] remainder is supposed to be in later [458:14] quarters. So here they kind of missed [458:16] the mark as they projected the [458:19] construction to be done by mid 2026. I [458:21] guess we're still technically in [458:22] mid2026. I don't know how that's [458:23] defined, but um I guess now the goal [458:26] would just to be to get it online before [458:27] the end of the year. So there you go. We [458:30] had a quick little overview of GPU cloud [458:32] economist economics together. And now [458:35] let's head into the finale, which is [458:38] putting an AI factory together. So how [458:41] do we put an AI factory as it's now [458:44] called together? Well, as the title [458:47] says, you start with the hardware, then [458:48] size the facility because it's the [458:49] hardware that is king. So what you do is [458:51] you work backward from processor demand [458:53] through conversion losses other rack [458:55] loads and facility sports. If you [458:57] remember the earlier example, we had a [458:59] 72 kW processor that required that much [459:02] power at the rails. Then we had to add [459:04] 12 kW of other rack loads and we finally [459:07] ended up with 90.26 kW of power required [459:11] at the DC bus and 93.05 [459:15] kW of power at the AC bus due to the [459:17] other conversion efficiencies. And so [459:20] these are just examples. It's not [459:22] actually what's happening in the Abene [459:24] data center for example. But that [459:26] exercise that we did of course only [459:28] looked at a specific rack only up to the [459:31] AC transmission to the bus. Whereas if [459:33] you're actually developing an entire [459:34] data center well you need to look at the [459:36] rack distribution backup and then the [459:38] medium voltage distribution as well. You [459:40] need to determine for the heating what [459:42] are the cold plates and heating uh [459:44] requirements for the remaining air heat. [459:46] Are you going to use a rear door heat [459:48] exchanger? Are you going to use a C A [459:51] computer room air handler? The world is [459:53] your oyster really with data centers. [459:54] It's like a massive mix and match. And [459:56] then also how are you going to connect [459:58] the GPU fabric so they can communicate. [460:01] As we mentioned in that previous [460:02] example, you do not want a data center [460:06] which is optimized for inference running [460:08] pre-training workloads because [460:10] pre-training workloads require a lot of [460:13] cross rack and crosscluster [460:15] communication as opposed to inference [460:17] which is usually scaled in a single [460:19] scale up domain. So in the NVL72 rack, [460:22] you can already run a pretty healthy [460:25] amount of inference and serve a lot of [460:27] customers with the high interactivity. [460:29] In any case, we can think of the [460:31] calculus for AI factory as taking into [460:33] consideration power, heat, data. And now [460:38] when it comes to planning the data [460:39] center, there's also the considerations [460:41] of how are we going to get the power to [460:44] the chips as fast as possible. One thing [460:46] we discussed in the previous chapter was [460:47] Abalene's gas plant having a role before [460:49] and after grid expansion where the gas [460:52] plant's role was bridge power at the [460:54] start and I think that's still the case. [460:56] I'm not sure what the status is. could [460:57] have already moved into long-term [460:59] backup. But anyways, at the beginning it [461:01] was 350 megawatts of on-site jet gas [461:03] generation as this sort of gas plant [461:06] bridge supply and then afterwards [461:08] long-term the plan is for it to become [461:11] the standby gas plant and that the grid [461:15] which is the energy reliability council [461:18] of Texas. So that's what supplies energy [461:20] in Texas that it now takes the uh [461:22] responsibility for powering the data [461:24] halls and the cooling. So you can see [461:26] this is one of the main methods that we [461:28] will be scaling up power in the [461:30] near-term future because connecting to [461:32] the grid is very hard with these massive [461:34] interconnect cues. Essentially the [461:37] waiting time required to get access from [461:39] the grid in many states in particular [461:42] Texas is going to be many many years in [461:44] most circumstances with these data [461:46] centers. So the best incentive here is [461:48] to get on-site gas generation ASAP. And [461:51] then once you have the on-site gas [461:53] generation or whatever is powering the [461:54] data center, we need to ask the question [461:56] for Abene. How would the campus reject [461:59] this heat? Here are three different [462:01] options. What did they pick? Well, in [462:03] Abene, they did aircooled chillers, [462:05] which uses refrigeration without having [462:08] any evaporative heat rejection. Because [462:11] in Texas, given a lot of Texas is a [462:14] desert, water is a scarce commodity. So [462:16] having a aircooled chiller is a good [462:19] option for avoiding water shortages. And [462:22] in fact, it's interesting to see what [462:24] Cruso said regarding their use of water. [462:27] So they said choosing to bear the extra [462:29] cost in the interest of responsible [462:30] water management was one of their [462:32] reasons why they chose this technology [462:34] aircooled chillers over for example a [462:37] water cooled chiller. But you know what [462:39] I think? I think they choose air cooled [462:41] chillers because if they use a water [462:43] cooled chiller then they realized they [462:44] would get a massive public backlash and [462:47] also it may have even been more [462:50] economically reasonable to use these air [462:54] cooled chillers that consume much more [462:55] electricity. Why do I think that? It's [462:58] because if you consider a future world [463:01] where there's many many data centers [463:03] then if they all compete for water then [463:06] water will be a scarce commodity and [463:08] whereas water is relatively finite um [463:11] energy if you are smart enough and are [463:14] relatively um ambitious can be obtained [463:17] in many many ways. For example, the sun [463:20] is almost an infinite source of energy [463:21] from the perspective of the earth. Um, [463:23] with that being said, it's probably [463:24] better to hedge your bets on paying only [463:26] electricity as opposed to needing both [463:29] water and electricity to cool your GPUs [463:32] in the future. So, I think this is [463:34] actually example of greenwashing. Let me [463:35] know if you guys have a different [463:37] opinion in the comments. But any case, I [463:39] really liked this comment in particular [463:41] when I read the Crusoe uh data center [463:43] report because I think that it is only a [463:45] halftruth. Okay. Now, what did they do [463:49] in Abene otherwise to get the compute [463:51] online fast? Well, Cruso built [463:53] electrical equipment while the campus [463:55] was being built on site in their [463:57] factories. In fact, the assembly floor [464:00] is in Tulsa, Oklahoma. And they just [464:04] opened a new assembly hall this month. [464:06] And in fact, [snorts] the uh report for [464:08] this for the second Tulsa manufacturing [464:11] facility was only published a few days [464:14] ago on September 14th. This may have [464:16] been a long time ago depending on when [464:17] you're watching this today. In any case, [464:18] the factory has made already more than [464:21] 200 switchboards that were directly [464:24] supplied to Abolene in order to expedite [464:27] construction and remove supply chain [464:30] dependencies on other companies that [464:31] Abene would otherwise have to wait for. [464:33] And so here you can see, I guess this is [464:35] a rendering of the factory floor that's [464:37] just been opened in Tulsa, Oklahoma by [464:40] Crusoe. So I really want to highlight [464:42] this. You're seeing a lot of these AI [464:44] data center companies becoming [464:45] verticalized to a great extent. Whereas [464:48] previously a data center company was [464:50] more of just a orchestrator saying okay [464:52] I'm going to get this from Seammens. I'm [464:53] going to get this from ABB. I'm going to [464:55] get this from Schneider Electric. Now [464:57] they are trying to verticalize and own [464:59] the stack so that they can control the [465:02] lead times which is essentially to say [465:04] the amount of time they have to wait for [465:07] getting their equipment. Okay. And when [465:09] it comes to the deal hierarchy, once [465:11] again when we talk about Cruso, Oracle [465:13] and OpenAI, the hierarchy is such that [465:16] Cruso builds and operates the campus. [465:18] Oracle provides the GPU compute capacity [465:21] for the cloud infrastructure and OpenAI [465:23] trains and runs the models for their AI [465:26] workloads. We also should remind [465:28] ourselves that it's not just Cruso [465:29] that's funding this. They have other [465:31] partners such as Blue Owl Funds and [465:33] Primary Digital Infrastructure which are [465:35] essentially financing the data center in [465:37] behalf or rather alongside Cruso such [465:40] that Crusoe reduces its own liability [465:43] and then this long-term rent supports [465:45] the returns of all these companies but [465:48] all of this is with the condition that [465:51] the open AI bill is being paid [465:54] continuously and if we consider a future [465:56] regime where compute is in much larger [465:59] surplus plus then this whole structure [466:01] may not be feasible or economical. [466:05] All right. Then with respect to the [466:08] premium price that's paid for getting [466:10] compute fast, this happened in the case [466:13] of Abene, which is to say that it ran [466:15] its first workloads while a campus kept [466:17] growing. So the first phade that was [466:19] completed in 2025 was able to support [466:23] OpenAI workloads once it was completed. [466:25] So we're talking a year ago in September [466:27] 2025. And this meant that the power, the [466:30] cooling, and the network were all under [466:31] operational conditions while the next [466:34] halls were still being built. And so [466:36] what we need to do in the meantime is [466:38] both connect these interfaces because [466:40] eventually open AAI will probably want [466:42] to have some cross data center [466:44] communication for their large [466:45] pre-training runs and then also we need [466:48] to keep the live phase operating while [466:50] the next hall is being built and while [466:52] this power cooling and network is not [466:54] yet online and this means having a lot [466:57] of orchestration and careful planning to [467:00] contingency planning as I mentioned to [467:02] ensure that there is no hiccups along [467:03] the road. Okay, so through this course [467:06] we start at the rack. We determine the [467:09] workload from the actual compute itself, [467:11] which is to say what is the power, the [467:13] cooling and the networking required. [467:16] Across a hall, these conditions become [467:19] distribution and cooling systems serving [467:22] rows of machines. And across the campus, [467:25] they require substations, outdoor [467:27] cooling, generation, land, and [467:29] financing. So interestingly a hardware [467:32] choice becomes becomes a facility design [467:36] and this is why personally I find data [467:38] centers so compelling is that depending [467:40] on the chip that you choose to put in [467:42] this tiny rack that will then go in this [467:44] tray that will then go in this uh data [467:46] hall you can have a completely different [467:49] end design. So that was the from watts [467:52] to tokens course. We started with [467:53] electricity at the site boundary and you [467:56] can now follow it all the way to [467:57] computation and follow the heat back [467:59] out. That is the physical system behind [468:02] every token. Thank you so much for [468:05] watching.