Kubernetes Operator Best Practices – Kubebuilder Deep Dive

[00:00] Learn the best practices for writing [00:02] Kubernetes operators using Cube Builder. [00:05] This includes strategies for managing [00:07] multiple controllers for parallelism, [00:10] resolving conflicts for custom resource [00:13] updates, implementing efficient retries, [00:16] and avoiding infinite reconciliation [00:19] loops. These resources provide guidance [00:21] on scaling operations to support [00:24] significantly more users effectively. I [00:26] usually start with a problem statement [00:28] in my videos but this time I want to do [00:31] something different. Um let's first [00:33] learn a concept and then we will go to a [00:36] problem statement and then we will see [00:38] um what we need to do. So um let me show [00:42] you something really quickly. Um I have [00:46] a cluster here and I do have my EC2 [00:51] instance um with a certain spec like [00:55] this. So this is something we have uh [00:57] seen in the video where we talked about [00:59] how to uh implement the status update, [01:02] how do you work with a generation, how [01:04] do you work with spec. It's just a [01:06] continuation of off of that. I will put [01:09] the link of the video uh in the [01:11] description. Now what happens is what I [01:15] wanted to show you is every time you [01:19] make changes to a resource, the [01:22] generation changes. If you have made [01:25] changes in the spec and also every [01:29] resource in Kubernetes has this thing [01:32] called a resource version. This resource [01:35] version is how CD knows about um your [01:39] object. This is how uh it is reflected [01:42] in the HCD and this is irres this is [01:46] going to change irrespective of whether [01:49] you updated your spec whether you [01:51] updated your label you updated your [01:53] annotations you put any changes in your [01:57] object whether it is spec or anywhere um [02:00] this is going to be changed whether the [02:02] generation will only be changed when you [02:05] actually update um the spec of your [02:09] operator of your resource. So let's do [02:12] uh some changes here. I'm going to say [02:15] uh testing and uh this is a change. Now [02:19] if I save and quit this, you see what [02:22] happened? We went from generation four [02:24] to five and there was a different number [02:27] for the resource version. Now to show [02:29] you something that it does not really [02:32] matter uh what change was there on the [02:35] on the object. If I just make a new [02:38] label and I would say um cloud uh GCP. [02:44] Now this is to show you that we are not [02:46] making any changes on the spec but [02:48] rather a metadata change but even that [02:51] if I save and quit you see um this [02:55] changed the resource version did change [02:58] It did not change the generation because [03:00] we didn't make changes to the spec of [03:02] our object but the resource version [03:05] changed. Just to quickly show you [03:07] something else. Um environment and uh I [03:11] would say production. [03:14] Now right now the resource version is [03:16] 42331 but if I save and continue uh [03:20] that's going to change to something else [03:22] and it is 42 345. [03:25] And that's something I wanted to show [03:27] you. It does not matter where you make [03:30] what change in your object. If it has [03:33] changed, it's going to get a new, you [03:35] know, the change is going to be uh [03:37] committed to HCD and every commit is [03:41] going to have a new resource version. So [03:43] now that out of the way, um let's move [03:46] to the problem which I was facing and [03:49] that got me to record this video. [03:53] Um now imagine this is what you were [03:57] doing and this is something what I did. [03:59] I felt the problem and imagine this is [04:01] what you are also doing. So you have a [04:04] requirement. Let's let's start with it [04:07] now. You have a requirement where you [04:09] want to make an operator, right? You [04:12] want to make an operator that is going [04:15] to work and that's eventually going to [04:18] give you a VM running somewhere. It [04:22] could be u in Amazon uh it could be uh [04:27] in GCP. It doesn't really matter. What [04:30] this VM is going to do is eventually [04:34] this virtual machine is going to be uh [04:37] added in your cluster as a worker. So [04:41] think of this as you're making a scaler. [04:44] You're you're building your own custom [04:46] autoscaler. And this is something what I [04:48] felt when I was working on a similar [04:51] project. [04:53] Now when you are writing an operator you [04:56] know you will be writing some custom [04:58] resources and for that you will be [05:01] needing some custom resource [05:02] definitions. So when I was working on it [05:05] I decided my custom resource definition [05:08] to very simply uh look like this. So the [05:11] kind was a node pool um and and this [05:14] would define the number of nodes that I [05:18] can have. So a node pool resource uh [05:21] would be defining um how many maximum [05:24] nodes can I get? What is my target node? [05:28] What's the name of my node pool? And [05:30] also a very simple um configuration like [05:34] how big the node should be, how much [05:37] disk do I need and how much memory um am [05:40] I going to need in that particular node [05:43] pool. You can think of this. This node [05:46] poolool is going to create me a node [05:50] which is going to be in one of the cloud [05:52] providers somewhere and that is going to [05:55] be then registered in my cluster. [05:58] The thing is a node pool can be created [06:02] manually [06:05] when I create this custom resource. [06:07] Let's say I have built my operator um [06:10] you know my operator has some knowledge [06:12] of creating a virtual machine. my [06:15] operator has the knowledge of [06:16] registering that into my cluster. [06:20] Now, if you really want to start using [06:22] it, when you create this custom [06:25] resource, you can make changes on these [06:29] target nodes. You could say from zero to [06:33] one and your operator is reacting on [06:37] that change. So, what it sees is it sees [06:40] right now I've got like zero. Let me [06:43] change uh to a different color. Right [06:45] now I've got zero nodes of this [06:48] configuration. The user has asked me to [06:50] create one node. One node and I'm going [06:54] to go and I will give them a node. This [06:58] is what a very simple operator that you [07:00] wrote. [07:02] You also have not just this spec here [07:05] but you also decided to give a status of [07:09] uh you know uh to someone who is looking [07:11] at your node pool. So this status has a [07:14] key called scale operation and this is [07:17] usually inactive if your operator is not [07:21] doing anything which means if your [07:24] operator is not busy creating a node or [07:26] registering that to the cluster it [07:29] doesn't change from inactive. However, [07:33] this scale operation changes. This scale [07:36] operation changes when um you know when [07:40] you are actively creating a VM in a [07:42] cloud provider from inactive to active. [07:47] And this is what you wrote an operator [07:49] for. This is something similar to what [07:52] we did in my um Kubernetes video of how [07:56] to build your own operator that manages [07:58] EC2 instances. similar to what I was [08:01] working on a project in my company. [08:04] The thing is you were able to create [08:08] these resources manually which is a node [08:11] because you have an operator you deploy [08:14] or you create this custom resource based [08:17] on the custom resource your operator [08:19] runs and it creates you a node and [08:21] everything is happy and healthy. [08:23] However, you decided to take this thing [08:27] one step further. You said that instead [08:32] of me deploying the custom resource and [08:36] then uh I to do do cube ctl [08:42] edit node pool and then I will go from [08:46] one and change this to two and save and [08:49] quit and eventually there goes another [08:52] node. Uh this was node number one. This [08:55] was node number two. I don't want to do [08:57] this. This is a scaler. This is not an [09:01] autocaler. So what you decided was um [09:04] that this cubectl edit node pool or in [09:08] the end the increment to this target [09:11] node, you know, this to this target node [09:15] should actually happen on a certain [09:17] event [09:19] and and you decided that this event [09:24] should be a pod being in a pending state [09:30] um because of CPU you or memory. Now you [09:35] know you know that a pod could be in a [09:37] pending state for so many reasons. The [09:40] pod could be in a pending state because [09:41] there's no secret that the pod is [09:43] referencing to or a persistent volume is [09:47] not available that the pod needs. The [09:50] thing is you have the logic where you [09:53] list on all the pending pods. You find [09:56] the ones which are actually pending [09:58] because of CPU or because of memory. you [10:01] see you know uh if the memory that the P [10:05] is asking for let's say if the P is [10:07] asking for memory uh which is around 28 [10:11] GB [10:12] um and then you have the logic that yes [10:15] because the pod is asking for memory [10:17] that is less than this 96 GB if I create [10:22] a node the node will be enough for my [10:26] pod to be deployed [10:28] you wanted to have this trigger [10:31] So you have this as an autoscaler. Now [10:34] what you do for all this, let me quickly [10:37] go and uh delete this because we don't [10:41] need it. What you decided was that I'm [10:45] going to write another um you know [10:49] controller. So there are two controllers [10:51] right now which are working in your [10:53] operator. [10:55] I think it makes more sense. Let me just [10:57] quickly say you can have one single [11:03] operator which can have multiple custom [11:06] resource uh multiple custom resource [11:09] definitions and also multiple [11:13] controllers there. So you said right now [11:17] I have uh one custom resource definition [11:21] which is called a node pool and there is [11:27] an operator that reacts on top of that [11:30] and that's going to be the node pool [11:34] operator which is eventually the one [11:36] that creates you a virtual machine or a [11:39] or a VM you know somewhere in the cloud. [11:42] You then decided because you wanted to [11:45] make this an autoscaler, you say I'm [11:48] going to have a new custom resource [11:50] definition [11:52] and that's called um you know uh that's [11:55] called an autoscaler. Let's say you [11:58] create a new custom resource definition [12:00] and you name this as autoscaler [12:03] um and then there is a auto scalar um [12:09] config um autoscaler operator. [12:13] This autoscaler custom resource is very [12:16] simple. It just says um how [12:20] many or you know how long to wait [12:25] before checking for pending pods. Let's [12:27] say frequency. For example, you can say [12:31] that my my autoscaler operator should [12:35] look for pods every five minutes. [12:40] You know this is the frequency that you [12:42] are configuring in your custom resource [12:45] definition. So this might look something [12:48] like uh let me just quickly go uh and [12:53] write a YAML for this. This would make [12:56] more sense. [12:59] Um so your kind will be um autocaler [13:06] uh version one. [13:11] Um then we have the meta data where wait [13:15] am I doing something wrong here. Um let [13:17] me quickly just get the API version and [13:20] kind and metadata. So this is what I was [13:22] missing because every resource needs [13:25] this. So I have my API version as [13:29] compute.ample.com example.com vi1 kind [13:32] is autoscaler and then we have the [13:34] metadata the name my autoscaler and then [13:39] you could have a spec essentially when [13:42] you are working with cube builder um you [13:45] create the spec you define the spec uh [13:48] of your of your custom resource in this [13:50] case I would have the list of name [13:52] spaces to target [13:56] you know these are my uh prod name [13:59] space. Then I have my search nameace and [14:03] then I have uh my um um real money maker [14:09] namespace. These are the name spaces in [14:11] which I want to target. If there is a [14:14] pending pod in these name spaces then [14:17] only I want to scale because there could [14:19] be people creating dev projects and I [14:22] don't really want to care about if [14:24] there's a pending pod because of CPU or [14:26] memory in the dev project. But if these [14:29] projects are there that has a pending [14:31] board we want to be um creating or [14:34] reacting on that event then you could [14:37] say uh frequency and you can say 5 [14:41] minutes. So every 5 minute your [14:44] autoscaler controller will check the [14:47] pending ports in the list of name spaces [14:50] to target and it will react on top of [14:53] that. That's simple. So you decided that [14:57] I'm going to write these two custom [14:59] resources and I'm going to write these [15:02] two different operators. [15:04] This operator creates the VM. This [15:07] second operator just finds [15:11] the pending pod. You know it just finds [15:15] the pending pods. [15:18] Now because what what would you want to [15:21] do if you have found a pending pod? you [15:24] would want to create a VM, right? [15:27] Because there's a pod pending, you need [15:29] compute and you would want to create a [15:31] VM. The thing here to notice is many [15:35] times you might find yourself in a [15:39] situation where you have custom [15:41] resources, you have multiple operators [15:44] and they talk to each other's custom [15:47] resource, not just their own but each [15:49] other's resource. In this case, for [15:51] example, the autoscaler operator once it [15:55] has found some pods, what this would do [15:59] is it rather than this creating a [16:01] virtual machine, it updates the custom [16:05] resource called node pool. [16:09] Essentially, think of this as cube [16:13] edit node pool and then you update the [16:16] number of target nodes. Let's say [16:20] whatever the number was uh two you [16:22] change this to three and because you [16:25] update the custom resource uh called [16:28] node poolool the node poolool controller [16:31] will see this change and it will [16:33] eventually create you a VM and then uh [16:37] your VM will be registered as a worker. [16:40] The thing here to understand is a [16:43] different operator which is our [16:45] autoscaler operator is using a different [16:49] custom resource. This is not you know uh [16:53] only working on its own custom resource [16:55] definition and you will find this [16:58] happening a lot and that's where things [17:02] get interesting. So let's draw a [17:05] timeline on what actually happened. So [17:09] you start from the very beginning uh [17:12] imagine a cluster where you have your [17:15] operator your resources all running. Now [17:18] what you do is you install or you create [17:21] a YAML for the node pool which is [17:26] essentially uh something like this which [17:29] we already have here. So I'm going to go [17:32] uh copy this and let's paste it over [17:34] here. Oh, okay. That was something I [17:38] wasn't expecting. Let me clear that. Uh [17:40] so this becomes a bit more um easier to [17:45] work with. Uh should have copied this [17:48] somewhere. [17:49] Um okay. So let me quickly just copy [17:53] that and keep a fresh copy of this in [17:55] case I need it in the future. So see [17:58] what happened. You created a node pool. [18:03] You created a node poolool resource [18:05] here. [18:06] and you define uh the target nodes as [18:09] node zero but you define the target node [18:13] as one [18:16] um or let's say it's zero to begin with [18:19] it is zero all right so you're not um [18:22] doing anything you just say I want the [18:25] resource to be there in my cluster I [18:27] don't want it to do anything but let's [18:30] let it be there um and let me quickly go [18:36] ahead and get this here. [18:40] Um I think we can just get from [18:43] carbon.now.sh [18:46] here. Uh this is my YAML. I'm going to [18:48] copy this image and here is my custom [18:52] resource definition or rather the custom [18:55] resource for uh the autoscaler. So these [18:58] are two uh CRs that you have created. I [19:02] hope this is making uh you know this is [19:05] this is going to be clear. [19:08] So do custom resources [19:11] eventually uh because you have two [19:13] custom resources you also have two [19:17] operators but the operators are not [19:19] doing anything. Um so the reason why [19:22] they're not doing anything for this one [19:25] you have the node pool operator for this [19:28] one you have the autoscaling operator [19:32] they're not doing anything because first [19:34] your target node is zero so nothing [19:37] needs to be done and also because you [19:39] don't have any pending pods in these [19:41] name spaces so nothing is being done [19:45] now let's say you manually went you know [19:49] uh and you said that I'm going to create [19:52] one new node from the node poolool [19:55] operator. What you do, the first thing [19:58] you go ahead and you do is cubectl [20:03] create um cubectl uh say edit [20:08] uh my node pool which is short form np [20:12] and you change this target node from uh [20:16] you know you change this target node [20:18] from one uh 0 to one. All good. This [20:23] operator starts working. [20:26] It is working and because you made a [20:29] change because you made a change in the [20:31] spec um a new version would be there a [20:35] new uh you know for the same object the [20:37] generation is going to change from 1 to [20:40] two and this is also going to be updated [20:42] to a number x y z that's all good um [20:46] this would be updated [20:49] now [20:50] coincidentally while this operator is [20:54] doing its job. [20:56] You found a pod which was pending in the [20:59] broad name space. [21:02] See what happens. [21:03] You found a pod which was pending in the [21:06] broad name space and you have configured [21:09] your autoscaler operator that if [21:13] pending is found you know you're going [21:17] to update [21:19] uh the custom resource uh you're going [21:22] to update the custom resource which is [21:24] in this case the node pool. So if you [21:28] found a pending pod, you update the [21:30] custom resource and just the same what [21:32] you're doing. Uh cube ctl edit node [21:37] poolool um yeah cubect node poolool. [21:41] Keep this in mind that this all happened [21:45] while your operator was working. While [21:49] your operator was working. [21:52] Now what you do is you edit the node [21:54] pool. Now this is going to move from one [21:58] to two because uh one from 0 to one was [22:05] you did this manually cubed edit node [22:08] pool and from 1 to two was done by this [22:11] particular operator while this was [22:14] running. Okay, while this was running, [22:18] now what's going to happen? Not just the [22:20] target node is changed, but the [22:23] generation also changes three. And this [22:27] goes from XY Z to A B C. [22:32] Okay. Um, and this is all done. It's [22:35] it's done. It's working. Once this [22:37] operator will finish, once this operator [22:41] will finish, you might want to update [22:44] this custom resource back because when [22:47] you start working, you you know, you [22:50] change this from inactive [22:53] to active because your operator was [22:55] running. And when the operator is done [22:58] creating a virtual machine, registering [23:00] that to your worker [23:03] uh to your to your cluster, you might [23:06] want to update the object from active [23:09] back to inactive because it's not [23:11] technically doing anything. And that's [23:14] where you're going to have a problem. [23:18] That's where you're going to have an [23:19] issue. [23:21] See when your node pool operator started [23:25] every time uh and this is this is how [23:28] it's going to make more sense. Let's say [23:31] you have a custom resource. [23:34] Uh it's u resource version is one [23:40] and you start an operator based on [23:46] uh this custom resource with a resource [23:47] version of one. Every time your operator [23:51] sees that there was a change on my [23:53] custom resource, the first thing it does [23:57] is it downloads or it caches [24:02] the object. [24:05] So if your custom resource had resource [24:08] version one in HCD here as well uh your [24:13] operator knows that the version latest [24:16] is one and the operator starts doing its [24:19] work whatever it is doing. Now someone [24:24] or you or another operator, another [24:28] controller, it updates this particular [24:31] custom resource [24:33] and this changes from 1 to two. [24:37] Your operator still thinks that the [24:39] latest version is one because it hasn't [24:42] gone to HCD to get the latest copy. It [24:45] it did not it didn't know that uh you [24:48] know the operator version has changed. [24:51] it doesn't even have an idea about it [24:53] yet because it's still running. [24:56] So while your operator has an older [24:58] version of the resource um you went [25:01] ahead and you updated this custom [25:04] resource in HCD. Now the resource [25:06] version is two. Now when your operator [25:09] is done and usually you might want to [25:13] update the object [25:16] imagine um this was a custom resource [25:19] for the EC2 instance um use case that we [25:22] have been talking about. Now you got the [25:25] spec the the resource version was one. [25:28] You create the VM. You got the public [25:31] IP. Maybe you want to add the public IP [25:34] in the status of that object because [25:37] every object has spec and then the [25:40] status. You want to update the public IP [25:42] in the status. You try to update the [25:46] object for which the resource version of [25:49] the object was still one. [25:53] And that's where the problem happens [25:54] because when you try to update the [25:55] object, you essentially want to commit [25:58] into HCD. [26:01] So you are telling HCD that hey or hey [26:05] CD I want you to update an object um [26:09] whose name is so and so and so. HCD [26:12] doesn't care about the name. It cares [26:13] about the resource version. So you are [26:15] saying update the object uh with a [26:18] resource version one. NH3D says there is [26:22] no such object whose resource version is [26:26] one. I have some whose resource version [26:29] is two. So it looks like you either have [26:33] an older copy of the object or you are [26:36] clearly doing something wrong. And [26:38] that's what you need to fix. And this is [26:41] the problem that happens when you are [26:44] working with more than one controllers [26:46] where each controller is you know uh [26:50] touching a different uh custom resource [26:52] and this is called a conflict. [26:58] You might have seen this error some you [27:00] know at some places you might have seen [27:02] this that the object that you are trying [27:05] uh you know u is old get the latest cop [27:08] copy of that object. So essentially if [27:11] you really want to update this object [27:13] what you would do is before writing to [27:17] HCD you know your because your object [27:20] update has failed what you technically [27:23] do you get the newer copy from HCD you [27:29] get the newer copy from HCD and now your [27:32] resource version is two and then you [27:36] will then update you will try retry try [27:40] the uh the update um object [27:45] um task or you will try to update the [27:48] object again and this time because you [27:52] are making a change in HCD for that [27:56] object its resource version will now [27:59] change from two to three see every time [28:03] anytime anyone makes any changes you're [28:06] going to have a new resource version [28:09] when you create the resource for the [28:11] first time. Let's say the resource [28:14] version was one and that's what your [28:16] operator started with. Now this guy made [28:21] some changes to the resource. The [28:23] resource version went to two. So you try [28:27] to update on an older version where in [28:30] city there was a newer version it [28:32] failed. You fetch [28:35] the new copy [28:38] and then you try to update [28:42] and because the last resource resource [28:44] version was two and you are doing an [28:46] update this changes to three. And that [28:50] is the whole story of me trying to [28:53] explain and come up with this situation. [28:55] I hope this has made some sense. If not [28:58] the comment section is open. You can [29:00] always ask these questions. But this [29:03] problem that can happen to you um and [29:07] and this happens because you're trying [29:09] to update an older version of the object [29:11] where the object has already been [29:13] updated by something someone external [29:16] but your operator did not know about [29:18] this. So to fix this there are two [29:22] different ways in which you can you know [29:25] uh in which you can fix this problem. Um [29:28] the first way is a simple way or I like [29:32] to call that a lazy way of doing it and [29:36] the second one is a pro way of doing [29:40] that. Now it doesn't make sense for me [29:42] to talk about these simple or lazy or [29:44] pro but rather let's go into the code [29:48] and let's see what different ways are [29:50] there where you can make your operator [29:53] intelligent that okay the object that [29:56] I'm trying to create it has been updated [29:58] by someone else somewhere let me get a [30:03] new copy and then I will try to make my [30:05] changes on the new copy so let's go into [30:08] the code and then let's let's see that [30:11] how it works. [30:13] All right. So, uh if you look at the [30:15] code here, um let me just quickly walk [30:18] you through how we are building our API. [30:21] And in this case, I have a very simple [30:24] API, a custom resource type which is [30:27] called EC2 instance. This is something [30:28] that I always use in all of my examples. [30:32] So, if you are working on this best [30:35] practices series, you already would be [30:37] very familiar with how do you build an [30:40] operator? Um, and there's a whole course [30:42] on cube simplify of how do you build [30:43] this on scratch. We'll probably put the [30:46] link in the description here. And uh, [30:48] here you can see I have a very simple [30:50] custom resource called fu and bar. It [30:53] has a spec which is just fu and bar. And [30:56] this is just a very simple string. [30:59] Look at the operator. Um, this [31:01] reconciled function is the main function [31:04] that runs whenever you make changes to [31:06] your custom resources. This is the one [31:09] that tells you what to do. Um, here's a [31:12] bit of a fancy print statement. So, we [31:14] can we can see in the logs of when the [31:16] reconcile is started from the beginning. [31:19] Uh, and we get the object that is being [31:23] worked upon into a variable of type EC2 [31:27] instance. [31:29] Now every object that you have it has a [31:32] spec it has a status that is something [31:34] you can define in um in here for uh for [31:38] my case the spec just contains fu and it [31:42] contains bar which is another um key in [31:46] my object. Um however every object gets [31:50] um some metadata. So every object would [31:53] have a UID, every object would have some [31:56] labels that you do not set in the [31:58] beginning. This is something which is [32:00] coming from the Kubernetes API server. [32:02] Similar to that, every object has a [32:05] generation and a resource version. So [32:08] while the resource version, think of [32:10] this as the version of the object in HCD [32:13] and the object generation is the [32:17] resource um uh it's a deployment version [32:20] of the object. So when you create the [32:22] object for the first time its generation [32:24] will be one. Any change in the spec that [32:28] you make increments the generation and [32:30] that's how you can know uh what is the [32:32] old version and what is a newer version [32:35] um that is being worked upon. [32:38] Now in in this function which is a [32:40] reconcile function it's fairly simple. [32:43] I'm just trying to simulate a [32:45] time-taking task that would be done. [32:47] Usually your operator which is in case [32:50] the EC2 operator in here um it is [32:54] supposed to be creating an EC2 instance [32:58] on Amazon. So usually this would go to [33:01] AWS make API calls using the SDK written [33:06] in Golang for AWS. It will authenticate [33:08] to Amazon. It will give the information [33:11] that I want to create an EC2 instance [33:13] with this name the security group. Um, [33:16] it should have these tags, these labels, [33:18] this is the AMI ID, all of this [33:20] information and eventually you're going [33:23] to get back uh some response from [33:24] Amazon. This is something we already [33:26] talked about in the in the course for [33:29] building operators from scratch. But I'm [33:31] not doing this all in here. I'm just [33:33] counting from 1 to 10 and I'm printing [33:35] I'm doing a task and the number of the [33:38] task and I'm sleeping uh time sleep. [33:41] Now, this is where the fun part begins. [33:44] thing while you are working in your um [33:48] operator while the reconcile function is [33:51] running. So see so what happens there's [33:53] a object right there's an object let's [33:56] say you just create this object its [33:59] generation would be one because it is [34:01] just created and its resource version [34:05] would be some random number which is [34:07] given by the cube API server. [34:10] Now your object was created and the [34:15] operator [34:17] fetches this object uh and it stores it [34:21] into the cache of the object and this is [34:24] what's being done here. So this is the [34:25] part which stores the object into the [34:28] cache of the reconciler. Right? You [34:31] start and you make some changes. Now the [34:33] operator runs and it keeps on making the [34:35] changes. However, what we discussed is [34:39] that you can have multiple operators [34:42] trying to write on the same resource. So [34:45] there could be um another operator or a [34:48] human that goes and does cubectl [34:51] edit um EC2 instance and it goes ahead [34:55] and probably change some aspects of the [34:58] custom resource. Now when you save uh [35:02] and quit you know when you save and quit [35:04] your changes the version is going to be [35:07] the generation is going to be from 1 to [35:09] two. Here the old object is with [35:13] generation one and it's XY Z and this [35:16] one's going to be A B C and um the old [35:21] uh the old object is generation one XY Z [35:24] resource version. This one is generation [35:27] 2 and ABC resource version. Now what [35:30] happens? Your operator is going to make [35:34] a change. Maybe it's going to update the [35:36] spec or maybe it's going to update the [35:39] status. Let's put it this way. You know, [35:42] your call to AWS to create an instance [35:45] gives you a public IP. Maybe you want to [35:49] put this public IP in the object status. [35:54] But this is where your um you know your [35:57] operator is going to cry about when you [35:59] say that update when you run the update [36:03] function or you make the update call [36:05] what you're telling to Kubernetes that I [36:08] want to put this update onto an object [36:12] whose resource version is XY Z and its [36:15] generation is one because that's what [36:17] you are aware of. Your operator does not [36:19] know that a change has been done outside [36:22] of its scope. It doesn't know that. And [36:25] when it goes to do it here, when you try [36:28] to do an update, you get an error. It [36:31] says the object that you're trying to [36:34] talk about no longer exists. I don't [36:36] have an object with this resource [36:38] version or this generation. There's a [36:41] newer object. There's there's a change [36:44] in that object that you do not know [36:46] about. And this is where um it gets [36:49] interesting. So see what's what's [36:51] happening here. we try to run some steps [36:54] in the operator trying to uh run some [36:57] steps and I have a very simple scanal in [37:00] so I can wait um this is going to run [37:03] and let me uh make my changes outside of [37:06] the operator with cubectl edit so this [37:09] would be similar to what any other um [37:12] another operator would make changes to [37:14] the same object and until I press enter [37:17] this just gives me a little bit of a [37:18] time until I press enter it will not [37:21] move ahead. So when I do enter uh what's [37:24] going to happen is I'm updating my [37:27] spec.bar [37:29] to a new string or to a different string [37:33] and when I try to update this is where [37:36] it's going to fail because I have [37:38] already made changes to my object [37:40] outside the version has changed. The [37:43] version that operator knows right now is [37:45] the old one and this is where it's going [37:47] to be a problem. Now if any error [37:50] happens in your operator, you usually [37:53] want to retry that. You usually want to [37:55] retry this. So this is where the return [37:58] function um comes into the picture. The [38:01] reconcile uh this this reconcile [38:03] function it has two return types or it [38:06] returns two values. One is the result. [38:11] The other one that we are more concerned [38:12] about and this is the lazy way of doing [38:15] things is the error. So when you have a [38:18] reconciler, [38:20] see the way it works is when you have a [38:22] reconcile function, if it's going to [38:26] return an error, you mean I mean if the [38:29] error that uh is returned is not nil, [38:34] Kubernetes is going to see okay the [38:36] reconciler wanted to do something but it [38:40] could not do that because uh and that is [38:42] why it returned an error. I would try to [38:46] run this reconciler on the object again [38:50] and this is the self-healing mechanism [38:52] of Kubernetes. Maybe you could not [38:54] create the Amazon instance because there [38:56] was a network problem you know and when [38:59] your reconciler exits it says the work [39:01] hasn't been done. Kubernetes will try [39:03] that again and this time the network [39:05] calls would go through or maybe you had [39:07] some permission issues and you fix the [39:09] permissions and this is how the [39:10] self-healing works with Kubernetes. [39:14] In our case, if we could not update the [39:17] status uh the oper the the custom [39:20] resource, it's going to return an error. [39:22] And this is the lazy way. The reason why [39:25] if there's no error, don't return [39:27] anything. The work has already been [39:29] done. We don't return anything. If [39:30] nothing is returned, the the reconciler [39:33] does not run again. It just waits for a [39:36] new change to a to a object that it is [39:39] responsible for. [39:41] But see what would happen because we [39:43] will not be able to update. There's an [39:45] error. The reconciler will run again [39:48] from the beginning and it will go [39:50] through all these 10 tasks. It will go [39:53] through printing the object and the [39:54] resource version. But this time it's [39:56] going to get the newer version because [39:59] it's this get object uh this get um [40:03] function gets the latest object that is [40:06] being worked upon. And this time it's [40:08] going to be version two which is some [40:10] change that was made outside the [40:12] operator. And this time uh it's going to [40:14] get the object print the version which [40:17] is new this time and run through all the [40:20] steps of your operator which it should [40:22] not be doing. There's no reason for [40:24] doing that. But because you returned an [40:27] error, it says I'm going to run through [40:29] everything in the reconciler. And this [40:31] is why I have been saying that your [40:34] reconciler need to be a tempotent. If no [40:37] changes are required you know to to be [40:41] made no changes should be made uh by the [40:44] reconciler. [40:46] Now this time everything will be fine [40:48] and because there's the same version of [40:51] the uh same generation and resource [40:54] version because you get the latest uh [40:56] one that is available you would be able [40:58] to update the spec. you would be able to [41:01] update the spec and without any errors [41:04] you will be returning a nil and that's [41:06] the lazy way because you run your [41:08] reconciler loop again just to get the [41:11] new version you're spending us uh you're [41:14] spending compute cycles you're spending [41:16] a lot of time um you're running your [41:18] reconciler you're wasting your compute [41:21] capacity for no reason instead of [41:24] managing the error here itself you [41:26] return the error you let Kubernetes run [41:29] your reconilider can get the new version [41:32] and uh eventually move on um move on [41:35] with the with the update and this is [41:37] what I can uh I want to show you. [41:40] Now before I can show you I also want to [41:42] talk about um I have changed my uh [41:45] development environment. Before I was [41:47] using kind but now I'm using wind or [41:51] wind as some people would call that [41:53] which is vcluster in docker and that is [41:56] going to be the base of every um [41:59] operator that I build every um [42:01] development environment that I ever [42:03] would be needing. bind is really really [42:06] cool and it gives you a lot of [42:07] functionality which I'm sure there are [42:08] other videos on cube simplify which you [42:10] can look at um but uh for for a very [42:14] simple example you could just do [42:16] vcluster create and it's going to give [42:19] you a vcluster in docker so exactly drop [42:23] in replacement for kind and I have my [42:26] vclustery file which if I can show you [42:29] it looks something like this so I'm [42:31] going to get my worker nodes as the [42:33] docker container. VIN also gives me [42:36] flexibility to create an Amazon EC2 [42:39] instance and attach that to my local VIN [42:43] cluster. [42:44] Um I'm going to be deploying the default [42:47] Flannel Celium um Flannel CNI and the [42:50] Kubernetes version that I'm using is [42:52] 1.3.4 core and you can do vcluster [42:55] create the name of the cluster in my [42:57] case this is zry the driver would be [43:00] docker and this works very well on mac [43:02] with obstack in case you're using this [43:04] you can you can totally use that uh I [43:07] already have this um cluster running [43:11] from from wind and you can see I've got [43:14] these containers and this is where my [43:16] kubernetes cluster is running [43:20] so um we have a we have a working [43:23] Kubernetes cluster. We also need our [43:26] custom resource definitions installed on [43:29] the cluster. And because we are using [43:31] cube builder and this is a make file, it [43:34] has some make targets make manifest, [43:37] make generate and make install. What [43:39] this is going to do is it's going to [43:41] create the CRD. it's going to uh apply [43:45] them on my cluster and I would be able [43:48] to do uh k get EC2 instances because [43:52] this was my um custom resource [43:54] definition or rather explain uh EC2 [43:58] instances and here you can see uh it [44:01] knows about this particular resource and [44:03] if I do spec you will see the fu and the [44:07] bar uh examples here and this is the [44:10] spec that I that I wanted to do. Now let [44:12] me quickly show you uh this in demo and [44:15] let me run my um [44:19] let me see I should be empty okay good [44:22] I'm running my main go and this is [44:24] running the operator on the vclustluster [44:28] um in in docker now see what happens as [44:32] soon as I create an object of type EC2 [44:36] instance let me just quickly show you [44:38] how does it look like so um it would be [44:41] config [44:43] examples or config samples compute. This [44:47] is my EC2 instance which I am creating [44:51] right. So let's do that. [44:54] If I do create what's going to happen is [44:58] my reconcile is started on the left [45:00] side. You can see the object the [45:02] generation is one because it's the first [45:04] time this has been created and the [45:07] resource version is something different [45:10] over here. Right? The resource version [45:12] is something different over here. Um, [45:15] however, it looks like it should be [45:18] stopping. Um, probably. Oh, yes. So, [45:22] what happened? This was supposed to stop [45:24] here so that I could go ahead and make [45:26] the changes myself. But because I hit an [45:29] enter, um, it was given as an input and [45:32] nothing uh, waited for me. It just moved [45:36] ahead. But let me quickly do a delete. [45:38] Uh, and then let's rerun this again. So [45:42] this time I'm going to make sure I do [45:44] not give any enter. I want this to wait [45:46] for me so I can make changes myself to [45:49] uh the object from outside the operator. [45:52] So if I do a create now see what happens [45:56] if I do k get ec2 instances hyphen o [45:59] yaml this is the generation and that's [46:02] the same generation my operator knows [46:04] about. So that's good that's perfect and [46:07] the resource version also is what it [46:09] knows. So right now both of them are [46:12] knowing the same object. [46:14] But what I'm going to do is I'm going to [46:17] edit my EC2 instance and I'm going to [46:19] say this change is made while the [46:24] operator was working on the resource [46:29] something like that. [46:31] Now it's edited and if you look at this [46:33] again this time the generation is [46:36] incremented by one and also the resource [46:39] version is changed. This tells you that [46:43] Kubernetes or API server or HCD sees [46:48] this object differently than what my [46:51] operator holds here right now. Now when [46:54] I press enter what's going to happen is [46:57] uh it will be at this stage when I press [47:00] enter it will try to update my bar it [47:03] will it will change uh whatever string [47:05] is there right now for bar which is [47:08] testing uh it will replace it with this [47:10] string comes from operator and when this [47:12] tries to update it's going to fail. So [47:15] let's press enter and now we will see. [47:18] Now see what happens. This is the [47:20] problem. It says um operation cannot be [47:25] fulfilled on EC2 instances. E2 instances [47:28] the object has been modified. Please [47:30] apply your changes to the latest version [47:32] and try again. Try again. The way you [47:35] are trying again now and this is a stack [47:37] trace of the of the error. The way you [47:40] are doing a retry now is that you simply [47:42] say panic. I'm returning an error. And [47:46] the way reconciler works is if you [47:48] return the error, the whole pro uh the [47:52] whole loop would be running again. And [47:55] this is what you see here. The [47:56] reconciler started. This time it got the [48:00] new version which is version two. Right? [48:03] It got the resource version which is [48:05] resource version here. It ran through [48:07] all these tasks. It ran through all [48:10] three all of these tasks [48:12] and eventually [48:15] it now knows the latest version that [48:17] that there is. And here you can see when [48:20] I press enter it did update the object [48:23] which is which is you know it it went [48:26] through because nobody else updated the [48:28] object while the controller was running. [48:30] It did not change the version. Um this [48:33] time when you did an update a new [48:35] generation was u was was changed or [48:39] rather a new generation was incremented [48:42] which is you see here any change on the [48:45] spec updates the generation. So when you [48:49] create the object for the first time [48:51] that was the generation I make the uh [48:54] generation one I made some changes [48:56] manually that incremented the generation [48:59] two and this time it went to generation [49:02] three because I was able to update my [49:06] object using um [49:09] using the the controller that we are [49:12] writing right now. This is a lazy way of [49:15] doing things. However, there's a better [49:18] way of handling errors so that you do [49:20] you don't just simply say, "Hey, there's [49:22] an error." You don't do that. Rather, [49:25] Kubernetes gives you a package called [49:28] errors. So if I show you um go package [49:34] um Kubernetes [49:36] package [49:38] errors and here's this errors package [49:42] and this is where Kubernetes has defined [49:44] or yeah Kubernetes has defined all the [49:47] errors that you can get. A lot of times [49:51] you might see problems like unauthorized [49:55] maybe you say it's forbidden it's a new [49:58] conflict that you have this um this [50:01] object already exists so I can't create [50:04] that all of these are already defined in [50:06] the package uh in the API uh called [50:10] errors and this is something that we can [50:11] use to figure out what type of error it [50:15] was and what to do on top of that. So [50:18] let's look at this errors and see how we [50:20] can make our operator smarter. So [50:23] instead of quitting and rerunning from [50:26] the beginning, it gets uh the latest uh [50:30] version here itself. When it fails to [50:32] update, it says okay, looks like the [50:35] operator outside, looks like the object [50:37] outside has been changed. Let me quickly [50:40] go and get the latest version right here [50:43] instead of running the reconciler all uh [50:46] again. So let's take a look and see how [50:48] this is uh this is done. All right. So [50:52] what we're looking into in here is to [50:54] try to update um our object if there was [50:58] a error. You know we want to retry if [51:01] there was an error trying to update our [51:03] custom resource because the version has [51:05] been updated or modified somewhere [51:08] external to our operator. The first [51:10] thing we need to know is how can we find [51:14] out what was the error. So this update [51:17] function here it does return an error [51:20] but it is on you to control that error. [51:23] It is on you to define how or what that [51:27] error was. It could be that the object [51:29] was no longer available. It could be [51:31] that you did not have permission issues [51:33] or you did not have permission. It could [51:35] be uh there was a 500 error that you [51:38] got. How do you classify these errors [51:40] that you are getting when you are trying [51:42] to update the resource and for that is [51:44] what we talked about the error package? [51:47] You could do if the error is a conflict [51:51] error something like this is how you can [51:54] say that if I got an error you know I'm [51:58] using the errors package and if it was a [52:01] conflict this error type is already [52:04] defined by Kubernetes in the errors [52:07] package. So if I show you that go [52:10] package errors here you can see this [52:14] package implements uh the functionality [52:17] to classify the error. Um now this isn't [52:20] the one uh it's for um kubernetes one [52:24] this one. [52:26] So this defines um already given [52:30] scenarios in which you can run into and [52:32] you can probably be given an error. For [52:33] example, you try to make an object which [52:37] is already available with that name. You [52:39] know, when you try to do cubectl create [52:40] on a resource and you try that again, [52:43] that's what you get returned. And this [52:44] is what the error is that it already is [52:48] available. One of the other one that [52:49] we're looking into is the is conflict [52:52] function. It checks that the error that [52:55] you got when you're trying to update [52:56] your object if it was indeed conflict [53:00] issue. you know, so you print out um [53:03] that the conflict error was there. Maybe [53:06] you would want to send an email to the [53:08] team that hey, I tried to run my [53:10] operator and I tried to run the [53:12] controller and when I tried to update my [53:14] object, there was an error or you might [53:17] want to uh handle uh it here, right? You [53:21] might have another errors as well. So, [53:23] it's not the only reason that you could [53:26] not update the object. Uh could be [53:28] because of conflict. it might as well be [53:31] that the object you are trying to work [53:33] on it has been deleted uh from you know [53:37] uh from Kubernetes itself. So when you [53:40] try to update that object which no [53:42] longer exists you're going to get an [53:44] error. And again here we can see if [53:47] errors is not found something like this. [53:51] And here is how you can make you know um [53:55] here's how you can make your um operator [53:58] more smarter in terms of how to do error [54:01] handling, how to do error management. In [54:03] our case, if there is a conflict, we [54:07] want to retry. We don't want to quit the [54:09] reconciler. We don't want to do that. We [54:11] want to stop uh there and retry rather [54:15] than exiting the reconciler returning an [54:18] error. and Kubernetes will see, hey, [54:20] this operator did not work properly. It [54:23] is returning me an error. So, I'm going [54:24] to rerun all the steps again just to be [54:27] able to update the function. [54:30] Essentially, at this stage, when you got [54:33] a conflict error, you want to get the [54:37] new object version and store in the [54:41] cache. and then uh you update um the [54:46] object [54:48] with the new version right that's what [54:51] you want to do and for that to do there [54:56] is a fairly simple function and there's [54:59] actually a package that kubernetes gives [55:01] you so that you can implement this retry [55:03] for without even that you could do very [55:06] simply so it's going to be the same code [55:08] that I try to get my object right if [55:11] there was an error. Let's get that again [55:13] and let's update. And this would be very [55:16] very simple. This is how you can [55:18] implement uh an error um conflict [55:21] functionality. However, the thing is you [55:25] can again get an error trying to update [55:29] your object. It's not like you got an [55:32] error once and the next time it's going [55:34] to work fine. You know, you might still [55:37] get an error. You might still get an [55:38] error here. And in the end, you're doing [55:41] the same thing. You are trying to just [55:43] exit the reconciler because you're [55:45] returning an error because you caught an [55:47] error when trying to update. And it's [55:49] kind of like you're going into a loop [55:51] here. So, Kubernetes has a retry package [55:55] which I can probably show you. [55:57] Kubernetes go retry package. It's [56:01] somewhere around here. It's in the [56:03] client go. And you can use these two [56:07] functions available. So one of them is [56:09] on error. Um the other one is retry on [56:12] conflict which is something we want to [56:15] only try if there was a conflict. The [56:18] the first one is pretty generic on any [56:20] error you want to try that again. This [56:22] one we want to try on conflict and it [56:25] has certain uh options to control how do [56:28] we want to retry. Um so this is [56:31] something that we can use. I would say [56:33] retry error and I'm going to use the [56:36] function retry dot return on conflict. [56:39] You can see I've already got the tab [56:41] completion and I'm not sure which [56:43] function which uh model am I using but [56:46] this is cursor um and let me just [56:48] quickly do uh this [56:52] retry error uh and um here we go. So [56:58] this bit makes your retrying a lot more [57:02] uh you know uh smarter rather than just [57:05] manually getting because we are still [57:07] doing the same we are getting the object [57:08] the new one from the cube API server we [57:12] try to update this is exactly what we [57:13] did even without this function but what [57:16] does it give it to you the one thing [57:18] that I really like about the retry [57:20] package and especially the retry on [57:22] conflict is that it does give two inputs [57:26] to this function. The first one is some [57:29] default back off or it is uh it is a set [57:32] of configuration that tells Kubernetes [57:35] or tells your reconciler how many times [57:39] do you want to retry before failing. So [57:42] if you look on the default backoff it [57:45] has steps as four. That being said it is [57:49] going to retry four times before giving [57:52] up and telling to Kubernetes hey I tried [57:55] I could not update it. So this is kind [57:57] of like a retry within the retry um on [58:01] conflict function. That's what it gives [58:03] you how many times do you want to try [58:05] again. The duration is the initial time [58:09] before which your retry function will [58:11] wait to try for the first time and then [58:14] the subsequent times. For example, um [58:17] it's 10 milliseconds. So for me, it's [58:20] going to start at 0 to 10 milliseconds [58:23] to retry. If it fails again then take [58:26] another 10 milliseconds then another 10 [58:29] then another 10. So 10 20 30 40 [58:32] milliseconds is how it's going to be. [58:36] The factor controls what is going to be [58:39] your duration or how effectively you're [58:43] going to retry. So in this case the [58:45] duration uh and the factor it's kind of [58:48] like a multiplier. So if I have a [58:50] duration of 10 seconds and the factor is [58:53] five for the first time when my operator [58:56] starts it's going to try on the 10th [58:59] millisecond then on the 50th millisecond [59:03] then on the 250 millisecond then after [59:06] 1.25 uh seconds is is it going to try? [59:10] So it's multiplying your duration with [59:13] the factor kind of like giving you [59:16] multiple tries within exponential back [59:19] off. Within every try you're going to [59:21] stop a little bit more and try again. [59:25] And this jitter is 0.1 up to one. That's [59:29] the maximum value. This gives you the [59:31] randomness between the time on which [59:34] you're going to uh retry your uh update [59:37] statement. This is especially useful [59:40] when you have more than one controllers. [59:43] They all have the same duration. They [59:45] all start at the same time with the same [59:47] factor. They all will try to write the [59:49] object at the same time. And this is [59:51] something we do not want. So jitter will [59:54] actually be um uh changing the effective [59:58] time. So if you would see the duration [60:00] and factor this would be something like [60:04] if your duration [60:06] uh if your duration is 10 and your [60:10] factor is let's say five it would have [60:14] been um it's going to do 10 milliseconds [60:18] then after 50 milliseconds then after [60:21] 250 mconds you're multiplying that by [60:23] the factor. But when you have jitter uh [60:27] which is uh jitter would be uh 0.1 or [60:31] it's 10% uh that the randomness that you [60:35] are involving. So it would be 10% of 10 [60:38] milliseconds [60:40] within that it's going to start right. [60:42] So it's not going to be exactly 10 [60:44] millconds but it's going to be probably [60:46] 8 millisecond 9 millisecond. Then for 50 [60:49] it would be 47 48 mconds. a randomness [60:52] is added. So it is not exactly this [60:56] particular time on which you are [60:57] retrying. This helps you when you have [60:59] more than one controllers running at the [61:01] same time uh and they all have the same [61:04] factor and the same duration. They try [61:05] to hit your cube API server at the same [61:07] time for updates and this randomness [61:10] would be uh allowing them not to do that [61:14] at one go. [61:16] So this retryon conflict function it [61:18] takes two inputs. One of them is the [61:20] configuration of your backoff which is [61:22] the step and what we talked about. The [61:25] another one is a function without any [61:27] inputs and this returns an error. [61:30] Essentially the return on conflict [61:33] function itself it returns an error. You [61:36] can see here in the function definition [61:38] this is the error it is returning and [61:40] this error comes from u this function [61:43] which essentially does nothing but it [61:45] tries to get the object right. it gets [61:49] the object and uh then tries to update [61:52] your operator or your custom resource [61:54] again. You don't have to have this line [61:57] because it's the same update that you [61:59] want from this from um you know the spec [62:01] that you have changed and then you try [62:03] to do an update. You return uh the error [62:08] if if it's if there is any because [62:10] update function also returns an error. [62:12] So essentially we are trying to get and [62:16] update the resource and we're returning [62:18] uh an error if there is any to the [62:20] return to the retry error uh variable [62:23] here and this is where we can see this [62:26] bit goes into a loop for your number of [62:30] steps. It waits for the the randomness [62:33] uh it waits for the duration. Then it [62:36] goes on every factor and added the [62:38] randomness. It defines the time for four [62:41] four uh steps or four retries. This is [62:44] something you can also very well [62:45] configure. You can give your own [62:47] configuration. Maybe you want to try for [62:49] eight times before you say okay I give [62:52] up now. There was an error and despite [62:55] trying for all these uh retries the [62:58] steps that you have like in our case is [63:00] four. If it is still not nil [63:03] that you know you can only retry. It's [63:06] not like it's a guarantee that your [63:07] update is going to go through, but you [63:09] are retrying. You tried again. You tried [63:11] again. Maybe 10 times you tried. And [63:13] here you can say um print ln. Uh I tried [63:19] I tried four times. Something is wrong [63:23] with the cubernetes cluster or API [63:27] server because that's is what you're [63:29] trying to hit with the with the r.update [63:31] function, right? And then you can simply [63:34] say uh do you want if you return an [63:36] error the reconciler will run again. [63:39] That's usually you can do that. It [63:42] depends on how do you want to choose to [63:43] build your operator. If I tried many [63:45] times already there is an error. Uh I [63:49] probably will just simply say nil that I [63:52] don't want my uh reconciler to exit and [63:55] retry again because I just just [63:57] retrieded. I would rather send something [64:00] as in send email because [64:03] it is something bigger. You know, you [64:07] retry then again and again and again [64:09] that you couldn't update. Maybe the API [64:11] server is no longer responsive. Um, [64:14] someone needs to take a look into this. [64:18] Um, so here you can see uh what's what's [64:21] being done. We will get rid of the is [64:22] not found object. And here it's sending [64:26] an email because something is bigger. So [64:28] if you could retry um you know if if you [64:32] could retry then we just uh ignore this. [64:36] We don't want to do anything [64:39] because if you could retry uh this auto [64:42] control. Oh all right. So in here you uh [64:47] were not able to update the object and [64:49] you send an email. However, if [64:51] everything goes fine, you know, you [64:53] could say successfully updated the [64:54] object after the conflict. And because [64:58] you changed the object, see this is very [65:00] important. Every time you make changes [65:04] to your object, the reconciler remembers [65:07] it. And when this one loop, this one [65:10] reconciler function exits. It sees that [65:12] the object was updated, it will run [65:15] again. And that's what we are saying [65:16] because the update uh because the update [65:18] was done to the object a new reconciler [65:21] will be triggered and we exit gracefully [65:24] without giving any errors giving another [65:27] run to the reconciler because the [65:29] operator because the object uh the [65:31] object was updated properly. [65:35] um if everything was fine. So this is uh [65:38] this is the condition if the error was [65:41] something else you know because you [65:42] could have um any other problems. This [65:45] is where you can also control this um [65:48] that uh fmt.print ln there was an error [65:52] other than the conflict. I'm going to [65:54] return the error. Um again you could [65:58] handle different cases. One of the case [66:00] was if there was a conflict, you could [66:03] handle other cases like uh if the object [66:05] was not found, maybe it was deleted. [66:07] That could be another reason um why you [66:10] might want to uh just um you know not [66:12] give an error. So you could say if [66:16] errors uh do is not found. If it's not [66:20] found, then I don't want to do anything. [66:22] You know, if it's not found, I can't do [66:25] anything. So I'm just returning [66:27] gracefully. We are not complaining. [66:29] We're not sending out an error from the [66:31] reconciler. We are not telling [66:32] Kubernetes try again because the object [66:35] does not exist. There's nothing to try. [66:36] If we don't have the object, we tried to [66:39] update, we failed. What can we do? We [66:42] can just simply ignore the failure. You [66:44] know, this is nothing that our operator [66:46] can control. [66:48] So you can have more than one um [66:49] conditions. And here also you can kind [66:51] of like have a catch all for errors. Uh [66:56] and here you can say um there was an [66:59] error other than the conflict or is not [67:02] found and uh you just then in this case [67:05] you really want to return an error and [67:07] you really want to retry the whole loop [67:10] for the reconciler because you already [67:12] have two cases covered. One of them was [67:14] if it was not found you do not do [67:17] anything. Nothing needs to be done. If [67:19] it was a conflict, we retry. And if we [67:23] couldn't retry properly, then also we [67:25] send an email. We don't really rerun [67:28] that again. But if if we could update uh [67:31] you know, if we could update our object, [67:34] we simply say, "Yep, the update went [67:36] through. When we retried, everything is [67:38] fine. We don't need to rerun the [67:40] reconciler because the object was [67:41] already updated." [67:44] And in the end if there was no errors at [67:47] all you can see here [67:49] um we just uh exit gracefully. So there [67:52] are many reasons or many places you will [67:54] find yourself. Do you want to return an [67:57] error? Do you want to return gracefully? [68:00] What do you want to do? Every place has [68:02] its own value. In this case we try to [68:05] update and if there was no error we [68:08] simply return because there was no [68:10] error. Nothing needs to be done. If in [68:12] case was there any error that is when [68:15] you are returning at different places [68:18] and handling that error differently. But [68:20] if there was no error in the first time [68:23] nothing needs to be done. We can just [68:25] simply exit out of the reconider and [68:27] wait for a new object change detected by [68:30] Kubernetes. [68:32] So um if I show you this in uh in action [68:35] now I'm just going to clean this up a [68:36] little bit. I don't want to look into is [68:39] not found errors and let's just do a go [68:43] run main go. I don't have any object [68:47] right now that this operator would be [68:49] looking into. So it's just going to wait [68:51] and I will do um create. As you can see [68:55] as soon as you create the object the [68:57] version is uh the object generation is [69:00] one and the resource version is 80,748. [69:04] It's going to wait because we ask it to [69:06] wait. What I'm going to do is simulate [69:09] that my object was changed outside. [69:13] So I would say this is now changed and [69:17] you can then see uh its version or [69:20] generation is two. The version is also [69:23] different. If I try again here you see [69:26] what happened. It found out that trying [69:30] to update the object there was a there [69:33] was a you know a conflict error and it's [69:36] it tries to retry there right then and [69:38] there it did not exit the whole [69:41] reconciler what you are seeing this one [69:44] is a new run because you update the [69:47] object in here and this is what it says [69:49] there was a conflict error it was [69:51] retrying successfully updated object [69:53] after retrying on conflict because [69:56] update was to the object object, a new [69:58] reconciler will be triggered. And this [70:00] again it makes sense because um you did [70:03] update the object and here you can see [70:05] you got the latest version and [70:08] everything uh worked as expected. So [70:11] this is something that you would be [70:13] actually doing and this is how you can [70:15] make your operator more smarter in terms [70:18] of not just retry but how do you handle [70:21] errors because there could be errors [70:23] when your operator when your reconciler [70:26] is running. So I think this is a bit [70:28] longer video but I really wanted to give [70:31] you an idea of why would you want to do [70:33] something uh especially when you have [70:35] multiple things juggling the same object [70:37] at the same time. uh in case of u when [70:41] you have more than one operators trying [70:43] to work on the object at one time uh you [70:46] it's always a good idea whenever you are [70:48] updating your object you know whether it [70:51] is a spec or a status it's always a good [70:54] idea to have a retry when you think that [70:57] this object could be modified by some [71:00] other controller if it's not there if [71:03] your operator owns that object nobody [71:07] else is changing it or no other process. [71:09] No other controller is updating that [71:12] particular object. You don't have to do [71:14] that. There's no need to deal with it. [71:16] But especially in in systems where there [71:18] are multiple actors running on the same [71:21] object, it is a very good idea to [71:23] implement this retry like we did. So, [71:26] and even then it's a good idea to handle [71:28] errors rather than just panicking and [71:31] telling Kubernetes please try again. I [71:33] mean rather than just returning an error [71:37] and let Kubernetes do it, it's it's a [71:39] good idea that you can do it on [71:41] yourself, right? When you try to update, [71:43] if there was an error, you handle that. [71:46] If there was a conflict, you retry. Hey [71:49] everyone, welcome to this video about um [71:52] operator best practices in the series. [71:55] And today we are going to talk about how [71:57] can you make your operator work a lot [72:00] faster compared to what you might have [72:03] been doing before. And for this let's [72:07] talk a little bit about what do I mean [72:09] when I say make the operator work [72:12] faster. [72:13] Rather than saying we want to uh make [72:16] our operator work faster, it should be [72:19] we want to make our operator work in [72:22] parallel. [72:25] This best practice does not talk about [72:28] how long does your operator take to [72:31] perform an operation. Um by now if you [72:34] have been following the codes and if you [72:36] have following uh you know the best [72:38] practices you would be knowing that we [72:41] always talk about this custom resource [72:43] called EC2 um EC2 instance right this is [72:49] instance [72:51] there we go right so we talk about a [72:54] resource a custom resource that we [72:56] created in the course which was EC2 [72:58] instance and our operator which was [73:02] running in this Kubernetes cluster was [73:05] responsible to reconcile on this [73:08] particular object. It goes to AWS. It [73:12] creates the instance. It gives you the [73:16] information about that instance such as [73:18] the private IP, the name of that [73:20] instance uh the the public IP that the [73:23] VM has uh has been given and it [73:25] essentially gives it back to you and you [73:27] update that to the status of your custom [73:31] resource and everything is happy. [73:34] When I talk about mating making making [73:36] your operator faster, what we are not [73:39] talking about is this process of [73:42] delivering your customers the virtual [73:45] machine or whatever time it takes uh for [73:48] you to do uh the task that your operator [73:51] is supposed to do. In my case, if the [73:53] time taken by my operator to talk to [73:56] Amazon is maybe let's say um 10 [73:59] milliseconds, it makes a call to Amazon. [74:02] Then Amazon does its magic and gives me [74:04] a virtual machine in 10 seconds. Let's [74:08] say it takes 10 seconds for my VM to be [74:10] really in the running and ready state [74:12] and essentially then I get back my [74:15] response another say 10 millisecond. [74:18] This is not the time we're talking about [74:20] to reduce this. This is something which [74:23] is very um programmatic. This is not uh [74:27] how it is related to an operator. This [74:30] is related to your own coding paradigms. [74:33] Maybe you are using an inefficient call [74:35] to Amazon which takes longer. Maybe you [74:37] are doing long polling. Um again it [74:40] doesn't change the time it takes for the [74:42] for the operation to work. But uh there [74:44] are ways in which you can make your [74:46] operations take less time but that's not [74:49] the the feature of this video. What we [74:52] want to talk about is when you develop [74:55] an operator, you usually do this because [74:58] there are people who want to use a [75:00] product. Imagine this that you are [75:04] working in a company and there are [75:06] people you know there's one guy there's [75:08] another guy and both of them they are [75:11] coming to you and they are saying that [75:13] hey Shbam they come to me and they say [75:16] hey Shabbam can you please give me an [75:19] EC2 instance and I think about that that [75:22] maybe these guys would be you know more [75:25] more benefited if I give them uh a [75:28] selfservice [75:30] kind of portal and that's where this is [75:34] my Kubernetes cluster where I create [75:37] them a custom resource which is again [75:40] called the EC2 instance I give them just [75:46] enough data that they need to populate [75:48] the custom resource with the name the [75:50] service account uh that they want to use [75:53] the IM role they want the EC2 instance [75:55] to have um the the private key they want [75:58] in the instance to be added so they can [76:00] SSH to the machine and essentially um [76:03] what would happen then my operator will [76:07] take this object. This is the operator [76:11] right? This is the operator. It will [76:12] take this object and this is going to [76:15] create me an Amazon um machine and I [76:19] would give it back to the user whoever [76:21] was requesting this virtual machine and [76:24] everything would be happy. [76:26] The problem starts that the the problem [76:29] starts when you are growing in terms of [76:33] the usage of your object. What do I mean [76:36] by that? When you have one person asking [76:40] for the instance, you're one operator. [76:42] You have one operator and this is [76:44] running and that's fine. What happens [76:46] when you have multiple people from [76:48] different teams coming all to you and [76:50] they say, "Hey, you know what? Um, we [76:52] need an instance as well." So imagine [76:56] there are 10 people. Imagine there are [76:59] 10 people [77:02] who submit a request for an object [77:06] called EC2 instance [77:10] and then again you have an operator that [77:14] reacts on this. Essentially what happens [77:16] you have a working Q. So you add 10 [77:20] objects of kind EC2 instance and then [77:24] your operator reacts on top of that and [77:28] it goes ahead and it creates the virtual [77:31] machine create the VM. It gives it back [77:36] to uh it updates the status of this [77:41] object. Then after it has processed one [77:46] out of these 10, this is interesting. [77:50] You have 10 resources to create. You [77:52] have one operator and every operator has [77:56] got something called a worker [77:59] and this worker is the one responsible [78:02] to look into the queue and see how many [78:05] objects do I have. If I have one, cool, [78:08] I will take it. I will process that and [78:10] I would move on. If I have 10 then also [78:15] it does the same thing. It looks on the [78:17] the queue of your objects that are being [78:19] updated. It takes one. It does not look [78:22] at how many it has. It takes one, [78:25] processes it, and it then moves on to [78:29] the second operator object, then the [78:31] third operator object. Because it works [78:33] as long as there are objects in this [78:36] queue. um it it's going to do that. But [78:39] this happens sequentially. [78:42] Imagine if here's a counter. Imagine if [78:46] here's a counter. Where are these there [78:48] are some objects, [78:51] you know, and there are also there's [78:54] another counter. You need to move this [78:56] object from here to this place. But you [79:00] only have one person working. And this [79:04] guy, even though there are multiple uh [79:06] objects available on your left counter, [79:09] he can only take one, then he goes back [79:12] and it places it on the right counter. [79:16] Right? This is what your operator is [79:18] doing. It keeps a look, it keeps [79:20] watching this particular slab and it [79:23] sees, oh, there are objects on top of [79:24] that, but I can only carry one at one [79:28] time. And this is the reason why this [79:31] happens is because your operator when it [79:34] is created it only has one worker. [79:39] Right? So if somehow you can tell [79:43] Kubernetes that you know what I assume [79:47] my object is going to be used a lot. I I [79:51] assume that there are going to be many [79:52] people trying to make changes to my [79:55] object trying to delete that trying to [79:57] create that. I don't want just one [80:00] person who's doing the the lift thing. I [80:03] might need multiple. And that's where [80:06] you can tell Kubernetes operators which [80:09] is we are using cube builder as a [80:11] framework. You can tell cube builder [80:13] that please go ahead and create me um [80:16] maybe two workers or or three workers or [80:19] four workers. [80:20] Essentially there's no limit to how many [80:23] workers you can create. I mean there's [80:26] always a limit because every worker [80:27] consumes uh compute and memory. Uh so [80:31] you don't want your operator to be using [80:33] gigabytes of memory and cores of of CPU. [80:36] You don't want like more workers than [80:38] the requirement because it doesn't make [80:40] sense. It's not a right optimization. [80:43] But uh you would be increasing the [80:45] number of workers, right? And this is [80:48] something you can do in your operator [80:49] itself in the reconciler where you run [80:53] your operator manager. That's something [80:55] where we can configure this. But let but [80:58] let's look at that in action. How does [81:00] it look like when I have 10 instances [81:03] and I only have one worker? How long [81:06] does it take for um you know um my [81:09] operator to be to be working? [81:12] So this is the this the code that you [81:14] have uh been looking at in every video [81:17] and you are very you should be very [81:19] familiar with that. This is our [81:21] reconciler function. If I quickly show [81:23] you the API uh that we created it was [81:27] EC2 instances and here you can see every [81:30] object has got um the metadata. It has [81:33] got the object metadata the type meta [81:36] and then it has the spec and also it has [81:40] the status. It's fairly simple. There's [81:42] no magic going on. We only have two [81:45] keys. One of them is fu. The other one [81:47] is bar. We don't have anything in our [81:50] status because it's not something where [81:52] we are looking forward in this video. [81:54] Now the controller here is also very [81:57] simple. This is the reconcile function [82:00] and this reconcile function is the one [82:02] that is running every time your object [82:05] gets updated or created. The object in [82:08] this case is the EC2 instance um custom [82:11] resource that we are working with. [82:15] So what we do is we register the time at [82:18] which my operator started and we're just [82:21] printing it. So we have a bit more [82:23] verbosity that uh the time it was [82:26] started was time dot now. So this is the [82:30] start time of the operator. the start [82:33] time of the operator. [82:37] Let's say not the operator but the [82:38] reconciler. That makes more sense. The [82:40] reconciliation [82:42] and um [82:44] right so we print out that we started [82:47] our reconciliation at this time and we [82:50] reg [82:53] instance. This is nothing new. You have [82:55] been seeing me doing this for a for a [82:57] very long time if you have been [82:58] following along along on the videos. For [83:00] operators, we create an we create a [83:04] variable of type EC2 instance which you [83:07] can see it only has a spec and it has a [83:09] status. The spec only has two things fu [83:12] and bar. We get the object from the [83:15] reconciler request because whenever your [83:19] object gets updated, a request is sent [83:22] to your reconciler. So we use the the [83:26] the reconciler and the get function to [83:29] get the object which is of type uh which [83:32] is coming from the request. We get the [83:34] namespace name. Essentially we get two [83:36] things from this. We get the name and [83:38] the object um the name space and the [83:41] object name. Uh from the request we [83:43] store that into our EC2 instance. And [83:47] here's where you will be uh simulating [83:51] simulating a task that takes 5 seconds [83:56] for example to complete. Think of this [84:00] as uh think of this as creating an EC2 [84:05] instance on Amazon. [84:08] Right? So I'm using cursor which is [84:10] pretty good in terms of uh helping you [84:12] with the development. But imagine [84:17] each object that your people are working [84:20] with that your customers are using it [84:23] takes around five seconds to complete [84:27] right around 5 seconds. I'm not saying [84:28] exactly five but let's say for to keep [84:31] it simple for the video we say every [84:33] anytime someone creates an object which [84:37] is of kind [84:40] EC2 instance [84:43] this will create an Amazon machine and [84:48] this particular operation takes 5 [84:51] seconds [84:53] now let's let's try to do this let's try [84:56] to run this and then we are just very [84:57] simply printing that the total time [85:00] taken by the reconciliation is we get [85:03] the time dots sinceense from I think it [85:06] makes sense to have it as a variable [85:07] because we want to store the time rather [85:10] than calculating it here because it will [85:12] be zero cuz time since now is is nothing [85:16] it's zero so let's do that start time [85:19] and essentially that is what we wanted [85:23] to do [85:24] so we very simply print out the time it [85:27] has taken since the starting of our [85:30] reconciler and then we return because we [85:33] are not really making any changes. We [85:35] are just seeing how the reconciler [85:38] function how the operator works if we [85:41] change some object and here's a same [85:46] setup with manager because we have an [85:47] operator manager and we set up our [85:50] reconciler with that manager. This is [85:52] something which is pretty standard. And [85:54] if I show you the object that we're [85:58] going to work with, this would be [85:59] something in config sample. And here you [86:03] go. So you can see this is kind of how [86:08] I'm simulating that multiple people are [86:12] trying to interact with my resource. So [86:15] this could be a person from team A who [86:18] gives the object that I want to create [86:20] an EC2 instance. This could be from team [86:22] B, team C, team D, team E. Could be [86:25] multiple files, could be same file. The [86:27] only important thing is all of these [86:30] objects that I'm giving to my Kubernetes [86:33] cluster at the same time is because this [86:35] is how people would be using your [86:37] object. There may be two people, three [86:39] people, four people in a team who would [86:42] be trying to talk to your operator at [86:44] the same time. No, the the only thing [86:48] difference is the names are not clashing [86:52] because we don't want to keep the names [86:53] clashing because this is not a name [86:55] score namespace resource. If I have [86:57] instance one, if I try to create that [86:59] again, I'm going to get a problem that [87:01] you're trying to create a resource which [87:02] is already existing. We don't want that. [87:04] Kubernetes 101. And here you can see [87:08] instance two. And here you can see three [87:10] and the same four and it's five. They [87:13] have different names. they have [87:15] different data. The data could be [87:16] anything. It could be the same. It [87:18] doesn't really matter. But the name [87:19] should be unique because it's not not [87:21] it's not a namespaced scope uh object. [87:25] What I'm trying to do here is just [87:28] trying to see or trying to tell you is [87:30] there would be imagine this is being [87:33] done by different people, right? So [87:36] let's run our operator. I will see first [87:39] if I have any EC2 instances in my [87:41] cluster. I do have one. I will delete [87:45] that. Um [87:48] I will delete that and I will just [87:51] install my custom resource definition [87:54] again. Um because I was working on [87:56] multiple versions of the operator but [87:58] this is the one that I want to use and [88:00] I'm using cluster um platform uh to run [88:04] my local development environment. [88:08] So I could say um go run main.go. So [88:12] this is running our operator and that's [88:15] where you will see this line which is [88:18] very important. It is starting the [88:21] controller. It is then starting your [88:23] worker and you only have one worker for [88:26] the controller kind of EC2 instance. [88:29] That means if someone creates an EC2 [88:32] instance, even if they give like 10 [88:35] instances, which is what I'm doing in in [88:37] this example, it is only going to have [88:40] it is only going to create one instance [88:43] at one time. And this is this is exactly [88:45] what you want. See, the thing is you do [88:49] not want um more than one people. So [88:53] imagine this that this guy called A and [88:57] this is called B. If A was already [89:00] working on this object, if A was already [89:03] trying to uh move it to destination, you [89:07] don't want B to uh interact with him. [89:11] Right? This is the same thing that [89:12] happens when in cricket you have a [89:14] cricket ball and there are two people [89:16] trying to catch that ball and they just [89:19] collide into each other and the the [89:21] catch drops, right? You don't want that. [89:24] What you want is if there were two [89:26] balls, which in cricket is not a good [89:29] example, but if there are two balls, [89:31] each one should be only looking at their [89:35] ball at the same time because you don't [89:37] want either of the balls to be dropped [89:39] on the ground. That is what we want. [89:42] We want to make our operator faster so [89:46] that not in terms of how long does it [89:49] take to create the WM but how long does [89:52] it take for the operator to react on [89:55] those 10 available objects. So we only [89:59] have one worker you can see and I'm [90:02] going to create all these five instances [90:04] and you will see how this is behaving. [90:07] So if I do uh create you can see here [90:12] start time of the reconciliation is this [90:14] and you see I created five instances but [90:17] what's happening is it's being done one [90:20] by one and the operator is smart the the [90:23] workers are smart they know that there [90:25] were five instances that were changed I [90:28] need to work on five instances but I can [90:31] only take one at a time I cannot take [90:34] multiple at one time and you don't also [90:36] want back. [90:38] So you see what's happening is every [90:42] time the reconciler starts [90:45] it calculates it prints out the time and [90:48] it prints out the time taken for that [90:51] reconciliation. [90:52] Every time the reconciler runs it runs [90:56] because an object was updated. It runs [90:59] because an object was changed. You would [91:02] be if you were thinking that every time [91:05] you see I'm taking five seconds for one [91:08] uh instance I have five instances my my [91:12] reconciler will take 25 seconds for all [91:14] five. That's not how it works. [91:18] Yes, for my loop to run, if you will [91:21] just count them, five five five signs, [91:24] it essentially took 25 seconds. But one [91:28] worker takes 5 seconds um and five times [91:33] is is when it's added to 25. It doesn't [91:36] change the time taken per [91:38] reconciliation. It still takes 5 seconds [91:41] for one object. Now what you can do in [91:44] this case is imagine if the time taken [91:47] was longer you know if if your operator [91:50] takes more time let's say 2 minutes or [91:53] which is not too bad but it's not also [91:56] not great not terrible quoting [91:58] shernobbile here however [92:02] let's do let's kick things up a notch [92:05] and I would say uh you know it doesn't [92:07] sleep for uh 1 second it sleeps for 10 [92:12] seconds [92:14] every time you are going to because I'm [92:16] creating five instances. So just trying [92:19] to simulate that now it takes 10 seconds [92:22] for each object and I've got five. So [92:25] I'm just printing it five times. So it's [92:27] going to be actually 50 seconds which is [92:30] not what we want. I want two into 10. So [92:32] five times it's going to run 2 seconds. [92:35] So 10 seconds in total is what it's [92:37] waiting for. If you stop it and if I run [92:40] that again, you're going to see the same [92:42] thing. See, now for each instance, [92:48] it hasn't even started one of the other [92:50] instances yet. It's just using instance [92:52] number one. Then it moves on to instance [92:55] number two. This is a serial um work of [92:59] your operator on your resources because [93:02] you only have one worker. The worker is [93:04] busy uh here with instance one. Then it [93:07] moves on to instance two then on to [93:10] three. People who have created their [93:13] object before the others it it's going [93:16] to be um you know um given priority to [93:20] them. Actually this doesn't make sense [93:22] because uh it is not really um 1 2 3 4. [93:27] It's not by the the creation time stamp. [93:29] But the thing is your worker is using [93:32] one object at one time and this is not [93:35] what an efficient use of the operator [93:37] framework is. You could say that in my [93:40] company I see a lot more usage. So if I [93:43] have like 10 objects it makes sense to [93:48] create three or four workers because [93:51] then they can have things in parallel. [93:54] They will still be completely separate [93:56] from each other. If I give 10 EC2 [93:58] instances, they're going to be creating [94:00] 10 different instances completely [94:02] separated because they do not work with [94:05] each other. All they do is they grab an [94:08] object and they do what they have been [94:11] told to do separately without working [94:14] with other other worker processes. And [94:17] this single thing can really help you um [94:23] to to work with your operators much more [94:25] efficiently. And that can be done in the [94:28] setup with manager function. What you [94:30] can do is you could say that um for the [94:35] EC2 instance that is what my operator [94:38] should be reacting on. I want it to [94:42] react with the options and then you have [94:44] options within your controller. [94:47] And here's you can see there are options [94:49] and one option is max recurrent [94:53] reconcilers. [94:54] Re reconciles that means it's [94:56] essentially a fancy word for number of [94:59] workers. How many maximum um instances [95:03] of the reconciler you can run and [95:05] because we know that one or one worker [95:08] runs one reconciler at one time. If you [95:11] have 10 then there will be 10 workers. [95:14] They can reconcile 10 times or they can [95:16] run this function at one time uh 10 [95:20] times. And that bit is going to make [95:24] huge difference for you. Let me put it [95:27] back to where it was for one second, [95:30] right? Because we wanted it to be 1 [95:32] second. And now you can see how it works [95:36] as a difference. What difference does it [95:38] make when we run this again? [95:41] Now you can see I run my operator and [95:44] because I now have 10 workers you can [95:47] see it's already doing things in [95:50] parallel. we get I'm starting with um a [95:54] reconciliation time and you can see this [95:56] is instance 2 3 2 5 it all you know each [96:01] of them took 5 seconds because we are [96:03] not reducing the time it takes [96:04] individually [96:06] but your other objects that were there [96:09] they are not waiting to be picked up by [96:12] an operator because your operator has [96:14] number of workers which are all busy [96:16] they they will be taking up the object [96:18] when it's [96:20] We don't want that. Think of this [96:23] practice as when you have a lot of use [96:27] for your um for your custom resource. [96:29] For example, pods. If I was to create [96:32] 100 pods and only one could be created [96:35] at one time, you're going to be waiting [96:38] before your pod could even be scheduled [96:40] on a node before it could it was even [96:42] picked by the operator. So number of [96:45] workers increasing that it really helps [96:48] your operator being a lot more parallel. [96:51] It helps you to pick those objects much [96:54] faster and essentially then whatever [96:57] time it's going to take they will [97:00] because we can't stop that. As I said if [97:03] they're going to take 5 seconds in each [97:06] iteration for each object they will do [97:08] that. That's not the point of the video. [97:11] You could you could do this by saying [97:13] okay um if I wanted to reduce the time [97:16] they take within each reconciler for [97:19] each worker I could say millisecond you [97:22] know this is I make my program more [97:24] efficient and they're going to now take [97:25] one millisecond but the point is if you [97:29] have a lot of operate if you have a lot [97:31] of objects that you're working with and [97:34] you do not want to wait for them to at [97:37] least be picked up and get started [97:40] because imagine Imagine uh this case [97:43] this is for humans [97:46] that let's say on an axis of time [97:52] um here time being zero and here time [97:55] being infinity [97:57] if creation of one instance [98:01] right if creation of one instance takes [98:05] 10 minutes I'm just giving you a very [98:07] simple example if this takes 10 minutes [98:09] it's bad uh Um again it it depends on [98:12] what you're trying to create and where [98:14] you are trying to create that. But let's [98:16] say um this one is 1 p.m. So 1300 hours [98:21] is here and then there's probably like [98:24] 2300. [98:27] If I was to start my operator creation [98:31] from 1300, [98:34] I would be getting my instance at 1310 [98:37] because it takes 10 minutes for my [98:40] instance to be created. [98:42] However, if there's a flood of objects, [98:46] right? If you give like 10 objects [98:50] and each one takes 10 minutes. So your [98:53] instance which was probably taken the [98:55] last it would be starting 10 minutes 10 [98:59] minutes 10 minutes 10 minutes 10 10 10 [99:02] 10 10 10 10 10 10 10 10 10 10 10 10 10 [99:03] 10 10 10 10 10 10 10 10 10 10 10 and [99:04] then here you would be starting your [99:07] your request would even be picked up uh [99:10] after like um you know it takes one [99:13] instance 10 minutes or 100 minutes after [99:15] an hour or more than an hour more than [99:17] one and a half hour later your instance [99:20] request would be picked up. So it would [99:22] be around 14 um uh would be about 1430 [99:28] 90 minutes or 100 minutes would be so [99:30] 1440 is when your instance would be [99:33] picked up but you made the request at 13 [99:36] at 1 p.m. requested and your request was [99:39] started to be picked up at 2 hour 40 [99:42] minutes and you know PM and then you [99:45] would get your instance in the same 10 [99:48] minutes because everyone got their [99:49] instances in 10 minutes but you had to [99:51] wait because there were only one worker. [99:55] If only you have like 10 workers and if [99:57] only you have 10 objects [100:02] all these objects would be picked up at [100:04] the same time. They would be worked on [100:07] by one one worker and you would get your [100:10] object. All of you who have requested [100:12] for the object, they will be getting [100:14] their object after like 10 minutes [100:16] because it takes 10 minutes to create [100:18] one instance. The whole idea is yes [100:22] there are ways in which you can make [100:24] your reconciler efficient which is what [100:26] I did from time second to time [100:28] millisecond that's always always [100:30] possible you know again it depends how [100:32] you have coded but the the main uh [100:36] motive of this video is to make your [100:38] operator behave in parallelism so you as [100:42] a user are not waiting for someone to be [100:46] free and pick up your request and start [100:49] working on top of that. And that could [100:51] be done with the controller options uh [100:54] using the max current reconciles. That [100:58] means um how many concurrent [101:02] um instances of this reconcile function [101:05] could be running. And because one worker [101:08] will only work on one object, if you [101:10] have 10, you could give 10. There's no [101:14] limit to how many workers you can have. [101:15] Of course, it's also a bad design. If [101:18] you have like 100 workers but only at a [101:21] time you get five requests, 10 requests, [101:24] 10 object changes, creations on your [101:26] custom resource, that's also a bad [101:28] decision. There's no one stopping you [101:30] from doing that. But you wouldn't be [101:31] making 100 rooms for uh for people, you [101:36] know, who are only maximum 10 at one [101:38] time. So the decisions like these would [101:41] actually help you to make your operators [101:44] much more faster in terms of [101:47] parallelism, right? And that's the [101:50] object that's the the the objective of [101:53] this video. If I would run that again [101:55] now, you can see it's efficient in terms [101:57] of how long does it take for the [101:59] instance creation. It's like 5 [102:01] milliseconds, but it still picked up all [102:05] of your instances at parallel because [102:07] you now have the worker count to be 10. [102:11] Small things like this can really make a [102:13] difference when you are using your [102:16] object when you're using your operator [102:18] at scale. All right. So for this best [102:21] practice before I talk about what we are [102:25] going to do here I rather want to um [102:29] draw a picture. So if you have gone [102:31] through my previous course of how to [102:33] build your own uh Kubernetes operator [102:35] from scratch um you must have heard me [102:38] saying this many many times that every [102:42] time you make an update to your object [102:47] um you are triggering a reconciler loop [102:52] and I've said this extensively in the [102:55] video you run the reconciler loop again [102:59] and this is Why you need to make things [103:02] ident [103:03] so that if the external state that your [103:06] object uh is managing or your operator [103:09] is managing if it matches the desired [103:12] state that the user has asked for you [103:14] simply do not do anything. Think about [103:17] uh anible in this case you write a [103:19] playbook that installs a server uh [103:22] that's installs an engineext uh [103:24] application server on on a Linux [103:27] machine. it starts the service and you [103:29] run that again. It does not reinstall [103:32] EngineX. It does it does not restart the [103:35] service. It just simply says things are [103:37] exactly how you want me um to make and [103:41] I'm not going to make any changes. The [103:43] same thing happens in your reconciler. [103:46] So any update that you are making [103:49] whether it was uh an AI u you know tool [103:53] that did it for you whether you did this [103:56] with cube c [103:59] or whether it was kit ops that makes the [104:02] changes for whatever reason there's an [104:04] update to your object an object has two [104:08] fields which is spec and the status [104:13] either one that you update [104:16] This triggers a new reconciler loop and [104:19] this is why um you know um your [104:21] reconciler should be ademptent that if [104:24] no change is needed I do not do [104:26] anything. I have talked about this [104:28] extensively uh in the in the YouTube [104:30] video where we talk about how do you [104:32] build an operator from scratch and I [104:34] will be putting a link of that in the [104:35] description. [104:37] Now let me uh let me draw a picture. uh [104:41] we made an EC2 operator that operator [104:46] creates an instance for you. So in that [104:49] operator we had an object um and we had [104:53] a simple spec you know things that you [104:55] need when you are creating um an [104:58] instance which is uh with uh in Amazon. [105:03] So, so you had basic things like you had [105:05] the instance name that you want. Um the [105:08] AMI ID um that you wanted to use. Uh you [105:12] also had then the security group. Um you [105:16] then talked about the storage that you [105:18] want to use. Um and you could be as uh [105:21] descriptive or as you know uh options [105:24] you can give as you want, but I'm just [105:26] going to keep these simple. This was a [105:28] spec and the status pad. couple of [105:32] fields like the public IP, what is the [105:35] public IP of my instance and it could be [105:39] um the creation [105:41] timestamp. [105:44] Now these things you will get or you [105:49] will you know uh use to create an Amazon [105:53] instance. [105:56] And here is your EC2 instance. [106:02] Now imagine you wanted to make this [106:06] operator a bit more reliable or you [106:08] wanted to make this operator a bit more [106:10] um intelligent. What you thought of is I [106:14] have the public IP, I have the creation [106:17] timestamp. Um I want to periodically [106:20] check if my instance is there or not to [106:24] do some sort of drift detection. and you [106:29] wanted the user to know when the last [106:32] check was made. So you introduce a new [106:35] field in the status. See, you're not [106:38] changing the instance. You're not [106:39] changing the instance's name. You're not [106:41] changing its AMI ID. You're not changing [106:44] anything anything on the instance. [106:46] Nothing is being changed. The spec is [106:48] exactly the same. What you are changing [106:52] or rather introducing a new field which [106:55] is called last [106:58] checked [106:59] at all you have made in your operator is [107:03] it goes to Amazon with that instance's [107:07] name sees this instance is there if it's [107:10] there it does nothing it only updates [107:15] your reconciler it does not do anything [107:17] if it's there and running because that's [107:19] what you want this is your desired [107:21] state. If it's not running, you can [107:23] start it. If it's deleted, you can maybe [107:26] send an email from your reconciler. [107:29] That's all uh you can do when it's not [107:31] there. But if it's there, all you want [107:34] it to do is to update your status of [107:37] last checked app. Uh and this could be, [107:40] you know, uh 5:00 p.m. um 5 p.m. 10 [107:44] minutes [107:46] and 20 seconds. This is when you made [107:49] the change and you want to do this [107:51] periodically [107:52] every every 2 minutes. Let's say you [107:55] want to make a check on Amazon that if [107:57] the instance is there. Every 2 minute [108:00] your reconciler is running not doing [108:03] anything on the spec because you don't [108:05] want to uh change anything on the spec [108:07] but it goes ahead and checks your um you [108:11] know it checks your instance and every 2 [108:13] minutes it's updating this specification [108:16] here which is last check. [108:19] The thing is you are updating the status [108:24] the resource is already in that state. [108:28] Now what what's kind of happening is [108:30] what I told you that when you make an [108:34] update to your object [108:38] when you make an an update to your [108:40] object um it triggers a reconiler [108:47] loop. [108:48] We talked about this and if you are [108:51] making changes to your object in the [108:54] reconciler loop you will be rerunning [108:58] you know you may update the object the [109:00] reconciler loop runs you update the [109:02] object the reconciler loop runs it does [109:04] not matter if the object was updated [109:08] externally from someone who's using [109:11] cubectdl manually or it was updated in [109:14] the reconciling loop itself within the [109:16] reconciler the only thing Kubernetes [109:19] cares about object was changed. I will [109:21] rerun the reconciler loop which is then [109:23] you have the steps for uh creating the [109:27] instance. [109:29] Then you have um you know um maybe [109:32] sending an email to the team with the [109:35] instance IP address um and all all of [109:38] the steps that you need to do when you [109:40] are creating an EC2 instance. [109:43] The problem is these steps should only [109:48] be running when you actually want to [109:51] make a change whether it is creating a [109:54] new instance which is in this case it [109:56] just creates an instance [109:58] just because you updated your object or [110:02] I should say just because you updated [110:04] your status [110:06] that the instance was there you need not [110:09] to run the whole reconciler loop you [110:12] should run the reconciler loop again if [110:15] I change the AMI ID in that case your [110:18] operator should you know delete the [110:21] instance and recreate that again you [110:24] need to make some changes in the [110:26] environment but if I just go ahead and [110:29] check hey if the instance is there if [110:32] it's there I just want to update the [110:34] metadata I did not change the name of [110:37] the instance I did not change the AMI ID [110:40] of the instance no security groups are [110:42] changed. I didn't increase the storage [110:44] from 50 to 100 gigs. Nothing is [110:47] changing. So why are you running your [110:50] reconciled loop and wasting your [110:52] precious CPU cycles? This is where you [110:55] are trapped in a loop where you update [110:57] the status which was supposed to be the [110:59] metadata. The reconciler runs again and [111:03] this loop continues forever. [111:06] So this is what I kind of wanted to um [111:09] show you. You know when you are creating [111:12] the instance you have a spec here and [111:15] then you have the status which is where [111:17] we have the last check that your [111:19] operator goes to Amazon sees if the [111:23] instance is there if it's there we don't [111:26] make changes to the spec but we make [111:28] changes to the status where we update it [111:31] with the last check that and the time [111:34] the thing that happens is because you [111:36] updated your object [111:39] there was a watch event which your [111:41] operator always uh watches on um you [111:45] know um your controller always watches [111:47] on the operator. What am I saying? Your [111:50] controller always watches on the object [111:53] that is changed and you rerun again [111:57] even though you did not really want to [111:59] make any change. your reconciled logic [112:02] should have just skipped it that hey you [112:04] know what guys the user did not ask me [112:07] to make any changes to the WM why why am [112:11] I even running okay I I know I have to [112:14] be a dempotent so that if I run I should [112:18] not make any changes but why run in the [112:20] first place if I was not you know [112:22] changing the spec if the user did not [112:24] change the spec there's no reason for me [112:27] to run the change was only at a status [112:29] level which was only the metadata [112:32] and this is the problem with uh the [112:35] reconciler or not the problem but the [112:36] way the reconcilers are created because [112:39] Kubernetes does not differentiate if the [112:42] change was in the spec which you most [112:45] care about or was it in the status which [112:48] you care about less to rerun your [112:50] reconciler it only says the object was [112:53] updated rerun the uh reconcil loop [112:56] that's it so I hope you got the problem [112:59] you're updating the status of the object [113:03] and this triggers an update. Um, and [113:07] this triggers a reconiler [113:11] and within the reconiler you updated the [113:13] status. You then got a update event. The [113:17] reconiler runs again. You again go [113:19] update the status. You then kubernetes [113:22] says hey an update is there. the [113:24] reconciler runs and this is going to [113:26] make your operator a living hell. [113:30] Yes. I mean see the thing is you could [113:33] say um Shubhham it's not a problem even [113:36] if my operate e even if my reconciler [113:40] runs you told me to make it Impo. So [113:43] even if it runs, nothing's going to be [113:46] um changed in my Amazon instance. And [113:49] that is correct. But that's the [113:51] difference between you and a staff [113:53] engineer or a principal engineer who [113:55] would stop you from running that [113:57] operator even in the first place because [114:00] you're going to be using CPU of your [114:02] cluster and if you do not design things [114:04] properly that you could eventually uh [114:07] you know uh waste your resources. [114:10] So this is what we have been doing. So [114:14] the user he makes a change to the API [114:17] server. Um there's a there's a cop [114:20] there's a component in your operator um [114:23] which is called a reflector and this [114:26] reflector is responsible to watch um [114:30] your op your uh object. [114:34] We are not going to go into details of [114:36] what and how reflectors are implemented. [114:38] Think about this as the deflector is the [114:41] one responsible to keep a keep an eye on [114:44] your object in the API server and it [114:47] catches these events. [114:50] It's the one that updates the informer. [114:52] So you go ahead and you update the cache [114:57] with the latest object that was updated [114:59] by the user and then uh the event [115:02] handler is the one responsible to add it [115:05] to the working queue where you add what [115:08] object has been changed and the reconil [115:11] gets the object from this cache. So it [115:13] doesn't go to the API server to get it. [115:15] And what it does is it does an update of [115:19] the status. [115:22] Eventually, here's where the loop gets [115:24] created, which is I just showed you [115:26] above. That's the loop that you're [115:28] making. We're just going into a bit more [115:29] detail here. So every time you make a [115:32] change to the object, the reflector gets [115:35] it. The reflector sees that there was a [115:37] change. The reflector talks to the [115:39] informer which gets the object and [115:42] stores it into the cache memory or the [115:45] cache of the operator. Then the event [115:49] handler is is is you know um informed [115:52] that hey you know what this object has [115:54] been changed please add it to the [115:56] working queue. And the reconciler [115:58] workers are looking at the working queue [116:01] seeing that new object has been changed. [116:04] It has been added to the working queue. [116:06] I need to start and run uh my [116:08] reconciler. [116:10] Then in the reconciler you update the [116:12] status and this whole story uh you know [116:14] the whole story goes on. I think it [116:17] makes more sense if I show this to you [116:20] in a demo. So let me go to uh my trusted [116:24] uh cursor and let me show you how this [116:27] looks like. So this is standard cube [116:29] builder. I'm sure you have gone through [116:32] the video so you are familiar with uh [116:34] how the cube builder u you know um how [116:36] the cube builder structure looks like. [116:40] The most important things that I wanted [116:41] to show you is my API and this is what I [116:44] have created. So I have a very simple um [116:48] new custom resource called EC2 instance. [116:51] It has two things. It has a spec of type [116:55] EC2 instance spec and then it has a [116:58] status of type EC2 instance status. Now [117:02] this now the spec is very simple. It's [117:04] just fu and bar. It is just the two fu [117:08] and bar keys that I want to use. We're [117:11] not creating a fully functional operator [117:13] in this video. We did this in a previous [117:16] course. This is a continuation from that [117:19] course. [117:21] The status is what we are more [117:23] interested in. And here we have the last [117:25] updated at which is a type of [117:28] metav.micro [117:29] time. This um this is a strct that lets [117:33] you add a time up to microsconds. We [117:35] will talk about that. And this is my [117:38] simple uh EC2 instance custom resource. [117:41] It just has a spec of bar and a status [117:45] which is last updated at. Now how does [117:48] my reconciler look? [117:50] So if I go to my reconciler here, you [117:53] can see um there are a couple of things [117:55] that I do not need. Um [118:00] uh wait uh I don't need the math library [118:03] as well. So here's my reconciler. [118:07] See this reconciler does something very [118:10] very simple. It does something very [118:13] simple. [118:14] it gets um uh it creates a variable of [118:18] type EC2 instance. So this is where we [118:21] will be storing the object that we get [118:22] from Kubernetes in this line here. This [118:27] course is this code is something that [118:29] I've already explained extensively uh [118:32] into our uh Kubernetes operator video. [118:34] So I'm just going to go through that [118:36] very quickly and show you essentially [118:38] what's happening. We create a variable [118:40] of type EC2 instance which is our strct [118:43] which is our custom resource. We get the [118:46] object from the request that was made to [118:49] my reconciler. You know API server said [118:52] hello this object has been updated. I [118:55] get the object. I stored it this into a [118:58] variable and I'm reading that variable [119:02] and as such. So f value is [119:05] instance.spec.fu two and bar value is [119:08] instance.spec.bar. [119:10] Now you can see here [119:13] so far I've just got the object and [119:15] iterated over its values. Now here's [119:18] where the interesting thing um here's [119:20] where uh things gets interesting. [119:23] What I wanted to make is I wanted to do [119:26] an update. [119:28] See I have this uh I have this EC2 [119:31] instance custom resource. It has two [119:34] fields spec and a status. I am only [119:38] reading on the spec. I did not make any [119:40] changes on the spec. What I am making [119:43] the change on is the status and the [119:47] field called last updated at. So I want [119:50] to set the last updated as field uh to [119:54] the current time till uh to to the [119:57] microsconds of the current time. And [119:59] this is where we get the time uh in UTC. [120:02] And then we convert that or rather we [120:06] format it uh into a um time which is [120:09] microsconds. I think it it shows [120:11] microsconds and then I'm doing a [120:13] status.date [120:14] function. What this does this line here [120:19] updates your object. So essentially [120:22] what's happening is you start your uh [120:25] reconciler, [120:27] you get the object. [120:32] Okay. Uh here's where you got the [120:33] object. Here you got the time and then [120:38] here's where you read um or they could [120:41] be interchanged, but that's not [120:42] something we need to worry about. What's [120:44] essentially happening here is where you [120:47] update [120:49] the status [120:52] and because you updated the status with [120:56] the R dot status [121:00] dot update function. [121:05] You're going to put this back to the [121:07] working Q [121:10] work Q. Think of this working queue as a [121:14] place where once your reconciler is done [121:17] he goes and sees is there something and [121:20] yes you added your object key the name [121:23] of the object that was updated here. Now [121:26] what's going to happen? You update the [121:28] status and then we are printing this [121:30] function last update is you know uh you [121:33] print something that last update is done [121:38] blah blah blah and then you return. Now [121:41] as soon as as soon as you do a return [121:46] your operator is going to look in the [121:49] working queue. [121:51] He's going to read from the cube and [121:53] this is going to write to the cube which [121:55] is the name of the object that was [121:57] updated. You read it, you rerun the [122:01] whole reconciler from the beginning and [122:03] here you see you update it again. You [122:05] write it, you finish, you go back to the [122:08] working queue, you run from the [122:10] beginning, you read it, you write it. [122:13] This is the loop that I've been talking [122:15] about. And let me show you this in um in [122:18] action. [122:20] So here you can see we have uh I [122:26] so there is no EC2 instances right now. [122:29] I have already installed this in my [122:31] cluster using cube builder and you can [122:33] simply use the make generate make [122:36] manifest um and make install. I've just [122:39] uh uh done a clean installation. So I'm [122:42] always using uninstall and install [122:43] because I was making changes to my spec [122:46] my status but it's installed. So you can [122:49] do cubectl explain EC2 instance status [122:53] and here you can see the status field [122:57] has last updated app and spec has got [123:01] bar and fu which is exactly the spec [123:04] that we are creating. All of my [123:06] terminals are connected to the same [123:10] cluster where we have installed these [123:11] custom resource definitions. [123:14] Now if I wanted to show you what we are [123:16] creating, you can see here it's a very [123:19] simple EC2 instance. It just has a bar [123:23] for testing. It has two for testing. [123:26] Nothing nothing fancy. So I'll run my [123:28] operator and because there is no EC2 uh [123:32] resource uh there's no EC2 instance [123:35] custom resource, it's just watching. It [123:38] just says okay I'm waiting for any [123:40] changes to happen on your operator on [123:43] your custom resource whether you update [123:45] it or you create that. Now I'm going to [123:48] create this in my cluster and you're [123:51] going to see something interesting which [123:53] I've been trying to show you. [123:55] Let's create that and you see what's [123:57] happening. [123:59] You see my operator is going crazy. It's [124:03] running and running and here you can see [124:05] the status update is happening which I [124:08] wanted this but my operator keeps on [124:11] running. My my reconciler has gone [124:14] crazy. It's gone into a loop where it [124:17] runs it updates the status then um it [124:22] registers in the working queue. It [124:24] exits, sees in the working queue, runs [124:27] it again, updates the status, puts it [124:30] into a working queue, exits, sees the [124:33] working queue. And this is the whole [124:35] shenanigan that's been going through. [124:36] And this is something you absolutely do [124:40] not want. You don't want to rerun your [124:43] reconcile it just because you made a [124:45] metadata change, which is the status. [124:48] Why would you want to do that? [124:50] Now something that I also want to u you [124:52] know um put into here is something [124:56] called generation. [124:58] See as your operator is making changes [125:01] to your resources um you do update the [125:05] last update act because this is what you [125:07] want to do but Kubernetes also updates [125:11] something called a resource version. [125:13] This is the version of the resource [125:14] which was there in HCD. So you always um [125:18] have a resource version which is telling [125:20] you the latest version uh that your [125:22] operator is uh in HCD. There's also [125:26] something called generation. [125:28] Now you can see my operator has been [125:30] running for so long and I've got about [125:33] so many resource version changes. [125:36] However, I only have generation as one. [125:40] Now let me show you something really [125:41] interesting. If I stop my operator and I [125:46] do a cubectl edit EC2 instances, I do a [125:51] bar and I say new testing data. I save [125:55] and quit that. [125:58] See what happens. The generation changes [126:03] uh this time not just with the resource [126:05] version. Resource version changes [126:07] anytime you make an object change. But [126:09] generation changes only when you update [126:12] the spec of Kubernetes of your object [126:16] not the status because we kept running [126:19] our operator for a very long time which [126:21] is you can see again you can see here my [126:25] status is changing my resource version [126:27] changes but the generation never changes [126:30] however as soon as I do a change [126:37] in this case I do testing you see my [126:40] generation is incremented by one. So [126:44] every time you update your status, the [126:48] generation does not change, but every [126:51] time you update your spec, the [126:54] generation is incremented. [126:57] And that's something you can make use of [127:00] when you are writing your operators to [127:02] not go into this forever running loop um [127:05] where you uh you know you update the [127:08] object then you read you update the [127:10] status you rerun and and getting [127:13] yourself um throwing those precious CPU [127:16] cycles away and here's what's going to [127:19] happen. See when you are working with um [127:24] with status updates you just see there [127:27] was a change you don't care about if the [127:29] change was in status you don't care [127:32] about if the change was in spec or [127:34] Kubernetes does not do that your [127:35] operator does not care about it right [127:38] now however in cube builder in [127:42] controller runtime there is an option [127:45] where you can say guys I only want to [127:49] trigger a rerun of my reconciler if the [127:53] generation was changed. That means if [127:56] the spec was changed, you always want to [128:00] rerun if the spec has changed. Because [128:02] when you change the spec, that is [128:04] generally to reflect the state that you [128:08] want in here. If I let's say um you know [128:11] um think about this if I have a [128:15] deployment [128:17] and this creates me it has um [128:22] spec [128:24] template.spec it has replicas. If I have [128:27] five replicas I'm going to get five [128:29] pods. If I change this to 10, I need to [128:34] um take an action which is incrementing [128:38] to five more pods. So I need to do [128:41] something externally. I mean not [128:45] external to Kubernetes in this case, but [128:47] you want to make some changes. In case [128:50] you want to add a new environment [128:52] variable, you need to rerun your pods [128:55] because the environment variable has [128:57] been added there or you change the value [128:59] to the environment variable. Um the pods [129:01] have to be recreated. Some some change [129:04] has to happen whether it is um [129:06] recreating the pods, whether it is [129:09] recreating the config map, whatever it [129:11] is, you need to react on that change. [129:15] But if you update the status, you do not [129:20] need to react. Many of the times you do [129:24] not need to react on uh these uh you [129:28] know these these uh updates of the [129:30] status and this is exactly what you can [129:33] um you're going to be in a problem and [129:35] the solution for that is to better [129:38] manage your status updates. So the same [129:42] flow continues. Somebody makes changes, [129:45] user or your own operator [129:49] makes changes via the API server to the [129:52] object, you get storing this into HCD. [129:58] Um, from there the reflector watches [130:00] this event, it gets the data, gives it [130:03] to the informer and you store that into [130:06] your cache. [130:08] Now because the operator was updated [130:11] your reconciler has to run. Now what [130:14] happens is you give it to the event [130:18] handler which again was responsible to [130:20] add this object into the queue. But now [130:23] you have a predicate filter. [130:26] You're making your operator a bit more [130:29] smarter. what you say, hey event, uh, [130:33] hey event handler, I want you to add [130:37] this into the working queue only if [130:40] there was a change in the spec, [130:43] not if there was a change on the status. [130:46] So I want you to ignore the noise or I [130:50] want you to simply skip running the [130:52] reconciler or rather one step further I [130:56] don't want you to be putting this object [130:59] into the working queue because once it [131:01] gets into into the working queue your [131:03] reconciler will take and run. So you're [131:06] simply telling with this predicate that [131:08] guys I only care about the changes in [131:12] the spec. [131:15] I do not care about uh the changes in [131:17] the status. That means if the generation [131:21] was changed then do your thing. If the [131:25] generation did not change do not rerun [131:28] that again. And there are there are some [131:31] other predicates as well. The one we are [131:33] talking about is the generation changed [131:36] predicate. It simply checks if the you [131:40] know um if the generation has changed. [131:44] that means the spec has changed. That [131:46] means I want to run it. But if the [131:49] generation did not change, that means [131:51] the spec did not change. Probably it was [131:53] just a status update and do nothing. And [131:57] that line, that one single configuration [132:00] is going to stop you from wasting your [132:02] precious cycles, your CPU cycles because [132:06] um you don't want to just write an [132:08] operator as a junior engineer. You want [132:10] to write an operator thinking it's not [132:13] just running, it is running efficiently. [132:16] Because if you if you really go into uh [132:19] operators, you want to keep them as [132:21] efficient as possible. Otherwise, your [132:23] cluster will be your resources are going [132:26] to be wasting or wasted on the operators [132:29] themselves rather than the data that [132:32] they are managing. So you have to be um [132:35] not careful but smart about it this [132:38] time. That's why it's called a best [132:40] practice. [132:42] So, how do you how do you work with [132:43] that? And here's where the magic [132:46] happens. When you are setting up your [132:49] controller with your manager, I think [132:51] let me just quickly talk about that. [132:53] You have a manager [132:58] and within a manager you can have [133:01] multiple APIs. [133:04] Usually you have an API and for that you [133:08] have a controller. [133:11] The same thing happens here. Um [133:14] let me copy that here and let me copy [133:18] this. [133:21] So you can make multiple APIs with cube [133:23] builder in the same project. Um, for [133:25] example, I could have my cube builder [133:28] domain [133:30] as cloud.com [133:32] and I can have APIs as uh EC2 [133:35] instance.cloud.com. [133:39] Uh, I can have one object is is EC2 [133:41] instance. I can have so this is going to [133:44] be my API number one here. So, one [133:47] controller that manages the EC2 instance [133:49] for me. Um, the other one manages [133:52] buckets. So buckets docloud.com. [133:56] The other one could be storage u or it [133:59] could be you know uh ebs.sto.cloud.com [134:03] and then you can have uh [134:04] esfs.sto.cloud.com. [134:06] You can you can make multiple. The thing [134:08] is all of these controllers have to be [134:12] registered with the manager. [134:16] This manager is responsible for running [134:18] multiple workers threads um routines for [134:22] your controllers. So this is the one [134:24] which is doing the heavy lifting. We [134:27] attach our controller um to the manager [134:32] uh using this function setup with [134:35] manager. [134:37] we define which manager we want to use. [134:40] Um and essentially we are telling that [134:43] um for this object you know for this [134:46] object I want you to attach to my [134:48] manager I want you to do this with an [134:52] event filter and here's the one that is [134:56] going to make your ch your life a lot [134:58] more easier. [135:00] So what you're telling is I want to [135:02] filter on some events previously. [135:06] Previously you were just updating um [135:10] whether it you were updating my working [135:12] queue even if it was a spec change or [135:17] either it was a status change. But this [135:19] time I don't want you to do that. I want [135:22] you to see if that was a spec change. I [135:24] want you to see if the generation has [135:26] changed. um then add this to my working [135:30] queue. If it hasn't changed, then don't [135:33] do that. Do not add it to the working [135:35] queue. Do not let my reconciler rerun [135:39] even if it's identic. [135:52] Your playbook is topnotch. uh it is [135:55] ident. [135:57] But why would you even want to run this [135:59] in the first place if there was no need [136:02] you know um if no need don't run it if [136:06] it runs it doesn't change okay that's a [136:08] good argument but the better argument is [136:10] if not needed do not run it um because [136:13] even if you run it even if no changes [136:15] were made you're wasting your compute [136:18] and that's what we want to stop [136:21] so with this event filter uh we have [136:23] couple of predicates available and this [136:25] generation change predicate It simply [136:28] says um if your uh generation did not [136:31] change uh don't do anything. So here is [136:35] what you can see. Now if I run my [136:38] controller now if I run my controller [136:41] the first time my controller runs it [136:43] will update the status because I want my [136:46] status to be updated. The next time [136:50] because there was a loop you saw before. [136:52] the next time we are you know our [136:54] controller is going to see hey you know [136:56] what the generation did not change that [136:59] change was only in the status so I'm not [137:02] going to rerun that I'll just wait so [137:05] let's do that now I hope this is going [137:08] to work just as we wanted you see we did [137:12] update our last updated at but that's it [137:18] our operator is is active uh is running [137:22] but not doing anything because of this [137:25] one predicate because my generation did [137:28] not change. [137:30] If I want to show you that again you see [137:33] uh right now our microsconds is 108. Uh [137:37] just to show you that the changes are [137:39] actually being made. This is what we [137:41] have currently uh right now. And if I [137:44] rerun my operator this will be updated. [137:47] So you can see here now um the time has [137:51] changed the status has been updated but [137:56] now see what happened [137:59] it's here. So see what happened um you [138:03] updated your object. Okay you updated [138:06] your object you updated this into the [138:09] cache. You have stored this into the [138:11] cache. Then the event handler before it [138:15] added this back to the working queue, it [138:18] saw that the generation has actually not [138:22] changed and that's why it did not add it [138:25] back to the working queue and the [138:27] reconciler did not run. Even though the [138:30] reconciler did update the status but it [138:33] did not do anything at all apart from [138:35] that. And that's what we wanted to do. [138:37] We wanted to light our operators so that [138:40] they do care about spec, but they do not [138:44] care much about the changes on the [138:47] status. This lets you have a free hand [138:50] of any sort of metadata that you want to [138:53] put into your status. You want it to be [138:55] more descriptive. You want to have more [138:57] fields in the status to make it as more [138:59] descriptive as possible. But you were [139:02] worrying or you were managing how do I [139:04] not rerun my reconciler if I'm doing a [139:07] status change. I have to be very [139:09] careful. But this one single predicate [139:12] does it for you. It says if no change in [139:16] the generation I'm not going to make I'm [139:18] not going to rerun that again. It did [139:21] run the first time. You see we did read [139:24] the value of it. We did read the value [139:27] of it. Um and here is my is my update. [139:33] So this is what I wanted to show you uh [139:35] how the generation works, how the how [139:37] the resource version works and [139:39] especially the generation was changed [139:42] when you update the spec in Kubernetes. [139:45] Uh the the generation does not change [139:47] when you change the status of your [139:50] operator uh in Kubernetes. So I hope [139:52] this has been really helpful and this is [139:55] going to help you create operators [139:58] better not create operators because [140:00] that's the course that helps you to do [140:02] this but this is going to help you write [140:04] operators thinking from a point of view [140:06] of a platform engineer who's not just a [140:09] junior but uh a staff engineer or a [140:12] principal engineer. Um these things are [140:15] really mattering when you write [140:17] operators and this is how you should [140:19] write this in a better way. [140:21] So, I hope this has been useful.