Code a Reinforcement Learning Library in C from Scratch (Full Course)

[00:00] Welcome to this deep dive into building [00:01] a custom reinforcement learning library [00:04] entirely from scratch in C. Throughout [00:07] this tutorial, you will construct an [00:09] autograd engine and implement the policy [00:12] gradient reinforce algorithm without [00:15] relying on any external machine learning [00:17] frameworks to put our custom library to [00:20] the test. We will also code a complete [00:23] snake game environment to train our AI [00:26] agent and watch it learn how to navigate [00:29] the grid. [00:30] >> Hi everyone, I hope you're doing good. [00:32] In this video, we will going to [00:33] implement reinforcement learning library [00:35] from scratch in C. So I hope you have [00:38] some understanding of the reinforcement [00:39] learning how the agent loop works and [00:42] how the agent takes action in the [00:44] environment and environment gives the [00:46] report to the agent and that how the [00:49] agent learns and get feedback uh and [00:52] also have some understanding of the [00:54] machine learning and stuff. So yeah I [00:57] think u if you have these all the [00:59] prerequisics then let's get started. I [01:01] think this is a very good video because [01:03] here we can actually go very deep and [01:05] implement things from very scratch in C [01:09] and uh we will implement the policy [01:11] gradient reinforcement reinforce [01:13] algorithm and we will implement the [01:17] model from scratch. We will implement [01:20] the environment RL environment from [01:23] scratch and also we will not implement [01:27] these things but also implement the [01:29] autograd engine which is like a [01:30] computational graph of the gradients [01:33] where that's how neural network trains. [01:37] So yeah, let's get started. U there's [01:40] some of the things that uh I already [01:42] defined which like base which is like [01:44] some [01:46] types here which I have already defined [01:49] and this files contains some random [01:51] numbers which will help us to use it in [01:56] further video. Uh so here are some [02:01] things I want to mention first like this [02:03] is the video you can probably see in the [02:05] screen this video is really good video [02:08] uh and I think this person actually [02:12] taught really good about like how to [02:14] build the machine learning library from [02:17] scratch. So this is kind of an [02:19] inspiration that I took and okay let's [02:20] go and implement the RL library from [02:23] scratch. So yeah, if you want to check [02:25] out this video, please check out that [02:27] too. And uh pretty good video. So let's [02:31] get started here. Uh I think first we [02:33] can uh let's define the autograd because [02:36] that is one of the fundamentals right [02:38] now and then we can uh go for like [02:41] defining model and then our our [02:43] environment. So let's get started. [02:45] Autograd C. Um okay. And uh here we have [02:50] this u what we need. Um so as you [02:54] probably [02:56] have the understanding of like how the [02:58] model works you have like forward pass [03:01] and then backward pass. So which [03:03] requires a lot of which requires a graph [03:06] uh to go from one direction and then [03:08] also in the reverse direction and that [03:11] graph is what we're going to be define [03:13] here like this the autograph which [03:15] computes the auto differentiation stuff [03:19] uh in the backward pass. So here we will [03:22] define the graph of the forward pass and [03:24] then in the backward pass when we are [03:26] going that backward pass basically [03:29] creates the traversal graph and identify [03:32] variables uh within that and then [03:36] computes the gradients and that's why we [03:38] call autograd which is like auto [03:40] differentiation stuff. So yeah uh let's [03:45] go and define the variable first like [03:49] what is variable you know variable is [03:52] basically the uh which is struck it [03:55] could be and uh it will contains index [04:00] u32 index [04:02] and then will have the number of inputs [04:08] you know and that number of inputs is [04:11] will be like a number of it's also a var [04:14] so inputs because one variable can [04:17] contains the inputs of it. So let's [04:19] suppose I have a + b equals to c that c [04:24] is also where a is also where b is also [04:26] where v variable is basically the [04:30] parameter you can say in model. So this [04:33] is what it is. It will have flags [04:36] you know [04:38] uh U32 flags [04:40] and uh this v inputs basically will have [04:45] number of inputs and then it will also [04:48] contain so basically this v inputs is [04:51] the number of inputs uh number of inputs [04:55] it will contain a basically the inputs [04:58] it can have like one input or two input [05:02] so because like for Some of the [05:03] operations like relu soft max requires [05:06] just one input but for the operations [05:10] like addition subtraction that requires [05:12] two inputs which like could be a and b [05:15] and any kind of variables. So yeah I [05:19] think uh what else we need like we need [05:22] value which is like a what which will be [05:26] a matrix. So here we will create a [05:29] matrix. We haven't defined the matrix [05:31] yet. So but we'll we will define later [05:35] and uh a gradient. So gradient is also [05:39] matrix [05:41] right. So these are the things. Uh what [05:43] else we doing in a variable can you [05:45] think of [05:46] um bar can have like columns and stuff? [05:50] I guess not really but a metric can [05:54] have. So we can define in the metrics. [05:56] So yeah [05:59] um you can define create metrics which [06:03] will have U32 [06:05] rows 32 columns [06:08] and um to and then we it will have like [06:12] a data right [06:15] and that data is could be 32 okay let's [06:20] define first this thing have this and [06:24] then a data I [06:26] So we can have like a U32 data or I can [06:30] say like F32. Yeah, we can now this is [06:34] like uh the actual definition like the [06:37] the initialization of the stuff we can [06:39] define the definition here also so it [06:42] won't get confused that defined earlier [06:45] or later. [06:46] So type def I hope it is recording it is [06:50] recording I guess. Yes. So, so right we [06:55] can check it if this is working or not. [07:00] Go to terminal C run. [07:04] So this is very confusing you know when [07:07] we are using Python and C at the same [07:11] time. Q32 will come from nice. Uh then [07:16] we have like strruct bar type. So this [07:20] is the bar we have defined. But like you [07:24] know we also need what is the type of [07:26] this V because it can have like addition [07:29] bar it can have soft max bar. It can [07:32] have like different operational types of [07:34] R. So with what type of V is a bar type [07:38] and this can have like this var type can [07:41] have um the types which is basically [07:45] shape. You can have bar function shape. [07:49] This shape you know this this function [07:52] basically will help in shaping this v [07:56] type [07:58] and this v type have can have like v [08:02] funk back forward function [08:07] and backward function. So then it will [08:11] have a good definition of our type like [08:13] what kind of function it has, what kind [08:15] of backup function it has [08:18] and obviously the name. So it will have [08:23] a name, right? [08:26] And that's it. That's uh that is the bar [08:28] type [08:30] and also the type. So now we also need [08:34] to define the function here which is [08:37] like this function and stuff right. So [08:41] after our definition of v we can define [08:45] or like there's not nothing needed for [08:48] that because like say here v32 [08:53] and uh function which will have like the [08:56] v32 columns 32 rows. This will have just [09:02] one. Okay. Now this is we have defined [09:07] it will have forward backward shape type [09:11] and stuff four errors which is like type [09:15] this should be I guess uh should be this [09:19] should be a pointer. [09:21] Got it. Uh the type will already be [09:25] defined by the name here. So I guess [09:28] that's it. And uh that also requires [09:31] like how many inputs it has this um this [09:36] part type will have. So we can also [09:38] define like the 32 number inputs [09:42] right it is working. Um we can now we [09:46] need to define some of the definitions [09:48] here which is the v it will have like v [09:53] soft max you know that will have me [09:57] arena okay I haven't defined the mem [10:00] arena right so I'll just what I'll just [10:04] do is like I will also include the arena [10:07] thing which is arena is basically which [10:11] defines the memory location allocations [10:14] and stuff and for this kind of projects. [10:18] So I will just going to go and I have [10:21] defined the arena here. This is the [10:22] arena. [10:24] So it will help us to actually create [10:27] the arena and stuff. And uh arena.h will [10:30] have the all the definitions. [10:34] So yeah um in autograph we can just like [10:37] import it like this. Now this [10:41] softmax will have me arena now which ask [10:46] for space and stuff model state model [10:50] and then war [10:53] input actually just input I will define [10:57] the model state so we will get the arena [11:00] from the arena itself and uh I'll define [11:03] the model state [11:05] but we can define it first it's not a [11:08] great big you. So strct model and uh in [11:14] this model we have you know model state [11:19] is that model state uh in this we'll [11:22] have like you know a lot of things uh [11:26] which help us to build the model the one [11:30] of the things that I actually getting in [11:33] my mind which is advantages. [11:35] So advantage to Y in reinforcement [11:38] learning. So we will store the advantage [11:40] here and this is basically going to be [11:44] float 32 right and uh this is the [11:50] advantage and it will this contains a [11:53] lot of things like number of bars [11:58] and number of bars will be again this [12:01] U32 [12:02] and it can have it can also have a lot [12:06] of different things I guess model state [12:09] not model it can have you I guess that's [12:13] it we will innovate it later I think we [12:17] can just find what are exactly required [12:19] for us so this is the model state uh we [12:23] can define it here again type dev straw [12:27] basically we can just define it here [12:30] like that bar soft max bar arena arena [12:37] in a state model where input and we can [12:42] copy it right over the where add now [12:48] this v a b this again subtract I guess [12:54] we also need type of it which is like [12:57] the v type I'm guessing v type type [13:02] subtraction is done now maximum [13:06] is the matrix multiplication you know [13:09] and model state reinforce and R we'll [13:12] get like a pro probabilities and art [13:15] stuff. So let's run this after this [13:20] draft. Okay nice now you know we also [13:24] need like backward pass and forward pass [13:28] thing. So void will basically you know [13:31] which is where soft max forward which [13:36] basically going to forward the soft max [13:39] and that require these things like me [13:42] arena and all stuffs inputs and this [13:47] will compute the I guess this will [13:49] compute the softmax and stuff yeah so [13:53] you don't need bar type you don't you [13:57] need this [14:00] and uh yeah why we are just doing [14:04] this is again going to be you know same [14:07] same going for the backward so [14:10] soft max [14:12] backward although like here we just need [14:17] don't need like because we we're not [14:19] computing anything we're just computing [14:20] gradients and we already have like the [14:24] uh calculation stuff so we are going [14:27] backward word and then [14:30] using this var we can just get the [14:34] output of it and we can compute the [14:37] gradients. Got it? V [14:41] backward. This is our backward pass [14:45] functions and stuff now. Okay. Now we [14:48] need to build function which can create [14:51] variables. you know we have to find this [14:53] is some abstraction for us this [14:56] variable. [14:57] Now we need to define variables um like [15:01] create variables. So a variable will a [15:04] function that returns variable create [15:07] bar arena arena model state and uh view [15:13] 32 rows 32 columns right and flags. Now [15:19] in arena you will find like this push [15:23] truck function which is here push truck [15:27] function which basically [15:29] can be used to allocate the memory for [15:32] the specific var first right so we will [15:38] use this this autograd you know I hope [15:42] I'm recording yeah all good so this auto [15:47] grad let's create where basic Basically [15:49] first it will do is like var out equals [15:53] to this is how you define it right like [15:56] this and uh and this out index plus [16:01] equals to 1 + = to 1 ++ I can say out [16:06] and then it will have now we need to [16:09] define create a matrix [16:12] so matrix m equals to create matrix a [16:17] function that can Create matrix. Now [16:21] before creating that we should have [16:22] created metrics. Create matrix that [16:26] contains like you know arena arena rows [16:32] columns and the matrix is this right [16:37] rows column data I guess like this is uh [16:40] let me confirm it will be a pointer to [16:43] be honest. [16:44] Um yeah it will be a pointer. So yeah, I [16:49] think this is uh this how you define [16:52] create metrics. Create matrix and uh you [16:56] basically going to push struck the [17:00] metrics. This push truck will contain [17:03] arena and that say out [17:08] columns return out data will be okay. So [17:14] yeah data we probably we can like define [17:17] data as as push truck this data is push [17:23] array because we already have this [17:26] function push array. [17:29] So we can use this arena tn. So arena [17:34] t is basically the tricks size of tricks [17:40] and then rows and columns rows. We can [17:44] just like push array paste like this. [17:50] Yeah, exactly. That's how we use this. [17:53] We can directly use this. Uh it's [17:55] getting confusing. [17:57] So this working we can try finding [18:01] create metrics. [18:04] We don't have arena. So create arena [18:08] create arena. Arena will take capacity. [18:13] So let's define 1,00 [18:16] and uh arena 25 25. [18:21] We can also print out the matrix which [18:24] is the matrix A as it has 0 0 0. So this [18:29] printed the first. Okay. And we just [18:33] print the rows you verify. That's great. [18:38] So this is uh we are this is here we are [18:41] right now. Uh instead of like defining [18:44] this we can also say like push strruct [18:47] and then uh it contains [18:51] and then [18:53] create a metric arena rows columns here [18:58] and uh we haven't defined the flags [19:00] right we haven't defined the flags so [19:03] flags are something you know we can [19:06] define here as enum flags which has var [19:13] flag none which is zero [19:17] where requires [19:19] have this value binary flag. So these [19:24] are the flags you know uh like what [19:29] exactly this variable have is it a [19:33] variable that contains gradient or is it [19:37] this variable is a parameter like it's [19:40] none or something like that just useful [19:43] computations [19:44] so that's what it is require right and [19:50] um yeah I hope it is recording thing my [19:53] voice is clear and uh yeah so this is [19:57] what so we are here right now define the [20:00] matrix and then matt will have you know [20:03] this will have a name value and then out [20:08] val equals to val out grad equals to if [20:13] it has a you know if I would just say [20:17] type defum [20:19] max it's just defining as 232 let's say [20:23] if flags if flags uh flags uh if it is a [20:28] bad lag cat then probably we also need [20:31] to add gradient into it create metrics [20:37] columns that's it I think inside this [20:41] there's nothing else uh there's inputs [20:45] so for that like I guess we also need to [20:47] define the inputs um zero [20:52] equals to a = to b. And it will only [20:56] happen that if this if dot number of [21:01] inputs contain greater than one then you [21:05] have to basically store it there. And [21:08] this number of inputs comes you know [21:11] comes from type number of inputs [21:13] contains them. Add this you know we also [21:16] need to add type here which is like [21:20] which we will get type [21:24] and uh we will also I guess like model [21:28] state is not required here. We'll just [21:31] remove this and we can also remove uh I [21:35] guess like when we are creating V like [21:38] first time we will create so we don't we [21:40] will not have like inputs and we cannot [21:42] define this. So I guess uh earlier [21:45] version was better which is like model [21:48] state and uh this thing and uh type we [21:52] can say null just say this. Yeah I uh [21:55] that's it. Now we can just return the [21:58] value we will define like the inputs and [22:01] stuff when we we will create a node you [22:04] know so out we we have defined and uh [22:09] type because we haven't [22:12] defined in here would be like this is [22:16] working I guess and uh then we have to [22:19] define node this creates the bar and we [22:23] can actually go and then try to create [22:26] it. Uh we can just go and where [22:31] B is a v [22:33] arena then model state is state. I don't [22:37] know like why did we needed a model [22:40] state here you know state model flags [22:45] will be like zero I would say. So yeah [22:49] it's working. Uh does it have a grab? it [22:54] won't have a grant because like we have [22:55] to write this well uh you know [23:00] this model state that I just remembered [23:02] like it will have a number of bars so [23:05] when we create this thing that's why we [23:08] need model state because [23:11] index we can say it number of this will [23:16] have a index right this will have a [23:19] index so number of V model [23:26] number of bars ++ means like as we [23:29] create this var that will also reflect [23:33] in model which is like a you know model [23:36] will have it will actually automatically [23:39] goes inside the model you know and then [23:42] it increments the number of bars [23:45] although like this limits the [23:47] possibilities like if you want to create [23:49] number of bars outside the model And but [23:52] yeah, I think for now it's good. So V [23:56] create V. This is done. This is working. [23:59] And then when we have to create a node, [24:01] you know, uh what does node will do? B [24:05] uh node basically the graph nodes of [24:07] graph you know because we have graph [24:09] from bottom to like operations like soft [24:14] max. This is it's itself is a node. This [24:18] itself is a node. This is a node. This [24:20] is a node. So as the computational graph [24:22] grows it will create a node based on [24:24] that. So that's why like we need to [24:27] create a node and for that like we have [24:30] to define [24:31] graph uh which is like again a strct [24:36] strct graph and this graph will contain [24:40] you know u [24:45] this graph contain the variables. So [24:49] because graph is contains like a notes [24:52] we can say like a notes but like instead [24:55] we say like v and that where is a [24:59] pointer the area of pointers. So you can [25:01] say like also this double asterric is [25:05] the [25:06] uh the pointer to the location of [25:09] pointers you know. So this will have [25:13] bars [25:14] and uh will have a size. What is the [25:18] size of the graph? That's it. Uh this is [25:20] the graph and uh matrix in the graph. [25:25] What exactly is happening here? Why does [25:29] I did something? Okay, I just did [25:32] something here crazy which is okay [25:35] because the model wise [25:38] is is is [snorts] is null [25:44] it's just definition is wrong but yeah I [25:48] think number of hours will be going to a [25:52] plus+ in this Um, [26:12] I'll take a little break, I guess. You [26:14] know, I should take a little break or [26:16] like I should go create something [26:19] really. Okay. So, so graph has no [26:22] problem with this and okay [26:26] now this is done this the graph is like [26:30] we have defined a graph now node that [26:33] node basically will give us the v itself [26:38] you know create node [26:42] and that contains v a and v b you know [26:47] and uh because a node itself is a soft [26:50] max let's suppose so it can contain bar [26:52] a and bar b and uh can have like a model [26:56] state can have flags no 32 max that's it [27:02] um so if inputs and it will have like a [27:08] type v type type and this type will be [27:13] basically a point so this uh will create [27:17] a node sorry create a [27:19] this bar basically arena and stuff will [27:22] have like the VA A and V B model state [27:28] for and then okay got it. Um [27:34] now this var type you know if type is [27:38] type equal equals to bar will have like [27:43] type equals type if now this type [27:47] contains you know this type contains [27:49] shape. So and that shape is basically [27:53] can have a b can have a and b. We have [27:58] seen it like that function contains a [28:01] and b a b columns and rows a b f this is [28:06] basically returns true because it's a um [28:11] this shape this [28:13] v fun type is a boolean itself. So if [28:19] this thing contains if it is true then [28:23] that means that the shape is right of a [28:26] and b compatible enough to the operation [28:28] whatever the operation uh based on the [28:31] type and uh if it is true then I guess [28:35] like we can just move forward with that. [28:38] Um so I have just do it like bar type [28:43] shape and before that I think I have to [28:46] define row equals to zero [28:50] column= to zero and then we have to [28:53] provide the reference of it and that [28:56] will have the things but what we will do [29:00] inside of this is like create a node and [29:03] then model state I guess like that's why [29:06] I like models [29:08] was required. This is the right way is [29:11] creating a new variable based this is [29:14] based on this the the the goal of this [29:17] create node is that you know to create a [29:21] new variable and uh but that variable is [29:26] already coming from something so [29:29] variable A and variable B you know so [29:32] that's why like it's not like [29:33] independent variable like this so but I [29:37] guess like we don't have to do that [29:39] Because this is already doing inside [29:42] this create v that requires model that's [29:46] why like model [29:48] rows column and flags we are making [29:51] complicated a lot with this because uh [29:55] you know so a lot of things we are [29:58] actually taking here which is not [30:00] required. So as if the goal of this node [30:04] is to create a new variable based on the [30:07] past A and B. So we don't need like a [30:10] lot of information. We already have some [30:12] information from A and B. So I guess [30:14] like we just remove this thing [30:17] arena we have model state v and this [30:21] this type only. Now this flag we can [30:25] just define you know we do flags as a [30:29] bar flag none first [30:32] right and then if flag if a isn't if a [30:38] isn't equals to null [30:40] and and a flags and required grabbed or [30:46] r or b isn't equals to null [30:50] and and E flag [30:54] and where flag requires graph. So if [30:58] this condition is true like means like A [31:02] has this either A has like the gradient [31:06] on or B has this gradient on then we can [31:10] say like black is requires that you know [31:14] I it requires only one of it because we [31:17] are using R here. [31:19] So this is how we can get like flag and [31:23] uh for the rows and columns [31:25] here we have defined it here rows and [31:27] columns and then inside this if this is [31:30] not true you know if if this type dot [31:34] shape is not true [31:36] then we can just return null and uh [31:40] after that we can like create this bar [31:43] because we now we have rows and columns. [31:47] So this shape will you know I haven't [31:49] defined the shape function but this [31:51] shape will do some kind of like [31:52] manipulation with this rows and column [31:54] that's why like we are passing the [31:55] reference so it will change the whole [31:57] this rows and columns and then we will [31:59] provide here so we don't need to pass [32:03] from the arguments we already getting it [32:07] by when you like because a and b you [32:11] know is a this new variable is coming [32:16] out from A and B. So either it is [32:19] exactly their shape or like some kind of [32:23] uh matrix multiplication they are doing. [32:25] So we can already compute like the out [32:27] shape and compute the rows and columns [32:29] of the out thing out shape. So this and [32:34] then flags right arena model rows [32:38] columns and flag and then where type is [32:41] type [32:42] we can define it like this. We already [32:45] had defined it doesn't make okay now if [32:49] so we can also define this because it is [32:51] coming out of A and B so A and B's are [32:54] basically [32:56] and we are the inputs of this U so if [32:59] var bar v bar v bar v bar v bar v bar v [33:00] bar v bar v bar v bar v inputs zero is [33:03] basically a and if var dot type number [33:09] of inputs greater than one are inputs [33:14] to [33:14] That's it. Uh this is this will create [33:17] the node. We will return the awesome. [33:20] This is this is great. Four errors which [33:24] is like a flag. It should be flags. [33:26] Okay. It should be flags in. Yeah. [33:30] Should be flags columns and rows. It [33:34] should [33:35] rows and columns. I guess this wasn't [33:38] the error. Incompatible [33:42] pointer to integer conversion is give 32 [33:44] star. What does it mean here which is it [33:49] should have is working. I guess I should [33:52] take a little break and then we can just [33:55] come back and start it. Okay. So this is [33:58] great. Uh next we have to implement uh [34:01] the graph. [34:03] So the basically the how we going to [34:07] build the graph is we can build the [34:09] graph using the output. So let's suppose [34:15] the first is input and the last is [34:18] output. So output could be like after [34:20] the soft max and in between there's a [34:23] matrix multiplication [34:25] value and all these layers but soft max [34:28] is basically going to be the last layers [34:32] and that soft max we can put at outwire [34:37] and then outwire that will back [34:39] propagate [34:41] and discover the nodes discover the [34:45] nodes and then eventually going to [34:47] figure it out. So I can just show you [34:49] like I basically like this is slides I [34:53] made. Um yeah like this is the graph [34:56] like this will have inputs [34:59] matt I I think it's visible right this [35:02] input matt activation stop. So this is [35:06] like the forward pass but you know [35:09] during the gradient computation stuff it [35:12] will going to be a little different [35:14] because uh we need to provide we need to [35:17] go backward. So we will provide the [35:20] output wire and then it will discover [35:22] the inputs because we are storing the [35:23] inputs also. So it will go to the all [35:26] inputs and then discover everything [35:29] and then finds out like okay which uh [35:32] finds out the dependency of the inputs [35:35] and then compute the gradients. Um and [35:38] uh yeah I think that's we have to do now [35:43] for that I guess uh there's this thing [35:46] called create graph return graph and uh [35:51] it will have outline. So there's this [35:54] algorithm called uh depth for depthwise [35:57] traversal algorithm which will take uh [36:01] the one node and then finds out the [36:05] inputs node and then traverse back you [36:09] know so it it will have like a stack so [36:12] it will have a stack and in that stack [36:14] we can put a lot of different things. So [36:18] it's like this uh you know stack will [36:21] have some you know we will put lot of [36:24] things uh more like a b c d e all these [36:31] computations and but it is going in the [36:34] forward direction sorry it's going in [36:37] the from backward direction so because [36:39] we want to create this graph we will [36:42] probably what we will do we will go out [36:45] and [36:47] Um using output graph we will okay let [36:51] me explain again like so in stack we [36:56] have this a lot of computation a b c and [37:00] d and all the stuff right so these [37:02] computation basically will have uh from [37:05] backward so we are going from outward [37:08] which is like out bar to the inputs [37:14] so this will go into the from out bar [37:17] which like the out bar wire will be here [37:19] and then it will discover the inputs [37:21] which dependent on the out. So that why [37:24] like it's it is not in the right order. [37:27] So we want in the right order for the [37:28] graph to execute the forward pass. Uh [37:31] and so there will be two kind of of [37:35] these things and uh this will be one is [37:39] stack and other will be out which will [37:43] have all the in the right order whatever [37:45] the right order will be. So the the [37:47] first it will the f it will contain [37:51] leaves [37:52] first and then contains here all the [37:55] output and stuff. uh I can explain you [37:57] better if I have like I have chat [38:02] jupitia asked about like to create the [38:05] computational graph so you can look at [38:07] those graph let me just pull this thing [38:11] so yeah so let's suppose like you know [38:14] you have this graph zed zed is the [38:17] output and then you will traverse back [38:20] to discover some of the inputs so zed is [38:22] dependent on x and y okay and then x is [38:25] dependent on a and b and y is dependent [38:27] on b and c. So you will see like you [38:30] know um you probably find that okay when [38:33] you go in the in this direction you will [38:35] see you will check the inputs y okay and [38:39] then uh you will add in the stack you [38:43] will add zed in the stack y in the stack [38:46] and then add c and b in the stack and [38:48] then check another input x then b and c. [38:52] So you will actually create stack like [38:54] this um from backward. [38:57] But the problem here is like there's a [38:59] lot of problems with this uh which is [39:01] like here you can see like this y is [39:04] also dependent on uh b and x is also [39:07] dependent on b. So if you just like go [39:08] in one order and then uh discover the [39:11] inputs then you will basically going to [39:15] have basically add this b twice. So you [39:20] will have two bs. So you have to make [39:22] sure that okay you [39:25] are not adding the the same parameter [39:27] same variable and you are also adding [39:31] these variable in the right order. So [39:34] then we can actually create the graph [39:36] create a graph. [39:38] So yeah, I think this is what we want to [39:41] do uh here right now to create this [39:45] graph like first we need to define some [39:48] of the things which is like graph and [39:52] this graph is empty here [39:56] and uh some of the variables like stack [40:00] that stack is [40:02] honestly the pointers of point location [40:06] it is a pointer to locate uh points the [40:09] location of pointers the stack and this [40:14] maloc size of bar model [40:19] number of bars and same as out and then [40:25] visited because we have to check like we [40:27] are not adding adding the same variable [40:30] twice. So visited will be [40:34] uh P32 boolean visited [40:38] um [40:53] well this will have this will also be a [40:56] pointer and like this and then basically [41:00] You know stack will have this uh initial [41:04] stack will contain stack size. [41:08] Stack size equals zero [41:11] and out size equals to zero. Stack stack [41:17] size [41:18] ++ equals to stack size stack size ++ [41:23] equals to outl. Okay. So this will [41:27] contains the outbire and then we will [41:29] traverse back you know wire. So I hope [41:33] you understood this like which is this [41:36] is a post increment thing. So it is [41:39] exactly like this stack height. So [41:42] initially it will get the value of like [41:45] zero with index stack and zero with [41:47] index we are storing the out bar and [41:48] then stack size increasing after that. [41:51] So stack size will become one. While [41:54] stack size is greater than zero, what we [41:58] have to do is like we have to [42:02] go and then pop out the stack. [42:06] So by current [42:10] and stack stack size, this will pop the [42:15] index. So this is a pre-increment. So it [42:19] will turn to zero and then it will [42:21] index. [42:23] So this current is stack stack minus [42:27] minus stack size. Then what will happen [42:31] that we have popped out the stack size [42:34] is zero. It has zero length. [42:38] But um we can actually see if this [42:42] current dot uh current if this index if [42:47] this um visited [42:50] current [42:52] dot index if it is visited then just add [42:57] in the output you know add the out add [42:59] in the output out [43:02] don't go further just continue because [43:04] like we already so as we have seen mean [43:08] like okay if it already visited this [43:11] kind of a thing then just add into the [43:16] outsides and continue [43:19] u otherwise like just go and do more [43:23] computation which is more computation [43:24] which will be like just discover more [43:26] inputs. So let's suppose what we are [43:29] trying to say is like you know if you [43:30] have not visited this thing you discover [43:34] more inputs and if you already visit [43:36] visited this variable okay don't [43:39] discover this okay we will add add this [43:42] thing into the uh output I hope you're [43:46] getting I I know this is not kind of [43:48] makes sense but like when we actually [43:50] run this create graph you can actually [43:53] understand we will see like what exactly [43:54] it's doing through the output [43:57] So if visited current index out out [44:01] size++ current continue [44:04] and uh and then then then then [44:07] we have this for use 32 I = to0 less [44:14] than input size. So we have to like find [44:17] out like the inputs which is the type is [44:21] not null type is not null dot type will [44:25] have number of inputs otherwise zero. So [44:31] if it is inputs I less than inputs I ++ [44:36] input is okay. So if this input if this [44:41] input is um if this input is less than [44:45] the input is we have to like iterate uh [44:49] for U32 a =0 J less than ST it will make [44:56] sense like I'll explain what I'm trying [44:57] to do here I J equal if it is like [45:01] equals to that means like it is like a [45:03] duplicated so because like as we have [45:05] seen like X also had this B variable [45:08] able Y also had this uh V B variable [45:12] dependent on. So both shared the same [45:16] variable. So we need to make sure that [45:18] okay one is only added. So first when we [45:23] discovered X we will add the B. Now when [45:27] we discovered Y's [45:30] variable the variable of Y we can [45:33] actually uh update B you know we can [45:36] replace the stack. So let's suppose like [45:38] this [45:40] it will add X Y by B let's suppose like [45:44] this B and A right so again when it [45:48] finds that you know Z now in next [45:52] iteration like it if it finds like Y is [45:57] also dependent on B you know B so this [46:01] condition matches okay this is actually [46:04] already in the stack [46:07] Right. This is already in the stack. So [46:10] it will going what we need to do is like [46:13] remove the old position thing. Just have [46:16] this in the here. So it will re [46:20] basically remove the old thing and then [46:23] have this in the right order. [46:26] So only one B is going to be there. [46:28] That's what we will going to do. Um it's [46:31] already there. How we can remove this is [46:35] basically we can shift the order. So [46:39] let's suppose when we discovered this [46:41] you know this is here. So we can just [46:44] remove this and uh shift the order. So [46:49] how we can like do this is basically we [46:52] will just go iterate k equals to j. So [46:56] we have discovered this at the j [46:58] position. So we'll start with J and J [47:02] will go to the stack size minus one [47:07] and uh K++ and this will what will what [47:13] will happen that stack of K equals to [47:18] stack K + one right so the K + one [47:23] position will become the K position so [47:28] Yeah. And then here we can just do the [47:33] stack size minus minus. So one position [47:37] like because one position is already [47:38] shifted to the left. So reducing the [47:42] stack size also and that's it. Um what [47:46] else is needed like we haven't stored [47:48] the inputs. So this was the input right? [47:53] and uh stack stack size ++ nice that's [47:59] it like this is the graph um so it will [48:03] run the this traversal until the stack [48:08] size is not less than zero and uh pop up [48:14] the st pop the [48:17] last [48:19] um last thing in the stack [48:22] and um [48:25] finds out if it's already visited. If [48:27] it's not then uh continue here. If it's [48:31] v visited already just add into the out [48:34] um stack and this input is basically [48:38] will discover the number of inputs and [48:41] then I trade over the inputs and then I [48:44] after I trading the inputs let's check [48:46] like whether we have already uh added [48:49] this in the graph in the stack if it's [48:53] not all if it's already added then [48:55] reorder the position and then remove the [48:57] last one. So kind of a graph it will [49:01] have you know and we can see like uh we [49:04] can print it that will give a better [49:07] idea. So first let's let me define the [49:10] model state you know I don't know like [49:13] why it is not getting initialized in the [49:16] right way I guess. Uh okay this is done [49:20] to be honest like I wanted to check like [49:23] this thing but we haven't like defined [49:25] the model yet like so we need to first [49:28] define the model and then we can use [49:31] that model to define a model state let's [49:33] define the model uh let's see I'm going [49:37] to basically copy some of the things [49:41] here just the imports because imports is [49:44] like difficult to manage stuff um in C [49:50] and uh yeah to include base arena matt [49:54] we don't have anything in that I guess [49:57] you can define a m dot c m okay okay [50:01] nice uh this should be let's define [50:04] somewhere else also okay so I'll be [50:07] doing because of this it should point it [50:11] the input is coming from the current so [50:14] not from here you So here we can define [50:18] like the actor model just [50:22] this will give us model actual model [50:26] because we are going to use re uh [50:28] reinforce [50:30] only not spo and stuff right now which I [50:34] thought to actually also implement this [50:35] but probably this were going to a lot [50:38] long so yeah let's go with like actor [50:42] actor model only it's like simple not do [50:46] much in this. We don't have a value [50:48] function in here which is great. Um but [50:52] it will be very um high variance. So [50:56] make sure like when we were going to [50:57] train we'll see like a very high [50:59] variance and it will train in a very [51:01] long time. So yeah I think create actor [51:05] model model state model it will have you [51:09] know the weights and stuff. So where [51:14] input arena I guess model uh the input [51:19] will be so we are going to create some [51:24] uh kind of we can create a graph later [51:27] on. We can create a graph later on. Uh [51:30] so when we have like a model state [51:32] defined what we can do right now like [51:35] and we can uh finish this function which [51:40] is these things. So you have actually [51:44] just noticed that it requires a var type [51:47] you know. So for that V type I'm going [51:50] to define the V type here which is our [51:54] soft max which will have number of [51:57] inputs equals to 1 dot name equals to [52:03] soft max and uh I hope like it's the [52:06] right syntax number of inputs [52:10] shape will be okay we haven't defined [52:13] the shape you know so there will be two [52:17] functions actually which is v32 which is [52:21] like boolean and bar shape this will [52:25] contain [52:27] the bar a b and column we will check if [52:31] in this implementation I'm not checking [52:33] too much thing making it simpler so [52:36] we'll only check like if a isn't equals [52:39] to a dot rows isn't equals to b rows or [52:45] our a columns isn't equals to B columns [52:50] return false because this is a same [52:53] shape thing. The function the goal we [52:55] are trying to achieve with this function [52:57] is that it should have a shame same [53:00] shape and uh this same shape will going [53:05] to have also update the rows and [53:07] columns. Because these rows and columns [53:09] here you can see like when we are [53:12] creating the node we are calling this [53:14] shape thing and we are passing the [53:16] reference. So this will change the rows [53:18] and columns here which which is defined [53:21] as 0 0. So it will change to the right [53:23] column and stuff. So this we will [53:27] dreference this and we will update this [53:30] as a row and similar to like where where [53:35] a b [53:37] now in u the matrix multiplication it's [53:40] kind of a different so if a rows isn't [53:44] equals to a columns isn't equals to b [53:48] columns. Oh my alarm is when did I have [53:51] alarm? I did have alarm like I I didn't [53:54] know like I had alarm. I always [53:58] I guess always had skipped. This is the [54:00] first time I'm discovering. Anyways, [54:03] it's 5:30 a.m. right now happening [54:05] though. So if a columns isn't equals to [54:09] uh b columns and uh so it should have [54:13] like this kind of because in matrix [54:16] multiplication you basically compare [54:17] with like the first dimension of the [54:19] matrix last dimension of the first [54:21] matrix and the second dimension of the [54:22] first matrix. If it compatible with this [54:24] then it can only do the matrix [54:26] multiplication otherwise it can know. So [54:29] return false and uh dreference this rows [54:34] and change the parameter will be a rows [54:40] and columns will be b column. This is it [54:46] now. Yeah we can pass these things into [54:49] here. Don't I I hope like I'm defining [54:52] kind of the same. I have to check and [54:55] then forward function which is going to [54:57] be you know soft max forward function [55:01] and uh dot backward function will be [55:04] soft max backward function. Okay. And uh [55:07] we can define the same thing with other [55:10] also v= [55:13] to dot name [55:16] although let's check like if it's [55:18] compiles. Wow. Because because because [55:21] because it is giving errors because the [55:25] thing is that it is a var you know so it [55:28] will have [55:30] valed [55:33] here so we can also this should be void [55:38] wait this should not be v okay so I [55:42] guess like the the error is basically [55:45] because of the signature is not matching [55:48] with this error is because the bar [55:51] softwax bar. Okay, so I found it like [55:55] what exactly the error is. Um you know [55:59] so this is just a definition [56:02] right now. Define it not coming right [56:06] now. This is good. Um now what we can do [56:11] um we can here we can here define these [56:14] functions you know um we can put [56:17] information inside this function where [56:20] softmax for [56:22] this will basically do the softmax [56:25] operation. [56:27] So you know soft max operation we can [56:30] just define it here [56:32] which is what is soft max operation if [56:34] if you just think of it for u32 inputs [56:41] uh right we'll have inputs [56:44] and uh so we can actually define size so [56:49] u 64 size equals to you know and iterate [56:53] and compute the softmax so I've already [56:56] like did some define the actual thing [56:59] like this is soft max and this is the [57:03] formula really this is this and policy [57:06] gradient like reinforce this is it so [57:08] you probably have familiar with this I [57:10] guess um and then what we're going to [57:12] just do is like we will iterate over [57:14] this Z to collect the Z I and then [57:18] compute each of the uh with each of this [57:21] element [57:23] and similar to all those goes you [57:26] This size will come from bar columns and [57:32] then go to size I ++ if now what we have [57:38] to do is like okay we have gone into [57:40] this size [57:42] now because if you check this formula I [57:45] hope I'm recording this if you check [57:49] this formula you are basically doing the [57:51] max of zed so first data like a max [57:57] let's say 32 f32 okay and then we can do [58:03] uh this should be the output of it so I [58:06] guess like we have defined it the right [58:08] way input is the data which is the I [58:14] will store the formula I check it was [58:18] the I guess the max value like this we [58:22] have to just do the uh sum and then We [58:25] can divide this uh with the thing right. [58:30] This is the this is this this is the [58:32] thing we have to do um var val and uh [58:36] this is only for max value then i then [58:39] we have to do the sum because you know [58:42] uh we also have the sum and then we'll [58:44] divide it with the um we will scale this [58:48] later but like let's compute the sum [58:52] first and then we will okay let me [58:54] explain this again I guess you haven't [58:56] uh really got what exactly I'm saying to [58:59] to compute this uh softmax forward uh [59:03] what we need to do is like you know we [59:05] have to compute these formula and it's [59:10] it's simple it's not really difficult so [59:12] because we are we have this V we can get [59:14] actually the inputs so like this Q3 we [59:19] can calculate the size which is the is [59:22] the which is the bar val columns and [59:28] multiply by R valu [59:33] like it this is not a mattable operation [59:36] so I'm just like doing val I'm just [59:38] resting on the val otherwise like I have [59:40] to I can take take this from input as [59:43] well so for 32 I zero I go size I ++ [59:52] now because if you see this it's [59:56] basically like we have to compute is in [59:59] element wise where the uler has the [60:04] power of like zi minus max of zed and [60:09] then it is normalized with some [60:11] summation although the this is exactly [60:16] the same like this equation and this [60:18] equation is same it's just like it is [60:22] summed over it. So the easy way to do is [60:26] like you basically I less than zero you [60:31] you do max value do max f32 max value is [60:38] let's initialize with this bar inputs [60:43] zero is also a val right input is also a [60:48] val I forgot that like input is also a [60:51] bar so it will have a data. Now this max [60:56] value going to be changed because like [60:59] we need to find the max value. It can [61:01] have max [61:04] value on like this thing. Now again go [61:08] for [61:10] size [61:11] plus then the sum you know. So sum will [61:15] be to do sum and which is the this [61:20] function which is we are trying to do is [61:22] and what is the function name uh it's x [61:27] f you know um the xf and it will [61:31] basically if I just and z i is basically [61:35] you know uh is the input [61:39] so this we're going to store in where [61:42] this we're going to store and var val [61:44] data I and so the upper part is done [61:49] like this will also compute the upper [61:51] part and then with the sum we can [61:55] compute this um and then we can actually [61:58] divide it you know so how we can divide [62:00] it divide with the sum so go with the [62:03] U32 I = 0 [62:07] size ++ then that's it this is our soft [62:13] max. This is our sort max and exactly [62:16] this is going to be for the value also [62:19] that [62:20] um you have the size then you iterate it [62:24] over it and uh in this iteration [62:29] I hope like I'm audible to yeah in this [62:32] iteration in this iteration we can just [62:35] like go and then see what exactly is [62:38] happening like because really is [62:40] basically turning all the negative thing [62:43] to zero. Z and then all the positive [62:45] thing to whatever the the value they [62:48] have. Right? So if the data if the V dot [62:55] inputs of zero it will also have the [62:58] only one input which is zero. V data I [63:03] is less than zero. then v input [63:11] v val data [63:14] i equals to zero. Else if it's not like [63:19] this [63:21] v val= to inputs whatever the input is [63:26] just store there and it should be [63:28] inputs. Now for the R operation, it's R [63:32] operation is good. It's not very [63:35] complicated. [63:37] So again the size will be this. And in [63:41] this size we can just say you know if [63:45] val data I = to bar inputs [63:50] zero val [63:52] data I plus v inputs zero val data I [64:01] should be one. So we have element wise [64:05] addition right. The same goes with the [64:09] sub also. We just going to subtract it [64:12] here. Now, Matt Mull is kind of very [64:16] very different because it will have you [64:18] know a lot of things which could be a [64:21] transpose non-transpose and stuff. So [64:25] for this one like we will define another [64:29] set of function. Okay. So, so the batmo [64:33] is kind of different here because it [64:36] will have a a transpose and different [64:39] kind of stuff. So, what I can do here is [64:42] like no here we can define [64:46] another set of things [64:48] which is matrix. This matrix [64:51] multiplication [64:54] we will have uh you know we are not [64:57] going to take anything out from the [65:00] function we can just actually [65:04] means no transpose in either side of it [65:07] so it will basically going to yeah I [65:10] guess like we don't need this kind of a [65:12] thing void because we are passing the [65:16] out metrics matrix and uh a matrix B [65:21] right and for do I = to0 I less than [65:27] size I ++ basically this uh this this [65:31] this this [65:34] a you know uh how we can compute the [65:36] size which is we don't need to like to [65:39] go in the same way like how we went [65:42] through in the last add and uh soft max [65:45] thing but rather we can call this [65:50] And then u32 [65:52] J = J less than matrix A [65:57] to columns and then A++ [66:02] for two we can say I64 because you know [66:07] maybe matrix is bigger than ready very [66:09] ready. [66:11] K equals to we need three iteration [66:14] because you know the first is the out [66:16] rows and then we are going indexing on [66:21] the actual matrix A which will help us [66:25] the second uh second iteration will help [66:28] us to indexing on A and B right [66:32] and um the out which is again going to [66:36] less than these three indexing which [66:39] help us to it's going to help us to do [66:41] the whole thing. Now I'm going to [66:44] explain again. So white mat [66:49] n this n is basically the uh not [66:54] transpose. So to have this out matrix [66:58] matrix out matrix [67:02] A matrix B you know and if basically we [67:08] can just U64 [67:11] I = to0 we are just using U64 here maybe [67:15] like matrix is very big although like it [67:18] doesn't make sense why we use only U32 [67:22] in earlier version but it's Okay, [67:25] earlier version means in soft max radio [67:28] we could also use the U64. It will go to [67:31] the a rows columns. Oh sorry a out rows [67:39] out plus i ++ you know. So out to the [67:44] rows plus again J = to0 J less than [67:50] columns J ++ for K =0 K less than it is [67:58] K K K K K K K K K K K K K K K K K K K K [67:59] K K K K K K K K K K K K K K K K K K K K [67:59] K Z K less than [68:02] out columns K ++. So basically these [68:07] three iteration we are going to do to [68:09] actually index the matrix. So if you [68:13] don't know like the matrix is basically [68:14] just the you know how we index things [68:18] because we are not using like any kind [68:19] of array we are just like indexing [68:21] through the the the actual memory. So if [68:26] you think of this is as a matrix then 1 [68:29] 2 3 4 5 6 7 8 9 you know this is a [68:32] matrix row if you want to access the [68:35] memory the location of this matrix the [68:38] row major you basically mostly use row [68:40] major basically and how you store the [68:44] how you also store in the memory of the [68:47] matrix which is the row major majority [68:50] of the time which is like your flattened [68:53] memory will be 1 2 3 4 5 6 7 like this. [68:56] It will be like this. And the column [68:58] major will be if you want to store with [69:00] the column major it will be like 1 5 93 [69:04] like like this 1 15 93 26 10 like this. [69:09] So here we are using like a row major [69:11] thing. So it has a formula to access the [69:16] do the indexing basically which is like [69:18] column plus rows number of columns [69:21] multiply by number of columns. [69:24] So row multiply by number of columns [69:26] will happen first and then you just add [69:29] with the column. This is exactly like [69:32] you know this is exactly like telling [69:35] you have this matrix M and then row and [69:40] column and like probably you have used [69:44] this in Python which is like slicing [69:47] this is row and column [69:49] right um so this is how you actually [69:53] store these things otherwise right now [69:56] we're going to use this formula which is [69:58] column row so here's a twist Let's [70:01] suppose if we want to transpose because [70:03] you know if you know more about [70:05] transpose if you don't know transpose is [70:07] basically transpose is like changing [70:10] this row to column. So this will become [70:15] column and row and then so this thing [70:19] will also change. So this will become [70:22] row and column right? So this will [70:26] change it. Now when this will change we [70:31] can able to define functions like with [70:34] if you want to multiply with transpose [70:36] or not transpose. So we can also change [70:38] the formula and use the right formula [70:41] indexing thing right. So transpose and [70:44] basically you it is exactly like a [70:48] transpose if you do a transpose on i on [70:52] k it is exactly like a of k and i. So [70:56] here the out of data will be I + K I + [71:04] K. So sorry I + J. J + I sorry if I just [71:08] say J + I because J is the column and [71:13] row should be here because if you just [71:15] using this column plus rows multiply by [71:20] number of columns. So ji [71:24] columns equals to you have this out a [71:29] data [71:31] and uh which is k + i which is so here [71:36] you can see like k is zero and i is zero [71:40] the first index [71:42] and then you just and if it's zero then [71:45] basically this whole thing cancels out [71:48] and you're left with like a k. So you K [71:52] is become like a column for you here and [71:55] row is zero. So here's like K is [71:59] something that I able going to index [72:01] over it [72:03] and uh you're going to do the dot [72:07] product with B then B dot product will [72:09] happen with the J and K. That's it. Like [72:14] this is the whole formula. This is this [72:16] is the thing. Now the cool part is that [72:19] you can just change the indexing you [72:22] know let me check like if it is correct. [72:25] Yeah it is actually correct. Now this [72:29] map N can be like NT which is like a [72:33] transpose the P matrix. So you just have [72:37] to do you know what you have to just do [72:39] is like you have to just change this [72:43] order just I + you know so if you just [72:48] change the order sorry uh I guess like I [72:52] change the A of it which should be the [72:57] here K + J then B is transpose. Now if [73:04] we want to transpose this t and n then [73:08] basically a should be transpose. So if a [73:11] is transpose [73:13] so the iteration we are doing over the k [73:17] over the a sorry will become rows here. [73:22] [snorts] So and this will also change i [73:26] + k. Why it will become why it will [73:29] become rows because you know it will [73:31] also it is transposing things you know [73:35] the A is transposing because right now [73:38] we were actually moving over the A. [73:42] We're using A to iterate but like A [73:45] wasn't changing b was changing. So our [73:49] iteration loop was exactly like this. [73:52] But like right now our A is affected [73:55] which was using which we were using in [73:58] the um [74:01] in the iteration. We can also use the B. [74:04] So that will create a non-effect thing. [74:07] But like right now it's good that we are [74:10] using only A for the iteration. [74:13] But as A changes, A is getting [74:15] transposed. So we have to also change [74:17] the iteration loops. Okay. So and then [74:20] we can have like a TT which is like both [74:23] are changed. So now it will become I + K [74:28] K + J then also become like rows. Now we [74:33] can just like define the final mat which [74:35] is like a case of it you know matrix A [74:38] and matrix B. Now this will basically [74:42] you know end. Okay. So this is like just [74:46] a zero if if all are zero then we will [74:49] use this n1 and if n1 is true we can use [74:53] this thing. So we can define like um [74:57] case thing like you know there's one [75:00] more thing like we can also say which is [75:04] zero out if you want to do the zero out [75:07] although like it's not very important [75:10] and then we can just get rid of both [75:12] because like if both are true then it's [75:14] already going to be tt. So zero out will [75:19] help us to basically initialize the [75:21] matrix with all zero. Um so f0 out. So I [75:28] initialize with R U64 [75:32] and uh transpose transpose A or [75:36] transpose B if it is kind of a situation [75:41] then switch and then I can just copy [75:44] this case that going to be. Okay. Uh [75:47] this is this now we can utilize the same [75:50] manual here. We can say let me just [75:54] check if we haven't made any mistake. [75:56] Yeah, this looks fine. Let me just fix [76:00] this quickly these errors. Let's [76:02] implement this reinforce forward which [76:05] will be the the computation of the log [76:09] and the returns right uh it will be the [76:15] logs and the returns. So the log of [76:19] probabilities basically will be I can [76:22] just copy it from here size right and in [76:27] that basically what we can do is uh to [76:29] make it more safer because we will get [76:32] like a probability from the inputs. So [76:35] max of either one that's zero one this [76:42] is the uh this is the max or basically [76:45] do the one will be the returns this is [76:50] rainforce forward okay now we have to [76:54] compute the backward passes and stuff so [76:56] backward passes the gradient we have to [76:59] compute it's very simple uh also like [77:02] it's not difficult I have [77:05] added what will be the loss. So you can [77:07] see like in the for the reinforce loss [77:12] the grain will be like GI times the RA I [77:15] which is the reward here divided by PI [77:19] and negative. So and really for it is x [77:24] if it is greater than zero then ti it's [77:28] like the the previous gradient or the [77:31] evaluation gradient of value or if it [77:35] will be going to zero and then soft max [77:40] gradient will be this. So yeah, I think [77:42] let's let's compute this first. We have [77:45] we can compute the reinforce [77:48] gradient and do like this and if I guess [77:53] this should be 64 to only use 64 and uh [77:58] and then over it and uh grad will be TI [78:02] is basically the grad. We can also uh do [78:07] similar kind of thing here as well. what [78:11] we have done here because I think you [78:15] know if and wire if this is like this [78:20] you know in the input [78:24] uh this is getting very confused you [78:26] know but the thing is that [78:29] I want the prob should be the uh [78:33] gradient [78:35] you know so input will have is input [78:38] will have this this zero index is [78:40] probability [78:42] you know so val I can say cra I guess [78:46] this is something like this I think you [78:48] know it's basically a transpose so if [78:51] you when you do when you do uh matrix [78:54] multiplication it will going to [78:57] transpose so for the gradients you know [79:00] I will do much better if I just if I can [79:03] just do it like this a and b and we have [79:07] to compute the gradients of a N and B. [79:11] So here you have seen like we haven't [79:13] computed the we haven't computed the [79:16] gradients of inputs two because with [79:20] input two is basically RT which is the [79:25] minus R it's like reward here. So we [79:29] don't need like uh to compute the [79:32] gradient of return. It'll just it is [79:34] just a weightage. So we can just leave [79:36] it and then we can only compute the [79:38] gradients of the probability [79:40] which is the probability of given action [79:44] and state probability taking action [79:47] given state you know. So if variable [79:52] this flag a we can just say a flags then [79:58] we can do match we can pass the ban [80:02] inputs zero b transpose will be because [80:06] it's a so transpose b will be transpose [80:10] here to calculate the gradient uh for [80:13] the a having the gradient of a will be [80:17] basically going to be the transpose of [80:19] AB. So AB [80:22] transpose B transpose [80:25] and this here A is transpose and B. I [80:31] hope you're getting it. So yeah for the [80:35] addition it is basically the addition. [80:39] Yeah. Now we have to do the same thing [80:43] which is like addition [80:45] and uh put the entire thing in here. It [80:49] will be not just like plus equals to but [80:52] rather okay here we have just did some [80:54] kind of a mistake. It just it will be [80:57] crap. No it is fine. Okay. So some [81:02] errors which I will solve here. 394. [81:07] This is for the A [81:10] and same goes for the B. So B it will be [81:15] this B is B going to be one one and [81:20] that's it right like for the subtraction [81:23] I think for the subtraction it is also [81:26] kind of a similar I guess it's it's it's [81:28] very similar because but here it will [81:32] going to be negative we can change this [81:35] to now again like this input equals to [81:41] only one input put okay and size will be [81:45] this size is you know this size is uh [81:49] and in this input what we're going to do [81:51] is like if equals = to zero is greater [81:55] than zero this otherwise this [81:59] y can also add one more thing which is [82:02] like a floating point right and it's [82:05] going to be plus equals to okay now we [82:09] only have left this soft max. So the [82:15] easiest part about this is that the [82:18] vector form this is uh our equation [82:22] which is like a dotproduct of g and s. S [82:24] is basically [82:26] the soft max [82:29] data and G minus G is the gradient of [82:32] the the bar [82:35] and uh G minus G multiply G dot product [82:39] with this and then soft max data again [82:44] dot product. [82:45] So let's compute this dot product. Going [82:49] to take this and uh also going to take [82:54] this I will say dot because you know we [82:57] want to use the softmax data. This is [83:00] the dot product it will do. Now again we [83:03] have to do but this time data dot [83:07] product of dot dot s minus s. Okay. So I [83:14] hope the computation is right. Okay, we [83:19] have implemented everything now like all [83:21] the things which we have defined as a [83:24] function. So the next what we have to do [83:27] is like we can start building up a model [83:31] you know because we have built all the [83:34] things required to build a model. So we [83:36] can start building model and for that [83:38] like [83:39] model C. Okay. So we have implemented [83:43] this autograd thing. Uh now let's create [83:47] environment um or we can create model [83:50] but I think uh I want to create [83:52] environment first. So environment let's [83:55] see is this the here we probably going [83:59] to do is uh create some strct and enums. [84:03] So let's first understand what kind of [84:06] environment we are trying to create. We [84:08] will create a snake game environment. I [84:10] think this is going to be a really [84:11] simple one and this uh which can [84:14] actually have some level of difficulty [84:16] also which have reward structure where [84:20] the snake will get reward and if it hits [84:24] the wall it will get penalty and stuff. [84:27] So this will going to be simple snake uh [84:31] simple environment. So let's go just [84:35] going to have like actions action left [84:39] uh left will be zero right will be one [84:43] up two down three right uh this is going [84:48] to be action now snake env this will [84:52] contain a lot of information like you [84:54] know state uh which which has a state uh [84:57] variable type so this state variable [85:00] type will be Drop right and this state [85:05] is not state it's snake [85:09] state food what will be the state of [85:12] food and uh score score will be at 32 [85:18] score and uh action will be POV like [85:22] what exactly the point of view because [85:24] snake can point because in snake game [85:27] you probably have seen like snake has a [85:28] point of view it goes in one direction [85:31] then we if [85:33] in keyboard like in keyboard phone when [85:36] you press the low button down button [85:39] then it POV changes and then it moves [85:42] and downward. So something like this [85:44] it's basically the point of view of the [85:46] snake. Now this is the action POV and [85:50] then what we'll have POV that's it I [85:53] think what else we can have this current [85:56] state will be snake state food then [85:59] score [86:01] this yeah and uh then we have like this [86:04] uh type defct I hope it's recording I [86:08] guess yeah it's recording type defv [86:13] then we have a replay buffer you know [86:15] because in reinforcement learning [86:17] You probably have like heard of this [86:19] kind of you play buffer where you do the [86:23] gradient accumulation later after doing [86:26] the roll outs. So you do the roll outs. [86:29] So I can explain you like this. You do [86:32] the roll outs. Roll outs you do the [86:36] rollouts is basically collection of [86:38] data. Collection of data. you do the [86:41] roll outs and then after doing the roll [86:43] outs you do the step gradient step using [86:48] that roll out. So you collect the [86:51] experience. So that's why like ARL is uh [86:53] is basically doesn't require training [86:57] data because it generates the training [86:58] data uh the pure AR thing. So this will [87:02] generate the data the experience from [87:04] the snake and then we will use this [87:06] experience to correct the agent's [87:08] action. So we need to store these [87:12] experience. Uh that's what we call like [87:13] replay buffer strct. This replay buffer [87:16] you know will have um a lot of things [87:20] also. This snake game will have like a [87:23] rows and columns and grid size and [87:28] action POV and number of steps it had [87:31] took. So use 64 steps. This replay will [87:36] have you know the first is like [87:39] trajectories and this trajectory is [87:41] itself a shock. So let's suppose the [87:45] buffer size is 512. [87:48] Okay. And uh episode length because [87:52] episode is basically snake is collecting [87:56] the uh states and actions in a single [87:59] game. So it goes to x0 y 0 like the [88:03] coordinates x1 y1. So it basically [88:06] navigating the environment. So how many [88:09] steps it is taking in a game is called [88:11] episode length because it cannot like [88:13] take forever. We have to like it can [88:15] take forever. It can just like take uh [88:17] until it's win or lose. But like let's [88:19] define a definite uh uh steps. So it [88:23] could be like 200, right? Or maybe like [88:27] we can define more let's go with like [88:30] 100 and uh this can be,24. [88:34] Now this trajectory you know this [88:37] trajectory contains a lot of information [88:39] which is like states state episode [88:43] length state it states which is like all [88:47] the states the snake went through and [88:51] this food states [88:54] episode length action [88:57] and all the action it took and uh action [89:00] what is the POV episode length [89:04] rewards and returns. Um F2 returns and [89:09] uh state next state because we have to [89:12] store the what next state it went [89:14] through. What will be what was the next [89:16] state after taking the action [89:19] and uh next states and done like is the [89:23] game done? Do we have a result of it? [89:26] So not then we can actually iterate [89:29] again and again and again. So this is [89:32] some kind of abstraction layer in [89:34] trajectory we can say 32 count right and [89:39] here we can also say length. Now this is [89:43] the abstraction [89:45] define [89:47] what exactly is exactly is this error [89:50] just pass the chance why we getting this [89:53] error now the error will go away now [89:56] create the env. So this basically create [89:59] TNV will take a because we are not like [90:02] defining this in arena now like it's [90:04] very array kind of thing. So here like [90:07] in trajectory and all stuff we haven't [90:10] used any kind of a pointer thing but [90:12] that's okay I guess like let's go with [90:15] grid size rows will be two side because [90:19] it the grid size grid will be a square [90:22] so we can like qrtf [90:26] and uh grid size right and this 32 call [90:32] equals to side same thing then This will [90:37] have like a columns and rows and then we [90:40] can define like snake take the push [90:44] truck which is the arena itself and the [90:50] size and all stuff. So I'm going to go [90:53] like there's other ways which is like [90:56] maloc you know we can just take and say [91:00] size of this thing size of CNV but like [91:07] let's go with like arena arena arena and [91:10] now 25 or 36 let's go with like 36 [91:16] okay created the env [91:21] you know snake dox equals to zero that's [91:25] how we can define n dot c 0 and uh [91:30] actual po right so because it's starting [91:32] from 0 0 so 0 0 will be like top left [91:39] bottom left [91:41] and then in the x and yax x and y graph [91:46] so 0 0 will be like a top b Sorry bottom [91:51] left and then it can POV towards the [91:55] right. So this will define the ENV if we [91:58] can actually enter the print enter. [92:04] Okay. Why we were doing this? Damn. What [92:09] exactly is this? It should be zero, [92:12] right? I think it should be zero. Um [92:17] I guess like you know we have defined in [92:20] a wrong way we can defined like this uh [92:22] which is much better this matter is like [92:26] this why it is getting like this long [92:30] with this damn it when we are returning [92:34] env that's it small small mistakes okay [92:38] so now this is done we have created the [92:41] env U this is our env [92:44] and uh we need to find the you know we [92:48] need to reset this state also. So after [92:51] episode end we need to reset it reset [92:55] again like this going to be like this [92:58] but we can make it more interesting here [93:01] which is like get random lock. So this [93:06] will give you the random food location [93:09] based on the number provided number [93:14] provided which could be uh so I'm not [93:16] doing any checks here so just to make it [93:18] simple but you can just think like okay [93:22] there will be a check which checks the [93:24] range within the range of like grid size [93:27] and rows and columns. So if there's [93:31] function random in the PRNG this file [93:36] which is which is which is which is this [93:39] function define it like this this will [93:42] be I32 because this can be negative. [93:45] Yeah. So that's why define this side [93:49] here why the Got it. Got it. Got it. And [93:54] same here now. Okay. Okay. So this reset [93:58] uh env is done again [94:01] you know you can see by passing it we [94:04] are getting random. Now taking action we [94:08] we need to make sure that we are taking [94:10] the right action in the environment the [94:13] valid action. So we have to define that [94:16] you know action this will [94:20] basically have snake game will be [94:23] action. [94:25] There will be one more action here. [94:27] Okay, that's it. I guess there can be [94:30] one more action like which could be none [94:33] like no action. Okay. And uh take [94:36] action. If a entaction [94:42] if action equal equals to write you know [94:45] action equal to none action if action is [94:50] none then action will become you know [94:53] POV env is POV like because we are not [94:56] taking the model decided to not take any [94:58] action. So the POV the whatever the POV [95:02] of the snake will become its action you [95:05] know. So just take action make push [95:10] snake in the POV direction because snake [95:13] will going to take action [95:16] at every tank step with its POV. What [95:20] the model has to do is to take action to [95:23] prevent it from hitting the wall or like [95:26] to get in the direction of food. So if [95:30] action is none action this this action [95:33] is right envo [95:48] to action [95:50] whatever the action was okay that's it [95:54] in the else we we can return action is [95:57] tide The snake position would become [96:01] should be one nice one. And if it's left [96:05] then it should be minus one. And if it's [96:08] up should be the y should be okay. Y [96:12] it's been a lot of time. So yeah. So [96:16] this is working right. You know what we [96:19] need? uh we need to build uh some sort [96:22] of [snorts] vector which is the some [96:26] sort of vector in the sense because you [96:28] know you have this inputs and math and [96:31] stuff like that. So this input will be [96:33] like some kind of embeddings and some [96:35] some one hot coded vector which has all [96:39] the information about the snake world or [96:42] snake env you know because we don't have [96:45] any kind of like input like text or [96:47] something like that but this snake env [96:51] world which has a lot of information [96:52] like where the snake is going what's POV [96:56] what's date and all stuffs it's actually [96:59] a lot of information so we will provide [97:01] this as input to the model and model [97:05] will actually find patterns and observe [97:08] it and produce an output based on that. [97:11] But we need to convert this input in a [97:13] right way so the model can actually [97:16] understand it. So uh the best way to do [97:18] so is to create a some kind of vector [97:22] right. So build state vector which [97:26] contains some informations like which [97:28] has like metrics and state and then [97:32] state of this will be state of snake and [97:36] food and uh action what will be the POV [97:41] what is the POV of the action and uh [97:44] column right uh now to compute the snake [97:48] I we know the formula of this this [97:51] column row in intro major thing. So in a [97:54] grid size like we are storing let's [97:56] suppose this is the state it will have [98:00] like for 0 0 0 [98:04] 0 0 0 0 and then for like grid size till [98:09] the 36th 36th of 36th of this thing [98:15] length or I can say 35th because if we [98:18] have like 36 grid size we have like this [98:21] kind of a state and snake eye will be [98:25] you know one here and one like if the x [98:30] and y is like this. [98:33] So this will have some kind of a [98:34] information in our flat vector and we [98:38] can do this same with the snake um with [98:42] the food also you know snake [98:46] position it's like a one hot encoded [98:49] thing and then for the POV we can for [98:54] the POV it can have again some kind of a [98:57] similar similar similar vector which can [99:01] actually have some information. Well, we [99:04] are going to create a very simple simple [99:07] vector not very like kind of a embedding [99:09] type dimensional thing. So just a one [99:12] hard code which has some information [99:13] which can actually help the model to [99:15] absorb it. U snake I which is uh this [99:19] columns [99:21] plus rows rows is basically U32 snake [99:26] columns will be snake doy right and uh [99:30] multiply by and then plus like it's [99:34] little bit weird I would say but I can [99:38] define it like this X state doy that's [99:43] kind of different here but like it's [99:45] Okay, columns is x. Sorry, I'm just got [99:49] confused in this state.x and then plus [99:54] state snake snake doy then multiply with [99:58] columns. This is the right thing exactly [100:00] with the food and then we can store in [100:03] the inmetrics which we are which we have [100:07] here which is the input matrix [100:10] data snake I to snake 1 0. So whatever [100:16] the index of the snake I will have like [100:17] a 1.0 [100:19] floating point and in data I size plus [100:26] 0. This is a flat vector which have like [100:28] a you know uh in Python or like in other [100:32] matrices we in numpy we actually see [100:35] shape like this you know where we have [100:37] this three three inputs and then the [100:41] grit size but here we we have platin [100:45] thing right so three into grid size so [100:50] this is exactly our input vector there [100:52] will be like a another function which [100:54] will help us to find the reports Because [100:56] you know we have defined the take action [100:59] but we haven't defined the reward [101:01] function which help us to get the [101:03] reward. So when the agent will take [101:05] action in this let's suppose if it finds [101:09] the food and uh finds the food of the [101:12] state uh then we have to give them a [101:15] reward and if it hits the wall then we [101:18] have to give them a negative reward. So [101:21] right this will give reward and it will [101:26] it will have snake. [101:29] Now if env.x X dot snake dox [101:36] greater than [101:38] less than zero or R env dot env snake [101:43] dot y less than zero r env if it is like [101:48] this you know then give them a negative [101:51] reward or something like that [101:54] right rv [101:57] snake dox is greater than the env rose [102:02] or or we could say to i32 [102:07] and if it's like greater than the envir [102:14] f32 give them uh basically minus min - 5 [102:19] okay 5 1 0 f now if it lands on the food [102:27] if snake dox [102:29] equal equals to envood.x [102:34] X you know and then envake [102:39] doy equal = to [102:42] food doy and reward give them reward [102:46] like + 10 f these are the two things [102:51] basically um now it is done um like it [102:55] will get the positive reward here but [102:59] like now we have to when it eats the [103:02] food when it actually lands the food we [103:04] have to give them a new location to this [103:07] uh food. So that's why like uh new food [103:11] will be we'll do do while do [103:15] while new food equals to state get [103:21] location [103:23] close. Okay. And if new food new food [103:27] dox equals equals to inv snake.x X [103:33] and and and means like a we don't want [103:36] like where where the position of the [103:38] snake is we don't want the food should [103:42] be again come at this position so we'll [103:44] just like okay if this is the condition [103:47] is true then you know run again [103:50] iteration and find new food a new [103:52] location that's it uh that's it I guess [103:55] and uh yeah have record and then turn [104:00] rebar. All done. We can actually train [104:02] this uh simulation this environment you [104:06] know we can simulate this environment [104:07] and train the model. Uh we haven't [104:09] defined the model yet but it's okay like [104:11] we will define later. [104:13] So here we can actually create a [104:15] training pipeline. Training pipeline [104:17] will be simple also. won't have much uh [104:20] there like you know in training pipeline [104:24] we we have this training of RL and in [104:28] training of RL you basically do rollouts [104:32] collect experience you will compute the [104:36] returns of this compute the returns of [104:39] the roll out and the buffer and then [104:44] sample experience from the buffer and [104:47] then train it this is the whole pipeline [104:50] of the RL training you know so void [104:52] train [104:54] arena I guess like there's no arena [104:57] model state model [105:00] and uh snake env [105:03] me arena there's this arena I don't know [105:05] why we need arena we need to create here [105:08] but okay uh the first thing we have to [105:12] do is like you know we'll do the forward [105:14] pass and we haven't defined the forward [105:15] pass you know because uh throw out will [105:19] do the forward pass and stuff. Well, I [105:22] will just you know uh for the ease of [105:25] this [105:27] what I will just going to do is like I [105:29] will paste this epox [105:33] size episode length and gamma which will [105:35] help us to basically these are the [105:37] hyperparameters [105:40] and uh let's see if it's compiling. [105:43] Okay. Uh we'll we'll see this later. [105:47] Here we need to do is like forward pass [105:50] right and backward pass right. This [105:53] forward pass basically will have arena I [105:56] guess arena. No this forward pass will [106:01] have you know graph. [106:03] Now U32 [106:06] is the size of the graph of dot. This [106:09] will be a pointer and then U32 [106:13] I = to0 I less than size I ++ F. Now [106:20] iterate the Iate the size. Uh what else [106:24] we need from here is access the V [106:29] because you know we the graph will have [106:32] that build graph we had here create [106:36] graph. This will give us the graph from [106:40] the forward pass [106:42] and we can use this to write where it [106:45] went. So we have to iterate over this I [106:50] compute the bars location right dot type [106:55] parent dot type which is like a itself [106:57] is a bar and if it doesn't have a type [107:02] type isn't is equal to null return [107:06] otherwise just to because it will have a [107:09] forward function type and this forward [107:12] function you know will need a bar which [107:15] is again see yeah that's it this is the [107:20] forward pass but the backward pass here [107:24] uh it will be exactly kind of similar to [107:27] this [107:28] but if uh if this is not true if this is [107:33] not true then what you have to do is [107:35] like continue and if [107:40] you have to continue again and if it is [107:44] like a bad require ire grad as if it's [107:46] parameter which a bar require grad then [107:51] we need to clear this thing uh it's grad [107:55] so this will clear the gradients the [107:58] gradients so clear the gradients and uh [108:01] then you can use fill uh I guess bill is [108:04] there then fill the graph r² minus one [108:09] the last variable should be 1 [108:14] right uh So fill the last variable last [108:17] gradient last variable's gradient to 1.0 [108:20] 0 and then traverse back. Basically I 32 [108:26] I 64 = to 0 less than I I should be [108:31] sorry I should be graph I g 0 I - [108:37] traverse back and then you know where [108:41] where and check if it has gradient [108:45] gradient then you know current equals to [108:48] zero then continue if current dot type [108:53] type [108:55] isn't equals to null and current type [108:59] back over isn't equals to null. I can [109:02] change to current because like we [109:06] already defined C uh okay so it's good [109:10] we have implemented the backward pass is [109:13] not defined uh you know why is not [109:17] defined because copy this there's some [109:20] function we can actually import we can [109:22] define in which basically fills and fill [109:25] the random we don't need this random [109:28] thing okay is compiling then in the in [109:32] the arena or in the ENV you can go and [109:36] train it proper. So this will we will do [109:40] like a roll out experience collection. [109:43] So 32 [109:46] I epoch equals to 0 goes to epoch [109:52] ++ right and then in this epoch we [109:56] probably will do forward pass. How how [110:00] we can do forward pass from the model [110:02] because we have model right. So but I [110:07] guess you know in here model state [110:11] when we have defined it has only these [110:15] things. So it will have forward graph VR [110:19] graph output. These are the things model [110:23] state will have. again go for J =0 [110:29] I less than roll out size I ++ this will [110:33] be roll out now we need to take a step [110:37] in the environment so let's generate an [110:40] action first let's see like if it is [110:43] working or not so we can call train [110:47] model state so this training function is [110:50] working right uh then what we have to do [110:53] is uh what we have to do is roll out [110:57] sizes and then then then then in here [111:01] you can say model dot forward graph is [111:05] equals to build create graph and uh you [111:11] know the create graph what the create [111:14] graph will take arena model state and [111:17] outvar the out bar will be you know and [111:21] the backward graph will The order output [111:26] this will be a cost that's why like we [111:30] need to define a cost here it is null so [111:33] it will probably will give this uh auto [111:38] graph return a pointer of graph and this [111:42] is not done yet we have defined this [111:45] graph as this I guess you know we don't [111:48] we can change the definition of the [111:50] model state that we don't need this and [111:54] uh this will graph dot because you know [111:58] we are actually adding in the out of it [112:01] here out so the graph dot varss will be [112:06] out now the graph bars is basically not [112:11] defined yet push truck push truck or [112:14] basically we can do there's this another [112:17] thing which is push array int uh it will [112:23] save some memory of it but basically [112:26] just not initialize this array like [112:29] others like you can see it is true true [112:32] means like it initializes everything [112:34] with zero but it doesn't so yeah v prop [112:42] isn't equals to null means it has space [112:46] allocation [112:48] equals to graph dot = to dot number of [112:53] hours equals to out size and then we can [112:56] just do me cpy to move the variables [113:01] locations pointers to here from out [113:06] size of will be and then we can just [113:08] free the stack because it was a [113:10] temporary location which visited is not [113:13] number of hours. So the issue is uh [113:16] issue was coming from here this this [113:19] kind of a issue because you were defined [113:23] after this I'm using here so that will [113:28] resolve issue and uh okay so this is [113:31] done uh now what else we need here we [113:34] haven't defined these functions you know [113:36] so by softmax which basically creates [113:39] the the variable of softmax so we [113:42] already have this create node so There's [113:45] not really needed [113:47] for any kind of logic. Just need to call [113:52] these functions, right? So we need to [113:55] call this return it and this uh takes up [114:00] some input which is arena model state [114:02] var variables. Okay, that's it. And then [114:06] v type you know a and b is basically [114:09] input null v type we have defined it [114:14] right we haven't defined that thing we [114:17] haven't defined the bar type really oh [114:21] wow we have defined the bar this thing [114:24] and rel for the rel dot number of inputs [114:29] equals to 1 dot shape equals to v same [114:34] dot [114:35] Forward view type it will be exactly [114:38] like this. So this will be M two inputs [114:42] where shape M forward audible. Yes. What [114:47] is this error? V type of max. We can [114:51] find this. Let me search like why [114:54] exactly I'm getting this error. Okay. [114:56] It's again like the same issue. I have [114:59] to put this in here. So this is a big [115:04] issue. So what I will just going to do [115:08] you know I'll just define definition [115:11] here. [115:12] So it would be better. No it will not [115:15] going to work I guess cuz it's started. [115:19] So okay we can complete this non noun [115:23] type. This one is this is nice. Uh so we [115:28] were completing the our [115:31] training pipeline in this we probably [115:33] you know take the action so let's [115:36] suppose the action for right now let's [115:39] take the action of this thing you know [115:42] print of action [115:45] you this is the action this action is [115:50] okay this is working I would say we have [115:53] like a five action 1 2 3 4 5 Right. So [115:56] it should be fine. Why sometimes it [115:59] fails? I just don't know. But it's okay. [116:02] Like we have this action and uh take [116:06] action and this is the envir [116:14] action x will be like taking this thing. [116:18] If it does not take action then um [117:01] Oh, got it. Got it. Got it. It can [117:04] happen because you know is game over. [117:07] What we need to do is like w [117:11] 32 [117:13] game env [117:15] snake.x [117:17] less than zero r. [117:20] It's exactly the same. It is working [117:23] right now. Much better way. But okay. So [117:27] if this is true then [117:31] okay. Yeah it's working. And when it [117:33] gets minus one it is it. Yeah. When it [117:37] gets minus one the game is over. Right. [117:40] Uh see I'm going because it is like it's [117:44] been [117:45] so long and we haven't finished yet. So, [117:48] I'm going to take this so fast and uh [117:52] let's go. Let's go finish this. Uh okay [117:55] this is uh how the game will over you [117:58] know and uh every time we have to reset [118:01] the state for every roll out size we [118:03] have to take this thing [118:06] and uh for the trajectory trajectory [118:09] define a trajectory trajectory which is [118:13] a buffer and this is this buffer you [118:17] know it's coming from here and uh this [118:21] has I okay now we can store it inside [118:25] this trajectory the information 32 I I [118:29] go to episode length okay length is I [118:34] already defined defined but it's okay I [118:36] ++ now in this um inside this we will [118:42] take the action now we have state equals [118:45] to env state next state food state okay [118:52] action pub is npv. Now we need to build [118:56] the state vector and inside this input [119:01] model val. I hope it will work I guess [119:06] because val is not a null state state [119:11] loop state and pnv [119:15] columns pnv rows like this it will going [119:20] to fail because we don't have model [119:22] right. So what I will going to do like I [119:25] will you know quickly I will just going [119:27] to copy and paste the model which [119:28] because I have a model [119:31] is simple it's nothing here so these are [119:36] the things it has create after critic [119:38] model which is basically arena and we'll [119:41] take model [119:43] and uh this is input and we are creating [119:45] a bar of size like 76 and one because [119:48] you know 36 is grit size and again 36 is [119:52] for the hour uh storing the food [119:54] position and four is for storing the POV [119:58] you know. So this is 76 [120:03] one bad none actually I we have five [120:08] actions but like eventually the actions [120:11] becomes four because you know if it is [120:14] non action then we are actually defining [120:16] it. You can see here action equals to. [120:21] So yeah, even if it's a action none, we [120:23] are actually [120:25] changing the action. So it becomes code [120:27] action. That's why like 76 by we have to [120:30] add one more. So uh this weight weight [120:34] zero is we are creating bar create for [120:37] this 128 76 it's 178 um size [120:44] and the bias. This is the bias and we [120:46] are providing the flag of it which is a [120:49] it's a parameter and also a require [120:51] gratuit for the input. It's required [120:54] none basically doesn't need to have [120:57] gradients and it's not a parameter also [121:00] it's just input. So and then we have [121:03] this uh weight one and bias one weight [121:06] two and bias two with like 128 128 [121:09] hidden dimensions and the bias will have [121:11] 128 one dimension. Okay. And then here [121:14] we are initializing this with like f rad [121:17] and all stuff. Uh which we had already [121:20] defined I guess or if not then I think [121:24] we have to define it here and we can [121:27] import stuff here. Okay [121:30] this this is that's why like it is so [121:34] let me just like think like what exactly [121:36] is the error. Okay. So the error is gone [121:39] but I have to change a lot of things [121:40] because the code I pasted it was [121:43] different what we have created. [121:46] So like different kind of function name [121:48] and like parameters required for that. [121:53] So yeah I think now we don't have any [121:56] error. So yeah I think this is uh [122:00] weights bias weights bias we created [122:02] like a three weights and three bias [122:06] corresponding to that and then this [122:08] basically randomly initialize this [122:10] things and um you know it will [122:13] initialize the layers and uh z a is [122:18] basically the the map of weight and [122:20] input and then we are adding v adds [122:24] using v add we are adding the the this [122:27] thing set not a and b not. So that's why [122:31] like we created this v mat where add v [122:34] which basically creates the nodes and [122:39] then do it will not going to compute [122:42] right now it will create a nodes that [122:44] what are the operations required okay so [122:47] it will create each node like this and [122:50] then similar to that we have another [122:54] matrix multiplication with weight one [122:56] and the input is a not which is the [122:58] previous uh output [123:00] it exactly like PyTorch we do in uh in [123:03] Python but like we we are basically [123:06] doing more scratch way and can see [123:10] layers in front of it. So Z 1B is [123:15] basically the bias again is added with [123:18] Z1 A B1 and then this is the radio layer [123:23] after to normalize this thing. And then [123:26] for the again doing the matrix [123:29] multiplication with the last weights and [123:31] then doing the addition bar add and then [123:35] we are basically doing the computing the [123:38] probabilities using v softmax and [123:41] defining a model output equals to [123:44] output. Okay. And advantage is basically [123:48] the it's actually none right now like it [123:52] doesn't have anything just create a bar [123:53] and does not have any information but we [123:56] will add the information inside this [123:58] advantage during the training and this [124:01] uh cost is basically the reinforce loss [124:04] we have so v reinforce and then [124:07] basically it will have like a bar type [124:09] reinforce and output and advantage it [124:11] will have then the cost function is this [124:14] right uh now Here we can now say no we [124:18] can now say create actor model arena [124:22] model now this will work okay so there's [124:28] something which is basically doing some [124:32] sort of you know we can use uh coding [124:35] agent I guess to other it will take a [124:38] lot of time [124:40] so we can just say like okay what [124:43] exactly is [124:46] okay. So these are basically the errors [124:49] uh which is like null but we have [124:53] created the model isn't it? Okay number [124:56] of bars equals to zero because number of [125:00] bars are basically kind of this [125:04] condition we can actually have to secure [125:07] the model. Okay, we have allocated more [125:10] memory. Now let's see. Okay, but this is [125:14] exact. Okay, I got it. That was mistake. [125:18] Okay, so the major issue was basically [125:22] here as I was like very confused on like [125:25] you know maybe like we needed 77 [125:31] dimension because like we had five [125:34] actions. [125:36] So I thought like no okay it's only four [125:39] I guess because like action is but it is [125:42] actually going to be 77 [125:45] uh that was making the issue here and [125:49] now it is actually working fine and uh [125:52] yeah this uh will be cost graph. Now if [125:57] you just run the training okay build the [126:01] state and uh now let's finish this [126:03] quickly. Um we have this uh arena env [126:10] there's this you know we can sample the [126:13] action uh using our probability [126:15] distribution right now but let's not [126:18] focus on that let's build the pipeline [126:19] first take the action we can take the [126:22] action now we need to store uh the store [126:26] the all these things so if uh before [126:30] this we have to like f2 [126:33] report at reward or if this thing is [126:38] reward. Uh now this give function we are [126:42] getting here reward then we can actually [126:46] score plus equals to reward right and [126:51] then done will be calculated like here [126:54] and then next state is basically n snake [126:59] after the action. The previous state was [127:02] the like this you know then trajectory [127:05] we can actually copy this and it will [127:07] actually store all the information [127:10] states food states POV actions reward [127:13] and all stuff this will store the [127:16] informations there's no words there's no [127:21] rewards okay there should be a rewards [127:25] yeah you can say there's st because it's [127:28] trajectory we that I'm creating over uh [127:32] done done is actually there's still more [127:36] errors [127:38] trajectory. Now this roll out is done. [127:41] You know after this roll out [127:44] this uh roll out is done. What we can do [127:47] is like we can compute the returns [127:50] compute the returns like because what [127:53] exactly the returns of each thing [127:56] returns that it's going trajectory of 0 [128:00] to episode length and we will compute [128:03] the returns which returns the sum of all [128:05] the reports at each step in a [128:08] trajectory. So I guess like this is [128:12] buffer count. It's not very efficient [128:15] way to do so but like it's okay. So this [128:18] trajectory gain the buffer trees F32 GG [128:24] is basically return [128:26] 0F or 32 [128:30] 32. So you know in advantage is you [128:33] basically go from you go from trajectory [128:37] length to zero and then computes the [128:39] trajec uh the returns at each step. So t [128:44] equals to trajectory rewards [128:48] rewards of t plus gamma returns returns [128:53] of t. I can uh basically paste this [128:58] algorithm. This will go to trajectory [129:00] and computes the using gamma. Gamma is [129:02] basically used for is a discount factor [129:05] and then trajectory returns is g and [129:08] then sum return sum. We haven't defined [129:11] any kind of return sum here but like so [129:14] we don't need this return sum return [129:18] square some yeah it is useful like [129:22] u standard deviation calculation [129:25] sample count uh that is also not needed [129:29] average return well we can actually [129:31] define this thing do average [129:35] return which will be outside of this [129:39] Let's print this. [129:42] Uh we will going to we also have to add [129:47] this in our buffer. We are getting [129:50] returned which is like a negative. So [129:53] our snake game basically snake is [129:56] getting killed every time. I don't think [129:59] I can find any positive here. Can you [130:02] find any positive? I don't think. Okay, [130:05] this is good. What else we need? [130:09] uh we have computed the returns [130:11] now it's time for to compute the [130:13] gradients and all stuff which is like [130:16] actual training thing isn't it so we [130:20] will going to go again and backward [130:24] pass in that backward pass you know we [130:28] have computed this thing which is [130:31] clearing the gradient clear the gradient [130:34] uh so again like we will do this before [130:38] computing in the gradients we have to [130:39] clear it. So 32 I = to zero. I will go [130:44] to graph dot [130:47] size I ++ [130:51] and then there's no model.graph. [130:54] Okay, this is clearing the gradient. Now [130:57] what we need to do is like again this is [131:00] a training phase. In this training [131:02] phase, you know, uh what you need to do [131:04] is like just go through all the buffer. [131:07] buffer count is basically the rollout [131:10] size. So U32 I = to U32 and then go to [131:16] the trajectory is the trajectory [131:19] iteration [131:21] but here instead of absolute length we [131:24] will go trajectory dotlength. Okay. [131:28] And uh again build the vector build this [131:32] vector. And this time the state is [131:35] coming from trajectory [131:39] trajectory dot state [131:42] of I and trajectory [131:45] state [131:47] pro trajectory. [131:48] Now this is done. Uh what we need to do [131:51] is like again clear the advantages. So [131:55] for U32 [131:58] I [132:00] K equals to0 K less than the model dot [132:06] advantages [132:07] this will go to the I we just want like [132:10] because we will compute the advantage so [132:12] just get rid of like the old advantages [132:15] thing. So advantage now to compute the [132:19] advantage it's simple advantage is [132:23] basically [132:25] trajectory [132:27] returns of t minus the baseline. The [132:31] baseline is basically the return mean [132:35] and some okay this is basically the [132:39] return mean. How can we compute this? We [132:42] can compute this uh average return. we [132:45] can not average return we can just say [132:48] to return mean okay and average return [132:52] is kind of so for the return mean [132:57] I can uh like there was this sample 32 [133:02] samples number of samples zero and uh [133:07] return sum average return is basically [133:10] something else you can just like remove [133:13] the average return error and put this uh [133:17] inside this number of samples [133:21] you know in a trajectory and uh return [133:26] sum plus equals to g that's it [133:30] and then after that u number of return [133:35] sum f32 return [133:39] mean is sum is sample number of samples [133:44] So this is our advantage [133:47] and uh yeah this is our advantage and [133:52] then we can just model advantage equals [133:55] to advantage right. Um this advantage is [133:59] basically val trajectory [134:03] actions. Okay, that makes more sense [134:06] because you know so there will be five [134:09] accents so five index for it but like it [134:12] will only store whatever the action it [134:14] it took. So it will only store the [134:17] advantage on the whatever the action it [134:19] was. So action actions now just do the [134:25] forward pass with given the model dot [134:30] forward graph and then do the backward [134:33] pass given the model backward pass cost [134:38] graph actually. Awesome. This is [134:40] actually computing the gradients. And [134:43] now after doing this uh after computing [134:46] this optimizer step we have to do the [134:50] optimizer step. Gradient scale is [134:53] learning rate divided by 32 samples [134:58] count right. Um now what we have just [135:03] done here have to do the same thing and [135:08] uh inside this the gradient current data [135:13] will be into gradient scale. This will [135:17] take the gradient. Well this will just [135:19] the gradient scale but we need to also [135:21] subtract this. So it will also going to [135:26] subscribe subtract this and subtract [135:30] current val width inside this the out [135:33] matrix will be current val current val [135:35] width [135:37] current gradient sub will be not defined [135:42] I guess because sub is where it is in [135:45] the autograph can say like this this is [135:49] actually working now and this is the [135:52] whole implementation of the [135:54] reinforcement learning from scratch and [135:55] see and I know probably I have made some [135:58] of the mistakes here like implementing [136:01] in life some of the few things probably [136:05] although like the library is pretty big [136:08] uh what I actually built but it's a very [136:10] minimal implementation of it where I [136:13] just try to get rid of lot of functions [136:16] and try to attend everything in a single [136:19] file like it's going 400 500 lines codes [136:22] in some files is so I try to attend [136:24] everything not try to manage everything [136:27] not try to check the [136:30] uh safety checks and all the stuff so it [136:32] was like very minimal implementation try [136:35] to simplify this and I hope this made [136:38] sense so I guess like probably I don't [136:42] like this will learn but it will going [136:45] to learn probably like if you just like [136:47] leave it for like 20 30 minutes it can [136:51] find [136:54] So you can see right now it has started [136:56] getting positive rewards after uh I [136:58] adjusted some of the things and improved [137:01] logging system and stuff. So you can see [137:03] like the returns are like positive [137:05] earlier it was negative um negative [137:09] negative negative negative then started [137:11] learning and I trading it I trading it I [137:13] trading and then become better so [137:15] reinforce is a very like unstable [137:19] algorithm you can say because it has a [137:21] very high standard deviation but still [137:25] it can learn uh it can perform really [137:27] good uh when you actually train for [137:30] longer time and it can figure it out. So [137:33] yeah, I think this was about the [137:35] reinforce algorithm. In this video, I [137:37] thought to actually implement also more [137:40] like um more algorithms like PO [137:44] and um the the versions of PO and stuff. [137:48] But I guess like it this video is going [137:51] very long. So I'll cover this in next [137:53] few videos or like in any other project. [137:57] But see you next time. Bye.