Claude Certified Architect - Foundations – Prepare for and pass the exam!

[00:00] Hey, this is Andrew bringing you another [00:02] tech certification course and this time [00:03] it's the Claude Architect Foundations [00:06] Certification. Uh, and so I think I [00:08] might be only one of the only providers [00:11] that actually has this course out and on [00:13] top of that I've put in the work and [00:15] done a lot of hands-on uh, labs in this [00:18] course. So you will not only just learn [00:20] the concepts, you'll be able to apply [00:22] them uh, very effectively. And I think [00:25] you're going to be really happy uh with [00:26] the content because even if you are not [00:29] utilizing claude and you're using some [00:31] kind of other agent or AI vertical the [00:34] knowledge here is extremely transferable [00:36] uh extremely deep and extremely [00:38] valuable. Uh so I hope you are excited [00:41] to jump into this course as much as I [00:43] had making it and I will see you soon. [00:46] Cha chia. [music] [00:50] Hey, this is Andrew Brown and we're at [00:52] the start of your journey asking the [00:53] most important question for us which is [00:54] what is the Claude Architect Foundations [00:56] course. So this is a uh certification [00:59] from Anthropic and it will help you [01:02] teach you the broad underlying concepts [01:04] of building agents expose you to clouds [01:06] models coding tools and SDKs. It's [01:07] focused on the adoption of workloads for [01:09] orgs or if you really need to understand [01:11] MCP. Anthropic created the idea of MCP [01:14] so they really want you to know it in [01:15] this course. not that hard to learn. The [01:17] exam code for this is CCA-f. There [01:20] technically is no official exam code, [01:22] but this is what everybody is going [01:23] with. Uh consider this certification if [01:26] you want to know how to effectively [01:27] build aic workflows. You're planning to [01:29] adopt claude as your primary AI driver [01:31] within your org or you are part of the [01:34] cloud the claude um partner network [01:38] because it might be a requirement for [01:40] you to advance within their network. Um [01:42] but we'll continue on here. You can see [01:43] the road map is really really expanding. [01:46] Um, and so I just want to give you some [01:48] uh tips here. So if you're going to take [01:50] the Claude AR certified architect [01:52] looking for my pen. Where's my pen? [01:54] There it is. If you're take if you're [01:56] thinking about taking this, take the [01:57] Claude code essentials. There's a 30% [01:59] overlap between them, but this one will [02:01] get your feet wet and so you're not [02:03] overwhelmed when you jump into this [02:04] course. This is not an easy course [02:06] despite them saying that it is. You need [02:08] to have developer experience and I will [02:10] help you through it. I'm a developer of [02:11] 20 years. So you are in very very good [02:13] hands. If cloud code essentials is too [02:15] much then you should be starting with [02:16] the genai essentials. And if you really [02:18] are struggling here then you should be [02:20] looking out for my cloud code from zero [02:22] which is expensive but it's really [02:24] designed to use the super plain language [02:26] and to break things down one to one so [02:28] that you can be successful. Um, in terms [02:30] of the ecosystem, I've thrown in my [02:32] Nvidia one in here because I believe [02:34] that understanding how um these LLMs [02:37] work uh like what takes it to deliver [02:39] them at scale is very very important in [02:41] data centers and that will help you [02:42] understand if you ever want to bring [02:44] your own model or just understand the [02:46] costs uh involved. But you can see [02:48] there's a lot going on there and uh you [02:51] do not need to take the claude certified [02:52] architect to do my uh claude code boot [02:54] camp when that comes out. Um but you [02:56] know it doesn't hurt, right? Let's talk [02:58] about how long it will take to pass. [02:59] Well, I would say if you're a beginner, [03:01] it's going to take 30 hours and it might [03:03] even take you longer because the [03:04] concepts here are so new, even for me, [03:07] that it was taking me time to basically [03:11] understand them. And so for experienced [03:12] people, if my course is 12 hours, even [03:15] if you two times it, you're going to be [03:16] taking 12 hours to do it. Um, so just [03:19] consider that. So maybe average time 12 [03:21] hours 50% lecture labs 50% practice [03:24] exams and you know study once one hour a [03:26] day should be a good pace for you. Don't [03:29] do this in a single day you'll burn out [03:30] and you won't keep the information but [03:32] do not take too long as it will leave [03:35] your mind. Okay. Um so what does it take [03:38] to pass exam? do the the video lectures, [03:41] but the main thing is do the hands-on [03:43] labs. And it's hard because I cannot [03:45] give you one-on-one instruction or one [03:48] uh onetoone instruction because LMS are [03:51] chaotic. They like to do things [03:53] differently, but you should watch my [03:55] videos and then try to figure it out [03:56] yourselves. What I'm providing you is my [03:58] developer knowledge, my developer [04:01] troubleshooting, and my judgment in [04:02] terms of what looks good and what [04:04] doesn't. And so you're watching them as [04:06] a reference to figure out how to apply [04:08] them and if you're getting the code that [04:09] you want. Uh the other thing that I [04:11] would just say is that um I do have the [04:13] code bases so you like it's all the code [04:16] is there for you. So if you just want to [04:17] run them and just inspect my code or [04:19] even try to prompt it to make it better [04:22] go ahead and do that but just understand [04:23] I cannot give one to one instructions [04:26] unless I spend a lot of time engineering [04:28] this stuff and then we're losing all the [04:30] learning value. Okay we got paid [04:32] practice exams on the platform. We [04:34] usually give you one for free. It's over [04:36] there on exampro.co at uhcca-f. [04:41] Where do you take this exam? Well, [04:42] Anthropic is using SkillJar to um serve [04:45] it. They technically have their own [04:47] course, but I'm going to tell you it's [04:49] not enough. It simply is not enough. Um [04:51] and it's very much research based where [04:53] I am giving you hands-on so you can [04:55] actually apply the knowledge. Um, and so [04:58] I it's a good pairing like if you want [05:00] to just round out and have a second [05:01] source, but uh you're going to struggle [05:03] if you just use them as your primary [05:05] source. And and I need to point out that [05:08] um this certification right now at the [05:10] time of this video is only available to [05:11] claude partner network. So everyone just [05:13] saw the name Claude certified architect [05:15] and said I want to go get it. But you [05:17] might not be able to get it because if [05:18] your company doesn't qualify for this [05:20] and they have crazy requirements to get [05:22] in, not crazy, but you have to have at [05:23] least 10 people in a certain size of [05:25] company, whatever, whatever, and you [05:27] have to be aligned. Um, uh, you might [05:29] not be able to get this. And if you [05:31] can't, Exampro has got you covered. Just [05:33] take ours and generate it out. Um, pass [05:35] it on our platform, our version of it. [05:37] And until you can get it, then you can, [05:40] you know, [05:41] uh, come back to here if this ever [05:43] becomes GA. and they might make this [05:45] generally available to everybody not [05:47] just in the partner network. So what is [05:49] the content outline? Well, there are [05:50] five domains. So we have agentic [05:52] architecture and orchestration, tool [05:54] design and MCP integration, cloud code [05:56] configuration and workflows, prompt [05:58] engineering, structured output, context [06:00] management and reliability. Uh notice [06:02] here we see claude code. This is not a [06:04] cloud code course, but the middle [06:06] section here has a lot of cloud code in [06:08] it. So 30% of this content comes from my [06:12] cloud code essentials. So if you've [06:13] already taken that course, you are doing [06:15] really well. So I think it's like [06:16] basically half of domain 2 and all of [06:19] domain 3 is coming from my cloud code [06:21] essentials. But the new content here in [06:23] domain one, especially domain five, are [06:25] extremely valuable, but they're [06:27] completely new concepts for most people. [06:29] And so uh the only way I can make this [06:32] content is I had to do it. In fact, I [06:34] found that often the docs were wrong, [06:37] the exam guide was wrong, the AI was [06:39] wrong. And so only by doing it by going [06:41] through the process did we find out the [06:43] real truth of these tool tools. Um and [06:45] so that will probably improve with time [06:47] but you know having the real truth is [06:50] you doing it for real. Okay. And so [06:52] that's why we have so many labs. Um but [06:55] yeah the domains are all weighted [06:56] differently as per usual. The passing [06:59] grade is 720. So it's much higher. The [07:02] other ones usually there's 70%. This [07:03] one's 72%. Um there are 60 questions and [07:07] you have 17 questions you can get wrong. [07:09] a huge pool of questions uh and lots of [07:12] opportunity to get things wrong. It's [07:14] just multiple choice. But despite being [07:15] multiple choice, the questions are very [07:18] verbose. Um it's like they're writing [07:21] solution architect professional [07:22] questions. I don't know why they did [07:23] that because it just makes it really [07:25] hard. Not in the sense that the question [07:27] they're asking is hard, but the ability [07:29] to parse it. And to me, that's a [07:31] mistake, but that's the choice they [07:33] made. Um the duration is two hours. You [07:35] have a lot of time, two minutes per [07:37] question. So take your time reading the [07:38] questions and absorbing what they're [07:40] trying to say. Your exam time is 120 [07:43] minutes. Your seat time is 150 minutes. [07:45] Technically, there's probably no seat [07:46] time because they're using Skill Jar. Um [07:49] as the as the delivery for their [07:52] certification. Uh but you should still [07:55] plan to review instructions. If there is [07:57] an online proctor, there probably isn't. [07:59] Uh and read and accept the NDAs or [08:01] complete the exam and provide feedback. [08:03] Okay. We don't know how long this course [08:06] is valid for unless it becomes generally [08:08] available to everybody, not just people [08:10] in the Claude uh partner network. We're [08:12] just not going to know. Um but anyway, [08:16] hopefully that gives you a clear idea [08:17] and I will see you soon. Cha chia. [08:24] Hey folks, it's Andrew and I just want [08:26] to give you considerations about this [08:28] course. So in terms of requirements, we [08:30] are not covering the setup of agent SDK [08:31] or cloud code. I have that in uh another [08:35] course which is my cloud code essentials [08:37] course. The reason I didn't include it [08:38] in here is um I just don't want this [08:41] course to get be bloated and have [08:43] repetitive content and so I'm trying to [08:46] uh reduce the amount of share and so I [08:48] felt that I have a very good robust uh [08:51] setup in another course and also if [08:53] setup does change I'll update in that [08:55] course and this will keep this course uh [08:57] fresh because I don't want people to [08:59] feel that this course is out of date [09:00] because the setup may have changed and [09:02] so that's been separated out for a [09:04] reason but it's still available to you [09:06] for free and paid. Um, so just go look [09:08] for that. The approach we're taking for [09:12] this is we're going to walk the exam [09:14] guide. I found that by just going [09:16] through and trying to do everything they [09:18] say in the order that they say it is [09:20] effective learning um for this [09:22] certification because of what they're [09:24] trying to show you. And I would say that [09:27] it's very important that we take an [09:28] implementation focused approach. I know [09:31] people are using the anthropic content [09:32] which is very conceptual based and [09:34] there's people failing because it [09:36] doesn't stick unless you do it. So for [09:39] me um I'm going to take an [09:40] implementation approach because I think [09:42] that will be the best result for you. Um [09:44] and so the idea here is we will look at [09:46] a code example and we will go implement [09:48] it. Okay. And I did mention that [09:49] anthropic does have free courses. They [09:52] are very conceptbased. So they have [09:54] really nice diagrams, but they're really [09:56] not showing you uh implementation um or [09:58] not enough implementation specifically [10:00] for this um certification. That might [10:02] change later on. Um but you know, I want [10:04] to cover the gaps that are not being [10:07] covered here. And uh you know, [10:09] implementation I think is going to be [10:11] what is going to be best for you, best [10:12] for everybody. Uh and so that's what I'm [10:14] going to do. Okay. Chowo chia. [10:21] All right. So in this um follow along [10:24] we're going to look at uh learning how [10:27] stop reason works and we're basically [10:29] learning also how to implement tool use. [10:32] All right. And so what I'm going to do [10:33] is make a new folder here. Um and I'm [10:35] going to call this one actually before I [10:37] do that just because I need to be [10:39] diligent about this. I'm say get init [10:41] add get add all get commit m [10:48] uh init hello world because I just want [10:50] to get us set up here. Okay. So, um I'm [10:53] going to push this repo later. I'll [10:54] probably make a video prior to this to [10:56] show you where it is. So, you already [10:58] know where this is, but at this point in [10:59] time, I don't actually have it pushed to [11:01] get. But anyway, we want a new folder [11:03] here. And the new folder is going to be [11:05] stop reason. Okay. And I want to [11:08] describe what it is that I want to [11:10] accomplish in here. So, um I'm going to [11:13] make a plan here. I just like to call [11:15] them plans. And I'm just kind of writing [11:17] up ahead. And we're going to ask Claude [11:19] to help us out because Claude knows a [11:21] good enough about agent SDK. It's not [11:23] always the best at it, but generally it [11:25] knows what's going on here. So what is [11:27] our what is our goal? Uh we want to show [11:31] the user how stop [11:35] reason works. [11:37] So um by showing so we want to show the [11:43] user how sub works within the claw agent [11:46] SDK. [11:49] Okay. So we need to see stop reason [11:55] for [11:56] uh tool use. We need to see stop reason [12:02] for uh end result. [12:06] We need to see how it's chained uh [12:11] appended. How the results [12:13] is appended. I want to use a weather uh [12:19] tool use example. [12:22] Uh we are implementing in Python. [12:27] Um we want to use the [12:31] uh lib [12:36] SDK parser [12:40] SDK parser [12:45] to format [12:48] logging information. Okay, so I'm going [12:50] to go ahead and do that. And by the way, [12:52] you know, if you get something [12:54] different, this repo's up there. You can [12:56] always copy my code and then try to run [12:59] the same thing if you're not getting the [13:00] exact same results, okay? because you're [13:02] just not going to be able to do it [13:03] unless we go the old school way and I [13:05] write this by hand. And I can write the [13:07] code by hand, but I'm assuming folks out [13:09] there, you want to leverage Claude as [13:12] much as you can and we will do that. [13:13] Okay. But I totally am comfortable doing [13:15] it the old school way. So I'm going to [13:17] go here and say um please execute the [13:22] plan for uh plan MD [13:26] um and generate the code into the stop [13:32] reason folder. [13:36] Okay, I probably should have told it [13:38] like the purpose of this is to show an [13:39] example, right? And so we're talking [13:41] about stop reason, but we're going to [13:42] also learn about tool use here while [13:44] this is going. Um, and so I'm going to [13:46] pause here and then we will see what we [13:48] come back with. If there's anything I [13:49] need to accept, I will accept it back in [13:51] just a moment. Okay, I'm just watching [13:53] this thing go. And right off the bat, [13:55] it's like it created a requirements txt [13:57] with anthropic. I'm like, wait, that's [13:59] not right. So, uh, I think we're going [14:01] to have to help it out here. So, [14:05] tab to amend. I'm going to hit tab to [14:06] amend and say, uh, look, I shouldn't you [14:10] be using [14:13] using the [14:17] for here [14:19] claude agent SDK. [14:24] This is what I'm confused about. And [14:25] we'll double check this here because, [14:27] you know, they're always changing stuff [14:28] on us and we want to keep nice up to [14:30] date here as possible. So I will just [14:32] double check here as we can't just fully [14:34] trust I guess we already have the tabs [14:35] open here. We can't fully trust this [14:37] stuff but it says claude agent SDK. So [14:38] that must be what it is. The claw the [14:40] claude code SDK was renamed cloud agent [14:43] SDK. So that is prop the proper name and [14:45] so it's confused on its implementation. [14:49] I'll go back over to here. I said the [14:51] claude agent SDK and so it's the first [14:55] thing we see here. So, I'm just going to [14:58] edit this requirements while this is [15:00] going because we'll probably keep using [15:01] this document. So, [15:05] uh yeah, that's fine. We'll come you [15:07] we'll come back to that in a moment. [15:08] We'll just say yes for the session so it [15:10] can work and so we'll be un under [15:12] uninterrupted. [15:14] And I'm going to go ahead and do this [15:16] here. Yeah, you can use Python. Sure. [15:19] And what I want to do here is just help [15:22] out with the requirements. So the first [15:23] thing is [15:28] use [15:30] must use the [15:32] agent cloud uh cloud agent SDK. [15:41] Okay. Must Python [15:45] 3. Uh I mean that's pretty obvious. [15:48] Python 3 uh must implement [15:52] must use [15:54] lib [15:56] uh [16:06] format messaging here. We'll go ahead [16:07] and do that. And this might help a [16:10] little bit. [16:13] We are creating [16:15] a code example to learn about stop [16:19] reason [16:23] scope. [16:25] We are learning about agent [16:29] claude agent STK and claude uh agent. [16:35] Yeah, we'll just say that that's fine. [16:37] Okay, [16:40] we don't need that. And so now we've [16:41] kind of adjusted that. [16:45] Here's trying to run it in an [16:47] environment. I don't know why it's doing [16:48] that. [16:54] Just use the local Python environment. [17:00] No, I don't need to do it that way. [17:02] Okay, that's silly. Um, and the reason I [17:04] say that is just because we could create [17:06] isol environments, but like we're just [17:07] installing that one thing, and I don't [17:09] want to be doing that all day every day. [17:10] I'm already on WSL 2. I don't need any [17:12] more isolation here. So, um, let's go [17:14] take a look at what it's produced and [17:16] see if this is even readable here. Okay, [17:20] so we're going to go all the way to the [17:21] top here. [17:23] And right away, I don't like this [17:25] because it's giving us um fixed weather [17:28] data. So, I guess I should have been [17:30] more particular and saying like, hey, go [17:31] use an API endpoint for that, which is [17:33] fine. But anyway, so here we have [17:37] Claudian uh options and the the client. [17:39] That's fine. We have a tool use block [17:41] and a tool result block. Okay. And they [17:44] have simulated weather results. We are [17:46] defining a tool. Get weather get current [17:48] weather for a city. And I mean this [17:51] looks like a decorator. So a decorator [17:53] in Python uh helps enrich functions. So [17:56] this has to be tightly coupled to this. [17:58] I'm not an expert in Python, but I know [17:59] decorators when I see one. And so this [18:02] is adding additional data that it needs [18:04] to know about this function here. And so [18:07] the function takes in args and then [18:08] we're going to parse the args. Right? So [18:11] here we're taking a city and it's going [18:14] to look based on the weather data if it [18:15] matches and then it's going to return [18:17] that information. So we have type text [18:20] um text result. Okay. Then we'll go down [18:23] below here. And so we have logger [18:24] helpers. Um I again I want to keep [18:27] everything within my logging tool. So [18:28] I'm not sure why we have that. We have [18:31] stop reason annotation. [18:34] So, I think what's going to help is [18:35] looking at the main loop here. So, we [18:37] have create SDK [18:39] MCP server. [18:42] Why do we need an MCP? Well, I guess we [18:44] could do it that way. [18:47] And so, I'm going to go over to here. [18:51] All right. I guess that's how it wants [18:52] to do it. That's fine. So, we have [18:53] weather tools. And so, we add a tool and [18:55] then we have MPC servers weather server [18:58] options. [19:00] So, you know, my thought as a as a [19:01] programmer, as a developer, is like, why [19:04] is it putting it into an MCP? I probably [19:07] wouldn't want to do that right off the [19:08] bat. So, there's that. [19:10] Here's explaining the stuff. I don't [19:13] need it to explain it. Okay, so here's [19:16] the problem with this this program. [19:18] Okay, first thing is like [19:22] I'm going to go say stop reason here. [19:23] Stop reason. Um [19:27] the problem with this program [19:30] is I don't want the program to actually [19:33] explain how it works. All we are doing [19:37] is outputting [19:40] key uh outputs to observe. Okay, that's [19:44] step one. Number one, uh the other thing [19:46] is you implemented [19:49] an MCP server. [19:51] Do we really need that? [19:54] to show tool use. [19:58] I mean, isn't there isn't there [19:59] something simpler? [20:04] Something simpler. And then the other [20:06] thing is like can't you use add API [20:11] endpoint [20:12] public to get weather data? [20:16] So, we'll go off and we'll do that. [20:18] While it's doing that, let's go look it [20:21] up ourselves, okay? because there's [20:23] nothing stopping us from reading the [20:24] code. But I wanted to show you like [20:28] because I keep telling people they're [20:29] like, "Yeah, you know, Claude can do [20:31] whatever." I'm like, "Yeah, but it [20:32] doesn't know itself." And you still have [20:35] to go look things up. So, what we'll do [20:38] here is we'll take a look [20:41] and we'll go to the API reference. That [20:43] might be a good starting point. This [20:44] usually what I do for projects is I go [20:46] there, say, "Yes, I really need to make [20:49] a ding or something so I can hear it as [20:51] it's doing that." We'll go over to here [20:53] and I'm going to see does it have any [20:54] information here that might help us. No, [20:57] not really. That's not going to be [20:59] useful. We'll go back to the developer [21:01] guide. [21:02] Um, [21:04] usually I just go straight to API [21:05] reference and usually that has all the [21:06] information I need. [21:08] But this does not look good. So, we'll [21:11] go back a second here. [21:14] Uhhuh. [21:16] Go back a second here. [21:19] I remember seeing a lot of stuff here [21:20] before. Where is it? [21:23] I guess we'll just scroll on down till [21:24] we find it here. So, what I'm looking [21:26] for is that tool use implementation. [21:30] I remember having like a bunch of boxes [21:32] and saying like, hey, if you need to do [21:33] this, you need to do that. Give me a [21:35] moment to go find that. Okay. All right. [21:37] So, one thing that's confusing like here [21:38] I'm under tools. We're seeing anthropic, [21:41] but there's also the agent SDK. So, [21:43] which is it? Right? And so, that's [21:45] something we're going to have to figure [21:46] out in right now. So, just give me a [21:48] moment to figure out why there's a [21:49] discrepancy there. [21:54] Hey, this is Andrew and in this video [21:56] we're going to take a look at setup for [21:58] both um agent SDK and Claude Code. [22:00] Technically, I have them already both [22:02] installed and so the process might look [22:04] a little bit different. If you need it [22:06] completely from scratch and seeing that [22:07] process, you can absolutely do that by [22:09] looking at my Claude Code essentials [22:11] course uh which shows multiple [22:13] installations on multiple machines. And [22:15] if for whatever reason this is out of [22:16] date, go to that other course because it [22:18] will have the most upto-date way of [22:21] doing that. So we'll go ahead and say [22:22] cloud code install. It's going to be [22:24] dependent on what kind of system you're [22:25] on. So I'm on WSL 2. And so that's going [22:28] to be with Linux. And so what I'm [22:29] looking for here is the installation [22:31] instructions. Um and so here they are. [22:34] And so we have these modes. So we have [22:36] native install, homebrew, and windget. [22:38] And they recommend it. In general, [22:40] people almost always recommend the in uh [22:42] the native installs because they're [22:44] going to install directly from the [22:46] source or they're going to compile and [22:48] be the latest up to date. If you take [22:49] these other ones, uh the releases might [22:52] be old and so you might run into an [22:54] issue where you're having weird API [22:56] issues because they're old and um uh [23:01] yeah, I would just say go with the [23:02] native install. So, I'm on WSL, so I'm [23:05] going to go ahead and grab this [23:07] installer here. I have a new folder I've [23:09] made for for this course and we'll go [23:11] ahead and install it. Again, it's [23:12] already installed, but we will run in [23:14] here so you have an idea of that [23:16] installation process. [23:19] So, it's just taking a moment there. [23:21] See, it already says that it's installed [23:22] and we have version 2.1. To launch up [23:25] Claude, we're going to type in Claude. [23:27] Okay. And it thinks I'm already logged [23:28] in. So, I'm just going to log out here [23:30] for a second. Log out. [23:32] I might not actually be logged in. I [23:34] actually didn't check. But uh what I'm [23:36] constantly doing to see is I'll type in [23:38] uh claude o status and that will tell [23:41] you whether you're logged in or not. Um [23:43] again in my cloud code essentials we [23:45] have a lot of stuff that covers [23:46] authentication and all these uh little [23:49] things. So if you really want to know [23:50] all of it you can go ahead and do that [23:52] over there. But we'll launch cloud. I'm [23:54] going to go ahead. It wants dark mode. [23:55] Sometimes it'll ask me on setup. If it [23:57] doesn't do this uh all these steps and [24:00] you just type for/lo but anyway I'm [24:02] going to go here to my subscription. [24:04] probably one of the best ways that you [24:05] can use claw because it's the most cost- [24:07] effective way right now for learning. [24:09] And so here it has opened up a new [24:11] window in a Chrome browser. It has a [24:13] weird looking Chrome here. Um, and so [24:16] I'm going to ignore this one because [24:18] it's weird. I'm going to go ahead and [24:20] copy that. Make a new link. Paste it in [24:22] here. It seemed like that was like an [24:24] isolate Chrome browser running. I'm [24:26] going to hit authorize. We'll go back [24:28] over to here. Copy the code. We will [24:30] paste it in. Hit enter. [24:33] Okay, now we are successfully logged in. [24:35] Enter. Enter. If you didn't see that uh [24:37] that whole setup, you just type in [24:39] login. It will do the same thing. Okay, [24:42] so that's part one of getting uh Claude [24:45] there. I just want to make sure that it [24:46] is actually working. And so I'm just [24:48] going to say uh I'll just talk to Claude [24:51] here. Can you create me a readme.md [24:55] file um that is completely empty [24:59] in this folder. [25:02] Okay. And so I'm going to go ask it to [25:04] do that. Just making sure that it works. [25:07] And we'll say yes, we can do that. [25:10] We're using the set model down below [25:12] here. We can change our models if we go [25:14] here and type in like haiku and things [25:16] like that. Um, again, as we work through [25:19] implementations, I will change things [25:21] and just follow along as we go here. And [25:24] I'll go over to here. And so, we should [25:26] have a folder or file. [25:30] Um, I'm not seeing it. So, I'm just [25:33] going to do code period. [25:36] Sometimes that helps it open up into [25:37] context. [25:41] And so, here it is. Okay. So, that [25:43] works. That's great. We'll go back over [25:45] to here. And so, step one is done. We [25:47] have Claude installed. It's working. Uh [25:49] the second part is the agent SDK. So [25:51] let's go ahead and take a look at that. [25:53] So um we'll go over to Claude uh agent [25:57] SDK. [26:00] Here it is over here. [26:02] And [26:04] uh yeah, it's been renamed. Yeah, that [26:07] doesn't matter here. And so we're just [26:09] looking for the install. Here it is. [26:10] Okay. So we have pip install agent SDK. [26:14] Okay. I'm going to go ahead and hit [26:17] enter [26:18] uh here. [26:20] Yes. [26:25] Okay. [26:26] And I don't think you can use the SDK. I [26:30] was thinking about like can you use [26:31] subscription? I don't know. I will [26:32] always use it with the key as I was just [26:35] thinking about that because um well [26:37] normally you'd be using this in a [26:39] production environment. So it would [26:40] never be tied to a subscription, but I [26:43] just want to check if we can use it in [26:44] development uh with subscription in just [26:46] a moment. All right. So the suggestion [26:49] is is that we can actually use your [26:51] subscription. I don't know if it's true, [26:52] but we'll have to validate it. Uh but [26:54] the idea is that if we are logged in [26:55] with the claude uh CLI, so if we go back [26:57] over to here, we'll have to test this in [26:59] a moment. I mean, we are testing it [27:01] right now, I should say. And so we'll do [27:02] this. And so it says that I'm logged in, [27:04] right? And so the idea is that if I was [27:07] to uh use any kind of code here, it [27:09] should be picked up. So I'm going to go [27:10] ahead here. We'll make a new folder and [27:12] we'll just say hello world. It'll be our [27:13] first hello world of using the agent [27:16] SDK. I'm probably going to try to use [27:18] the agent SDK more than um cloud code. I [27:21] guess we'll find out when we go through [27:22] this uh course. But just because I we [27:25] can use cloud code to help us write code [27:26] for agent SDK, but um I think that this [27:29] course especially this first section is [27:32] really about implementation for um [27:35] companies and things like that. So I [27:36] think we'll benefit better driving it [27:38] that way. But anyway, so um here we have [27:41] an example code. I'm going to go ahead [27:43] and copy this here and we will take a [27:46] look at what we're looking at. Well, [27:47] first of all, I got to make sure I have [27:48] a file. So, we'll go here and this will [27:50] be hello.py or just main.py, I suppose. [27:53] Main.py. And we'll go ahead and paste [27:56] this. [27:58] Okay, I'm just going to zoom out here [27:59] for a second. And so, we have uh the [28:02] agent SDK. Okay, here we have a message [28:05] fix uh find and fix the bug in the [28:07] off.py. So, I'm going to change this to [28:09] like hello world.rb. [28:14] And I'm going to I'm going to make a [28:16] mistake. puts puts hello [28:20] world. [28:22] Okay. And so I have a few mistakes here, [28:25] right? So what are the mistakes? Uh it's [28:28] should be puts, not puts. Uh it should [28:30] have two um double quotations or singles [28:34] and then it shouldn't have a parentheses [28:36] or it should. Right? So we'll go over to [28:37] here and say find and fix the bug in the [28:40] hello world.b. [28:44] Okay. And so we are giving it the option [28:46] to read, edit, and use bash in order to [28:48] accomplish its tasks. And then it will [28:50] print out uh the results down below [28:53] here. I'm just going to fix indentation. [28:55] I like two spaces, not four spaces, [28:57] which is what the creator Python wanted [28:59] you to use, but was never enforced that [29:01] way. And so this should do something if [29:04] it works. So I'm going to go ahead and [29:06] run it. Uh if you don't know how to run [29:07] it, you could just ask cloud to do it. I [29:09] one thing I would like is the [29:11] requirements.txt. So, I'm going to go [29:12] over to here and just say, um, hey [29:15] Claude, we don't say hey Claude, but uh, [29:17] in my hello world, [29:20] uh, I want to add a requirements.txt [29:24] that my main.py will need. Can you [29:29] figure out what I need? Okay. And I [29:32] mean, we already know what it is. It's [29:33] right over here. Um, it's this. And so [29:37] probably would just be that in an empty [29:39] file, but I figured, you know, let's try [29:41] to just get a little bit warmed up here [29:45] uh for cloud API. Sure. Okay. [29:53] And we'll give it a second here to get [29:55] that implemented. [29:57] Again, just if you're really really [29:58] rusty with this stuff, we're just trying [30:00] to get going here. That looks good to [30:01] me. We'll say yes. That's all I was [30:03] expecting. And so there it is. And so [30:06] I'm going to go ahead here and we're [30:07] just going to make a readme. [30:11] And this one's going to be [30:14] um you know cd hello world. [30:18] You write your own uh documentation. I [30:21] sure do if that's what you're thinking. [30:23] Python hello uh hello or just main.py. [30:26] So I have main.py. Okay. So what I'm [30:29] going to do here is just quit out of [30:30] here. No reason to waste credits for [30:33] such such a simple action. and we will [30:35] cd into our hello world folder. I'm [30:37] gonna try to keep these um lowercase [30:39] underscore. [30:41] Just gonna go ahead and fix that there. [30:42] There we go. Uh it won't let me do it. [30:44] Well, that one's just going to be like [30:45] that. I'll try to not do that in the [30:47] future, folks. Um but we'll go ahead and [30:49] we'll just type in python main.py. [30:52] And it's saying there's no file there. [30:54] Uh I think there is. [30:59] Oh, it did change. It did change. It's [31:01] not in my UI. Let's go ahead and give [31:03] this a refresh. It was weird. like we [31:04] don't we can't change it. And guess [31:05] what? It did change it. That's fine. I I [31:07] want that. Okay. So, we'll say uh Python [31:10] main.py. There we go. [31:13] And I don't see any output. So, we'll go [31:14] back over to here. We'll just print some [31:17] stuff. So, let's just say print [31:19] uh running uh [31:23] fix agent. [31:26] And we'll go ahead and try that again. [31:29] We're getting no output. Oh, [31:31] interesting. Um, [31:34] that's kind of strange for such a simple [31:35] app. And I'm just going to say here, [31:38] running main.py [31:41] does not print to uh terminal. What's [31:45] wrong? And I'll go in the front here, [31:47] just type in debug. Debug is just a um I [31:50] guess it's a skill that specializes in [31:52] debugging, but this should work. I'm not [31:54] sure why it's not uh picking it up. It's [31:56] kind of embarrassing that this basic [31:58] script is not working right off the bat. [31:59] Like this print should be working. [32:00] Absolutely. Um, my primary language is [32:03] Ruby, so sometimes I have a lack of [32:06] confidence in Python, but uh, we'll give [32:09] it a second and it'll tell us what the [32:10] problem is. Okay. [32:13] And Oh, right. Yes, yes, yes, yes. Well, [32:15] did you even have that over there? [32:18] Oh, I missed it. Okay. And that makes [32:20] sense with async.io. I actually have a [32:22] coding course I never publish where I [32:24] cover async.io. So basically um if you [32:27] have asynchronous stuff you have to then [32:28] run it within the um uh the as async [32:32] tool is how it works. So that's good [32:34] that it caught that. We'll go ahead and [32:35] say yes. I don't need it to run it but [32:37] it might try to run it for us. [32:40] Great. And so I'm just going to stop [32:41] this here. We'll type in clear and we'll [32:44] go back and hit up. [32:47] And it now executes [32:50] um [32:52] which is fine. So, here we have a bunch [32:55] of stuff, but it's kind of hard to read. [32:57] So, I'm going to go and take this here, [32:59] and I'm going to feed it back. Oops. [33:03] Feed it back here. We're going to get [33:04] set up here. So, I'm going to go ahead [33:06] and just say, um, back to Claude. I'll [33:09] go back a level. This is where we really [33:11] should be. Every time you trigger [33:12] Claude, it creates a project in your [33:14] home directory. So, I wanted to try to [33:16] stick in the same place. So I could go [33:18] ahead and just say um so I got back uh [33:21] data for [33:23] my um [33:26] hello world. Hello world [33:29] main pi [33:31] but it's not human readable. [33:35] I'm going to change this to plan. I hit [33:36] shift shift tab. So now and hit it [33:39] again. So I'm on plan mode. uh uh can [33:42] you tell me how we can make the uh this [33:45] output useful to a human [33:49] and can we create our own uh Python uh [33:54] library for parsing it? [33:57] Okay. And so I've given it the format [34:00] and I mean it might come up with [34:01] something really simple like uh pretty [34:03] print where it will just print the [34:04] structure. But what I'm hoping is that [34:06] maybe we'll have this little file and [34:07] then we can uh use it in other places, [34:09] right? So, I know it seems like we're [34:11] getting a little bit already advanced, [34:12] but that's just how we roll here. Let's [34:13] go ahead and hit yes. I just wanted to [34:15] have full access there. Notice why I [34:17] said yes always. If I go over to here, [34:19] it added this. It says always let cloud [34:21] API to do stuff, which I don't remember [34:24] getting cloud API before. I just [34:25] remember it working. So, I'm not sure [34:26] why it needs to do that now, but uh [34:29] that's fine. [34:33] And so it's off to the races. Uh trying [34:35] to execute here, which is fine. I don't [34:38] want that. Microsoft Foundry is always [34:40] popping up on me. They're like, "Hey, do [34:41] you want to use Microsoft Foundry?" No, [34:42] I'm using Claude right now. Go away. Uh [34:45] what is it? Uh what's the insult on [34:48] Microsoft these days? Mic Microsop [34:51] Microsop. [laughter] [34:53] And so, um [34:56] what we're looking for here, it's really [34:58] taking a long time here. It's not that [35:00] complicated what I'm asking for it to [35:01] do. I'll pause here and we'll see what [35:03] it comes back with its plan. Okay. All [35:06] right. And so I think we got a plan [35:08] back. Let's take a look at the plan. And [35:11] we have the hello world main pie script [35:13] uses the claude agent SDK and calls the [35:15] print on each yielded by query. This [35:18] produces raw Python object repo strings. [35:21] Um the user wants human readable [35:23] terminal output a reusable Python [35:24] library. That's true. And so here it's [35:26] saying SDK uh init.py [35:30] um formatter main.py. PI and then we [35:33] have SDK parser format. So here it's [35:35] suggesting the format. So each handles [35:37] these parts. It puts the parser there. [35:39] Export a single format message. And by [35:42] the way, you know, if you're like, I [35:43] don't have the exact same code as you, [35:44] Andrew. This repo will be on GitHub. I [35:47] will show it earlier in the course. So [35:48] you can grab it and then you can just [35:50] grab the parts to have par. So So like [35:53] this thing, if I'm using it throughout [35:54] the course, which I don't know yet at [35:55] this point of time, then you could grab [35:57] that code and use it. Okay. But it says [35:59] export a single format message string. [36:02] And that looks good. The only thing I [36:03] would say is like it's pretty close. [36:07] Good. But I plan to use this [36:11] library in multiple uh folder projects. [36:16] So maybe it should live in um a lib [36:21] directory at the top of the project. and [36:24] make sure to give me a readme [36:29] so I know how to import it for uh [36:34] projects [36:36] within this repo. Okay, so we'll go [36:39] ahead and do that. And that's the only [36:42] change I'd like. And so yeah, you have [36:43] to read your plans if you want to get [36:46] code that you like. Um and so we'll give [36:48] it a moment there. I didn't tell it to [36:49] go execute. It's supposed to update the [36:51] plan, but we'll give it a moment to come [36:53] back here. Okay. All right. Our plan is [36:55] back. And by the way, people don't [36:57] complain if you see me doing this. When [36:59] I made my clock to essentials, people [37:00] were like, "Andrew, you're right. You're [37:02] using the thing and then you're pausing, [37:04] whatever, whatever." Like, "What else [37:05] would I be doing? This is what we're [37:06] going to be doing." So, you know, don't [37:08] don't give me those comments, okay? [37:10] Because that's the work. Anyway, let's [37:12] take a look and see the plan updated. [37:13] So, here is the lib SDK parser and we [37:16] have the net formatter and the readme. [37:18] Uh, I don't know what the output is. [37:19] Like, it'd be nice to be like, what does [37:21] the output look like? Um, but I would [37:23] rather just test it and find out. [37:25] Explain what the library does. Yeah, I [37:28] don't know what the output looks like. [37:29] And we have a verification uh set up [37:31] here. So, we'll go ahead and say yes. [37:33] Um, auto accept. And so, now it's going [37:36] to go ahead and implement it. And we'll [37:38] be back here in just a moment. Okay. [37:41] Okay. So, it looks like the code has [37:42] been implemented. And so, now we have, [37:44] we'll give this a refresh here. We now [37:45] have our lib directory. Um, here's what [37:48] it's done. So, did it actually test it? [37:50] now prints out formatted message to use [37:52] as another project. Whatever, whatever. [37:54] Let's take a look at the read me. Did it [37:55] do what I wanted here and give me the [37:57] simple instructions here. I mean, it [37:59] gave me a lot of information that did [38:00] not ask for. Um, converts noisy output [38:03] like this. Well, hey, at least it's [38:04] going to tell us. Oh, to this. Okay, so [38:06] now we can actually see what it looks [38:07] like. Um, and we have import options and [38:09] stuff like that. Another good reason [38:11] that we did this is that then the AI can [38:12] pick up on this. The only thing that I [38:14] will do here is just change the readme [38:16] to a clawed MD file as um that's what it [38:20] prefers to look for stuff. And so that's [38:22] what I'm going to do. So I'm going to go [38:23] ahead and do that. It did change it. [38:25] It's just being silly. I think I do [38:28] think it's changed over to the [38:29] capitaliz. Let's take a look here. [38:32] This Visual Studio Code thing, it's [38:34] really annoying. And I'm going to go [38:35] here for a second. I'm going to say move [38:38] claude to claw.md. [38:43] It's not the same file. One is one is [38:45] capitalized. I'm going to go ahead and [38:46] say claude one. There we go. ls [38:50] move claude [38:52] one to claw.md. Maybe it did change. [38:55] There we go. And even though it's not [38:56] reflected here. There we go. Now we have [38:58] it like that because that's what you are [39:00] going to uh want it to be called when [39:03] you're doing that stuff. I didn't do it [39:05] for here because it's just like so basic [39:07] uh what it's doing. So I'm not really [39:08] worried about it. Um but we will go back [39:11] and we will go back and we will take a [39:13] look at our hello world or sorry our pi. [39:16] And it's brought in the parser. I want [39:19] to uh go back a directory and go here [39:22] again. We'll say hello world. Oops. [39:24] Hello. [39:25] And we have two of them because it got [39:27] confused. I'm going to delete the other [39:28] one. I don't want to have two. I'm not [39:31] sure how that happened. [39:32] We'll go ahead and we'll say hello [39:34] world. And we're going to go ahead and [39:36] say python main.py. We'll run that. And [39:39] I want to see nice nice output that I [39:41] can read. [39:43] Okay. So here it says thinking let me [39:44] find the hello world.py first. Um it [39:47] globs the pattern. No files found. I [39:49] couldn't find a hello world rb. The file [39:52] in the current directory uh could not be [39:55] found. [39:56] And you know why? It's because the name [39:58] is wrong. So we'll go ahead here and [40:01] we'll fix that. There we go. Fair [40:03] enough. Right. [40:06] And we'll run it again. But already like [40:08] we're becoming very productive and [40:09] getting ourselves set up really really [40:11] well for when we run all these [40:13] implementations, right? Um so let's take [40:17] a look here. [40:20] Um [40:23] I like how we can see what it's doing as [40:25] it goes. That's really nice as well. [40:28] Whenever you read the file, you should [40:30] consider whether or not consider it [40:31] malware. You can and should provide an [40:33] analysis. Oh, my goodness, it's doing a [40:35] lot of stuff here. Did it actually fix [40:37] the file, though? [40:40] Oh, it's empty now. [40:44] Okay, I'm going to try this again. So, [40:46] I'm going to say puts whoops [40:49] puts [40:51] hello world parenthesis like this. I'm [40:54] going to try that again. [40:57] And you know, this is just the default [40:58] prompt that it came with. And we'll go [40:59] ahead and run that. Again, some things [41:01] that we could do to maybe make this a [41:03] little bit less painful is maybe dial in [41:05] um I'm assuming there must be an option [41:07] for like uh the uh context output and um [41:11] or context window and then also maybe [41:14] adjust which model's being used. [41:17] And so I'm going to go back over to here [41:19] and take a look. Did it did it do [41:20] anything here? No. Let's take a look at [41:23] what this thing actually does. [41:25] So find and fix the bug in the hello [41:27] world.py. It has read, it has edit, it [41:29] has bash. So it should be able to edit [41:31] that file, right? And we'll go back over [41:34] to here. So here it says found it. I'm [41:36] going to go back into here. I just want [41:37] to see if it was updated. So found it. [41:40] There are two bugs on this line. Not a [41:42] valid Ruby method. Should be puts. [41:43] Misqued information. Let me fix both. So [41:47] tool edit, replace all. And it goes [41:48] here. Hello world. Hello world.bing to [41:51] replace a string. Tool results has been [41:53] successfully updated. I'm going to close [41:54] this out. Make sure that it actually was [41:56] updated. It doesn't look updated. Fixed. [41:58] Here's a summary of the bugs that were [41:59] corrected. [42:01] Okay. And the only other thing I might [42:03] do here because I'm thinking like, okay, [42:06] we had something that happens here, but [42:08] I want this logger to log out a time [42:10] stamp because then I can feed that data [42:11] back in here for debug to Claude when [42:13] things aren't working correctly. Again, [42:15] let's just get set up as best we can [42:16] here so our lives are easy. And I'm [42:19] going to go and say um [42:22] uh here for the SDK parser um the SDK [42:28] parser [42:30] the [42:32] lib SDK parser. [42:37] I'll just sell it. I want my lib SDK [42:41] parser [42:43] uh to dump logs [42:46] in a log directory [42:50] uh relative [42:53] to the project that is using it. So um [43:01] let's say I run my hello world [43:06] hello world uh main.py. [43:09] I want a log [43:12] in the um [43:15] hello world [43:17] logs [43:20] and I will have like time stamp like [43:22] human [43:24] ISO 20 ISO [43:27] date um human readable [43:31] time stamp.log [43:33] log. [43:35] Okay, we are for each run [43:39] of main we get a new log. Okay, so um [43:47] that's what I want. And [43:53] so when I run so I run in hello main. So [43:56] basically anywhere. So yeah. Okay, there [43:58] we go. So we'll go ahead and hit enter. [43:59] And I want it to go and update that. And [44:01] so that will give us an easier way [44:03] because right now it's not working and I [44:05] want to know why it's not editing. And [44:06] so that will uh make that work really [44:09] well. I'm going to say yes again for [44:10] these. Um I tend to just kind of accept [44:12] these as they go. Um it's just a means [44:15] to how we can do that. It's weird [44:17] because I said yes and then I thought oh [44:19] maybe for that session that's what it [44:20] was saying. Okay. But we'll be back here [44:22] with the newer code and then we'll test [44:24] it out. So here we have a code update [44:26] and again remember this thing here. uh [44:28] you can make your own but if you want to [44:29] have par with me so things are one to [44:31] one when you're doing the other parts I [44:33] mean your code's not going to be the [44:35] same but at least this part will be the [44:36] same then uh you know just grab this [44:38] from my codebase and put it in your your [44:40] stuff here so capture the time stamp get [44:43] the logs and write um sure yeah that [44:48] seems fine we'll go ahead and do that [44:52] and [44:53] yeah and so now it's just we have format [44:55] message log message right so we have two [44:57] different things [45:00] which is fine and then it's bringing it [45:02] in here. So we have log message. Okay, [45:04] sure. [45:09] And we will go ahead and print that. [45:10] Yep. [45:12] It's going to test it as well. [45:18] So that seems fine. I'm going to go and [45:21] quit out of this. We're already in the [45:23] hello world. We'll go ahead and run [45:25] that. And [45:28] I want to go here, give this a refresh. [45:30] We do have logs. So there it is. And so [45:32] now if that file has not changed, we can [45:34] now feed it back into it. And so that's [45:36] going to make it really, really useful [45:37] for debugging. Okay. So here I'm going [45:39] to go back into I'm just going to go [45:41] back a directory clot. I'm trying to [45:42] stay in that one. And I'm going to say [45:45] debug. Um so [45:48] I'm going to provide the log. [45:50] So say um [45:54] describe the issue. So um I am using [45:56] agent SDK. It's supposed to [46:03] edit a file. It does not. Um here [46:09] uh here [46:13] check the logs and help me. So we'll go [46:17] here and say hello world [46:20] logs [46:22] like that. Okay. And the idea is now [46:25] like it has logs. So like it has all the [46:27] information about what it needs to to [46:29] figure that out. Um and so we will wait [46:32] a moment here and see what we get. Okay. [46:35] All right. Let's take a look here. So um [46:37] what do we have? So the edit actually [46:39] did work. The logs show the edit was [46:41] successful. The file currently confirms [46:43] this. It already has this. The likely [46:45] reason it doesn't edit uh when you run [46:47] it is because the bug is already gone. [46:49] No, that's not the problem. So, I go [46:51] over to here. [46:53] Oh, it's fixed. What? [laughter] [46:56] Well, whatever. At least we have this [46:59] set up really really nice. I'm not sure [47:00] what happened there. Maybe it's just [47:01] like a um a caching issue here with [47:03] Visual Studio Code, but it did force us [47:06] to have this nice setup and have a [47:08] debugging process. So I would say that [47:10] we are set up for um agent SDK and cloud [47:15] code. So if you're at this point too, [47:17] then you are set up as well. Um if you [47:19] need more setup instruction, if this is [47:20] not enough, I got more in that cloud [47:22] code essentials course which is very [47:24] specific about the tooling where here [47:26] we're looking at um implementation of [47:29] the stuff that um anthropic wants us to [47:32] learn. Okay, there you go. [47:40] Let's talk about what is claude code. So [47:43] claude code is an agentic coding tool [47:45] and it reads your codebase, edits files, [47:47] run commands and integrates with your [47:49] development tools. You can uh find cloud [47:51] code at code.cloud.com. [47:54] It's available in your terminal ID, [47:56] desktop app, browser, and more. So it's [47:57] across multiple services which we'll [47:59] talk about. Uh and so it's going to look [48:02] a bit different and work a bit different [48:03] depending on where you use it. Here is [48:05] the most common way to use it which is [48:07] via the terminal and probably the most [48:08] natural way to use it because it is a [48:10] CLI tool. Uh its use cases include [48:13] things like automating your workflow uh [48:15] uh uh you keep putting off. Build [48:17] features and fix bugs. Create commits [48:19] and pull requests. Connect your tools [48:21] with MCP. Customize with instruction [48:23] skills and hooks. Run agent teams and [48:26] build custom agents. Pipe scripts. [48:28] Automate with the CLI and work from [48:30] anywhere. The thing that people are [48:32] really interested in is the agent teams [48:34] where you basically have a bunch of um [48:36] agents working together in a team uh to [48:40] accomplish a task which we definitely [48:41] cover in this course. Um so look forward [48:43] to that. Uh the main thing is that cloud [48:46] code is a CLI program. It's a code [48:48] harness. It's an agentic coding tool but [48:51] at the at the end of it it's just a CLI [48:53] program. Okay. And so by nature it runs [48:56] in your terminal. Um, cloud code is [48:58] specialized to write code, but it can [49:00] also write docs, run builds, search [49:02] files, research topics, and more. And [49:05] cloud code uses an agentic loop, [49:08] multiple models, specifically the claude [49:10] code models. We have tool calls uh to [49:13] achieve your desired goal. We are going [49:15] to dive into all of these things. Okay. [49:23] All right. Let's take a look at the [49:24] agentic loop. If you've learned about [49:26] Asians, you probably already know this, [49:27] but we'll cover it anyway and [49:29] specifically in the context of cloud [49:31] code. Um, so there's some very specific [49:33] things or configurations for it. Um, and [49:36] so with an agentic loop, the idea is [49:38] that you are going to invoke cloud code. [49:40] You're going to ask it to do something [49:42] and that's going to start the loop. Uh, [49:45] and so the loop is over here. And the [49:47] idea is that at the start of your [49:49] request, there's going to be a very [49:51] specific model being used. It's going to [49:53] be opus haiku or sonnet probably sonnet [49:56] as that is the default and it will [49:59] continue on from there. Okay. And there [50:03] are three phases that um uh the cloud [50:07] code utilizes to execute its agentic [50:10] loop. We have gather context, take [50:12] action and verify results. And it loops [50:15] in on itself. So it's going to [50:17] continuously do this until it achieves [50:18] its goal. Um, all of these things can [50:21] call out to tools. So that is the idea [50:25] of where it will call out to something [50:27] that it doesn't know how to do um or [50:30] knowledge that it does not have and it's [50:32] going to reach out to be able to do more [50:35] or just interact with your system, [50:36] right? Because the LM is in its own [50:38] little box. It would have to call out to [50:40] a function in order to interact with uh [50:43] your developer environment. Okay? And [50:46] the great thing is that at any time you [50:48] can interrupt this loop and give it [50:49] corrections and then eventually it will [50:52] decide that it has met the goal or needs [50:55] more feedback or it doesn't want to run [50:57] anymore to not waste your money. Uh and [51:00] that's that. But we will dive in a [51:01] little bit more on the tool calls and [51:03] the models for this. Uh okay [51:11] let's make sure we understand what tools [51:13] are. So tools are code functions that an [51:17] agent is aware of and can invoke to [51:19] complete their tasks. And when I say [51:20] they're code functions, I literally mean [51:22] it's just a code function. If you know [51:24] cloud computing and you've ever uh used [51:27] a llama before where it's an isolate [51:29] piece of code, this is what it is. Um, [51:31] and so the reason why uh an LLM needs to [51:35] interact with a external function is [51:38] because it can only do so much within uh [51:40] itself, right? It can just produce text. [51:43] And so it needs some way of interacting [51:46] with external programs or maybe there's [51:49] something that is uh something that is [51:51] deterministic that you can use code for [51:53] and that's what it will use tools for. [51:56] So let's use an example of uh a tool [51:59] that might be called in each stage of [52:01] the gentic loop for clawed code. So [52:04] let's say it's trying to gather context. [52:06] Um and let's say the prompt is trying to [52:08] fix a failing Python unit test. So for [52:11] gather context, it may attempt to read a [52:13] file. So it's going to try to read the [52:15] test payment.py. The um agent knows what [52:20] tools are available to it. It knows the [52:22] name of the function, what its inputs [52:24] are, and what it expects back as output. [52:26] Okay, so that's already rigged up for [52:29] it. But here for gathering context, it's [52:32] going to read a file, which is the test [52:34] file. And now it's going to take an [52:35] action. So it's going to go ahead and [52:37] try to edit the file. So it will need to [52:40] know where that file is and what change [52:42] to be made. Then it's going to verify [52:45] the results. So it will run the command [52:47] pi test and then it will expect an [52:49] output and track what that output is. [52:53] Okay. But uh that's just a very simple [52:55] example of one prompt through there. If [52:57] it helps, we're going to just look at [52:59] more examples. Not the functions [53:00] themselves, but just what could be in [53:02] each category. So for gather context it [53:04] could be reading files, searching APIs, [53:08] querying databases, uh codebase [53:10] scanners. Technically this is rag right [53:12] here. So retrieve uh retrieval augmented [53:16] uh generation where it's going out and [53:18] grabbing data from a database to enrich [53:20] its context. That's a rag. Okay. Uh we [53:24] have system state queries for that. And [53:27] I don't really even talk about rags in [53:28] this course because it is the concept of [53:30] agents and stuff like that, but it just [53:31] kind of seamlessly happens here. So we [53:33] don't even think about it when we're [53:34] working with cloud code. Okay. But for [53:36] take actions, it can edit code as we [53:38] saw, run a command, write files, call [53:41] APIs, execute scripts. We have verify [53:44] results. So we can run a test, we can [53:47] compile code, we can query output, we [53:50] can expect logs, we can compare results. [53:52] So the cloud code has built-in tools uh [53:56] in five categories. They have file [53:58] operations, search and find, execution, [54:01] run, web search, uh code, uh code [54:05] intelligence. So within cloud code, can [54:07] you make your own tools? Um I believe so [54:11] like in the context of skills you can. I [54:13] literally can't remember at this point, [54:15] but if we can it will be covered in the [54:17] course. If we can't then you just won't [54:18] hear me mention it. Um but anyway that [54:20] is what tool call is and it's happening [54:22] underneath and it's really driving a lot [54:25] of the interactions without tools no [54:27] code would be changed right nothing [54:29] would be happening it would just be [54:30] talking to you but there you go okay [54:37] all right let's talk about u models in [54:40] particular for the agentic loop [54:41] specifically for clawed code and that's [54:43] the fact that when it executes it's [54:46] going to use the same model for all [54:48] phases of the agent authentic loop and [54:50] so you have that opportunity to change [54:52] it at that point in time. Uh you have [54:55] different model choices. You have the [54:58] default option which is just going to do [54:59] its best guess in terms of what it [55:01] should use. You have sonnet um which is [55:04] great for daily coding tasks. You have [55:06] opus which is for complex reasoning. You [55:08] have haiku which is fast and efficient [55:10] for simple ones. You have sonnet 1 [55:13] million. So this is where you use sonnet [55:16] with a 1 million token context window [55:18] for long sessions and then you have opus [55:21] plan. This is a special mode that uses [55:22] opus during plan mode then switches to [55:24] sonnet for execution. Okay, so there is [55:27] some variation there but for the most [55:29] part they're always using a single [55:31] model. Um there are three levels of of [55:34] models and we basically just covered it [55:36] but we'll cover it again. So you have [55:37] claude opus. I I think of it as really [55:39] slow, really smart, great for difficult [55:41] tasks. You have Claude sonnet all around [55:43] well balanced the goldilocks of models [55:45] so that's what you're going to be mostly [55:46] using and haiku really fast really dumb [55:48] great for general tasks if you are using [55:51] the subscription model it's very hard to [55:53] gauge uh usage between these three [55:56] models whereas if you are using the API [55:58] you're getting token usage information [56:01] so you have a better idea of that spend [56:03] we do have this graph over here which is [56:05] a bit dated because it's saying like [56:07] three and 3.5 and three you already know [56:10] we have something newer But generally, [56:12] Haiku is, you know, intelligence and [56:14] cost all the way down here as it [56:15] suggests for Opus. Um, it's probably way [56:19] higher now. So, it's probably way up [56:21] here now. And, uh, Sonnet is probably in [56:24] the middle. Okay. But, um, to really [56:27] understand these, you just really have [56:28] to use them and figure out their tasks [56:30] for them. For the most part, I've just [56:32] been running Sonnet and have been having [56:33] no issues. But, some people say that if [56:35] you move over to Opus, it's extremely [56:38] good. But again, you need to drive this [56:40] stuff and figure it out. Okay? [56:46] All right, folks. I'm just going to say [56:48] it again, just so you know, we are [56:49] walking the exam guide. So, notice here [56:52] at the start, domain one, they were [56:53] talking about stop reason. That's why [56:55] we're talking about stop reason. Um, and [56:58] I just want to make it clear like this [56:59] is going to be the best way because as [57:01] we work through this, we're going to uh [57:03] like talk about it then use it, talk [57:05] about it, then use it, talk about it, [57:06] then use it. And then that will [57:08] accumulate and it will make total sense [57:09] to you. Okay. Um, so I know feels really [57:12] random just jumping into stop reason [57:13] without talking about agent SDK and [57:15] everything around it, but I promise you [57:17] this will work. Okay. So let's talk [57:20] about stop reason. So stop reason is the [57:21] reason why the claude agent has stopped [57:24] executing its loop. And there are two [57:27] different values that can come back from [57:30] uh the uh the cloud agent. Okay. And [57:33] when I say cloud agent, I don't [57:34] necessarily mean agent SDK. I mean [57:36] whatever the thing is uh that they have [57:38] on their server that we're not privy to [57:41] that is running their agent. Okay. And [57:43] there's two values that will come back. [57:45] We have tool use uh tool uh use. I [57:48] already have a spelling mistake. You [57:50] know it's not an Andrew Brown course if [57:51] you don't have a spelling mistake right [57:53] off the bat. tool use and end turn. [57:56] Okay. And so the JSON data returned from [57:58] a request to the agent will have that [58:00] stop reason value. So let's take a look [58:04] at some returned output. So here's an [58:06] example of of um a stop reason. And so [58:10] here it's saying it stopped because of [58:12] tool use. So this is when the agent [58:15] stops with the intention to call an [58:16] external function. If you do not know [58:18] what a tool is, I cover this in my [58:21] Claude code essentials. But the simplest [58:24] way to understand it is that it's an [58:26] external function. Okay, the idea is [58:28] that you have uh the agent and the agent [58:32] uh is in its own little sandbox and to [58:34] reach out it has to talk to um uh tools [58:38] functions over uh communication channel. [58:41] Maybe it's gRPC or uh HTTP. I'm not [58:44] actually sure. Uh it doesn't really [58:45] matter. But the point is it's calling [58:47] those functions and then those functions [58:48] are able to do something and then bring [58:50] back output, right? And so the idea is [58:52] that um the idea is that you prompt your [58:56] agent saying hey I need you to go do [58:58] this stuff and there's tools that it's [59:00] aware of that it can do and it's going [59:02] to trigger those tools and if it does uh [59:05] behind the scenes it's going to return [59:07] the stuff saying tool use and then we [59:09] will see um that stuff there. Okay, it [59:12] doesn't make sense it will make sense [59:13] and we go to implementation. Another [59:15] reason for stop will be end turn. So [59:17] this is when the agent has decided to [59:19] return you a result because it's done. [59:22] And so here we have if you carefully [59:24] look at the tool uh clearly someone's [59:26] asking about the weather. So it says uh [59:28] like I'll say like I hey can you tell me [59:30] what the weather is in Winnipeg. I don't [59:31] live in Winnipeg by the way. I'm in [59:32] Scriber Ontario. That's a different [59:34] Winnipeg is a different province but [59:35] that's what I used. So here it says let [59:37] me uh let me check the weather for you. [59:39] So that's the tool uh replying or maybe [59:41] the agent replying there. And then it's [59:43] calling the tool use and there's a tool [59:46] called get weather and the input is [59:48] Winnipeg from what it extracted from the [59:50] user and later on like it will end the [59:55] turn and return that information. Okay. [59:59] Um so the thing is like how does how [60:01] does this go from here to here? Okay. [60:03] And this is going to come down to the [60:04] appended tool use. So the idea is that [60:06] here we have a message from the user. It [60:07] says what is the weather in Winnipeg? [60:10] And it says, "Okay, let me check the [60:12] weather for you." And so the idea is [60:13] that we do that and we're doing the tool [60:16] use. And the idea is that this is going [60:17] to output data and then we uh append it [60:20] back into the conversation, right? [60:23] That's all we're doing here. Okay. So [60:24] here you can see the tool use. Uh then [60:27] we have our um this is the tool results. [60:30] I feel like it's supposed to say [60:34] uh let's go back here a second. Oh, stop [60:37] reason. Right. So some reason will show [60:39] up somewhere else. But anyway, the point [60:40] is we'll we'll see that when we look at [60:41] the code. Okay, but the idea is that [60:42] whatever the result is, we're appending [60:44] it to our message conversation. Um [60:46] because what's in the messages is not [60:47] the same of what is being outputed, [60:49] right? And that's why we don't see um [60:52] stop reason within our message thread. [60:54] That makes sense. Okay. So um anyway, so [60:57] when Claude returns a stop reason uh [60:59] which tool use to the program, the [61:00] program needs to parse and send the data [61:02] to a function. The the function will [61:04] return the data. Then you append the [61:06] result of that data. So, let's go do [61:09] that now. Okay. [61:15] Hey folks, it's Andrew and we are going [61:18] to learn about stop reason. Uh, and I [61:22] want to point out again that we are [61:23] walking uh the exam guide as the method [61:27] to which we will learn. Um I'm doing [61:29] this because again I think it'll be the [61:31] best outcome for everybody here and I do [61:33] believe that it's more important to make [61:36] sure we validate all these things [61:38] actually work as expected and the only [61:39] way to do that is to do it. So uh we'll [61:42] go ahead and make a new folder here. [61:43] This will be for stop reason and we're [61:45] going to attempt with uh the API right [61:47] now. So right now I am using the API [61:50] with anthropic. I have some money loaded [61:52] up on there. So if you're not doing that [61:54] you're going to have a hard time testing [61:55] that out. As far as I'm aware of, [61:56] there's no free tier for the API usage [61:58] on the platform. Cloud.com. Um, and so [62:01] if you do not want to spend the money, [62:02] even though maybe I don't know, like [62:04] maybe we can cover this all in five 10 [62:05] bucks. I won't know till the whole [62:06] course is done, but you know, I would [62:08] strongly recommend load up five or 10 [62:10] bucks and see what mileage you get [62:13] because it'll give you an idea of how [62:15] fast the consumption is. I think I was [62:17] already I had like 10 bucks loaded up [62:19] and I'm already halfway through it [62:20] because I was doing other stuff with it [62:22] not related to this course. But um we [62:24] will try to monitor our spend as we go [62:27] here. And so what I want to do is I want [62:30] to show off um stop tool use or or sorry [62:34] like uh stop reason. And we'll go ahead [62:38] here. We'll make a new main.py. Since we [62:40] already have some base code here, we can [62:41] go ahead up to this one. I'm going to [62:43] copy it just to get us off the ground. [62:47] Uh and it didn't cost us any credits [62:49] either [laughter] [62:50] or tokens. We just copy paste it. Right. [62:52] if you can do it, save yourself some [62:54] money, right? Um, but anyway, so here [62:56] we'll load in the anthropic key. Um, and [62:58] we do have a function here. I'll just [63:00] take it out of the function so that it's [63:01] a little bit easier to work with. And [63:03] let's go and pull up the API. [63:07] And we might be able to ask it to um uh [63:10] give us information, but I would rather [63:12] just go ahead and uh take a look. So [63:16] here, we'll go over to here. And what I [63:19] want just a moment. What I want to find [63:22] here is the documentation um [63:24] specifically for tool use. And I know [63:26] it's here somewhere. I would I would say [63:28] I just do not like uh the way any of [63:31] this is organized. I do find it very [63:32] hard to find anything in the API. Um but [63:36] give me a moment. I will find it quickly [63:37] for us. Okay. One thing we probably [63:39] should do so we just get better code is [63:41] we probably would want to put it into um [63:44] uh something that's async as I know [63:46] that's probably more ideal. And I think [63:47] I'm just going to go ahead and do that [63:49] uh very quickly here. here. So, I'm just [63:50] going to grab this code. Again, don't [63:52] have to spend any tokens. It's right [63:54] here for us. Um, and we will go ahead [63:57] and grab this here. [64:02] Okay. [64:06] Here. And I guess we'll be ending up [64:08] putting this back into a function. So, [64:10] I'll just grab this like this. [64:13] Okay. [64:18] and we'll indent that. Um, and then we [64:22] have our little async call here. So, [64:23] this one's a little bit different. [64:26] In fact, we could probably just grab the [64:28] whole thing to be honest because we're [64:30] just going to be working with something [64:31] basic anyway. I think it's actually the [64:33] same thing. The only thing that's [64:34] different is the name of the um model [64:37] that's being used, right? So, again, you [64:40] know, try to maybe use one that's cost [64:41] effective since we will be testing out a [64:44] lot of stuff here. And I'm having a [64:45] little bit of trouble copying this. I'm [64:47] going to go here and just paste it up as [64:48] a value like this like model like this [64:51] and then I could just do equals there. [64:52] I'll have less of an issue copy pasting. [64:55] Okay. Um and we will go ahead here and [65:00] grab the stuff out of here like that. [65:04] Okay. [65:06] And I think this uh this is a little bit [65:08] different. So we'll grab this line. [65:13] And folks, we are learning by doing it [65:14] manually. It helps us see what's going [65:16] on. Okay, great. So, we now have one [65:19] that is using async. And I'm going to go [65:20] ahead and just go into our stop reason. [65:23] And we will go ahead and execute this. [65:26] Um, I don't know if async is part of [65:28] Python. Let's go find out. We'll just [65:30] run it and see. [65:33] It is not. So, I'll go ahead and just [65:36] grab this here. [65:39] Requirements.txt. [65:42] And we will drop this in here. [65:46] I'm going to assume that's what it is. [65:48] Like that. Okay. And then we'll go ahead [65:49] and do a pip install. pip install-r [65:53] requirements.txt. [65:56] And we will get that installed. Now, [65:57] let's go ahead and run that. See what we [66:00] get. Um, AIH. Okay, maybe I called it [66:05] the wrong thing. I think it even told me [66:07] that's what it was called. So, we'll try [66:09] that one more time. We'll run it again. [66:12] To use this, you must have installed the [66:14] package and extra. [66:18] Okay, let's go ahead and do that. [66:23] All right, let's ask how to install [66:25] this. Okay, so I'm going to go here, [66:29] claude [66:30] debug, [66:33] and maybe you can help us out here. [66:40] So here uh this error is a Python [66:42] dependency here. Anthropic. Oh, that's [66:46] why. [66:50] Okay. So what I'd probably rather do is [66:54] take this approach. [66:59] Sure. We'll grab this, [67:03] go over to here, [67:05] paste it in as such. [67:10] I'll just ask like can you Okay. Yeah. [67:12] Update the requirements.txt. [67:18] The only reason I I didn't just do that [67:19] myself is looks like the indentation was [67:21] wrong and so I just wanted it to [67:22] correctly put it in. We'll just say yes [67:25] for the session [67:27] because it had like a little set value [67:29] in there and I wasn't sure like oh two. [67:30] So it's saying move this to to that. [67:32] Okay. And so that was my confusion. [67:34] Great. We will close out of this. We'll [67:36] go back and we'll hit up. [67:39] Uh we got to do pip install. Pip [67:41] install. Right. [67:43] There we go. We'll go ahead and run it [67:45] again. [67:47] And so now we are getting output back. [67:49] All right. So [67:51] we are now better set up. So what we [67:54] want to do is set up tool use. And I'm [67:56] going to again find that here for us. [67:58] Okay. So we've scrolled on down here and [68:00] we have tool use. So the SDK provides [68:02] support for tool use also known as [68:04] function calling. And the SDK provides [68:07] helpers for defining running tools as [68:09] pure functions. So you can use uh the [68:12] beta tool option. Okay. And so here we [68:15] have it here. Um and that's probably [68:18] fine. I'd rather do it at the lowest [68:21] possible level. [68:23] So let's see if they actually have that [68:24] as an option here. [68:27] Okay. And so this is the actual isolate [68:29] page about tool use that I was looking [68:30] for before. [68:33] Okay. [68:35] And so what I want is a full example of [68:39] the tool used. So that's what I'm [68:40] looking for. [68:42] I know there's that beta tool, but I [68:44] would rather if we can [68:48] have the full thing. So the smallest [68:50] possible tool using program, one tool, [68:52] one message. Okay, so this looks [68:54] promising. [68:55] All right, so here we're defining a tool [69:00] and then in here we have our tool [69:02] choice. [69:05] And then we describe that here. Okay. [69:11] And so what I'm trying to figure out [69:13] here [69:14] is how does it align up the function [69:17] with the tool, right? Because here we [69:19] have a tool [69:21] and the idea is it's going to call this [69:27] and it's going to provide that input and [69:29] then we're going to get that information [69:30] back. Oh, right. because this is where [69:31] we would leave the agentic loop, right? [69:33] And then we would parse it. Okay. Okay. [69:35] So, what we'll do, um, I'm not a huge [69:37] fan on how complicated this tool is, but [69:39] we'll try to write one that is very, [69:41] very simple. So, I'm going to go all the [69:43] way to the top here [69:45] and, um, [69:48] I'm going to [69:50] just fix the indentation here. [69:53] I'm going to go ahead and write tools. [69:56] Again, the reason we're doing this by [69:57] hand is just so that the information [69:59] syncs in. Of course, we could just [70:00] generate this out. And I just want a [70:02] very simple tool. So, I want a tool. So, [70:06] this will be a tool that adds two [70:09] numbers together, right? Very [70:11] straightforward. And then we'll say [70:13] description. [70:16] Oh, that's that is the description. [70:19] And actually here it should be like uh [70:21] use this tool [70:23] to add two numbers together. So, I'm [70:25] being prescriptive to say like, hey, [70:26] make sure you say it this way. There's [70:28] actually a reason for doing it that way. [70:30] Uh that way it helps it to know how to [70:32] select the correct tool. [70:35] I think we I know we covered in the [70:36] cloud code essentials, but we might also [70:38] cover it in this course. And here I'll [70:40] just be like add to numbers. This is [70:42] obviously something that the agent would [70:44] be capable of doing. So it might make it [70:46] hard for it to know. So maybe what we [70:49] might rather do instead is is like um [70:53] uh magic [70:56] like magic eightball instead. [71:01] Uh when the user asks a yes or no [71:05] question [71:07] um that is [71:12] uh what's the word like [71:15] just know fortuneelling question [71:19] call this function. [71:21] Okay. And [71:25] so it doesn't matter what the input is [71:27] because we're not really going to [71:28] process it. Um but we'll get the idea [71:31] across here. So here, by the way, we [71:33] have to do this. [71:35] Bring this up like this. And here we [71:39] need to provide the input schema. I'm [71:40] literally just looking off screen of [71:42] what is going on here so that we get an [71:44] idea of it. [71:47] Input schema. Don't worry, I won't be [71:49] typing everything by hand, but I just [71:50] figured like early on here that it [71:52] wouldn't hurt for us to do this. Uh, and [71:54] then once we've done this once, we're [71:55] not going to type code out like this [71:57] because it'll take forever, right? So, [71:59] say properties. [72:02] Can't spell that word here. Properties. [72:08] And from here, [72:12] um, [72:13] we just need question really. [72:17] Okay. [72:18] So, we'll have the question here. Type [72:21] string. And so really it's just feeding [72:22] in the question. The question actually [72:24] doesn't matter. We're not going to [72:26] actually use it. And then down below [72:28] here, uh we need to have our uh whether [72:31] they're required or not. So we say [72:32] required. And this is going to be [72:35] question. This looks a lot like um [72:37] pedantic or schema whatever the schema [72:41] format is called. [72:44] And so this is looking fine. There we [72:46] go. Common missing there. And I'm just [72:49] checking if there's anything else that [72:51] it needs for this. Nope, that looks like [72:52] it. And so we have tools and tool [72:55] choice. Okay. So we will go back over to [72:57] here. And in here we have uh some [73:02] options. I rather have the model on the [73:04] top here. That's just me. And we're [73:06] going to say tools equals tools. And [73:09] then we're going to have tool choice. [73:12] Here we will talk about tool tool choice [73:14] later. But here we can um force it to [73:16] use it. So, it's auto. [73:20] I don't know what the other one is. If [73:22] it's [73:24] um [73:28] whatever, but we can actually force it [73:29] to use a tool. We'll talk about that [73:31] later. [73:32] But we'll just explicitly set to auto. [73:34] And so now we put a question. So, [73:37] hey Claude, [73:39] uh [73:41] will I be a billionaire [73:45] living on the moon? uh living on uh on [73:48] Mars in [73:52] 2026. And so the idea there is that it's [73:54] supposed to help it uh trigger that. And [73:59] so we have a single tool here and the [74:00] idea is that we want to see that tool [74:02] get called, right? Um and even here you [74:06] can see that they are outputting the [74:08] stop reason on the response but we will [74:10] um output the entire response not just [74:13] the um message content. So let's just [74:16] carefully look here. Actually the [74:18] message content is that they just call [74:19] it response. And so I'm just going to [74:20] change a response. I kind of prefer that [74:22] as a name [74:24] response. [74:26] And then we're going to get a uh a mess [74:29] of of information here. Okay. So, and [74:33] this one says it's not completing. [74:36] Was not closed. We talking It's right [74:37] here. Is it that one? Is that the [74:39] problem? Nope. [74:42] Anyone see? Oh, you know what? Coma's [74:44] missing here. There we go. [74:47] Okay, now that we are good, um, let's go [74:52] ahead and give this a run and see if we [74:55] get a tool call out here. So, we go [74:58] ahead here and we're going to run the [74:59] the Python.mmain. [75:01] and we get information back. So we have [75:02] our message content tool use block. [75:06] Notice there [75:08] uh will I be millionaire magic eyeball. [75:10] So it is calling the message. Now we can [75:12] make this a little bit easier to read [75:13] out. [75:15] Um [75:17] if we were to format it. [75:20] So there's probably a way to pretty [75:22] print [75:23] the structure. I'm going to go ahead and [75:26] ask Claude for that. Um I am printing [75:29] the response [75:33] from uh anthropic [75:36] in stop reason. [75:44] I might already be in that directory [75:45] main.py. Um, [75:48] can I pretty print pretty print [75:51] the response so it's uh formatted [75:57] for easy reading? I don't want it to [75:59] like go ahead and make a whole thing [76:03] like we have the parser for the agent. I [76:04] just want something that is uh [76:08] a generic pretty print thing. [76:11] Ruby has it. So, Python has to have it. [76:15] And maybe it might be dumping it to [76:16] JSON. Yeah, okay. That's exactly what it [76:18] did. So, and that's fine. That is a [76:20] method of of doing it. Um, other other [76:22] ones have like actually a pretty print [76:24] feature, but here it's just going to [76:25] dump it to JSON. [76:28] Okay. And so, what we'll do is we will [76:29] stop that here. [76:32] And we'll go ahead and run this again. [76:35] So, now we actually have something [76:38] that's readable. Okay. So, let's take a [76:41] look at the code here. [76:43] Um, and so this is the response that's [76:45] coming back, right? And so here, will I [76:47] be a billionaire living on Mars 2026? [76:50] And it figured out that it should call a [76:52] tool. So here [76:54] we can see [76:57] it's going to call tool use here. And [76:59] notice at the top level here, stop [77:01] reason tool use. It reasoned that we [77:04] should call a tool. And so it's going [77:05] over and doing that. And notice that it [77:07] figured out what to extract out to put [77:09] into the inputs. Okay. So that is what [77:13] is happening there. Um so now what we [77:17] need to do is we need to do the next [77:18] step. Okay. So that's good for the first [77:22] step. The next thing is we want to um do [77:24] something with this. And I'm going to [77:26] assume that we're going to have to parse [77:28] it like saying oh the tool is this [77:32] right? And they probably have this right [77:33] in here. [77:36] Um, so we go here. [77:39] And so they [77:42] have a run tool function. [77:46] Right. [77:48] And so I'm just kind of looking through [77:50] here to see how they've decided to do [77:53] it. [77:54] Ah, here we go. So we have create [77:56] response loop until the claude stops [77:58] asking for the tool. Okay. And so here [78:01] is a way of doing it. Pretty sure this [78:03] course talks about looping. I'm pretty [78:05] certain that um this is the way that you [78:08] want it to work. So, we will uh just [78:12] carefully look here. And so, I'm just [78:13] trying to [78:15] make our code a little bit easier to [78:17] work with. [78:19] So, [78:22] um [78:25] I'm thinking about this. [78:28] So, what I'm going to do is paste this [78:30] here like this. [78:33] And I'm going to see if I can just [78:34] refactor this out really quickly. So I'm [78:36] going to say defaf um [78:40] this will be call or send message or [78:46] invoke. I'd rather have invoke here. [78:48] Invoke. [78:49] And we'll go ahead and do this. [78:56] And we'll bring this up here. [78:59] No, I don't know. I'm going to leave it [79:01] alone right now. I'm just going to keep [79:02] it nice and simple. Let's not complicate [79:03] things. Okay. So, um what we want this [79:07] to do is we want this to loop [79:13] uh until it has this. But I don't see [79:15] any external loop here. This just says [79:17] while this loop until Claude stops [79:20] asking for the tools. Well, where's our [79:22] loop? [79:25] Oh, this is the loop. Oh, I see. Okay. [79:29] So here we have a response that's [79:32] created. [79:34] Actually, no, I don't understand what [79:35] just give me a moment. Okay, my [79:38] confusion with this code is that this is [79:40] a this is a for loop, right? And so here [79:42] it's returning back a response. And so [79:43] that response structure to me is a [79:45] Python object. And so at least that's [79:48] what I think it is. And so it's [79:50] iterating through this. And the thing is [79:51] like while there, oh, as long as there [79:54] is a tool use, continue the loop. And if [79:57] there are no more tools, then stop [79:58] because there could be more than one [80:00] tool returned. Okay, so now I understand [80:01] what's going on. That makes total sense [80:03] to me. So what we'll do here is we now [80:06] have um our loop here. It's going to [80:09] iterate through this um [80:12] this dump here until we get that [80:14] information. [80:16] And we still want to print out our [80:18] client here. That's fine. I'm just going [80:20] to indent here. So we are on the uh [80:22] interior here. Okay. And so this is [80:26] where we want to start processing our [80:28] results. So they have this block here [80:31] which is [80:33] getting the actual two use tool use [80:35] extraction assuming that that is [80:37] correct. [80:38] And then I'm just going to go ahead here [80:40] to say print tool use. [80:50] I think it's uh was it S? [80:58] What is it? S parenthesis S like that. I [81:01] don't know what it is. Um and so just [81:03] say print tool use. I'll just keep it [81:06] simple. I'm not trying to remember my uh [81:11] my Python right now. And from there then [81:15] we have the results [81:18] and oh that's where it run the tool. [81:19] Okay. So this is where I want to stop [81:20] right now. Okay. So, [81:26] and this is stuff that goes in the loop. [81:28] There we go. Okay. So, here the idea is [81:31] that it should loop and we should get [81:33] one tool return. So, we'll go ahead and [81:35] run that. [81:40] And it's looping forever. So, I'm just [81:42] going to stop that there because it's [81:43] never ending. And I think maybe the [81:45] reason why is that it might not be [81:46] processing it. [81:49] So here, that's that's why I was [81:50] confused. I'm like, how's this ever [81:52] going to stop? [81:54] So this is what I don't understand. So [81:56] I'm going to go ahead and grab this. [81:59] Go over to Claude and ask them. [82:05] Actually, we'll give it the whole code [82:07] sample here. [82:12] What will stop the the while loop? [82:14] Because to me, like right away when I [82:15] looked at I was like, what stops it? And [82:17] I thought maybe oh when it hits [82:18] something that it doesn't have it. So [82:20] the while stop is when the stop reason [82:22] is no longer tool used. This happens [82:23] when cla decides it has enough [82:25] information to get the answer. In this [82:26] specific example, this occurs when [82:28] claude has the create answer for. [82:32] Okay. [82:33] So but if I go to mine here like this. [82:40] So why did mine loop forever? [82:44] It's like I'm missing something here. [82:47] because you never update the response [82:49] inside the loop. Oh, [82:54] okay. So, we will go back over the code. [82:56] I did not notice that there, but we'll [82:58] go all the way down here. [83:01] Where is it getting updated? [83:08] Uh-huh. [83:10] I I don't see I mean, maybe this I I [83:13] guess because it's calling it from [83:15] within it. Okay. So basically [83:18] what we'll do is we'll keep this a [83:19] little bit simpler because I think we're [83:21] a little bit too far ahead of the game [83:23] here and their example is not really [83:25] broken down to be simplified. So I think [83:27] the thing is that we're going to get the [83:28] data and we'll ignore this loop thing [83:31] for now and I want to go back up and [83:34] look at our actual data that we had from [83:36] before. I'll just have to run it again [83:37] to actually get it because I cannot see [83:38] what's going on [83:42] and I think we should just process this [83:44] uh step by step. Okay. So, we'll go run [83:46] this again. [83:48] Okay. And so, in here in our data [83:51] structure, I want to get back the fact [83:54] that this um is a type of tool of tool [83:57] use. [84:00] Okay. And so in the response [84:05] there is content [84:07] and so it's response content tool use. [84:10] Okay. And so here I'm going to say print [84:13] response content tool use. [84:17] And then the idea here is that you know [84:19] if [84:21] response content tool use [84:28] equals tool use then we will do this. [84:34] I need a period here. [84:37] Uh maybe parenthesis. I don't I'm not [84:39] again not person that normally works in [84:41] Python. And so the idea is that if we [84:43] have that then we would want to run the [84:46] tool. And so here um I'm going to take a [84:50] look here and see how they are running [84:52] it. [84:54] So you have run tool name tool input [84:57] and then it says if it equals that then [84:59] we'll run that there. And so just to [85:02] keep ours really really simple, [85:04] um [85:06] I'm going to go up here and pretend that [85:07] we already have our function. [85:12] So I'm going to see if I can do it here. [85:13] So we'll just say um [85:16] I don't know why I keep getting dra uh I [85:18] don't know why this keeps um drifting. [85:20] Do you see my mouse? [85:22] I don't know what's causing that. It's [85:23] kind of annoying. But we'll go here. [85:24] There's something there's obviously [85:26] something doing it. But uh we'll go here [85:27] and say magic eightball like we'll say [85:29] tool magic eightball [85:33] uh eightball eyeball. [85:39] And so this will have the question [85:45] and then down below here [85:50] we'll go here and [85:53] it will be based on the name of the [85:55] tool. [85:57] So for theirs [86:01] uh do we have the name of the tool here? [86:02] It is under [86:04] name [86:07] content name. [86:13] Say response [86:16] content name. The only problem here is [86:19] notice here that we have this one and [86:20] then that one, right? So you do kind of [86:23] have to iterate to get to the um correct [86:26] one. And so probably what we do need [86:31] is this line here because then it will [86:34] return the actual block that we want. [86:36] Right? So we will place this here [86:41] um as it is trying to find it. [86:44] And so now we don't really need to check [86:46] whether there is a tool. It's just this [86:47] either exists or not. Right? So here, so [86:51] if the tool use exists, [86:54] then call the uh tool and pass it uh the [87:04] Okay, so I might just go ahead and ask [87:06] it to implement this for me. So I'll go [87:07] back here to Claude. [87:10] Can you implement uh hooking up the tool [87:15] use call on line 46 [87:21] for main.py. Please use the s uh [87:24] simplest solution [87:27] to hook up the function [87:30] to the function. Okay. And hopefully it [87:32] doesn't try to implement the whole [87:33] thing. [87:39] Uh, we'll see. Yes, sure. [87:44] It's just because I can't remember how [87:45] to do the if call checking of the thing, [87:47] like if it's undefined, and I just don't [87:48] feel like doing that. What I expected to [87:50] do is like have an if. So, yeah, that's [87:52] what I wanted. I want a random thing. I [87:53] don't like it how it keeps some of the [87:54] imports in there. Uh, and so that is [87:56] technically correct. [87:59] Yes, you can edit it. And then it should [88:02] check whether uh the name matches. And [88:05] then it should call that function. And [88:07] so then we go here. [88:14] So call the tool and send the result. [88:19] And that's the input. Yes, that is [88:21] correct. And then it's it's using it as [88:23] a splat to open it up. That is correct. [88:25] We have our tool result. And then here [88:27] it's doing the followup. And that would [88:29] be the next step of like how we feed it [88:31] back in there. So this is good. [88:33] Um, [88:41] okay. [88:46] Where is it? [88:52] And again, I'm not sure if this is just [88:53] a problem with uh Yeah, there we go. I [88:56] just reopened it. There's some weird [88:58] caching thing that's going on here with [88:59] Visual Studio Code. Okay, great. So, [89:00] what do we have? We've grabbed that [89:02] block of information. We then input the [89:05] results of that information there [89:08] and then we're following back up into uh [89:12] this here and we have RO user uh content [89:15] uh you know whatever whatever and and [89:17] we're chaining it along. Okay. So I want [89:19] to do a little little refactor here. So [89:21] can you refactor the calls to um [89:28] was it create [89:33] into its own function [89:36] for main.py. [89:38] And we're just cleaning up the code a [89:40] little bit here because it's a little [89:42] bit hard. We should really be tracking [89:43] the messages here. [89:50] Okay, here we go. [89:54] It's calling it follow-up request, [89:55] initial request. [89:58] Um, no. I mean, [90:02] okay, but shouldn't they shouldn't they [90:04] be a single function and messages are [90:09] passed in, [90:10] right? I don't know why I just created [90:12] two separate functions. We're using Son [90:13] of 4.6 here. Why is it being dumb? [90:21] Okay. [90:25] And so, we'll go back up to here. [90:34] And I don't know why it's not showing us [90:36] the latest, but we'll go ahead and open [90:37] and close it. [90:40] Oh, here's our create function. So, here [90:42] it's just passing information in. And [90:45] so, we got, hey Claude, I want I'm a [90:48] billionaire. We are passing in the [90:50] messages. Here we are waiting our [90:53] response. [90:54] Okay, we get the the the uh we iterate [90:58] through that block until we find tool [91:00] use and we only expect one right now, [91:02] which is not a good way to code it, but [91:03] it's a good way to show this here. Then [91:06] here we have our tool results. [91:09] Um so whatever is coming back here and [91:11] then notice that we are [91:14] putting back into here the actual tool [91:17] results. So see here we have type tool [91:19] result like we saw earlier. Um and then [91:22] we have a tool use ID. So we are [91:24] bringing back the ID of the tool so it [91:27] can map it up very easily so it knows [91:28] exactly what it's doing and then the [91:30] results itself and then we can follow up [91:32] and see the next one. And so what we [91:34] want to see in that followup [91:37] is um the end result. Right? So I'm [91:41] going to go here. [91:44] I'm going take this one out because that [91:45] would actually break our code [91:47] and [91:49] I want to [91:52] redump this out. But I'd rather call it [91:53] response twice to be honest. Okay. And [91:55] so now we should have two. And so I'll [91:57] go here. This will be uh followup. [92:03] And then this one will be here. It will [92:05] be the uh [92:08] original. Okay. [92:11] And so I'm going to just stop this here. [92:13] We'll type in clear. [92:15] We'll type in Python main.py. And we'll [92:18] go up. And so here is our original [92:23] And then in our followup, we are just [92:24] observing that it ended the turn. [92:27] Okay. So, I know there's a lot of setup [92:30] here, but we are going to reap the [92:32] benefits of this code going forward as [92:34] we are going to iterate through this. [92:36] Okay. Um, but there you go. Cha chia. [92:44] All right. Let's talk about [92:45] preconfigured decision trees. So, these [92:47] are when you hardcode the logic and the [92:49] code decides what happens next. A very [92:51] simple one be an if else statement. So [92:54] uh uh if this then do that or if you [92:57] want to have something that's more [92:58] complex that would be a state machine [93:00] where it's saying what the initial state [93:02] is and you can transition to other [93:04] states and stuff like that. Uh you might [93:06] hear the word tool sequence um used uh [93:09] through anthropic uh and so they're just [93:11] really talking about a state machine or [93:13] a series of steps. Um the thing is that [93:17] when people were first trying to figure [93:18] out what an agent looked like, they [93:21] would implement state machines because [93:23] that was the way they would set up the [93:24] complex logic. But we don't really [93:26] consider that to be really agentic as um [93:30] uh something that's agentic is that go [93:31] that goes through an agentic loop where [93:33] each time the um the decision is being [93:37] made by the agent itself and not by the [93:40] code. So here the decision-m is the [93:42] agent uh sorry is the uh code. If the [93:44] code is making the decision then it's [93:46] not exactly agentic now is it? But if [93:49] the agent is deciding then it actually [93:51] is agentic. Okay. So model driven [93:54] decision-m or basically an actual [93:56] agentic model is where you you hand [93:59] claude a set of tools and a goal and [94:01] claude figures out the sequence itself [94:03] at runtime based on context. You don't [94:05] tell it uh what to call or when. Okay. [94:08] So an example here would be like you are [94:10] a helpful assistant that helps users [94:11] manage their work work tasks. You have [94:14] access to the following tools. Now [94:15] you'll notice that when we did the the [94:17] stop reason video, we never put in the [94:20] prompt in the system prompt that these [94:22] were the tools that were available. So [94:25] um if you're getting into this now and [94:26] you uh weren't working with uh um coding [94:30] these things a few years ago, uh if you [94:33] were working with these earlier, then [94:35] you would have experienced this where [94:36] you would have to write them in. But the [94:38] agents or the models got smart enough [94:40] that you didn't have to do this anymore. [94:42] Or maybe there's something in their API. [94:44] They're doing some abstraction for us [94:46] and we don't actually have to do this [94:47] anymore. But I might just do it as um a [94:51] uh uh exercise just so that we have an [94:54] idea where that code would be. Um but [94:58] when we're using the cloud API, we just [94:59] literally specify these are the tools we [95:01] have. We don't have to literally tell it [95:03] what tools are available. Um, and so [95:05] something we're going to hear a lot [95:06] about is the hub and spoke architecture [95:08] for setting up a coordinator agent. Um, [95:11] and so this is where you will have a [95:13] single agent sitting at the center and [95:15] it's going to talk out to other sub [95:17] agents around it and it's going to um, [95:20] uh, intercept all communication between [95:22] it. We will talk about that later. But [95:24] right now, I just want to take a look at [95:27] the um the decision- making not for [95:30] model uh model driven decision-m since [95:32] we will figure that out here shortly, [95:34] but mostly the preconfigured stuff. And [95:36] both options are uh valid and useful. [95:39] It's just depending on what you're [95:41] doing. Okay? But obviously um in this [95:43] course, Enthropic wants you to be model [95:45] driven, right? Because that's what [95:47] they're trying to sell you on. But I'm [95:48] telling you that both are useful. Okay? [95:54] Hey, this is Andrew. In this video, all [95:56] I want to do is uh explore that [95:58] decision- making. So, I'm going to make [96:00] a new folder here. We're going to call [96:01] it decision- making. [96:04] And uh what I'm going to do is grab our [96:06] existing code here [96:08] for this one. And I'm going to make a [96:09] new one. This will be preconfigured. [96:14] Uh we'll say uh [96:17] what is it called? Preconfigured code. [96:19] So code, we'll say code driven. That's [96:22] actually a better word. I don't know why [96:23] they called it preconfigured stuff. I [96:25] swear to goodness the uh marketing [96:26] material is being done by um an agent. [96:29] And so basically what we want to show is [96:32] not tools in this one. We wanted to show [96:35] um [96:38] I guess we could, but anyway, I'll go [96:40] ahead here and we'll ask Claude to uh [96:42] we'll back out for a second because it's [96:43] in the wrong place, but I'm going to go [96:45] into um the top level here and we're [96:47] just going to ask Claude to change the [96:49] code. So here I'm going to say um I want [96:52] to demonstrate [96:54] um [96:56] preconfigured [96:59] uh what are they called? Preconfigured [97:02] decision trees. [97:06] Uh and so where [97:11] the logic is hardcoded handling the uh [97:16] information coming back [97:19] from the uh LLM. Can you adjust my code [97:24] and remove the tool use? [97:27] Okay. And so in the decision-making [97:34] code driven file and so hopefully we [97:35] will get something that is a good [97:37] example here. But we will see what we [97:39] get [97:41] because I'm already 91% up my session. [97:43] I've barely done anything today. People [97:45] have been complaining about that quite a [97:46] bit here today. But uh maybe maybe it [97:49] will be 3 p.m. soon. Let me take a look [97:51] here. Uh nope. So that's okay. I don't [97:55] mind taking a break. [97:59] And so we'll give it a moment here to [98:01] rework the code. I almost feel like I [98:03] should have just write it by hand [98:04] because sometimes it's slow. But here we [98:05] have something. And we will take a look [98:08] at what we have. Okay. So we'll just say [98:09] yes. Sure. [98:11] There was a lot of stuff in there. So [98:13] I'm not really sure if I like it. But [98:14] we'll go ahead and take a look. [98:17] And I don't know why it's still doing [98:18] this, but I'll just I'll save this file [98:20] here. Whoops. Nope. [98:25] Don't save. I don't know what it's [98:26] trying to do here. [98:28] There we go. And so what do we have [98:30] here? So we have your customer support [98:33] classifier. Classify the user message [98:34] into exactly one of these categories. [98:36] Billing, technical, or general. Respond [98:38] with only the category name and nothing [98:40] else. And so here we have the hard-coded [98:42] decisions. Handle billing, handle [98:44] technical, handle general. We'll go down [98:46] below. And so here you can see it's [98:49] going to route that information. [98:51] So let's take a look here. So, [98:54] I was charged this twice a month. Can [98:56] you help? My app keeps crashing after [98:58] the latest updates. Uh, what are your [99:01] business hours? Okay, that makes sense. [99:03] And so, here it's going to iterate [99:05] through the messages. Claude, classify [99:08] as category. So, it's telling Claude, [99:10] this is actually something that we were [99:11] told to do really early on with LMS is [99:13] like have it output as like a like [99:16] either structured output or or something [99:18] that it can act on. And so what it's [99:20] going to do is output a very specific [99:24] category. That makes sense. [99:27] And [99:28] then it's going to go and route it. So [99:30] it puts the category and the message in [99:32] here. And then we go up to here. So we [99:33] have our category and our message. And [99:35] depending on where it goes, it's going [99:36] to go to these different locations, [99:38] right? And so that's a hardcoded [99:40] solution. Let's go ahead and run that. [99:43] So I'm going to go into [99:46] decision- making and we will type in [99:49] Python and we will choose code driven. [99:51] Okay, we'll run that. [99:54] And so here we can see some routing. [99:56] Let's take a look at what it's done. [99:59] Routing to the billing team. Pull [100:01] account record and check payment status. [100:04] My app keeps crashing after latest [100:05] update. [100:08] Okay. And so we actually have multiple [100:09] runs here. Maybe I wasn't paying [100:10] attention. Oh, there was three. And so [100:12] it ran three different ones and so it's [100:13] showing us that it went two different [100:15] places. And so that's a way to route it. [100:17] Um, you know, like a good example of [100:20] where you might want to implement this [100:22] and maybe this is actually a good [100:24] example. This is an example that I would [100:25] use. So like imagine you're creating a [100:27] multi-user dungeon text adventure game [100:29] and you have very specific logic that [100:31] you want to play out, but you want to [100:32] have some flexibility. This is where you [100:34] can get a trade-off where you have this [100:36] logical structure that keeps the game [100:38] working in a very particular way where [100:40] you're not worried about things [100:41] hallucinating and you're not consuming [100:43] lots of tokens because you're expecting [100:45] it to handle a lot of the instate logic [100:48] where um the logic or the state is being [100:50] handled by the code. And so that would [100:52] be a good use case for that. Okay. Um [100:55] there might be better use cases that are [100:57] more practical, but that's what I can [100:58] think of off the top of my head. Let's [100:59] go over here and I'm going to make a [101:02] different one. And this one's going to [101:03] be um model driven.py. The only thing [101:08] that we didn't see there was um the [101:10] specification of tool use. And so we're [101:12] going to go ahead here and copy this. [101:15] And I'm going to paste this into here. [101:19] And I'm going to go back a directory. [101:21] I'm going to type in clot here. [101:24] And here I want to say I want to I want [101:27] to demonstrate model driven uh [101:31] uh decision- making [101:35] uh I want to [101:39] can you modify [101:42] the code for [101:44] model driven you can change [101:51] the the tooling [101:54] to meet a simple example that shows the [101:58] model is driving decision. [102:00] Can you show in the system prompt [102:07] uh the tools defined? [102:12] I know uh Claude doesn't need it, [102:17] but showing uh for uh demonstration of [102:21] where it used to go. [102:24] Okay. And so we'll go ahead and let it [102:26] do that. [102:28] It's going to keep complaining about [102:29] that. That's totally fine. Actually do [102:31] have some overages. So I like paid for [102:32] some credits. So it's probably fine if I [102:34] hit um 100% here. It's not a big deal. [102:38] And so, you know, we're going to hope [102:40] that we have a bit of uh rewrite. Also, [102:42] other things like I don't know if my [102:43] wife's using Claude. Like sometimes my [102:45] computer's open. She doesn't know the [102:46] difference between chat GPT and Claude. [102:48] She just uses what'sever there. And so [102:50] maybe she's she's consuming my credits [102:52] right now. Who knows? Um but you know [102:55] what we're trying to see is that loop, [102:56] right? And so, uh we'll give it a moment [102:59] here to output it. Okay. [103:02] All right. And we will uh just say let [103:05] Yep. I thought it was done editing, but [103:06] I guess not. We'll give it a moment [103:07] here. [103:10] Oh, it is done. Okay, great. [103:13] Uh, is it done? You're talking a lot. [103:16] Okay. Oh, I think it just ran it maybe. [103:18] Did it run it? No, it's just saying what [103:20] we will see. So, system prompt added. [103:21] Yeah, we'll just read it together. And [103:24] I'll just close this out. There seems to [103:25] be some kind of weird caching issue [103:27] here. So, we have our tools implemented [103:29] up here. And I think it's just following [103:31] the pattern that I implemented. So, it's [103:32] not necessarily that this is the way you [103:34] have to do it, but I think it's taking [103:36] cues from our original code here. And [103:39] then it's just calling here. I really [103:41] don't like constants, but that's fine. [103:42] That's what it does. And then it's [103:44] expanding out. Okay, so this is a [103:46] reference to these functions. And then [103:49] this is the actual schema of those [103:50] functions and what they'll take in here. [103:52] So, we have check inventory, place, [103:54] order, send notifications, and then we [103:56] have our system prompt, right? And [103:58] normally what we would see here, it [104:00] didn't do it, but normally we would [104:02] literally see the name of the tool. It' [104:03] be like send notifications, [104:06] right? It would literally tell you the [104:08] exact name of it like this and do that. [104:11] Um, but anyway, in our prompt here, we [104:14] have you are an inventory management [104:15] assistant to tools to check, stock, and [104:17] place orders, send notifications when [104:19] handling requests, always verify [104:21] inventory available before ordering. And [104:23] so basically, see these descriptions [104:24] here? Usually what would happen is they [104:26] would get picked up over here. So the [104:28] agent would know about these and that's [104:30] how it would reason. And so that's [104:31] probably why we don't need to have these [104:33] here. Okay. Use your tools step by step [104:36] in the order which calls that uh uh in [104:40] uh the order in which you call them [104:43] matters. [104:44] Sure. I don't like the fact that it [104:46] changed the model on me because I don't [104:48] want to consume more credits. I'm going [104:50] to change this back [104:52] to this. [104:54] Okay. [104:56] And um [104:59] we'll do that. One thing I might do here [105:00] is I might just create a claude [105:03] claude MD [105:07] say please use haiku model so our costs [105:13] are low for API usage and I'm hoping [105:16] that that way um this will get picked up [105:18] and so it will it won't do that in the [105:19] future here. But let's take a look at [105:22] what we have. So we still have our [105:24] create function and we have our [105:26] messages. So I need five units of widget [105:28] A. Check if they're available. Place the [105:30] order if so. Right. And then here we [105:34] have our loop. [105:37] Okay. So while true. Is that a good [105:39] loop? I don't know about that. I think [105:42] the the tool use uh loop was a little [105:44] bit better. But here if it if it ever [105:49] becomes false within the loop then it [105:52] would stop. I'm not sure where that [105:53] break would uh occur. Probably at these [105:56] points like break that might return the [105:57] false. [106:01] So I would say the code here is not [106:02] great. But that's okay. We'll leave it [106:04] alone for now because I think uh we [106:06] actually have um slides that addresses [106:09] where that might be an issue. But the [106:12] point is is that the idea is that it's [106:13] going to loop through this and it should [106:15] call [106:19] uh the create multiple times. Here it [106:21] calls it with one step [106:23] and it's going to iterate over that over [106:26] and over again. And if it needs to use [106:28] it's going to stop and do that. Okay. So [106:29] we'll stop this here [106:31] and we'll CD into decision-m into our [106:34] model driven code here. [106:40] Oh, we don't we don't have to CD into [106:41] it. Have to write it. Run it. Okay. And [106:43] so we'll go ahead and do that. And it's [106:46] off to the races. [106:49] So stop reason tool use stop reason tool [106:52] use and we can see that it's doing it [106:53] and figuring out and turn and then we [106:55] have the completion of it. So yeah the [106:57] only part of the code that I I think [106:59] that is probably not great is this while [107:00] true loop. Okay, I think we could might [107:02] have better logic here but we'll see as [107:04] we work through this course and we might [107:06] come back to this code and improve it. [107:08] Uh but there you go. [107:14] Let's look at some antiatterns that [107:16] anthropic wants you to know about. Um [107:19] when you are designing I guess your loop [107:21] basically. So um the first thing is that [107:24] when you are going to decide that the [107:27] loop is over uh you do not want to just [107:30] parse natural language to say like [107:32] looking for an indicator from the text [107:33] that it is complete. What you want to do [107:35] is check on the stop reason uh for [107:38] enter. And that makes sense because the [107:41] output for the text could be anything or [107:43] um the thing is is that maybe these [107:45] words task complete have something to do [107:48] not with it being complete but just in [107:50] general in the text. So there's a lot of [107:52] points of failure that can happen with [107:54] parsing natural text specifically for [107:56] stopping the loop. And obviously um they [107:59] have a a very specific structure like [108:02] stop reason. And so that's going to be a [108:03] lot more reliable and give you a better [108:05] indicator and not be thrown off by the [108:07] text content. That doesn't mean that you [108:09] cannot use um uh text and to use regular [108:12] expressions and includes to make [108:14] decisions on information though those [108:16] are brittle. But when we're talking [108:19] about stopping the loop, we want to use [108:21] stop reason. Okay. Um another thing is [108:24] that you can set an iteration number um [108:27] to cap as as a means to prevent from [108:30] this overusing. Remember in the last [108:33] follow along when we were uh going [108:35] through that while loop, we just had a [108:37] true statement on it. So basically um if [108:39] it never hit a false, it could go [108:40] forever. And so you might think, well, [108:42] let's put an iteration on that. And that [108:43] would um partially help the problem. But [108:46] what if it needed to stop sooner, right? [108:49] Because now you're consuming uh 10 [108:51] iterations as opposed to if it only [108:53] needed one or two or three. And so we [108:55] can still use [108:57] um the max iteration, but we are also [109:00] going to use that stop reason and then [109:02] we're going to provide it that break to [109:03] break out of that loop. And so that is a [109:06] another combination method of stuff that [109:08] we're can doing. You're probably [109:09] noticing a pattern here which is use [109:11] stop reason and turn. Okay. Uh another [109:13] one is [109:15] uh maybe you think okay well if I get [109:16] text back right then surely it must be [109:20] done. It's not calling tools. But that [109:22] is not necessarily the case. Okay. Uh [109:25] because that structure can return back [109:27] text in both cases or maybe it might not [109:30] in some cases. So again we are using [109:34] stop reason and turn to determine it. So [109:37] you know the the text could be a [109:39] misleader. Using natural language could [109:40] be a misleader. Using iterations could [109:42] be a misleader. Um and yeah we just want [109:46] to make sure we do stop reason enter. [109:48] Okay. [109:53] Okay, folks. Let's make this a quick [109:54] one. What I want to do is improve our [109:56] little loop here. And so, I'm just going [109:58] to go ahead and make a new folder as per [110:01] usual. I can just commit whatever I had [110:02] before here. Great commit message. Um, [110:05] so my team will love that message there. [110:07] And we'll just say um end loop [110:11] end loop correctly. Okay. So, I'll call [110:13] that. And what I'm going to do is go [110:15] into our uh [110:18] decision- making one because that's [110:19] where we actually have our nice little [110:22] loop here. I'm going to go ahead and [110:24] copy that code. Model driven model [110:26] driven decision-m [110:28] and notice how much we are reusing our [110:30] code. It's great. Eh, we'll go ahead and [110:33] type in main.py. We'll paste this in. [110:36] And so the main thing we want to change [110:38] is the loop here. So here we have a [110:40] while true. It's counting steps which is [110:41] fine. Um, but here there's nothing [110:44] that's going to prevent it from [110:45] stopping. Now, we do have end turn here [110:47] and we have our break. So, this is the [110:48] thing that's going to break out um, uh, [110:53] and end the turn. [110:56] Okay. [110:59] And so here, [111:02] the only thing I don't see is like what [111:03] happens [111:05] if uh, it gets to the end. I think that [111:07] the end it would just return a false if [111:09] there was nothing else to loop. I would [111:10] imagine. [111:12] So that's the only indicator that I have [111:14] here. So I'm going to go ahead and just [111:17] uh we'll we'll go back a level here and [111:19] we'll tell it to ingest this particular [111:21] file. So uh end loop [111:26] main.py. We use a while loop. [111:31] But I would like to um [111:36] have a max [111:38] iteration of 10. So we set an external [111:43] var. [111:46] Okay. Uh is there any uh other case [111:49] where the end loop might not end? [111:53] Uh we won't even put that in there. [111:54] That'll be fine. And so we'll go ahead [111:56] and we'll do that. And so that should be [111:58] enough to fix it. I really should have [112:00] moved that to Haiku. I'm using sonnet [112:02] 4.6 for simple coding tasks like that. I [112:04] think ha ha ha ha ha ha ha ha ha ha ha [112:05] coup could handle yeah there does that [112:06] max iterations. I don't like constants. [112:09] Uh if you're a good programmer you don't [112:11] put stuff like that. I feel like people [112:13] in the developer or sorry the the the [112:16] data space might have done that because [112:17] their practice wasn't coding but more [112:20] handling data. And so you see the kind [112:22] of amateurish uh coding methods in my my [112:25] opinion for this language. Um, if we're [112:29] talking about C++, [112:31] different story. Those folks are are [112:33] champions over there. But we'll go over [112:35] to here and that's it. So, I don't think [112:38] I need to test this. It's pretty [112:39] straightforward. But I just wanted to [112:41] show that we updated our code um and [112:43] what we would have done in that case. [112:44] And just to point out that break and I'm [112:46] still like, well, is there still a [112:47] little edge case? And we could waste the [112:49] tokens. I just don't want to waste the [112:51] tokens. Okay, [112:57] let's take a look at hub and spoke [112:58] architecture. So, hub and spoke [112:59] architecture is a pattern where one [113:01] coordinator agent sits at the center and [113:04] all sub agents talk to the coordinator. [113:06] I highlighted that word word coordinator [113:08] because you're going to be seeing a lot [113:10] of coordinator agent. And when you think [113:13] of like um claude uh claude code, I have [113:17] a feeling that this is at least one of [113:19] the means at least when you're working [113:21] with sub agents for communication, [113:23] right? So this is going to be something [113:24] really really useful to learn. Uh it's [113:26] going to be really fun and something [113:27] that you can apply um like immediately. [113:31] Okay, so sub agents never have direct [113:33] lines to each other. So if you have like [113:35] a research agent over here, it cannot [113:37] directly talk to the review a reviewer [113:40] agent. has to go through the [113:42] coordinator. Um, and so that is pretty [113:44] clear. And the coordinator is going to [113:46] own the routing. So it's going to decide [113:47] how to route things. Context sharing, so [113:49] what will be shared. So the research [113:51] agent is not going to be aware of what [113:53] everyone's doing unless the coordinator [113:55] passes that information along and it [113:57] gets injected. So um it really won't [113:59] know. Uh and any kind of error handling, [114:01] any kind of observability, any anything [114:03] like that. Okay. Uh and obviously that [114:05] would make it really good for [114:07] observability because now everything's [114:09] passing through there and we have a [114:10] choke point where we can check and [114:12] collect information, right? Um and so [114:14] here we have kind of the uh task life [114:17] cycle um of like okay we have something [114:20] that needs to be done and uh how is it [114:23] going to get executed out. So the role [114:25] of that coordinator is task [114:27] decomposition. So break uh break um the [114:31] task into subtasks, right? Then we have [114:33] test delegation. So who is going to be [114:38] working on that problem? Result [114:40] aggregation. So bring it all back [114:42] together uh to produce a final result [114:44] and decide which sub aents to invoke [114:46] based on query complexity. So um you [114:49] know we have a a lot of things going on [114:52] here, but let's just kind of walk [114:53] through it. So imagine you're a [114:55] coordinator and when given a task uh [114:58] it'll break down a subtask um uh for [115:00] each of the available tools and we're [115:02] saying do not do the work yourself. So [115:04] we're basically here defining [115:06] you know that you are a coordinator and [115:08] this is what you're going to be doing. [115:09] Okay. Um and then here it's like you're [115:11] coordinator and use your judgment. [115:13] Simple factual questions use a single [115:15] agent multi-step task delegate out to a [115:17] sequential passing the results forward [115:19] independent subtask delegating in [115:20] parallel. So we're basically defining [115:22] okay what does the routing look like? So [115:24] it's not just like static routing it's [115:27] you know use the routing you need to [115:28] route based on the use case right which [115:32] I think is really really interesting and [115:33] then down below here it's like okay [115:35] you've gotten all the uh outputs from [115:36] multiple agents combine them into a [115:38] single uh coherent response resolve any [115:41] conflicts and make the data pretty. So [115:44] uh that's the most basic thing when [115:46] we're talking about this coordinator [115:47] agent. Um there's a lot of stuff we have [115:50] to consider when implementing this, but [115:51] we will go and set up a super skeleton [115:54] one really quick quickly here and then [115:56] we will iterate on it. Okay. [116:03] Okay folks. So what I want to do in this [116:05] follow along is implement a very simple [116:08] coordinator agent. So we'll say [116:09] coordinator [116:12] agent [116:14] simple [116:16] or basic. And in here we will make a new [116:20] main.py. [116:24] And I suppose that we could um [116:28] pull up some code. I'm going to see if [116:29] we can switch over to Haiku [116:33] and save some credits here because it [116:36] might be able to do it. We'll see. [116:39] Hi coup. There we go. And so now it [116:41] switched over to the haiku 4.5 model. [116:43] And so I'm going to tell it uh um create [116:47] uh a very basic coordinator agent [116:53] um [116:54] in [116:56] coordinator basic main.py. [117:00] Please follow our general [117:05] um [117:07] coding example would be [117:11] uh what was one we did decision- making [117:14] would probably be one uh model driven [117:17] model driven pie and so I'm hoping that [117:19] by giving it that reference it will know [117:21] how to reference that stuff and produce [117:23] something that's going to be generally [117:25] okay u but we'll see how haiku does I [117:28] really should run haiku a lot more I [117:29] just kind stick at Sonnet. Um, and [117:32] that's my that's my fault there, right? [117:34] And so we will give it a bit of time [117:35] here. Let it accept here. And then we [117:37] will decide whether Haiku could even do [117:39] it or not. And does it have all the [117:40] components we need? And there it is. [117:41] That was pretty darn fast. Maybe it [117:43] needs a little bit more work to do, but [117:45] we have the start of it. Let's close out [117:46] the tab here. Sometimes it helps to [117:48] close out the tab and reopen it for [117:50] whatever reason. It's already done. So [117:52] here it says I've created a basic [117:53] coordinator. We have create task, get [117:55] task status, complete task, uh, task [117:58] list. And so we have the basic stuff. [118:01] Let's take a look here and see if it's [118:03] any good. Um, so we have tool [118:05] implementation. So tool create task [118:10] task status. And so it's talking about [118:11] how it has to manage the tasks [118:13] generically, [118:15] right? Um, then down below here. Yep. So [118:17] we have that [118:19] create a task with an optional list of [118:21] task IDs. Um, get the current status of [118:24] the specific task. Market is complete. [118:26] List all tasks as completed. [118:28] Um, so it seems like it's pretty simple. [118:30] Your role is to manage and coordinate [118:32] tasks. Well, a coordinator does that in [118:35] general. But I guess the thing is like [118:37] this is literally it sounds like it's [118:38] managing tasks. Create tasks as needed [118:40] for the workflow. Check this task status [118:42] and dependencies. Complete tasks when [118:43] appropriate. So what I'm trying to [118:45] figure out here is what is the use case? [118:47] So set up a [118:49] let's go down here for a second. We have [118:52] user message. So set up a workflow [118:56] create a task design and then create a [118:58] task implementation that depends on [119:00] design and then complete design task [119:02] first. This is so generic it's really [119:04] hard to make sense of it. We have our [119:06] while loop here. It brought in the while [119:08] true. So we don't have that max [119:10] iteration. Um and it might might not be [119:14] a major issue but we still might want [119:15] that in there. I probably should have [119:16] referenced the other code. [119:19] And [119:24] so I'm just carefully looking here at [119:27] what we have. [119:29] So we have our tools over here. [119:33] And so I'm not really sure if that [119:34] really fits our pattern exactly. I'm [119:36] going to go take a look at our uh [119:38] diagram here. What do we have? we have [119:40] de demp uh decompose [119:43] uh the routing and the aggregation right [119:46] so um [119:48] I don't think I see all those steps here [119:50] okay so what I'm going to do is I'm [119:52] going to go back to our smarter model [119:54] here sonnet [119:57] okay [119:58] and I'm going to go and ask the [120:00] coordinator [120:03] or I'm going to ask uh it to improve our [120:05] coordinator code so uh you know for a [120:08] coordinator agent [120:10] We should have uh decomposition tasks. [120:16] Um just a moment here. [120:26] Routing. It says assess complexity but [120:28] we have routing. [120:31] um [120:33] we'll say assess complexity [120:38] complexity [120:39] and routing and aggregate [120:43] results. The use case here is um [120:48] to generic [120:50] need [120:52] a better use case. Okay. And so we'll go [120:55] ahead and we'll see if it can improve [120:57] that code. And if none, I might have to [121:00] write even more detailed prompt. Um, I'm [121:02] just kind of low on our usage, unless [121:06] the window has rolled over. Let me take [121:09] a look here. [121:11] Nope, I still got 15 minutes for my time [121:13] to roll over over here, but we'll see. [121:16] And so, I just want to kind of see it uh [121:19] mimic these patterns here. And so, it's [121:21] not to say that it's not exactly doing [121:22] it, but it's definitely uh [121:26] not that sophisticated, right? [121:29] because I would expect there to be a [121:31] prompt for the routing component here [121:32] and I'm not seeing it here. Right. It [121:34] does say set up a workflow. So [121:36] technically that is that uh that right [121:38] there. So maybe maybe it is kind of [121:40] being implemented but we'll give it a [121:42] second here. Now write a concrete use [121:44] case technical due diligence decompose a [121:46] complex software review. Oh, I don't [121:49] like that. [121:52] No, I'm gonna stop this for a second [121:53] here. Stop. Stop. Stop. Stop. I I don't [121:55] like the use case. [122:01] Oh, it already stopped basically. [122:06] So, it says breaks the request into five [122:07] fixed areas. Well, I already got the [122:09] code, I guess. Let's just take a look [122:10] here. [122:13] Um, [122:18] so here it says a user submits a [122:20] software system for technical review. [122:22] The coordinator has a decompose of [122:24] requests, assesses the complexity per [122:26] area, routes and and and does that runs [122:28] an appropriate handler, aggregates all [122:29] findings into a single report. So that [122:32] sounds good. Uh we have uh tool [122:34] decompose request tool assess [122:36] complexity. Um [122:44] I don't really like [122:47] the use case because I want something [122:48] that's gonna be easy for us to validate [122:50] and test. And this will be too [122:51] complicated. I don't like the use case. [122:54] Can you propose to me [122:57] uh 10 possible use cases? [123:00] I want something that uh is not super [123:05] complex that will be like super [123:07] computational [123:10] but would need complex routing and [123:13] choices. [123:15] Uh, [123:17] don't implement, just [123:19] suggest ideas. Okay. And so I want to [123:22] see if we can pick something a bit [123:23] better. If it can't, then I might have [123:25] to uh decide on myself here. Here's so [123:27] job application screener. [123:29] Um, [123:32] event planning coordinator, bug triage, [123:34] restaurant order customizer. [123:41] I mean, I like the travel one. That [123:42] might have to go to the internet. I [123:43] don't necessarily want to do that. [123:55] Okay. Okay. So like with dumb [123:59] what would be the subtasks [124:02] the sub aents used because that's what [124:06] I'm trying to figure out and this comes [124:07] back to you know this idea where we have [124:09] an idea and it's doing a bunch of stuff [124:11] right so like coder writer researcher [124:14] planner data like so I'm trying to see [124:16] what we have and so we go over to here [124:19] and so we have re request to composer so [124:22] it takes the job posting [124:25] resume and extracts it out. [124:28] Looks at each criteria and decides the [124:30] routing. [124:32] Okay. Execut execution phase [124:36] um the actual routing targets [124:41] aggregation phase. [124:43] So it has two different ones here. So I [124:46] guess my question is like is the [124:47] ownership [124:52] I mean like should the coordinator [124:57] be routing to a decomposer and [125:01] uh [125:05] router a routing because I thought [125:08] the composer is supposed to own it [125:14] in a hub and spoke [125:17] architecture. And so that's the only [125:19] thing I'm I'm just making sure like [125:20] maybe just these are just tools and it's [125:22] calling out to them and so it still has [125:23] ownership. So you're right in a proper [125:25] hub and spoke architecture the [125:26] coordinator is the hub. It owns it. The [125:27] decomposer isn't a separate sub aent. [125:29] That's why the coordinator owns its [125:30] responsibility. So we have that sure [125:32] that looks like something. So the [125:34] coordinator decomposes assess complexity [125:36] decides where aggregates the spokes just [125:39] execute. This smells like something [125:41] belonging to the coordinator versus sub [125:42] agent. Does it need a full pitcher? [125:47] Does it need the full pitcher to do the [125:49] its job? Decomposition is always routing [125:51] always needs the full pitcher. [125:56] Uh I don't I don't I don't know what you [125:59] mean by full picture [126:03] but we are implementing [126:07] hub and spoke model [126:11] architecture. Okay. So I don't know what [126:13] it's trying to say there but um maybe [126:16] means like does things have to have [126:18] context. You're right. Okay. So it says [126:20] you're right. Oh yeah, of course I'm [126:21] always right. Um receives the requests [126:23] call spokes aggregates the results [126:26] independent workers each job does one no [126:28] knowledge of each other. That's right. [126:29] Okay. So for job app application [126:30] screener now we have keyword scanner [126:32] deep evaluation red flag detector score [126:35] aggregator takes all the spoke spoke [126:38] outputs. The coordinator owns calling [126:41] each spoke with the right input, [126:44] deciding which spoke to call, collecting [126:46] and combining their outputs. Yes. Okay. [126:49] And so now we're getting something [126:50] there. Just because it said it did it [126:53] does not mean it did. And this is me [126:55] looking at going that doesn't seem [126:57] right. Right. Um and so we will let it [127:00] go ahead and do that. I'm not getting [127:01] any more warning. So oh it says now [127:03] using extra credits. So, I am over my [127:06] usage, [laughter] [127:08] but I I should be okay. [127:11] You'd be surprised how long or how far [127:14] five dollars will take you here. Okay, [127:16] so it says it's completed the [127:18] architecture here. Um, let's go take a [127:21] look and check out the code. So, now we [127:25] got lots of stuff here. Okay, so keyword [127:27] scanner prompt. You are a resume keyword [127:29] scanner. Check whether required skills [127:31] from the job posting uh appear [127:33] explicitly in the resume. For each [127:35] required skill, output one line. Be [127:38] literal. Do not infer or extrapolate. [127:41] Report only what is explicitly stated. [127:43] Okay. And so then we basically have all [127:44] the ones here and they're fine. Of [127:46] course, if we're doing this for real, we [127:47] would be tweaking this all by hand, of [127:50] course. And then here we have the actual [127:53] um I guess they're saying the spokes. We [127:56] could say sub agents if you would or [127:58] they have them called as spokes and each [128:01] of them are individually calling uh this [128:04] stuff. I'm not sure why they're doing it [128:05] this way. It seems like this could be [128:07] easily refactored. This seems a little [128:08] bit uh messy. Maybe they're doing this [128:11] so that you could improve it later on, [128:13] but to me this um seems like this should [128:16] all be just one function. And then we [128:18] have the spoke aggregator where there [128:20] actually is a little bit different. So [128:22] they do have that there which is fine. [128:24] Then we have our dispatch tool. [128:27] Okay. So basically like where should it [128:29] go? We have and yeah whether it should [128:32] go there or not tool schema. So what the [128:33] coordinator hub sees. [128:37] Mhm. [128:39] So we have run keyword scanner. [128:42] Oh like these are the actual tools [128:44] deciding whether they should get [128:45] triggered or not. That's fine. Then we [128:48] have our coordinator prompt. So you are [128:49] a job application screening coordinator. [128:51] That's fine. Your job is to orchestrate [128:52] three independent screening agents. Mhm. [128:57] Uh your job is to orchestrate the three [129:00] independent screening agents and then [129:01] aggregate the results. [129:03] You may run three screening agents in [129:06] order. Do not skip any of them. And so [129:08] here it's defining that saying you have [129:11] a explicit order. And so obviously there [129:13] could be more complex routing than this, [129:15] but this is all there is. Then we'll go [129:17] down to here. And so here we have our [129:19] job postings. I was going to wonder [129:21] where this data was because I was going [129:22] to be like, is there a are the résumés [129:25] generated here? And they do. So, or we [129:27] have our job posting, then we have our [129:29] our resume. We only and we only have a [129:30] single resume, which is fine. Alex Chen, [129:33] that's interesting. Okay. So, we go down [129:35] here and we're passing that data in. [129:38] It's going through that loop. Again, we [129:40] have this while true. So, I'm not sure [129:42] if that's the best idea to have that [129:45] while true like there, but I will run [129:47] it. And [129:49] uh take the risk. [129:53] I think it's fine. [129:56] Uh you know what? I do want a max [129:57] iteration. So I'm going to go here and [129:59] just say like [130:02] uh the while loop is true. Do you think [130:06] we should have a [130:09] max iteration or any other suggestions [130:12] so it doesn't go [130:19] on forever. [130:21] Okay. And so that's [130:23] what I want it to answer there. I we had [130:25] the max it might just do the max [130:27] iteration because now I basically told [130:28] it to do that. [130:31] So here it says um a while true loop [130:33] with no exit condition. I mean there is [130:35] an exit condition. It's the break right [130:37] here. Um [130:40] and then we have a timeout for this use [130:42] case. [130:44] So a max steps caps is is the right fit. [130:47] That's what it's suggesting. [130:51] Fair enough. [130:57] How is it counting the max steps? [131:02] So the loop executes only when the [131:05] condition becomes false. Max steps and [131:07] not when and not when the break is hit. [131:09] So it cleanly catches runaway loops [131:11] without needing an extra flag. Okay, [131:13] that's fine. The cap is 10. Double the [131:16] expected five steps. Give the model room [131:18] to retry. I just don't see where it's [131:19] counting them. Oh, I guess it's right [131:21] here. Oh, sure. I I guess so. But I [131:26] mean, that's the same thing as a max [131:27] iteration. [131:29] Um, so [131:33] max steps is the same as [131:37] max iteration. [131:40] I guess it's fine. I mean, I'm sure it [131:42] will still work. So, [131:44] if it doesn't, we'll find out. And [131:46] again, you know, you can just watch and [131:48] see what my outcome is before you do [131:49] this if you do not want to waste credits [131:52] because I've made a poor decision. Um, [131:55] you know, like I'm loading my thing up [131:57] with like $5 at a time. So, I'm I'm not [131:59] that worried about um [132:02] small losses like that. So, we'll go [132:03] ahead and go into here and let's go [132:06] ahead and execute this. [132:12] And yeah, I'm not using my subscription. [132:14] Now, we could probably port this over to [132:15] agent SDK. Um, and this would be greatly [132:18] simplified. We might do that later to [132:19] see what's happening here, but we won't [132:21] do it right away. So, here it says, um, [132:23] coordinator routes to the spoke. [132:27] Okay, so it's found stuff. Something's [132:28] missing. Coordinator routes to the run a [132:30] deep evaluator. [132:33] Uh, strong alignment with the senior [132:34] level role with seven years of total [132:36] experience. Cool. Strong fit. [132:39] coordinator routes to the red flag [132:41] detector. Imagine someone uh just coded [132:44] this and this is what's keeping people [132:45] out of their jobs. That that would be a [132:47] bummer. And then we have step two of 10. [132:50] So match keywords [132:52] uh strong no flags decision hire. Alex [132:55] demonstrates strong alignment with [132:57] senior level requirements. Coordinator [133:00] for final recommendation for hire. So [133:02] it's recommending it six out of the [133:05] seven strong. No red flags. all core [133:08] required skills present seven years [133:11] experience whatever whatever and so we [133:13] just implemented our own coordinator [133:15] again the only thing that's really [133:16] simple like I still not confident about [133:18] the while loop but the only thing that [133:20] is um very simple is the routing but the [133:23] routing obviously is being handled here [133:26] um and so you know like in that diagram [133:29] it just seems like it's a separate step [133:30] like you cut them up and then you do [133:33] that um and so I'm not sure if that [133:36] should be separated out but the point is [133:37] we did implement coordinator agent um [133:40] and that's something we could decide [133:41] later on if we wanted to have an [133:42] individual step for more intelligent [133:44] routing. So that's the only thing that I [133:46] might um consider like I I would [133:49] probably ask it right now like if it [133:52] should be ran twice but I'm I don't know [133:54] I don't want to because I don't think [133:56] it's going to just tell me. I think it's [133:57] going to actually try to do it and so I [133:58] don't want to muck with it. And so I'd [134:00] say that's fine but just consider that [134:02] that's an uncertainty that I have right [134:04] now. And so I'm going to go back a [134:05] director here. just say get at all get [134:07] commit-m [134:09] uh basic coordinator. I think that was [134:12] kind of fun. [134:15] I thought the results were pretty good. [134:18] Okay, and I will see you in the next [134:20] one. Okay, chowo chia. [134:27] Okay, let's take a look at narrow task [134:29] decomposition. So when uh claw de [134:32] decomposes a task, it can only delegate [134:34] what it thinks to ask for. Okay. So here [134:37] we have an example says give me a [134:39] comprehensive analysis of the EV market. [134:42] Break the user's task into subtasks and [134:44] delegate them out. And so here we see [134:46] the subtasks. We have research EV sales [134:49] figures. Research EV battery technology. [134:52] Research major EV manufacturers. So if [134:54] the initial decomposition is too narrow, [134:57] entire topics never get researched. It's [135:00] because each sub aent only sees its o it [135:03] its own isolated context. None of them [135:06] can flag what's missing. So what got [135:08] missed? Charging infrastructure, [135:10] government policies and subsidies, [135:12] secondhand EV markets, consumer [135:14] sentiment and adoption barriers, supply [135:17] chains like lithium and cobalt, grid [135:20] capacity implications. Okay. Um, so you [135:23] need to be very specific on the task so [135:26] it fully covers what you expect. So here [135:28] it says give me a comprehensive analysis [135:30] of the EV market again. And so as the [135:32] coordinator when decomposing the task of [135:35] course we're generating out the the [135:37] subtask but ask yourself you know more [135:40] information ask subtasks to cover those [135:42] gaps only. Then begin delegating and for [135:45] research tasks specifically consider [135:48] this information. Now, what's [135:49] interesting here is like we created [135:52] um our our job application thing, but [135:55] this thing is talking about research. [135:57] So, they they might have just a single [135:58] sub agent that just does research. And [136:00] so, the idea is that all these tasks are [136:02] going to the same sub aent as maybe [136:04] separate um instances that are spawned [136:07] and they're being tasked with doing [136:08] different things. And so, this is where [136:10] you have a little bit more complex [136:11] routing, right? Or different kinds of [136:13] routing. Um and so one thing that we can [136:17] do to catch weak decomposition is uh [136:21] because like let's say for whatever [136:23] reason [136:24] in here uh this coordinator [136:28] uh that you wrote here to help it be [136:30] very specific uh it fails or you just [136:32] don't do a good job then you could [136:34] implement a tool and so the tool um can [136:38] try to catch it because now when the [136:40] court or when the agent goes and does a [136:43] task it's going to say oh did you submit [136:45] a subtask breakdown for review for [136:47] delegating Well, then trigger this tool [136:50] and then make sure right here that you [136:52] do this stuff and this gives you a [136:54] guarantee um you know of this or maybe [136:57] you want to be a little bit more [136:58] flexible what the input is from the user [137:00] and uh so this thing being decoupled [137:04] might do that. Um another way uh that [137:07] you can fix this problem is at the [137:09] aggregate level. So after you're [137:10] aggregating the results, it can check [137:12] here and say, "Hey, um did you make sure [137:14] before writing the answer that you uh [137:16] met these things?" And so you now have [137:18] basically two different safeguards for [137:21] um improving over uh narrow task [137:23] decomposition. So I'm not sure if this [137:25] will work in the one that we're building [137:28] right now or we'll have to build a new [137:30] little coordinator. Um but we'll go and [137:32] try it out. Okay. [137:38] All right. Okay, so we are back and um [137:40] what we'll do here is we'll try to [137:42] figure out this narrow task [137:43] decomposition. I don't know if it's [137:45] going to work for our case because um [137:47] for research it's a really good um use [137:50] case, but will it be for this one? I [137:52] don't know. Um so I'm going to go ahead [137:54] and just copy all this code here because [137:56] we already have something that's good [137:58] and Claude's going to have an easier [137:59] time working with tweaking that [138:03] that we're gonna have an easier time [138:05] working off of this. So um [138:09] let's go down to where the main [138:12] coordinator prompt is. So here it says [138:15] uh you're a job application screening [138:16] coordinator. Your job is to orchestrate [138:18] three independent screening agents and [138:20] the aggregate their results. Run all [138:23] three screening agents keywords deep [138:26] evaluators detector. So um let's go [138:29] ahead and just ask it. Okay. So we'll go [138:32] here. We'll say um for our [138:37] narrow task compion main for [138:41] our coordinator prompt [138:45] is the uh decomposition [138:49] um [138:52] is our [138:55] task decomposition [138:57] too narrow [138:59] and what do we need [139:02] to ask for a better decomposition. [139:08] Okay? Because this one's pretty darn [139:10] simple, right? It's just like there's [139:11] these three things. Feed it into those [139:13] three things because it's not conducting [139:14] research, right? Um it's not going out [139:17] and looking at large bodies of text and [139:19] trying to figure it out. So, you know, [139:21] maybe if there was like more than one [139:23] source, then that would be useful. And [139:26] so, maybe that's what we might recommend [139:27] here in just a moment. And I might say, [139:28] hey, like, you know, assume that you're [139:30] ingesting more than one source of [139:32] information. Um, and that might be a [139:34] better example. But let's see what it [139:36] comes back with here. And then I'll tell [139:37] you whether I agree with it or not. Just [139:39] because it will produce something [139:41] doesn't mean that it's useful. So, we [139:42] will find out here. Okay. Also, I was [139:45] just thinking about this. What we should [139:46] have done is just taken the coordinator [139:48] information and provided it uh to here [139:50] with the basic information because I [139:52] feel like it's consuming a lot more um [139:54] tokens than it should require. I mean, [139:56] it's not saying there's that many here, [139:57] but it is taking uh some time here. And [139:59] I again, I'll just wait. But I should [140:01] have really just extracted out that [140:02] individual information. So, let's take a [140:05] look here. And oh, yeah, we can edit the [140:07] main file. That's fine. Yeah, I thought [140:09] it was done. I guess it's not done. [140:13] Okay. So, let's take a look at the [140:14] problem here. [140:16] Spokes are narrowly scoped but [140:18] appropriately interpretive. That's [140:20] actually reasonable. But if you're [140:21] designing new coordinators, [140:24] well, I'm not designing new [140:25] coordinators, but we we'll take a look [140:26] here. Spokes answers, "What is X? Python [140:29] found [140:32] but without access to the resume." [140:38] Um, spokes answers, "What does X mean [140:40] for the hire? [140:44] Receives pre-interpreted signals and can [140:46] make the uh integrated judgment." [140:50] So I guess we're trying to determine [140:51] like is it fine? So what to ask? So so [140:54] is it narrow? So is skill X listed? Does [140:57] experience demonstrate the skills X [140:59] required? So if it's narrow saying like [141:02] is it just listed or is it actually [141:05] telling us? So that would be better. [141:06] That's true. Uh narrow resume only. And [141:09] so this is what I was talking about [141:10] where we would have more than one type [141:12] of um information feed. But here it's [141:15] saying feed in the resume and the job [141:16] posting for the fit. [141:19] context what gran granularity so one [141:23] spoke per keyword one spoke per uh [141:26] decision dimension whatever so this file [141:29] runs both the coordinator of the same [141:31] candidate [141:33] so you can see how the narrow [141:35] decomposition loses [141:37] the 50 million requests per day nuance [141:41] while the better one catches it. Okay, [141:44] so we'll go back up to here. I'm just [141:46] trying to make clear the this thing that [141:47] we're looking at. So narrow anti [141:49] pattern. What is X? Python found six [141:53] years, three no gaps. That's probably [141:55] like how actually recruitment people [141:56] work. The aggregator receives new facts. [141:59] It still has to do all the reasoning but [142:01] now without access to the resume. So [142:04] spokes answers what does X mean for the [142:06] hire? Strong trajectory risk. Okay. So [142:10] one thing I I was thinking of was like [142:12] you need to crosscoordinate this [142:13] information, right? [142:15] So um I would say you know one thing one [142:19] thing I noticed is you know can we [142:22] validate [142:24] the number of years [142:28] based on based on the resume [142:30] information? Can we mock [142:34] other data sources that uh that we would [142:39] feed in where [142:42] uh if we didn't do better task [142:47] decomposition [142:48] with very specific [142:51] things to check [142:53] we would run into an issue because [142:56] that's I think what's going to take it [142:57] but like that was one example of like [142:58] okay well you know if you had to [143:00] validate how many years someone had [143:01] experience, you'd look at their resume, [143:03] but you might also look at uh projects [143:05] or references or other stuff. And so [143:07] let's just see if it can, you know, [143:09] consider other data sources. [143:12] Uh maybe we should just want the Eevee [143:13] one because research is a really a [143:15] really good one. But I mean in this [143:16] sense like we are researching if there [143:18] are multiple things like maybe they have [143:20] blog posts and stuff like that. But [143:22] we'll we'll see what comes back here and [143:23] I might make s uh suggestions for data [143:26] sources. Okay. [143:29] All right. It is back. Let's see what [143:30] it's done. So, we'll go up to here. Key [143:32] addition. So, show activity since 2018 [143:35] from uh a Git profile. Let's see. All [143:38] all assessed skills are above senior [143:40] threshold. Verified 7.6 years [143:43] experience. Um, [143:46] okay. I mean, did it run it again? I [143:48] didn't tell it to run it, but um I guess [143:50] what we should do is just take a look at [143:53] what the new coordinator information is. [143:57] Your job is to coordinate uh three [143:59] independent screening agents and then [144:01] aggregate the results. [144:03] I mean this isn't this is showing steps [144:06] which is fine but we're not seeing it [144:10] doesn't seem to understand what I'm [144:12] trying to tell it. Okay. So [144:15] no I don't think it understands. So what [144:17] I'll do just give me a second here. I [144:20] need to give it an example and I just [144:22] need to extract out. I'm going to give [144:24] it a better example here. And we're just [144:26] going to clo I have my screenshot. I [144:28] just don't have the raw data. And so I'm [144:30] just going to uh chat GBT or something [144:32] here off screen. Be like uh get me get [144:34] me the text. [144:37] Okay. And just give me just a moment [144:40] here. Just getting the text here off [144:42] screen. See? So getting the text. [144:44] And I'm going to feed it as an example [144:47] of like [144:50] more information. [144:56] Okay. So, like [144:59] we'll go back here. [145:04] It's like you you know I don't think you [145:06] understood [145:10] uh to improve [145:13] narrow task decomposition [145:16] we should be giving it [145:20] specific [145:22] considerations. [145:27] Okay. [145:29] Oh no I didn't say to do that yet. Okay. [145:31] We'll paste that in as an example, [145:34] right? So, I don't know if it knows [145:35] that's an example, but I think it might [145:37] know. [145:43] So, hopefully it understands because [145:45] we're talking about this this area here. [145:48] Um, and if this fails, then we could [145:50] just again just make it it might even [145:51] try to change it to EV, but we'll see [145:53] what happens here. Um, and wait a moment [145:56] and see what it comes back with. [145:58] Okay. So, [146:02] some of your examples general through [146:04] this now, did it change it to Eevee [146:06] stuff or is it actually changing it to [146:08] uh a better part here? So, let's take a [146:10] look here. So, what did it change? [146:18] Let's take a look here. So, here's what [146:19] changed the domain. EV. No, I didn't [146:22] want you to change the domain. I just [146:25] wanted you to use that as an example of [146:29] uh specific task decomposition. [146:33] Okay, so there it's already kind of [146:35] messed up and I had a feeling that it [146:36] would do that because it literally did [146:38] not put eg or stuff in there and maybe [146:40] it's just that what we're trying to do [146:42] does not work for our use case, right? [146:44] Maybe. But like I I still think it is [146:46] because you are doing research, you're [146:48] collecting information and and gathering [146:49] it. But we're just assuming that we [146:51] already have these things and doing [146:53] analysis on that information. But uh you [146:55] know when you're doing broad research [146:57] and there's a lot of information, then [146:58] it can do do more there. So revert the [147:00] domain name, but we'll do [147:02] self-reflection structure into the [147:03] hiring coordinator. Um and so we'll take [147:07] a look at what it has and maybe we still [147:08] will do the Eevee example separately. [147:12] Um, [147:14] I mean it still has these in here, so [147:16] I'm not sure what it was saying. I just [147:17] don't say it because I'm not trying to [147:18] change that right now. [147:23] Yeah. So like you are a research [147:24] coordinator [laughter] [147:27] and uh I don't know this is the way that [147:29] makes money where you tell it something [147:31] and it doesn't do the right thing and [147:33] it's making more stuff here, but we'll [147:34] wait a little bit. Okay. All right. So [147:36] it's back and so we say okay um back to [147:39] original domain right in the pattern. [147:41] What changed? Narrow coordinator mirrors [147:43] the agent basic tells the model exactly [147:45] the three checks. Fix the pipeline. No [147:47] self-reflection. But that's not what I [147:49] want. Better coordination. Same domain, [147:50] same spokes, general initial screening [147:52] angles. What am I missing? Fill the [147:54] gaps. [147:56] And you know, I think I think it's [147:57] struggling here. Let's go back down [147:59] here. So, we'll go and take a look here [148:01] again. Well, here we have the narrow [148:04] coordinator. So, it says here, screen [148:05] the uh the uh candidate by running the [148:07] following checks. And then down below [148:10] here we have better coordinator. So [148:12] generate an initial list of screening [148:14] angles. Ask yourself what perspective [148:16] stakeholders or dimensions are missing. [148:18] Add screening angles to cover those gaps [148:21] only. Then begin delegating to screening [148:23] agent tool. For hiring decisions [148:25] specifically consider technical skills [148:26] and soft skills, hard requirements, what [148:29] candidate has done and etc. After all [148:32] screening angles are covered. Synthesize [148:34] a report here. So now I would imagine [148:37] that it's basically just hitting a [148:39] single agent. Yes, it is. And so before [148:42] we had those separated out tasks, right? [148:46] But just as I thought, it's like in [148:48] order to do it, um [148:52] the idea is that you say you're you're a [148:54] screening agent and then you are [148:55] contextualizing each one of it. So in a [148:58] sense each of these are basically [149:00] turning that into a specialized um a [149:03] specialized one as before we literally [149:05] had three separated ones out right. So [149:08] that I think that's what we're getting [149:09] at. So that is what we want. That's [149:11] actually good. So we'll go all the way [149:12] down here and what we'll do is we'll go [149:15] it says both the screen agent is [149:17] identical both the only variable is the [149:18] coordinator prompt. So with a hiring [149:21] specific checklist of what the [149:22] coordinator opportunity looks for. So [149:23] let's go ahead and run that. I believe [149:26] that's going to give us a better result. [149:27] Okay. So, we'll go here. We'll say [149:30] python main.py. [149:33] We'll run it. [149:36] And [149:38] so, here it's going through it. So, [149:42] and we're seeing the numbered values of [149:44] what it's checking for. [149:48] Okay. [149:51] Does the resume demonstrate all the [149:52] required skills? Does the candidate [149:53] experience depth? Are the there any red [149:56] flags? [150:00] Has some rails experience though [150:01] limited? Um [150:05] uh in the 58 senior range, [150:08] no employment gaps or job hopping [150:10] detected. The career directory is [150:12] logical, etc. Um and so we have that [150:15] there. [150:18] This is the narrow one, right? So fixed [150:20] checklist, no gap check. Okay. So, let's [150:21] go down to the more complex one. Um, so [150:25] we'll go down to this one. So, now we [150:27] have way more information. So, instead [150:29] of those three individualized things, [150:32] um, and remember there those were three [150:33] separate things before, now we have, um, [150:36] these. I just want to compare the old [150:38] one quickly here because I just can't [150:39] fully remember. We go here. [150:43] Yeah. Notice that we have three [150:44] individualized prompts. And even with [150:47] the narrow one, I guess it's still only [150:49] passing it through those three. And so, [150:52] um, that's interesting. But anyway, so [150:54] here, does the candidate demonstrate [150:56] mastery of, etc.? Okay, so we go down [150:57] here. [150:59] And [151:01] I guess we can't really see the [151:02] individualized results. So that would be [151:03] something you might want to do is like [151:06] output all of them and then save them [151:08] and then save the generated uh final one [151:11] to to exactly see what it is. So core [151:13] stack matches. Excellent. Okay. Okay. [151:17] Risk and gaps questions for interviews. [151:20] Final final recommendation. Now we have [151:21] a maybe. Alex is qualified candidate but [151:23] has some gaps for a true senior role. [151:26] Higher if your team values this pass if [151:28] whatever. Bottom line, Alex is a strong [151:29] backend engineer. And then we have [151:31] coverage. And so it's way better in [151:35] terms of its information. But really to [151:38] test this you'd actually have to um you [151:40] know create sample data, right? and test [151:42] it and then and then adjust and say hey [151:44] look uh this is not how I would have [151:46] judged it right based on that [151:47] information but this is the example that [151:49] we wanted but really that works when you [151:52] know there's a generic research agent [151:54] and then these individualized things are [151:56] going in and kind of helping to [151:57] specialize that research agent for its [151:59] task uh but yeah that was cool [152:07] all right let's talk about dynamic [152:09] selection so the idea here is that when [152:10] you have your coordinator and you sub [152:12] agents. Um you might uh find that if you [152:16] run the entire pipeline for every single [152:19] possible spoke uh in a sequence that you [152:22] are consuming as much as you can. And so [152:25] with your coordinator, you probably want [152:27] to tell it to think about what kind of [152:30] pathing it needs or what kind of routing [152:32] it should have and give it ideas of kind [152:35] of routing that it can perform under [152:37] certain circumstances so that it's doing [152:39] exactly what it needs to do. Even here [152:41] it's saying like you know only invoke it [152:43] if it makes sense. Um and a way we can [152:45] catch that problem. So if we have a a [152:47] poorly designed um uh dynamic selection [152:52] system, you can set up a tool just like [152:54] how we talked about with the narrow task [152:56] decomposition. You could set up a tool [152:58] that says hey did you do a good job [153:00] here? You can do the same thing uh with [153:01] a tool as well. Um, and I mean it's [153:05] really going to depend depend on what [153:07] we're doing, but that's something that [153:09] uh you know we'll want to consider. [153:10] Okay. [153:15] Hey folks, we are back and we are going [153:17] to try to do some dynamic selection. So [153:20] um you know for ours uh our our job [153:24] application [153:25] um basically we are taking in um [153:29] information from from one thing but [153:31] basically uh it's just checking [153:33] everything and so [153:36] you know the question is like can we [153:37] even think of any kind of select dynamic [153:39] selection that would be need to be [153:40] performed for a job application because [153:42] I feel like it would be more if you were [153:45] to ask certain questions to the agent [153:47] along with this coordinator then that's [153:50] where I would want to choose different [153:52] types of pathing. And so I'm not exactly [153:54] sure. We'll let it help us think of an [153:57] idea, but I do want to remind folks that [153:58] we are just doing this to learn. If you [154:00] are doing this for real, write these [154:02] things by hand yourself. Use your brain. [154:06] That's how you're going to get the best [154:07] result. Garbage in means garbage out. So [154:10] just because this thing works doesn't [154:12] mean that this is welld designigned. [154:13] We're just going through this uh to [154:16] learn these concepts, right? Okay. Um, [154:18] I'm not saying that this is the best, [154:20] but anyway, we'll go ahead here and make [154:22] a new folder. This one will be called [154:24] dynamic selection. [154:26] Okay. And I'm going to go ahead and make [154:28] a new main.py file. [154:32] And I'm going to go ahead and select [154:34] this code [154:38] and we're going to copy this [154:41] and we'll paste this into here. And so I [154:44] need to give it a concrete example of [154:46] what we're talking about for dynamic [154:47] selection. I do not have my text here. I [154:51] put it in there. I'm going to get uh [154:53] chatbt to extract it out just how we did [154:55] with the narrow narrow one. So just ask [154:58] it to, [154:59] you know, extract out the text for me [155:02] here. I'm just doing this offscreen here [155:04] because I need to give it a practical [155:06] example and try to describe what we're [155:08] doing here. Going to CD back a couple [155:11] and we'll open that up. [155:13] And so we'll go here and just say, you [155:17] know, I want to implement dynamic [155:22] dynamic selection [155:24] for my uh [155:27] uh uh coordinator. [155:32] um so that it's not running the entire [155:36] pipeline [155:38] but trying to choose the best uh things [155:42] to run based on use case. [155:46] Here is an example [155:49] of good dynamic selection [155:53] where we have different pipelines. [155:57] Okay, you can use to uh help you. Okay, [156:03] and the other thing is like edit the [156:08] dynamic selection main.py file. Okay. [156:12] So, it's going to go off and do that and [156:14] uh we'll see what it comes back with. [156:15] So, hopefully something that is useful, [156:17] but you know, if we don't kind of guide [156:19] it and say like these are the use cases, [156:21] you know, I'd be surprised if it doesn't [156:23] come back with anything good. But, we [156:25] will try here just for learning [156:27] purposes. Okay. All right. It's come [156:29] back with something. Let's take a look [156:30] at what we have. Um, so dynamic [156:34] coordinator. Oh, we still have that [156:35] narrow coordinator in there. We should [156:37] really remove that out of there because [156:39] it probably is confusing. We now have [156:41] like three coordinators. Um [156:44] I don't want to have three. [156:48] So [156:51] we'll go here. I'm just going to tell [156:52] like look I I only need a single [156:56] coordinator prompt. So [156:59] uh well I'm being lazy here. If we don't [157:03] need the other ones, we can just delete [157:04] them out. Right? We have this narrow [157:05] one. So we know this one is not [157:10] something we want. So we'll take that [157:11] out. [157:14] Then audits the gaps for delegating [157:16] specific domains. And then here we have [157:18] the dynamic one. So we'll take this one [157:20] out. [157:21] Okay, look at that. We wasted no tokens. [157:24] There's no reason we can't do that. We [157:26] don't have to prompt everything. Then [157:27] we'll go down here and take a look. So [157:29] this coordinator reads uh the roles then [157:30] decides which dimensions actually [157:32] matter. So routing the logic. So strong [157:36] technical match or whatever whatever [157:37] let's take a look and see what we have. [157:39] So routing guidance adapt to what you [157:42] observe don't apply mechanically. So [157:45] simple factual match skip keyword scan [157:49] go straight to this non-traditional [157:52] background transfer skills. Oh this is [157:54] cool. I like this. Never invoke a [157:55] screening agent unless it's answers a [157:58] real question. So I think that actually [158:00] um worked out perfectly. That's a great [158:03] example of of that. And so we'll go [158:05] ahead here and I'm just going to go and [158:07] run it. So we'll CD into dynamic [158:10] selection. [158:11] This has actually been quite fun. Um is [158:13] it useful? I don't know. Depends on what [158:16] you're building. And oh yeah, we don't [158:18] have the narrow coordinator. So I'll go [158:20] here. Just make sure. Narrow [158:21] coordinator. [158:23] Um [158:24] I want to get rid of these other ones. I [158:26] don't want to waste all that here. [158:30] And so, uh, I'm just going to go back a [158:32] step and just say, [158:34] uh, this should be fine. Let's just do [158:36] that. I didn't realize there's more to [158:38] rip out. I think it'll still work, [158:40] though. This is all three coordinators. [158:42] There's only one though, right? Because [158:44] I ripped them out. [158:47] So, here it says, describe the most [158:48] complex [158:50] screening angles delegated. [158:58] Okay. And the only the only way to [159:00] really know if this is different. [159:05] What happened here? [159:08] Oh, we have narrow QS. So, we still have [159:10] some of that remaining remaining code [159:11] there. So, it's just some of this stuff. [159:14] And so, I'll go ahead and try that [159:15] again. [159:20] I'm not sure if that will work if it's [159:21] just a single item, but I'm hoping that [159:23] it does. [159:25] Um, but here, um, you know, my best [159:28] guess is that it's choosing exactly what [159:29] it needs. Because if we go back up to [159:30] here, that's what it looks like it's [159:32] doing. [159:40] Oh my goodness. So, I'm going to go back [159:41] a directory here. Um, because this is [159:44] very frustrating. [159:47] Uh, remove the better. [159:51] I only have a [159:53] single coordinator. [159:55] But I removed some [159:57] of the other code [160:00] uh because I only really need a single [160:03] coordinator here. Can you fix fix the [160:07] code? You know what's funny is that [160:10] sometimes like I will type things and [160:11] every single letter will be wrong and it [160:13] still knows what I'm saying because it [160:14] like it does the off shift which I think [160:16] is really cool. As someone that's [160:18] dyslexic, as long as it understands me, [160:20] I I love that. Um, [160:25] yeah. And so I'm just asking it to clean [160:27] it up. I just want to make sure it's in [160:28] a working state before we move on here. [160:30] All right. So it thinks it's cleaned it [160:32] up. We'll go into here again [160:37] and we'll run this again. [160:41] So screening angles detected. [160:44] What I'm trying to determine when it [160:46] runs here is like how is it selecting [160:47] stuff? So in your current role at the [160:48] fintech, what is the scale of your [160:50] system that you worked on? You [160:51] transitioned from this, have you [160:53] designed or refactored it? So what it [160:55] looks like it's doing is it's actually [160:57] uh generating out uh possible angles [161:00] based on this information. And so it's [161:03] literally creating dynamic routing on [161:05] the fly. So it's not like here is a list [161:08] that we had before here, but literally [161:10] like here are things that you can check [161:12] and then choose what you want to put in [161:13] here. So it's not always applying the [161:15] same thing. [161:17] Then we'll go back up to here. [161:19] It's still a conditional maybe. But [161:21] yeah, we're getting something that is uh [161:22] it's again very interesting to see this [161:24] play out. Um again, you know, we don't [161:28] know if it's actually useful, but it's [161:30] fun to see the system working. Um and [161:33] there you go. Okay, [161:39] let's take a look at partitioning [161:40] research. So if you give three research [161:43] agents the same brief, you get three [161:46] overlapping answers and wasted tokens. [161:48] So if you're trying to paralyze things, [161:51] right, and say research CV market, [161:53] research CV market, and they're all [161:54] doing the same thing, that's going to be [161:56] nonsense, right? So carve up the scope [161:58] so each agent owns a distinct slice. And [162:01] so here we are seeing partition [162:03] information where um we are creating [162:06] structured data and we're providing [162:07] detailed information like topic cover [162:10] excluded things like that and providing [162:12] that information there and as per usual [162:15] we can create a tool that would check [162:17] and make sure that we're dividing the [162:18] research scope into non-over overlapping [162:20] assignments before delegation. What's [162:22] really interesting here is that it's [162:24] making a structure. Um and I mean you [162:27] know in the last thing that we did [162:28] technically it is already kind of [162:29] assembling um its own way of doing stuff [162:32] but um I suppose what we could do is we [162:35] could generate out into partitions um in [162:38] this sense and make sure that it's even [162:40] more detailed in terms of what it's [162:41] covering as an intermediate step um to [162:44] make sure that it's not doing the exact [162:46] same thing. But this one's more focused [162:47] on very specific things that it's [162:49] researching. Um but yeah, the question [162:51] is like does our current one even if [162:54] it's not the exact same one is it having [162:56] overlapping tasks and that's what we [162:57] don't know and so maybe um putting a [162:59] structured structure a structure with [163:01] partitions might help uh but we'll have [163:04] to experiment. Okay. [163:10] Okay. So let's see if we can implement [163:13] partitioning in here. Um so what I'm [163:16] going to do is make a new folder here [163:17] called research partitioning. [163:20] partitioning [163:22] and we'll go ahead and make ourselves [163:24] another new main.py file. We're having [163:27] lots of fun here. And so we'll grab this [163:29] uh wasn't this one. This is the last one [163:31] we worked on, right? Going to grab this [163:33] here. [163:35] Copy it and um go all the way down [163:42] and oh wait, no, no, this one's empty. [163:43] Here we go. Okay, so now we are back [163:46] with our dynamic one. So here we have [163:49] offscreen here I just told it to extract [163:51] it out right and [163:55] um I'm going to just say here like you [163:56] know I want to make sure I want to make [164:00] sure um my [164:04] uh research what are they called [164:10] what did they call them here [164:12] my research agent agents aren't wasting [164:16] credits [164:18] uh tokens by having [164:22] time by having uh overlapping [164:25] tasks. [164:27] And so I would like to have another [164:33] step where we have uh partitions. [164:38] Um [164:40] and I mean like the thing is like you [164:41] could manually make this up but I rather [164:43] just generate it out so it makes easier [164:44] for us. So we have partitions uh uh have [164:48] a step where we generate out [164:52] partitions [164:54] based on a JSON structure [164:58] and then we can determine [165:00] if there is if they are truly not [165:07] doing the same task. Make sure to print [165:10] out the structure. So the human can uh [165:16] see it on the run of the coordinator [165:19] agent. [165:22] Update the [165:25] research partitioning main.py. [165:28] And here is an example [165:31] of partitioning [165:37] uh from a different use case. [165:41] Okay. And so I'm gonna copy this over. [165:44] Bring it on over here, [165:47] right? And I'm hoping that this will [165:50] work, right? But I mean, like this also [165:53] could just be like a static way, like if [165:54] you were just statically uh building a [165:57] research agent, um that this would be a [165:59] means to which you could do it and you [166:01] don't have to delegate so much out to [166:02] the agent, if you will. But then now [166:05] we're kind of relying more on um code [166:09] driven logic. But you can mix them. By [166:11] the way, we didn't we didn't mention [166:12] that. But you can take a hybrid approach [166:14] where some of it is the coordinator and [166:16] some of it is code driven. There's [166:18] nothing wrong with it. There's no rules [166:19] here, folks. There's no 100% bad. It's [166:22] what you want to do, right? Um and so we [166:25] will see if it can come up with [166:26] something here. And then we will review [166:28] that code. Okay. [166:31] Okay. Okay, let's take a look and see [166:33] what it thinks it's doing here. So, [166:37] uh, narrow. We still got this language [166:39] like narrow versus better. I probably [166:40] should get that out of there. [166:46] I really should be taking this out so [166:48] that uh it's not getting as complicated [166:51] here. So, screening agent prompt. You [166:54] are a specialist hiring analyst. You [166:56] will be given specific screening uh [166:58] questions about a candidate. Answer the [167:01] questions two to three focus sentences. [167:03] Be concrete and specific. So really [167:05] changed it in this case. Um, [167:09] oh no, no, this is fine. This is still [167:10] the same. Runs a specialized agent. [167:13] Calls once per screening, etc., etc. Um, [167:18] I don't want Oh, I do not want multiple [167:21] agents. Look, I don't want more more [167:24] than one coordinator. Uh, we don't need [167:31] the narrow [167:33] coordinator. Okay. And so it's just [167:35] because we copied it and I had some of [167:36] the code still lying around and that's [167:38] what's Oh, no, no, no, no, no, no, no, [167:40] no, no, no, no. I just realized I was [167:43] editing the wrong file. [laughter] [167:46] Okay, so I went back there and I'm going [167:48] to make sure I didn't muck that one up. [167:51] Uh, yeah, I don't want to muck with this [167:54] one. [167:57] Oh, now I don't know. [168:01] Did I break it? Good. [168:03] Um, [168:12] I think I can just go ahead here and [168:14] discard the changes. I think it'll be [168:15] fine. Okay, so we'll go over to here and [168:17] this is the one we actually wanted. And [168:20] so it still has that logic in here which [168:22] is kind of a problem. But I we'll see if [168:24] it actually is an issue [168:27] because we only have one, right? And I'm [168:28] just going to remove I just don't want [168:29] it to get confused [168:36] and I don't want to explain any of that [168:37] here. So I'm just going to take that [168:39] out. So let's take a look here. Says for [168:42] both coordinators. So spoke system [168:44] prompts. I'm just going to take that [168:45] out. [168:47] It keeps talking about like [168:50] okay [168:54] and now let's take a look here. So you [168:55] are a partitioning uh a screen [168:57] partitioning planner given a job posting [169:00] resume output a JSON array of non-over [169:02] overlappinging screen partitions. Each [169:05] partition object must have an agent a [169:08] scope um rules design partitions so that [169:11] together they cover all relevant hiring [169:13] questions. No two partitions may share [169:16] the same cover uh cover aspect only [169:20] partitions that are genuinely needed for [169:22] this candidate. I feel like uh we lost [169:25] uh information here. [169:29] We have oh this is the planner part. [169:31] Okay. So this is actually a separate [169:32] part. Okay. So that just generates the [169:34] partition. All right. And then down [169:35] below here we have um the actual dynamic [169:40] coordinator. So you you're here. Invoke [169:43] exactly one screen partition call per [169:44] partition. No more less formulate the [169:47] questions for each cell. Do not invent [169:49] additional screening angles beyond the [169:51] partitions provided. They were designed [169:53] to cover all relevant dimensions without [169:55] overlap. And so I mean like that's [169:56] another way where it's just specifying [169:59] it in a different [170:01] way, but I guess it's generating out the [170:03] partitions. So in the other one we [170:06] literally listed out possible things and [170:08] here it's generating them out. Oh, here [170:10] it is. Um, [170:15] is this good? [170:17] Because before we had a list, right? So, [170:19] we go back over to our um we'll just [170:22] split this for a second and we go back [170:24] to our dynamic selection here, [170:31] right? And so here [170:34] we had this, but we lost our routing [170:36] guidance. [170:39] So, this is what I'm going to ask. [170:43] I'm going to resume the last [170:44] conversation we had. [170:47] Got make this a bit larger here. [170:59] No. So, it doesn't we don't have it [171:00] anymore, unfortunately. Or maybe I ran [171:02] into a subfolder and that's my problem. [171:03] So I'm going to go here and ask it like [171:07] we have uh we have a dynamic [171:12] what is it that we have? We have a [171:15] uh [171:17] research partition [171:19] so that we don't have uh overlapping uh [171:22] researchers doing the same thing. [171:27] Did we lose [171:29] uh selective routing [171:32] based on task? Uh and do we need to [171:37] bring that back in while preserving [171:44] our uh partitioning? [171:48] Okay. And so I'm going to go point to [171:52] dynamic selection [171:55] has the original [171:58] prompt that had routing. [172:02] And then here we have research [172:04] partitioning [172:08] is our um new prompt with partitioning [172:12] but the routing was removed. [172:15] And so how would it know to do [172:22] routing [172:24] like do uh like how would it know to [172:28] choose the appropriate [172:31] dynamic selection? [172:34] Okay. And so that's where I think [172:37] there's a bit of an issue. [172:39] Okay. Because sure like we now it will [172:43] generate out that structure and things [172:44] but how does it know to drive what to [172:47] generate [172:48] because it doesn't say that right and so [172:51] D the both selects the angles matters [172:54] and delegates it has routing that skips [172:56] beyond the scan and strong matches [172:58] now owns the selection step but it has [173:00] no routing rules it just generates [173:02] non-over overlapping partitions well it [173:03] guides which uh on which one will skip [173:08] and so here. [173:14] Okay. [173:20] So, I hit tab here. Okay. You are moving [173:23] it back in, but are you thinking about [173:28] how it will work or are you just dumping [173:31] it back in? [173:34] Okay. [173:38] because I could have done that. That [173:41] doesn't mean that's a smart approach. [173:44] And maybe that's all we need to do. But [173:46] let's take a look and see what it comes [173:48] up with. [173:50] I'm telling you, these things aren't [173:51] smart, folks. You got to think for it. [173:54] So, here we go. Fair point. Let me think [173:56] about it. So, we have partitioning that [173:57] sees the resume and then the dynamic [173:59] coordinator which sees the partitions. [174:01] Where does the routing actually live [174:02] now? So, in dynamic selection, the [174:04] coordinator was doing two of one things. [174:05] deciding what to check and calling [174:07] agents. Uh and the research partitioning [174:09] now splits routing partitioning [174:10] planner's [174:12] job and then the coordinator's job. So [174:14] the routing rules do belong to the [174:17] partitioning planner but they need to be [174:18] reframed. In the original one it says [174:20] never invoke the rules unless here we [174:22] should say only create partition if the [174:24] rules govern partition creation. The [174:26] coordinator in the new architecture [174:28] should stay as is. It's correct to be [174:30] dumb selection because the decision was [174:32] already made upstream. Uh giving it [174:34] routing logic. Again, we create a [174:37] conflict of two places fighting over [174:38] what gets evaluated. Um the partition [174:40] planner currently says only include the [174:43] partition genuinely needed. That's [174:45] vague. Adding a specific horistic [174:48] routing. I mean like I would have [174:49] thought and you might agree with me is [174:52] that in the dynam coordinator we had [174:55] that list. Uh but the thing is is like [174:58] it was saying like okay if you do this [175:00] then then do this or do that. But maybe [175:02] the problem was is that when we looked [175:04] at the EV research it literally had [175:06] pipelines where in this dynamic router [175:08] it just had things that you could choose [175:10] from that you might want to consider. [175:13] Okay but it wouldn't run them all. So [175:15] here it says we receive a set of [175:17] pre-planning partitions as JSON. Invoke [175:20] exactly one screening agent. [175:24] Uhhuh. [175:27] Okay. Well, let's just see what we get. [175:30] Okay. [175:32] I'm not sure if I like it, but [175:35] we're trying here, right? [175:38] And we'll go main.py. We'll run it [175:43] and see what happens. [175:46] So, what we have? We have [175:49] core stack proficiency [175:58] assess rest API capabilities. [176:01] Okay. So we have here um core sacrifices [176:04] evaluate mastery of required [176:05] technologies uh directly matching the [176:07] job stack. [176:11] Uhhuh. Access REST API design [176:14] capabilities. [176:16] Evaluate exposure to scaling patterns [176:18] and nice to have technologies. Confirm [176:21] senior level experience. [176:27] And then here we have screening angles [176:29] delegated. Does the candidate [176:30] demonstrate mastery of required stuff? [176:33] Uh okay. [176:38] And here we're getting partials. So we [176:40] have a maybe recommendation. Alex is [176:41] qualified to mid to senior [176:46] and we have different coverage. So I [176:47] still don't know if this is better. I [176:49] mean like we should be dumping all these [176:51] logs out and then comparing them and [176:52] then and doing stuff. So obviously we're [176:55] just trying to meet the requirements of [176:57] uh learning this stuff and kind of [176:59] having a sense of it. But is it good? Is [177:01] it is another question that will take [177:03] more time and I'm going to keep [177:04] repeating that because I just want you [177:05] to know just because we're doing it [177:06] doesn't mean it's great. Um, and you [177:09] should be thinking about like, okay, if [177:10] I had these three, four different ways. [177:12] Um, you know, determine usage, determine [177:15] outcomes, have your examples. Don't have [177:17] them for you here. That'd be a lot of [177:18] work for me to set up for you. Um, but, [177:21] uh, no, it's interesting trying to try [177:23] out these techniques and apply them. [177:24] Okay, [177:30] let's take a look at a refinement loop. [177:32] So, the idea right now is that um, [177:34] everything's been one shot. The idea is [177:36] it goes through it. it produces an [177:38] evaluation and then then it's over. But [177:40] what if we could feed it back in the [177:41] loop and refine it until we are happy [177:43] with it and that's the idea behind [177:44] refinement loop to make our research [177:46] system really really good. Um so if you [177:50] look at this prompt here for our [177:51] coordinator the idea here is that we are [177:53] telling it that we can have up to [177:55] maximum four refinement iterations and [177:58] that we are going to redelegate the [177:59] information uh back into here. And so [178:02] you'll notice here we have like an [178:03] evaluation coverage and when to submit [178:06] final uh and uh uh creating the [178:09] synthesis and things like that. And so [178:12] we will go ahead and try to apply [178:14] refinement loop to our um our agent. [178:18] Okay. [178:24] Hey folks, it's Andrew. In this video, [178:25] we're going to implement our own [178:27] refinement loop. Uh, so what we'll do as [178:29] per usual, I'm going to go ahead and [178:30] make a new folder. This will be my [178:32] refinement [178:34] loop. Okay. And then what we're going to [178:38] do is we're going to go ahead, let's [178:41] just grab our code here, main.py. We're [178:43] going to go grab our last one, which was [178:44] research partitioning [178:47] because we're building off of it every [178:49] single time, trying to make this thing a [178:50] little bit better. And we are going to [178:53] implement refinement loop. I need to [178:56] extract the text out because again I I [178:59] don't have it on on hand. But uh let me [179:01] go grab it from that slide. Okay, there [179:04] we go. I grabbed it. And if you want to [179:05] grab it too, all you got to do is take a [179:06] screenshot, feed it to Claude or Chat [179:08] GPT and extract it out, folks. Um [179:11] because you can you can make sure you do [179:13] that. Okay, it's not hard. Just build up [179:17] those skills. Okay. Uh so I'm going to [179:19] go ahead here and just CD out here. [179:21] We're going to go into our claude and [179:25] um you know I need to implement a [179:28] refinement loop [179:30] um [179:31] uh in my agent for [179:36] research partitioning main. Here is uh a [179:41] example [179:44] from another [179:47] use case you can uh use as inspiration [179:53] as guidance. [179:55] Okay. And so I'm going to paste that in [179:56] there. And so the idea though is that [179:58] with that information I'm hoping that it [180:01] can develop that refinement loop in [180:03] here. So we will see what it produces. [180:06] Okay. All right. So in here we have um [180:09] changes. Let's take a look. And well, [180:11] it's still trying to edit stuff. So yes. [180:13] Um and let's see [180:17] what we have. Okay, so it is bringing in [180:22] um evaluate coverage. Okay, so we have [180:26] that uh submit final. So it's setting [180:29] different states based on whether you [180:31] know high or maybe or pass. only call [180:33] this when the evaluation confirms uh [180:35] sufficient coverage and final [180:37] recommendation. So we have that in our [180:39] loop here. It is adding the evaluation [180:42] agent. Okay. [180:45] And we have some tweaks here. So we have [180:46] initial screening invoke exactly one [180:48] screening agent call per partition. [180:52] Formulate each question. That's fine. [180:54] Phase two evaluate coverage after all [180:55] initial partitions agents have reported [180:58] call uh evaluation coverage with plain [181:00] text. Then here we have refinement max [181:02] reiterations if the evaluation coverage [181:05] returns sufficient falls. Invoke [181:07] screening agents to fill only identified [181:09] gaps. Uh call submit final etc etc. Do [181:12] not call the submit file before [181:14] evaluation uh if it's only once. Okay. [181:17] So here it's obviously done a lot. I'm [181:20] kind of curious to think like maybe it's [181:21] just like you're brute forcing to make [181:22] it either that you really want this [181:24] person or you really don't want this [181:25] person. It'd be interesting to have a [181:27] larger data set like let's say 100 [181:28] applicants and you ran it through and to [181:30] see if it just skewed it to one location [181:32] or one side or not. Um, but it would be [181:35] very interesting to find out. But we'll [181:37] go back up to [181:40] here and so we can see the changes and [181:44] let's go and run this thing. Notice as [181:47] we are progressing it's becoming easier [181:49] and easier for us to update our agent. [181:52] And so far we've been just using the [181:54] anthropic um SDK, not using the agent [181:57] SDK. The agent SDK is awesome, but um we [182:00] will just continue on here. It'd be [182:02] interesting to convert it over and see [182:03] what the code looks like. And we'll [182:04] probably do that. U let's go ahead and [182:07] do python main.py and we'll go ahead and [182:09] run that. And the idea is it's going to [182:11] run. It says dynamic coordinator. [182:13] Obviously, it's the uh refinement one. [182:14] We don't change those names. And so here [182:17] reads candidates first routes to the [182:19] relevant checks only. So evaluate depth [182:22] access to database caching verify API [182:25] confirm senior level experience [182:28] and is the candidate senior level. Okay, [182:31] great. So now we're going into iteration [182:33] one. [182:35] Okay, [182:37] so we have coverage score, code quality, [182:39] practices, no evidence, etc., etc. And [182:41] so it is going again here [182:46] asking questions. [182:50] They are I think they are different [182:51] questions. [182:53] It's hard because we have the this up [182:54] here, right? And then down below. [182:57] Oh, look the the coverage score is going [182:59] down now. Interesting. [183:04] And so we are done and over with. We'll [183:07] go and uh look up here. So it did two [183:10] iterations [183:12] and their score went down. [183:15] So yeah, that's an iteration loop. Is [183:17] that good? I don't know. It takes a lot [183:20] of work to evaluate this stuff. We would [183:22] spend hours, hours upon hours tweaking [183:25] this to figure out is this valuable [183:29] information? Is our data set good? etc., [183:31] etc. There's no magic here folks. We can [183:35] uh code these out very quickly, but to [183:37] make sure they actually work good is a [183:38] different story. I'm going to keep [183:39] repeating that because it's true. Uh but [183:42] that is what the refinement look uh [183:44] refinement loop looks like. Okay. [183:55] Hey folks, we are back and I'm going to [183:57] make a new folder and this will be uh [183:59] coordinator observability because the [184:02] idea here [184:04] is that our coordinator should act as an [184:07] observable layer and so I want to make [184:09] sure that it actually is doing that. So, [184:10] let's go back here and we'll uh uh go [184:15] into um well, we'll type claude here, [184:17] right? And we already have the [184:20] refinement loop over here. So, I'm going [184:21] to go ahead and grab all this code. [184:25] Okay, I'm going to grab all this code [184:27] and we'll make a new file called [184:28] main.py. You're starting to get the [184:30] pattern here of what we're doing, right? [184:32] Very repetitive, but it really is good [184:34] in iteration. So, um, what I want to [184:37] figure out is, do we actually have those [184:40] values? Is the coordinator acting like a [184:42] coordinator? So, um, [184:46] I'm just thinking about this for a [184:47] second. So, what we want to do, so we're [184:49] going to say, uh, I have an agent, [184:54] uh, a coordinator agent, [184:57] uh, here. So, we'll say, uh, coordinator [185:01] observability [185:03] Here [185:06] I'm going to put this into plan mode. [185:08] Here are the questions I have. is my [185:11] coordinator [185:13] operating from observability level [185:18] observability and uh [185:21] uh I'm trying to think of the word like [185:24] and controlling controlling flow of [185:27] information [185:28] uh you know is it uh managing you know [185:33] like is it I'm trying to think like here [185:35] I have a coordinator agent right [185:38] here are the questions I have is my [185:39] coordinator They're operating uh [185:40] operating from uh uh operating with an [185:43] observability layer. [185:47] So we can capture [185:51] uh any errors all messages that that [185:54] that are being uh sent to our spokes [185:59] sub agents. [186:02] Is it controlling [186:05] context and what is passed [186:08] uh to my um [186:12] uh spokes and only those sub agents [186:20] can talk to the coordinator. [186:27] Right. [186:30] Is there something I am missing [186:34] to make my coordinator [186:39] like a coord coordinator, right? That's [186:42] what we want to know. And we'll go ahead [186:43] and hit plan there. You're thinking, [186:44] well, you know, you're just writing [186:46] whatever out. But that's that's a fine [186:48] prompt. You can't make perfect prompts [186:49] here, folks. I mean, you can make better [186:51] prompts yourself and spend time [186:53] engineering them. But this is good [186:55] enough to start getting some [186:56] information. So let's see what it comes [186:58] back with. And I'd be curious to uh [187:00] layer something here on uh there. So I'm [187:02] just curious what we can do [187:05] um to make our observability better. Um [187:08] but we will wait for that generation. [187:10] Okay. [187:12] Okay. So let's take a look and see what [187:14] it thinks. Um so we go all the way at [187:17] the top here. So the users coordinator [187:20] uses a partitionbased hubspoke pattern [187:22] for job screening. They want to know [187:23] does it have proper observability? Does [187:25] it capture all messages of spoke? Does [187:26] it control past the spokes? Is uh is the [187:29] spoke coordinator communication [187:31] isolated? What is missing to be a good [187:32] coordinator? So it says observability is [187:34] weak. I absolutely agree. We haven't [187:35] done anything for it. Um could we do [187:39] something better than just Oh yes, says [187:41] only print statements. I agree with [187:42] that. Um uh error handling anywhere. It [187:45] says JSON loads. Yeah, that makes sense. [187:47] API level tracing, no token counts, no [187:49] latency, no request IDs. Yeah, token [187:51] counts would be really useful. uh no [187:53] persistent audit trail of all output and [187:56] epheral std out. We have message capture [187:58] partial. So delegated list. We are [188:00] printing some stuff right because I told [188:02] it to do that. But um tracks questions [188:05] sent to spokes. Spoke responses are [188:07] never stored. They go straight into the [188:09] tool results and are lost after the [188:11] coordinator. No record of partition. [188:13] Each question [188:15] addressed X. No log of coordinator's own [188:18] reasoning. Texts between tool calls. [188:20] Mhm. every spoke receives the full job [188:23] posting [188:25] and so saying context control loose [188:27] resume regardless of its partition I [188:30] mean it would need it anyway right like [188:31] all of them need it so we're not really [188:33] giving them separate data uh regardless [188:36] of its partition scope so cover exclude [188:40] is only advisory only passed by JSON to [188:43] the coordinator prompt but not enforced [188:45] at the spoke level the coordinator could [188:48] ask any question no mechanism validates [188:51] the question stays within the assigned [188:53] partition. [188:55] Fair enough. Spoke isolation one [188:57] direction one direction law enforce. So [188:59] spokes are stateless functions called by [189:00] the coordinator. No cross no cross spoke [189:04] communication. So that's good. But [189:06] spokes have no awareness of their [189:07] assigned partitions. They can't reject [189:09] out of scope questions. Oh, that's [189:11] actually a good good point. No. Uh only [189:14] these folks can talk to coordinator [189:16] mechanism. It's just a single plane one. [189:19] So where are our gaps? Can't debug or [189:21] audit the run. Silent crashes on [189:23] failures. Can't replay or inspect what's [189:25] wrong. Coordinator doesn't know if all [189:26] dimensions are covered mid-run. [189:28] Recommendation may be issued before all [189:30] uh angles are addressed. Spokes are all [189:33] data even if relevant. All partitions [189:36] even if some obviously irrelevant. A [189:37] single pass cannot fill gaps during it. [189:39] And also our data like if really if we [189:42] may want to make this better, we would [189:43] be sending like different payloads of [189:45] different information to specific [189:46] research agents. But right now [189:48] everyone's getting the same job posting. [189:50] Um, so here they have recommend fixes. [189:52] So structured logging with time stamp [189:54] levels. That seems good. Uh, that's [189:58] fine. Sure. And it's going to log that [189:59] out. Error handling. So wrap the JSON [190:02] low generate the partition. Uh, persist [190:05] spoke inputs output. So extend beyond [190:07] the tracking. Add coverage evaluation [190:10] tool at add explicit gates. Force the [190:12] coordinator to call submit final. [190:15] And so these are fine. [190:18] Um [190:20] the only challenge here is I don't I [190:21] feel like there's a lot of tasks here. [190:22] So I'm just going to go here and say [190:24] there is a lot of of work here. I am [190:29] concerned [190:30] uh you will [190:33] uh not be able to remember all the [190:35] tasks. [190:37] Can you create this uh plan in a readme [190:42] with a task checklist and can you check [190:47] off the tasks [190:51] as you complete them? [190:54] Okay. And so the goal there is to help [190:56] it out a bit. Um that's not exactly [190:58] spectrum development, but the only thing [191:00] is like if we really wanted this to [191:02] drive and be really good, we would want [191:04] something that would clear context each [191:05] time. But we're not set up that way. I'm [191:08] not here to roll a a small spec driven [191:11] development little thing for us here. So [191:13] that's totally fine. And um [191:17] read me with tasks and checklist. [191:20] As long as it knows where it's doing [191:21] that, that's fine. But where's it going [191:23] to put that file? Yeah, I'm just going [191:25] to trust that it can do it. Let's go [191:27] ahead and let it rip. [191:31] Okay. Um, and so the idea again here is [191:33] to improve observability and we will uh [191:36] see how that goes. Another thing is like [191:38] I would imagine that if you were to put [191:40] this into production, you want to [191:41] productionize it, you might want to [191:42] contain these sub aents into containers [191:45] and then you might want to use hotel as [191:48] another observability layer. That's how [191:50] I'm kind of thinking about it. But we're [191:52] keeping it all monolith for now and [191:54] we're not going to overly complicate it [191:56] at this stage. Um, and I'm going to let [191:58] it go and burn away all my credit. Look [192:00] at that. 6.2,000 2,000 credits. Wow. [192:03] Let's go over here. It's like the worst [192:05] time to do this. People are like there's [192:07] on Twitter they're like, "Oh, it's down [192:09] and the usages are gone and stuff like [192:11] that." It's me. It's me. I'm the I'm the [192:14] problem. So, we go over here and right [192:17] off the bat like we are uh Oh, resets in [192:20] 9 minutes though. That's really good. [192:21] But only 33%. Well, let's burn burn, [192:23] baby. Though my week is is is uh getting [192:26] up to use very quickly there. Um, but [192:29] anyway, [192:32] oh yeah, it's it's going up. So, yeah, [192:35] we're going to consume tokens like it's [192:37] nobody's business. Um, but I think it'll [192:39] be worth it for this stage of the uh [192:42] thing as we are continually refining it. [192:44] Okay. Uh, that was fast. I feel like it [192:47] should have taken longer than that. All [192:48] six fixes are implemented. I mean, I [192:50] would rather it have been more granular [192:52] and clearing context between it so that [192:54] we would have um, you know, better [192:57] stuff, but that's fine. So, is my [192:58] coordinator operating with [192:59] observability? No, it had only print [193:01] statements, etc., etc. What was I [193:03] missing? Error handling, midrun gap [193:05] detection, no exit gate. And I'm not [193:06] even saying like this is the best, but [193:09] um you know, it's pretty good for us [193:12] throwing things here together. Let's see [193:14] if we can see what code I mean, it [193:15] probably will just tell us what code [193:16] changes were made. I suppose that's the [193:19] easiest way to check. [193:21] And um I'm just going to go all the way [193:24] up here. Let's take a look here. So, [193:25] we've added logging. Okay. [193:29] And we are implementing the logger [193:33] here. It says scope to each partition [193:36] part agent name uh is the names the [193:39] question belongs to from the partition [193:40] JSON. Okay. So it's being very [193:42] particular to make sure that it's [193:44] scoped. That's good. Evaluate coverage. [193:46] So midrun gap detection tool. Evaluate [193:48] whether the screening finds are [193:49] efficient to make a confident [193:51] recommendation. [193:52] Um confident that all partitions agents [193:55] have reported. return a coverage score, [193:58] etc., etc. [193:59] Okay. Submit final explicit exit gate. [194:03] Submit the final hiring recommendation [194:05] only. This call this only after evaluate [194:08] coverage. Fair enough. So, we'll go down [194:10] below here. [194:13] Mhm. Uh, fix error handling. So, here we [194:16] have [194:20] uh here's down the error handling down [194:23] here below. Fair enough. That's very [194:25] basic. That's not really that important. [194:28] And then we have the screening agent. [194:32] So we are seeing [194:36] oh to scope it into the boundary. Right. [194:38] So making sure that it's scoped. Fair [194:41] enough. [194:43] Here we have rule changes. [194:46] And so it's about passing that [194:48] information and it's talking about that [194:49] evaluation coverage in the final [194:51] recommendation. [194:54] Okay. And then we got logs, logs, [195:04] and more logic. Now, this thing is [195:06] pretty wild. I would probably want to [195:09] take it farther and refactor it, but I [195:11] don't really want to uh like this is [195:14] just this is a mess. Like this is not [195:16] how you should have your codebase, but I [195:19] don't want to go overboard at this [195:21] stage. I just want to make sure that [195:22] this works. Okay. So, what we're going [195:24] to do because I'm expecting logging to [195:26] appear, right? Um, [195:31] and so I would probably say like we [195:33] could just run it, but the other [195:35] challenge would be like we need actual [195:37] ways to test that this stuff works. So, [195:39] probably would would have been better. [195:41] Um, but it would have taken a like this [195:43] would have took an hour or two and you [195:44] folks don't want to wait around that [195:45] long to test that. But what I would have [195:47] done if we had the time and you wanted [195:49] to go through it, what I would do is I [195:51] would stage um examples and and and I [195:55] would want to see if like we could [195:56] pollute um uh pollute the context [196:00] between agents and make sure that it is [196:04] only receiving proper questions and it [196:06] rejects it and those would be things [196:07] that we test for. So we really are [196:09] skipping a lot of that stuff and that's [196:12] stuff that you would still have to do. [196:14] But just because I'm not doing it here [196:15] doesn't mean that I wouldn't do it. Um [196:16] it's just I'm not doing it because you [196:18] folks don't like when I make like two [196:20] three hour videos. Um even though that's [196:22] the actual effort for the work and I [196:24] can't really just you know fake that [196:27] along, right? So we'll go ahead and [196:28] we'll run that. I think it's called [196:29] observability wrong which is fine. And [196:31] so what I'm interested in is do we get [196:33] any logging? Where's our logging? I [196:35] don't see it. [196:38] Okay. [196:39] Oh, I mean it's just going to st out. [196:41] So, it's not logging to anywhere in [196:43] particular, [196:50] which is fine. Uh, so, you know, I'd [196:52] probably just have it logged to like [196:54] into in a log directory. And that's the [196:57] only that might be missing here. [197:05] Okay. [197:09] And I'm just going to wait for it to [197:10] run. Oh, there we whether it's done. And [197:12] so we have our final information. Did it [197:14] call that final evaluation step? [197:18] Yeah, final recommendation. There it is. [197:21] So there you go. That's all it took to [197:23] improve it. Definitely better than what [197:25] we had before. Um but yeah, I would [197:28] probably want to refactor this. So [197:30] that's like you shouldn't have one big [197:32] dumb file like this. Um and so we might [197:35] do that in a separate video, especially [197:36] if we want to convert it over to the [197:38] agent SDK to compare. That might be [197:40] something we might want to do. Okay. Um, [197:43] but yeah, now we've added observability. [197:49] Hey, this is Andrew. In this video, [197:51] we're going to look at how to handle [197:53] errors when we have multiple agents [197:55] being coordinated. Um, and it's [197:57] interesting because they have this later [197:59] in 5 uh 3, but we covered about sub [198:03] agent stuff earlier. So, I'm not sure [198:04] why they uh like we have sub agent [198:06] failure recovery here where we have this [198:08] example. Um, so maybe what we'll do is [198:10] just um go ahead and say make a new [198:13] folder. We'll say coordinator [198:16] um failure handle handling. And so I'm [198:19] going to go ahead and find that folder [198:22] here. We'll go ahead and say main.py. [198:24] And we'll go back down to our um one [198:28] where it was with the sub aent. [198:31] This one here. [198:35] Okay. We'll go ahead and copy this and [198:37] we'll go all the way up to this one and [198:40] then we'll take a look at what we have. [198:41] So what is it that this agent is doing [198:43] is I don't remember this is is errors [198:46] and so it's about checking for inventory [198:48] placing an order tool notifications. [198:51] Um and so what we want to do is go plan [198:55] MD and uh so [198:58] uh coordinator [199:01] error handle error handle [199:04] handling um [199:08] uh uh test. Okay. So we want to turn our [199:14] sub agent into a coordinator with [199:18] multiple sub aents. [199:22] The goal is to demonstrate um error [199:27] handling by sub aents. [199:30] So we should have at least uh three [199:35] different sub aents. Um [199:40] our goal is to do the following. [199:46] uh return back structured [199:50] uh error context. [199:52] DG failure type [199:56] attempts. [199:59] Uh [200:02] approaches [200:07] be clear [200:10] about the well I mean like it says [200:12] distinguish between access failures [200:14] invalid empty requests. I think like the [200:16] type would indicate that. So that's kind [200:18] of redundant here. Um we don't want [200:21] generic errors. So avoid [200:25] avoid generic errors. [200:28] Says while while silently suppressing [200:30] errors or terminating [200:34] uh why silently suppressing errors or [200:36] terminating entire workflows in single [200:38] failures are both antiatterns for [200:40] obvious reasons because that would be [200:41] bad. You know returning structured error [200:44] context including failure type when [200:46] attempted and partial results [200:47] distinguishing access failed uh for [200:49] valid empty ones. having a sub agent [200:51] implement local recovery for trans [200:53] transit failures. We already have that [200:55] technically. Um structuring synthesis [200:57] output with coverage annotation [201:00] indicating which findings are well [201:01] supported versus which ones. I'm not [201:02] sure what they're trying to ask for [201:03] that. Um so I would say like we have a [201:06] lot of this but basically we're just [201:07] doing this. So um what we'll do is go [201:11] ahead here. I'm just going to save this [201:12] file. We'll open up Claude and I'm going [201:15] to say main implement my plan. [201:20] Here we go. And off to the races it [201:22] goes. We'll be back here in just a [201:24] moment. But you can see like as we're [201:26] reading through this like some of this [201:27] is just fluff and so we need to really [201:29] identify what it is that we're trying to [201:31] solve for. Um but we'll be back here in [201:34] just a moment. Okay. All right. So it's [201:36] back and it's done a bunch of work. So [201:38] it says beta tool decorator function [201:40] replace the JSON tools. Uh, make tools [201:43] factory creates tool closures. [201:46] Oh, what the heck is this? This doesn't [201:47] sound right. Maybe I'm in the wrong [201:49] context. Oh, yep, yep, yep, yep, yep. [201:52] Here's a problem. So, we are we are not [201:54] in the correct location whatsoever. Um, [201:58] and [202:00] I have just mucked up my last one here. [202:03] So, let me just give this a refresh. [202:06] I'm gonna have to undo that. One second [202:08] here. [202:12] Last one here. Um, [202:16] sorry. Can you undo the changes? [202:21] I'm in the wrong folder. [202:25] Okay. So, I'll just undo those changes [202:26] and then I'll go into the correct [202:27] folder. Just be back here in just a [202:28] moment. Okay. All right. So, I was able [202:31] to revert the changes and I just have to [202:33] go into the correct directory um, which [202:35] is coordinator coordinator fail failure [202:39] handling. We'll go ahead and do this. [202:41] I'm getting a little bit tired of uh low [202:43] quality code here. So, I'm going to go [202:44] over to Opus. [202:46] And we are on X high. I don't want to be [202:48] on X high. So, we'll go effort. Um and [202:51] we'll switch this over to high. They [202:53] keep changing it on us here. You got to [202:55] be very careful when you're using it [202:56] because it just like for usage otherwise [202:59] you burn out on stuff really quickly. [203:00] So, please implement uh plan MD. Okay. [203:05] And we will wait for it. Okay. All [203:07] right. So, let's take a look here. Um [203:09] so, end to end works. The coordinator [203:10] delegates to all three sub agents each [203:12] return structure JSON report [203:14] notification sub agent recovered from [203:15] the permissions error. What was [203:17] implemented? Three sub aents. So [203:18] inventory a uh inventory agent order [203:20] agent notification agent. I'm not saying [203:23] that we should be breaking them up like [203:24] this but it's just for our example as [203:27] these are obviously very simple tool [203:28] calls. Coordinator delegates each uh via [203:31] the tool calls. We have structured error [203:33] context. Okay. No generic errors. So [203:37] which we already had before demonstrate [203:40] recovery. So the notification sub agent [203:41] hits a permission error on the executive [203:44] team and recovers it. So it's still [203:45] sticking to our original uh one when we [203:48] utilize this if you remember this use [203:49] case and we will go over to here and [203:53] we'll just take a look here. So we have [203:54] make error make. Okay. So every error is [203:57] typed. Cool. Okay. [204:00] And then uh domain tools. And so we have [204:04] inventory up here and our main tools. [204:08] And so there they are. And then here, [204:12] scope tool set. Each sub agent only sees [204:14] its own tools. Okay. And so here, these [204:17] are the tools that it's allowed to use. [204:19] So check inventory, place order, and [204:21] notification. [204:22] Um, [204:24] sure. [204:26] Okay. So this is just tool handles to [204:28] map to that. Okay. And so they each only [204:29] get one tool, which is fine. We have [204:32] structured report format. So returns [204:33] only a JSON object. Um and so here we [204:36] have status, sub aent, summary, [204:38] attempts, failures and then its [204:41] information and more information here. [204:43] Then we have our inventory agent. So [204:45] here you can see it has its detailed [204:46] rules on how it should recover. Um and [204:49] same thing for this one. Same thing for [204:51] this one. Then we go down to here. We [204:54] see our coordinator. So we have [204:56] coordinator tools, right? That makes [204:58] sense. And then we have the coordinator. [204:59] So you delegate to the three specialized [205:02] sub aents. Each sub aent returns a [205:03] structure JSON report plans the workflow [205:05] from the user request from delegating. [205:08] Okay, makes sense. After all [205:09] delegations, give the user uh user a [205:11] concise summary outcome. [205:14] Okay, and then we have our max steps. [205:16] And so really that's all we're doing is [205:17] we're just structuring the data and [205:19] making sure we are covering our use [205:21] cases here. And then we have our sub [205:23] aent registry down here wondering how [205:25] they're spawning them. And then down [205:28] below [205:30] it has run sub agent. So let's just take [205:32] a look at what that is. [205:35] Oh, it's up here. Okay. And so then [205:36] it'll just spin one up here. [205:40] Um, and I'm just trying to look here [205:42] what it's calling. [205:46] Is this using a sub agent? [205:49] Double checking. Double checking. [205:54] Or is it just putting everything through [205:55] a single pipeline? [206:01] So report await sub agent. [206:07] So the only question I have here [206:10] you know based on this code you know is [206:13] is our sub aents [206:17] actually implemented [206:20] as sub agents like they have their own [206:23] context [206:25] and are unaware of um any other [206:31] agent. They are independently running [206:39] uh running separately [206:45] also like what are we using? Are we [206:46] using the um agent SDK? [206:50] I mean like should we not use the agent [206:55] SDK? Uh and that way we can leverage um [207:01] agent definitions [207:06] check docs and so I'm going to see if [207:09] that makes a difference. There could be [207:11] like a reason why we don't want to do [207:13] that. We'll find out here in just a [207:15] moment. So yes, be back here in just a [207:18] second. All right, let's see what it [207:20] considers because maybe there's a reason [207:22] why. Um [207:25] so here yes architecturally per uh the [207:27] agent SDK own doc sub agents are defined [207:29] for properties concept isolation tool [207:31] restriction specialized instructions we [207:33] have handrolled as functionality the [207:35] same pattern the agent SDK agent uh [207:36] definition abstracts what the agent SDK [207:39] gives us what we don't have this is [207:40] actually a good layout so hand roll [207:42] coordination built-in stuff 50 lines of [207:45] manual response context handling a query [207:48] resume sub aent mint conversation [207:51] okay so we can't do that parallel sub [207:54] sub aents uh built-in file system [207:56] structured reports and so we don't get [207:58] sequential sequential or sorry we don't [208:00] get concurrent stuff for this use case [208:02] sequential is fine stick with the manual [208:05] pattern hide the uh exact mechanics [208:08] we're teaching dropout or structure the [208:10] agent SDK shines when you want file [208:12] system shell sub aents that's what it's [208:15] built for um so for a domain specific [208:17] coordinator then we should use the rise [208:20] to fair enough sure why not Um but uh [208:24] what we'll do is go ahead here like they [208:25] they I mean obviously agent SDK is [208:27] leveraging cloud code stuff. So things [208:30] that are code code related make sense [208:32] but you know sometimes we will get some [208:34] conveniences utilizing it as well. So [208:37] now that that is there let's go ahead [208:39] and see if we can run it. Before we do [208:40] that I want to make sure do we have [208:43] end dot? We do. Excellent. So we'll go [208:46] ahead and say python main.py [208:49] and we'll run it. [208:52] And here we see the the coordinator. So [208:55] a coordinator delegates to three [208:56] specialized sub aents. Each sub aent has [208:59] its own scoop, domain and structure. The [209:00] executive team notification triggers [209:02] permission error. So user request I need [209:04] five units of widget A and it goes here [209:07] and then it returns back the report that [209:11] it was successful and then we go to this [209:12] one and this one's successful and we are [209:15] looking for the failure [209:18] is probably this one here. So here uh we [209:21] send a notification [209:23] and it comes back with [209:27] oh that that looks successful too. [209:28] Executive team agent [209:34] direct uh direct executive team [209:36] notification denied. So I'm looking for [209:38] it in here. [209:42] And [209:45] where's the bubble up? [209:48] So, I'm going to go here [209:50] because I can't see it. It's just it's [209:52] just very hard to see. So, I'll go here. [209:54] Where is the error uh information being [209:59] shown here? Actually, you know what? [210:01] Let's make it so that it absolutely does [210:03] break. Um so, what I'll do here is I'll [210:06] just change the inventory to zero. That [210:09] might be a better example. And so, we'll [210:11] go here. [210:13] Oh, no. Not yet. And then we'll just run [210:14] it again. see what happens because this [210:16] will be probably a lot more clear and [210:18] fail quicker. [210:20] And so here we have an error and here it [210:23] reports back. Whoops. We reports back [210:26] status completed sub aent inventory [210:28] summary. And so that is the information [210:29] that we're expecting to see um for that [210:33] to understand if we're running into an [210:34] issue. So I would say that's pretty [210:36] good. And so hopefully it's very clear [210:39] what it is that we need to do. And I'll [210:41] bring up the exam guide just one more [210:43] time. [210:44] So the idea here is just providing [210:47] structured data that bubbles up, right? [210:48] And being very clear on what to provide [210:51] in terms of information. But yeah, there [210:53] you go. [210:59] Hey folks, it's Andrew and in this [211:00] video, what I want to do is I want to [211:02] refactor our coordinator that we've been [211:04] working on up to this point as I'm going [211:06] to want to port it over maybe to agent [211:08] SDK to just take a look and see what [211:10] that looks like. And so we're just going [211:11] to say um uh coordinator refactor here. [211:17] And I'm going to go ahead and grab the [211:19] code here. And we are going to ask it to [211:23] refactor. Um and let's just see if we [211:26] can make this a little bit more [211:28] maintainable. Okay, if you are not a [211:30] programmer, you might not know that this [211:32] is not good code. Okay. And it like it [211:36] works for this point that we've been [211:37] able to hold this all into memory. But [211:39] if we came back later, we wouldn't be [211:41] able to really make sense of it. And [211:43] just because the agent can make sense of [211:45] it and summarize it to it, that's not [211:46] good enough. We need to make it so that [211:49] it is more human readable. Um, and that [211:51] is what we are going to do. So, I'm just [211:54] going to uh cd out of here and we're [211:56] going to go into claude and uh we are [212:00] going to get some refactoring going on. [212:02] So, what I'm going to do is I'm going to [212:03] make a new file called refactor [212:06] um refactor [212:09] md and I'm going to go say refactor um [212:13] tasks. So, this is this document [212:18] um is the tasks I want completed to [212:22] refactor [212:24] uh this uh [212:26] our coordinator agent. Uh currently all [212:31] code sits in the main.py and we need to [212:35] uh [212:37] break it into multiple files. Okay. [212:42] So uh let's go ahead and start making [212:46] some tasks. I'm just going to make my [212:47] observations of uh what I don't like. So [212:51] the first thing is um the prompts. So [212:55] all prompts should be uh stored [213:00] as um [213:03] all prompts should be stored all prompts [213:06] should be stored as markdown files [213:10] in a prompts [213:13] directory. [213:15] Okay, so that's step number one. The [213:17] other thing is like tools. See how tools [213:18] is very uh wieldy. So uh tools should be [213:23] uh individually [213:26] defined as their own files in the tools [213:32] directory. [213:35] We should have py files for each actual [213:39] tool code [213:41] and the um tools dot the tools. I mean [213:47] like can we that's just JSON, right? [213:50] Um uh can we I don't think there's [213:52] anything special about this and the um [213:55] tools JSON [213:58] for the long tool [214:02] right I think it will understand what [214:03] that is for the uh [214:08] gets passed [214:09] to create so that is definitely [214:11] something I would like fixed what else [214:14] what else um [214:18] junior partition S [214:21] we do have the partition system. [214:26] So we'll say partition uh generation [214:29] should be [214:32] in lib as its own file. That's something [214:35] else I would do. [214:38] Um that's a function that is that run [214:42] coordinator. [214:43] Um [214:48] the logging is inconsistent. I don't [214:50] like how the logging is. So um we should [214:54] have a logger that um refactors [214:59] all the logs to be consistent in a file [215:03] in a file called logger [215:06] in our logger pi in our uh lib [215:11] directory. [215:13] That'd be another thing I would want. [215:16] Um, [215:19] yeah. So, I think that's a start. And so [215:22] I'm going to go ahead and just say uh [215:27] coordinator, [215:32] I want you to refactor [215:36] my code based on that markdown files [215:41] tasks for the [215:44] main.py in the coordinator refactor. [215:48] Okay. And so we will let it go do that [215:51] and we'll see if we can really reduce [215:54] and organize that codebase because it [215:55] should be really easy for us to read, [215:57] right? Like I know we can make sense of [215:59] it because we've been carrying forward [216:00] it, but we really need to be 100% like [216:03] yes, absolutely I know what I'm looking [216:05] at. Okay. Um so I'm just going to accept [216:09] everything that goes along here and then [216:10] we will take a look and see what we [216:12] have. So it's already off to the races. [216:14] We have our prompt, our partition [216:15] planner, our screening agent MD. For me, [216:18] like I probably want these to be [216:20] templates that you can inject stuff [216:21] into, but there's no reason for that [216:22] right now. We don't really need dynamic [216:24] injection, but it's definitely something [216:26] that would be uh interesting to do. We [216:29] have our tools directory. I think we [216:30] only have the one agent here. And then [216:33] we have our tools JSON. So, that is uh [216:36] working out pretty well so far. It's [216:38] going pretty quick, too. Man, these [216:40] things are getting really, really [216:41] better. Here we have our logger. And [216:43] then we are going to have our [216:44] partitions.py. I really don't like that [216:46] we have constants. I do not like using [216:48] constants whatsoever. I think they're [216:50] just it's just bad bad code. Um but [216:52] we'll continue on here. And [216:55] check it out in a moment. I'm going to [216:56] just check how my usage is going. And oh [217:00] it doesn't matter. It just reset. I'm [217:01] back at 2%. Look at that. Lucky me. Eh. [217:05] Okay. So we are just chilling out here [217:07] waiting for this to generate. I'm going [217:10] to pause here and we will come back here [217:13] in a moment. I think it might be done. I [217:16] didn't even really wait that long. And [217:17] so we'll go up here and take a look. And [217:20] so we have our main.py. We have our [217:22] prompts. We have our tools. We have our [217:24] libs. Let's go take a look here and see [217:26] if this is reduced enough. [217:29] Okay. And I probably should have told it [217:32] to check box off these. [217:35] Uh which is fine. So, I'll just go here [217:38] and just check them off myself. I just [217:40] didn't feel like telling it to do that. [217:41] I don't know. I assumed it would be [217:44] fine. [217:46] I could also ask it like, "Hey, is there [217:48] anything else that we could do to [217:50] refactor to make it more human readable, [217:52] but I don't feel like it would know [217:53] because it's not a human [217:57] and it's trained on garbage repos." [217:59] Okay, so let's go into our main, [218:02] wherever that is. Hold on here. Charming [218:06] and I I usually just have a sense of [218:07] like is this readable, right? Um [218:11] and [218:14] it's still yick. It's really really [218:16] long. [218:21] So there's still some stuff in here that [218:23] needs to be refactored. We'll go over to [218:24] here. Um [218:28] so say coverage report [218:34] Coverage report should be its own [218:39] file in lib [218:42] called coverage report. [218:47] Okay. Um [218:53] the other thing is like the data. So [218:57] right now we have hardcoded data. [219:06] Make a data folder and store data [219:10] artifacts [219:12] and load them into the app. [219:17] Okay, that's another thing. Um, [219:31] I really dislike the logging. [219:40] Yeah, we got trace append. [219:44] It's still very very verbose. [219:49] And there are still things. It's like [219:51] I'm noticing here like um [219:57] there are uh there are templates for uh [220:02] content for messages that should really [220:06] be uh templated uh files [220:10] that take variables [220:15] and load it in. [220:17] Maybe um [220:24] okay like is that a prompt? I mean like [220:26] we have this technically are prompts. So [220:28] technically technically they are prompts [220:34] are prompts for content [220:37] and so uh prompts. [220:40] So move them [220:43] them to prompts folders. [220:47] Okay. So, there's that. There's a lot of [220:49] those. Okay. And so, we'll go back here. [220:51] We'll save the file. [220:54] Um, all the way down to here. [220:57] There are new tasks in the refactor [221:07] MD. Okay. And so, we're going to have it [221:08] go off and do those tasks. [221:19] And so we'll give it a moment there. I'm [221:21] going to pause. And I mean, I really [221:23] should tell it to check them off. I [221:24] didn't tell it to do that, but we'll [221:26] come back and take a look. And then [221:27] we'll ask it to do a general refactor. [221:29] I'm still like I really don't like this. [221:30] Like we see how we have like double [221:31] lines and stuff like that. I don't need [221:33] that kind of level logging. Um but I [221:35] would have to explain to it um why [221:37] that's an issue. [221:39] Yeah, it's still just making them MD [221:40] files. is not marking them whether [221:42] they're templates or not, but we'll just [221:43] treat them as templates. And here we're [221:46] getting a lot more in here, so that's [221:47] better. [221:50] But I just know what good code looks [221:51] like. And I I know that's not good code. [221:53] Um, but there's only so much you can do [221:54] with Python. Certain languages have um [221:58] the ability to have better readability. [221:59] Like Ruby's really really good at that. [222:01] I'd love to port this over to Ruby. I [222:03] just didn't check if the agent SDK is [222:05] available. I don't think it is available [222:06] in Ruby. I just think that the [222:08] Santhropic one is. And so if the agent [222:10] SDK was in Ruby, I would absolutely be [222:11] using it over the Python one as I really [222:14] do not like Python code and um it it [222:17] just you just can't get it to be [222:19] extremely human readable. Um [222:21] unfortunately we were all kind of using [222:22] it because of the way the industry is um [222:25] as they've adopted it not because it's [222:26] good just because of mass adoption in [222:29] the uh data data science and stuff like [222:31] that. So, it just became de facto. Oh, [222:33] look. It's already done. And so, we have [222:36] uh that there. And so, we will again [222:38] look at the main file. I'm just going to [222:40] close it and reopen it. Sometimes it [222:41] doesn't always show me right away. It [222:43] didn't check box these off. I really [222:45] should tell it to checkbox them off when [222:47] it does that. So, we'll go ahead and [222:48] save that. We'll go back over to our [222:50] main Pi and we'll scroll up. And again, [222:53] I'm looking that looking at this for uh [222:55] for refactoability. [222:57] And [222:59] I would say like they haven't done a [223:01] good job with logging. So I'd say [223:06] you haven't done a good job refactoring [223:10] the logging. Right? So for example, [223:14] we have log log info partition like [223:17] partition. [223:19] Uh you should be making [223:22] helper functions. [223:24] Uh so these logs [223:30] you like log partition [223:36] um or you know like log right log warn [223:40] and they will add [223:46] uh the you know [223:49] tags. The uh other thing is [223:54] um you have superolous [223:58] logging that is great for human [224:02] readability [224:04] but we want to focus on logging good for [224:10] for uh logs and uh we should be [224:14] outputting [224:16] logs to a log folder relative to the uh [224:20] folder. [224:21] of this agent. Okay. And so that, you [224:24] know, that's one thing that's really [224:25] bothering me. [224:28] I really hate constants. So that'll be [224:31] another thing that we fix here in just a [224:32] moment. But again, we're just trying to [224:35] get this to be in shape. Um, [224:40] did also move this out of here? Like [224:42] what's this big thing? Like why is the [224:44] tool use so large here? [224:48] Okay. Um, and while that is sinking, [224:51] let's go review our other parts of code. [224:58] Okay. I mean, this is fine. [225:01] I think I wouldn't mind if we had like [225:03] this is a big JSON object, but I [225:04] wouldn't mind if we had a shorter syntax [225:06] for this. I don't want to do that right [225:08] now because it's totally possible that [225:11] um uh we will be able to do that in [225:15] agent SDK where it probably already has [225:16] like a shorthand to make that code [225:19] smaller and so I don't want to uh muck [225:22] with that. [225:24] And we'll look at the partition here. [225:26] Really hate those constants. [225:29] And also I really just like how it's [225:31] loading in the prompt template. So there [225:33] should be a way to uh manage that. [225:37] Look, look at all this logger logic. Oh, [225:38] no, that's the logger file. Yeah, here [225:41] now we're starting to get those things [225:42] that I'm that I was asking for. That's [225:43] good. Um, [225:49] okay. [225:54] The other thing that that's and I mean [225:55] we don't need to do this, but like [225:56] technically, you know, we have all these [225:57] sub agents they're calling create. We [225:59] could technically delegate them out to [226:00] different models if we needed to. Um, or [226:03] we could even say like, hey, can you try [226:04] to choose the best model as it's working [226:06] through there? But yeah, I guess the [226:07] next thing on my task is to fix that. [226:09] Um, like I'm not updating the refactor. [226:12] We could keep updating that as a means [226:14] to keep keep track of stuff, but I just [226:16] don't care. Um, and so [226:20] yeah, I want to fix those constants. And [226:22] I want to get something that loads in [226:24] the templates. [226:26] I'm just going to take a look here at [226:27] our usage. [226:30] 9% doing okay over here. [226:35] Okay. And so now that is uh fixed up. [226:38] We'll take a look here again. I'm [226:39] looking at my main seeing if it's [226:41] shorter. [226:43] Yeah, it's looking Yeah, this is way [226:44] less messier. [226:46] Um I don't like using constants. [226:53] Eg. Like [226:56] this is a var. Please don't use these [227:01] in the folder for the coordinator [227:05] refactor. Fix the code. Okay, so that's [227:08] something I really dislike [227:12] and so we will get that improvement [227:13] there. [227:16] This this to me is like there's a big [227:18] issue with the loop. So I feel that we [227:20] need to give it a better instructions on [227:21] like how to better refactor the loop. I [227:23] mean it's using just a big elf is elf [227:25] blocks block. There might be some kind [227:27] of uh state flow machine or something [227:30] that could improve that loop. Um as I'm [227:33] not happy about it. Before we do that, I [227:35] want to fix the template reading and [227:36] loading of files. [227:39] Um [227:42] and so there it's just making basic [227:44] changes for names [227:46] right there. So those are getting [227:47] changed. Good. [227:49] And it'll be done here in probably just [227:51] a moment. It's just updating the [227:53] main.py. And then we will [227:56] have those fix. Come on. Come on. Hurry [227:58] up. Hurry up. [228:07] And also like loading these templates [228:09] and populating them probably needs to be [228:11] um its own thing. Yeah, great. Thanks. [228:14] Okay. Another thing is uh loading [228:17] loading files and templates where you uh [228:21] inject variables. [228:24] Um you can make a new uh uh template [228:30] template um [228:33] template file in the [228:37] uh lib directory. And this [228:41] should [228:42] uh refactor having [228:47] you know large [228:50] load code eg like this. Okay. And so [228:54] that's another thing that's kind of [228:55] bothering me. So we will get that [228:57] cleaned up as well. [229:01] Um [229:03] there's other things like this like see [229:04] how this is like something's happening [229:06] here. So that should be refactored out [229:08] into a function. Uh like everything here [229:11] like just the units of code is is just [229:13] not explainable. [229:16] So the run coordinator definitely needs [229:18] to be broken up into tons of functions. [229:19] Every single of these if else blocks [229:21] should be functions. [229:23] Um [229:25] and I would probably prefer stateless [229:27] classes. I really prefer stateless [229:28] classes as that makes it really easy to [229:30] track inputs and outputs of stuff. Um, [229:34] and Python's pretty good for that [229:35] because of the way it defines uh these [229:37] label tags. I can't remember what they [229:39] called the prop named properties. And so [229:41] that will be good. We're making a lot of [229:43] changes here. So there's a high chance [229:45] this might not work, but that's fine. [229:47] Uh, we can always work through that. Uh, [229:49] it's fine. You're fine, Claude. You're [229:51] fine. Okay. And so now that's loaded. [229:54] I'm not sure if uh that actually [229:56] changed. We'll go back over to here. And [229:59] so with that now when it needs it, I [230:01] feel Yeah. like load prompt. exactly [230:02] like this. Um so yeah the big problem [230:06] still is the run coordinator. [230:10] So the run coordinator [230:13] file is giant [230:17] uh we should refactor [230:21] um [230:23] into a stateless [230:26] uh uh a stateless function [230:29] and [230:32] all the parts of code [230:35] uh should be chunked [230:38] into functions. [230:40] So the functions basically act as [230:44] documentation. [230:46] You know for example the contents of if [230:50] if uh if blocks are really [230:54] uh should be uh function calls right [230:57] we'll go ahead and we'll see if it [230:58] understands what I'm trying to say. But [230:59] like when you write good code, you know, [231:01] like this would be a function, this [231:04] would be a function, this would be a [231:06] function. Um, whatever this is, right? [231:11] These if blocks and we'll see if it [231:14] understands what I'm talking about. Um, [231:17] but I feel like that's a really [231:18] important component. In fact, this run [231:20] coordinator could also be in the lib [231:23] directory. Um, and we might suggest it [231:26] to move that in a moment. But right now, [231:28] I want to see if it could even handle [231:29] what I'm asking it to do. It might not [231:31] understand. Uh, cuz if I don't have like [231:34] an example, it's just not going to know [231:36] what I'm trying to ask for. Again, [231:37] checking my coverage here. Uh, we're at [231:40] 11% usage. Some folks were suggesting [231:42] that like when it's high peak usage, it [231:45] depends on like if you overlap with eur [231:46] European time. I'm not uh obviously in [231:49] Europe and so um they said like just try [231:51] a bit later when everyone's sleeping and [231:53] it's way later so uh you know it would [231:56] be maybe u off peak usage time but [231:58] anyway we'll just wait here and see what [232:00] happens. Okay, [232:02] it's back with functions and again we [232:04] could call plan to ask it to do this [232:05] before but I don't really care that [232:07] much. So we have plan partitions [232:08] validate index partitions run. I've seen [232:10] stuff like with partition information. [232:12] Almost wondering if that should be [232:13] really part of the partitions pi, right? [232:17] Um [232:19] log coordinator reasoning handle this [232:22] information. And so those partition [232:24] ones, there's three with partition ones, [232:28] right? And so we'll go over to here. [232:31] Um [232:34] I mean like sure you could do it that [232:36] way. That's not what I asked for. I need [232:39] to verify this. So, I got to go over [232:41] here like what does a stateless [232:45] class look like in Python? Can it be a [232:50] class with static methods? [232:54] Okay, because that's what I was asking [232:55] for. [232:59] Yeah. Okay. So, look, [233:02] I don't It did not do what I wanted. I [233:04] mean, close. So, okay. We'll go all the [233:08] way down here. [233:12] you know, I wanted a stateless class [233:17] that is a class [233:20] with static functions. Okay? [233:25] Right? So, you didn't uh and I noticed [233:32] some of the functions were handling [233:35] partition logic. [233:38] Is that something that should really be [233:41] part of the partitions [233:44] uh pi? Right. So that's something I'm [233:47] noticing [233:50] as we are shaping that code. [233:54] Okay. [234:04] And so we're going to give that a moment [234:06] to shuffle those things around. [234:11] Now, is it thinking about that or is it [234:13] just shoving things over there? Index by [234:15] agent for partition. Sure. [234:18] Build initial messages. Again, is that [234:21] for partition? Is it actually asking [234:22] that question? Does it belong over [234:24] there? Or is it just that it's handling [234:25] partition data because it moved it and [234:28] didn't actually question whether it goes [234:30] there or not? [234:33] Um, [234:36] but any we'll go over to here and we'll [234:38] take a look of what's changed. What's is [234:41] still working? [234:44] So now this is looking a lot better. [234:47] Um, [234:49] because now we can see what is going in, [234:51] what's going out, right? [234:57] Okay. [235:06] And so we have all these steps. [235:09] So we have call. So create a message [235:13] log reasoning [235:17] again like it does this logging stuff [235:19] belong with the logger. Handle screening [235:22] agent. [235:25] Handle evaluation coverage. Handle [235:27] files. handle submit uh final [235:31] process tool calls [235:34] run. Did they put these at the bottom? [235:36] They did. Sometimes people put these at [235:37] the top or sometimes they put at the [235:38] bottom, but like the one that obviously [235:40] is the big one is this one here. And so [235:42] the idea is that we should be able to [235:43] easily see what it's doing. So we have [235:45] generate partitions, partitions, [235:47] validate overlap, index agents, right? [235:50] And so this should be self-documenting [235:52] as you read it. We go down here. We call [235:54] the coordinator. We do the log [235:56] reasoning. [235:58] Why are these functions? Are these just [236:00] loose functions? They are. Well, no, [236:03] they're part of the partition. And so I [236:04] would go over to here and I would say, [236:06] you know, give the um [236:10] give the partitions pi [236:14] the partitions pi like [236:20] all of our lib directory. Now I'll say [236:23] our partitions pi [236:25] should be a stateless class [236:31] I so a class with static functions [236:37] please update and that's just the way I [236:39] prefer it. Okay I like to group them [236:42] into domain. I don't like having loose [236:43] functions where we don't know where [236:45] they're coming from and who who respects [236:46] them or owns them. um people in the data [236:49] space are very much used to just [236:50] randomly importing a bunch of stuff. So [236:52] they have a uh less sensitivity to to [236:55] that kind of thing. But to me as a very [236:57] professional developer, I I want to see [236:59] where that stuff is coming from. We [237:01] still have some of our if else stuff [237:02] here. And notice in here like these [237:05] again, these should be functions, right? [237:08] All they all they're doing is calling [237:10] these things, but still I want these as [237:12] functions. If there's an if else [237:14] statement in here, especially in our [237:15] main loop, that's what it should be. We [237:17] have a range of 1 to 31. So that's kind [237:19] of defining how many steps it can take. [237:21] Um I would rather that uplifted as a [237:23] variable, [237:25] but we're not going to go too crazy on [237:27] this. I just want to get it in enough [237:28] shape here and mostly just to show you [237:31] like what good code looks like um and [237:34] what you should be doing before you move [237:35] on stuff. You might say, "Well, Andrew, [237:37] why? Like this is more work to read." [237:38] Yeah, but if if you want to write test [237:40] code for this, then you have an input [237:42] and an output and you know exactly what [237:43] to mock going in there and out of there. [237:46] The only thing that I would also change [237:47] is like if these are complex um objects, [237:50] I'd want them to be dumber and only pass [237:53] in really dumb data so that we could [237:54] mock it a lot easier. And this is pain [237:56] points if you've spent a lot of time [237:58] writing uh code. And you might say, [238:00] well, you know, the AI can write the [238:01] test code for us, but that doesn't make [238:03] it good test or reliable tests. And and [238:05] you'll only know that by uh writing that [238:07] stuff. But we'll go over here. We'll [238:09] take a look at our partitions. And so [238:11] that is fine. There's still lots of [238:13] little improvements to be made. like I'm [238:14] looking at like why is that like that? I [238:16] don't like hardcoded stuff like that. Um [238:18] there's just a bunch of little things [238:20] but I'll just say we'll move the [238:21] coordinator over. So uh the coordinator [238:24] uh can be in its uh coordinate class [238:29] should be u in a its own file [238:34] in the lib directory. [238:37] Okay. And we'll move that over there. [238:39] That'll be the last thing we do here. [238:41] Um, [238:43] and then what we will do is we'll just [238:45] run it, make sure it still works. And [238:47] then we'll call this, you know, doneish, [238:51] right? But again, you know, if this was [238:53] something that I would want to put in [238:54] production, I would take the time and [238:56] fine-tune it. I take the time and [238:58] fine-tune it and and because that's [239:00] about getting um [239:03] uh the word I'm looking for is um [239:06] uh technical ownership, right? that you [239:08] have ownership of the code and and you [239:10] know exactly what it's doing. And when [239:12] you shape it like that, then you have a [239:13] better sense of it. So now the [239:14] coordinator is over there. I want to [239:16] just run it to make sure it works. So [239:18] I'm going to CD into the coordinator [239:19] refactor and we're going to go ahead and [239:21] run python or python.mmain.py. [239:25] Okay. So we're going to run that and we [239:27] will take a look and hopefully it still [239:29] works. [239:33] There we go. I wonder if it's going to [239:36] make the log file. [239:38] We do get our logs. Excellent. [239:39] coordinator.log. Okay. And see, that's [239:41] what I meant when I said I want it to be [239:43] nice and uh and whatever. I might even [239:46] suggest like I'd probably prefer to log [239:48] out JSON structure because then we could [239:49] parse that information. [239:51] Um [239:53] yeah, I think I that's what I would [239:54] prefer especially like if you're data [239:56] driven, you have JSONL data as logs. [239:58] It's super super useful. Um, so instead [240:00] of having like coordinator or delegate, [240:02] um, I would probably just have JSON [240:03] objects and then I could parse it and [240:05] ingest it into something else. But [240:06] again, these are little tricks that you [240:08] learn building applications of all [240:10] kinds. Um, but the point is that it is [240:14] running. We just want to see it to [240:15] completion and then we will call this [240:17] done and then we will move on because [240:20] the next section of stuff we are looking [240:21] at is um, stuff that I feel like agent [240:25] SDK is going to be uh, very useful for. [240:28] um they'll all have to decide on that. [240:30] And so it did run, worked great. The [240:32] only thing that I'd probably ask it to [240:34] do, which it's not doing right now, is I [240:35] would have it dump the coverage report [240:37] into its own file. So that'll be the [240:39] last thing that we do here. Okay. And so [240:42] I'm going to go here [240:44] because that would actually make it [240:45] useful, right? So, I'm going to go and [240:47] just say, um, you know, for my [240:50] coordinator refactor, [240:54] uh, it currently generates [240:57] it generates [240:59] a coverage report [241:02] in the logs, but it really should be [241:04] outputting [241:07] outputting um, uh, the a a report [241:11] timestamped [241:12] uh, in a reports directory. [241:16] um and formatted nicely for uh human to [241:22] read. Right. And so that's the last [241:26] thing I would absolutely say we need to [241:27] do. I just realized that that's a little [241:29] bit um gross on how it currently is. Uh [241:32] and we never looked at our data, but [241:34] yeah, we have our job posting and stuff. [241:36] And this we could enrich these later, [241:37] but they're fine. It's really excellent [241:40] data here. [241:42] Uh, we could have made a research that [241:44] would go and grab job postings and make [241:45] it for us. Not that anyone really should [241:47] care about job postings anymore because [241:49] agents are just going at it, but we'll [241:51] wait for this to finish. Okay. And then [241:53] we might run this one more time. Okay, [241:55] there we go. And so, um, it says it's [241:57] there. The other thing is that I don't [241:58] think we're logging uh, usage. [242:01] So, that would be nice to be able to log [242:04] that information out, but again, these [242:06] might be things we get for free when we [242:07] use the agent SDK. So, I'm not exactly [242:10] sure. Um, and so now that it's done, I'm [242:12] going to go ahead and run this one more [242:13] time. Clear. [242:15] And I have no idea how many um, credits [242:17] I'm burning. Like again, I haven't hit [242:19] my I have like $5 or whatever. I haven't [242:21] hit it yet. And Beco is not going to get [242:22] mad if I load up another $5. So, so far [242:25] it is not a pain problem. People don't [242:27] know Beo is the other Andrew. Andrew B. [242:30] I'm Andrew B. And so we call him Beo. So [242:32] it's not confusing. He's definitely a [242:33] real person. He's not um an agent. Or is [242:36] he? We don't know. No one ever sees him. [242:39] Uh, so we're going to run this again. [242:41] I'm going to pause here and then I just [242:42] want to confirm the reports are there. [242:44] But again, you can just see my thoughts [242:45] of like what would be good to do. Okay, [242:48] we still have the coverage report being [242:50] logged here, which I don't like, but [242:52] that's fine as long as we got a I didn't [242:54] we didn't sell tell it to not do that [242:55] there, but we'll go here. And then here [242:57] is our report. We can go ahead and view [242:59] it over like this. And so there is our [243:02] final coverage assessment. [243:05] Um, I really don't like how long it's [243:08] written this stuff. Like if you were a [243:09] human, would you want to read this much [243:10] information? Probably not. Or you'd want [243:13] it summarized in a different way, but we [243:15] never gave it a coverage report [243:16] template. So that's fine. We will [243:18] consider this done. We'll say get add [243:21] all get commit refactor. [243:24] But that wasn't bad for a quick [243:26] refactor. Still lots of work to be done [243:27] there, right? Um, I'll see you in the [243:29] next one. Chowo chia. [243:35] Hey folks, it's Andrew. We are back and [243:37] it's time for us to port our coordinator [243:39] application over to agent SDK. And the [243:42] reason why is that we're going to be [243:43] getting into um uh specific agent SDK [243:49] um arguments. And if we want to know how [243:52] they work, we need to have an example [243:55] over there. And I think we should just [243:56] continue this project forward. And I [243:58] think it's not a bad idea. So what we [244:01] are going to do um is we're going to [244:03] call this uh port to to agent [244:09] SDK. [244:11] Okay. And so what I'm going to do here [244:13] is I'm going to grab the contents of all [244:15] this. Not the logs. We don't need the [244:16] logs or the reports, but we will grab [244:18] this, this, this, this, this, and this. [244:22] Rightclick, copy. We'll go down over to [244:25] our port agent SDK. We will paste this [244:28] stuff in [244:30] and we're are going to let it rip and [244:34] see if it will allow us to port it over [244:36] in one go here. So [244:40] I need to port the my codebase [244:45] uh of anthropic SDK based on uh for my [244:50] agent that uses directly the anthropic [244:55] SDK to use [244:57] Claude agent SDK for this folder [245:02] port SDK. [245:05] And so we're going to ask it to go ahead [245:07] and do that. That's a big thing. Again, [245:08] we probably should have put it in plan [245:10] mode and ask it what it can do. Uh but [245:12] I'm just going to go off to the races [245:14] and do that. And if it works, we will [245:16] explore it. And um we'll have time to [245:19] look at the codebase quite a bit as we [245:21] walk through other features. Okay. All [245:23] right. Let's take a look here and see [245:24] what we have. So, we have the run [245:28] updated. I'm not sure why it did that. [245:31] It's not really that big of a deal. Uh, [245:35] we've removed the async, anthropic, and [245:37] coordinator. These are now internal to [245:39] the coordinator. Sure, [245:42] it has a complete rewrite. I was [245:43] expecting that. That I assume that would [245:46] be the largest rewrite for us. [245:49] And I guess all those are unchanged. [245:50] That's really interesting. And then we [245:52] need to do a uh a update here. I mean, [245:56] you know, you know, can you make the [245:58] requirements txt for me? Because that's [246:01] what it should have done, but we never [246:02] copied it from a prior one. That's [246:04] probably why. Um, yeah, we didn't. So, [246:07] let's go take a look at the the major [246:09] changes. So, we'll look at the main.py [246:11] and um here we can see async anthropic. [246:16] Oh, so there's where it's different. [246:17] Default AIO uh HP client. That's why [246:20] there was a change there. This is the [246:21] new one, right? There we go. Okay. Um, [246:24] and so [246:26] this more or less looks the same, but [246:30] we'll go into our coordinator directory [246:32] here [246:34] and let's see if we can [246:38] make the difference here. [246:47] Okay. So, I'm going to do scroll up [246:49] here. What I might do just so that we [246:51] can really clearly see the difference, [246:54] we might refactor a smaller one because [246:55] it's very hard to see the changes. They [246:57] don't even show us the code changes [246:58] here, right? Um, so what I'm going to [247:01] do, I'm going to make another repo where [247:04] we have uh [247:07] make another folder here. Let's see. [247:08] Port [247:11] to [247:13] anthropic uh port to agent SDK small [247:22] And the reason I want to do that again [247:24] is to really clearly see the difference. [247:26] And so I'm trying to think of one that [247:27] we were doing before like narrow task [247:29] decomposition. [247:31] Yeah. Where we have this one. This one's [247:33] a lot simpler, right? And we actually [247:35] might want to go one step before that [247:37] where we are using tool use. Um [247:42] could be decision- making model driven. [247:43] Right? So this one here [247:48] is a very simple one with multiple [247:49] tools. So what we we'll do is we'll copy [247:51] this over here. I'm going to go into [247:53] this directory just so we can clearly [247:55] see [247:57] um the difference. And then also maybe [247:59] just have another one that we can work [248:00] on though I don't really like this use [248:02] case per se. Okay. And so um I'm going [248:05] to go and say okay great. Can we can we [248:08] convert the code for port to agent? [248:13] uh SDK small [248:19] over to agent SDK. And again, we haven't [248:22] tested if these actually work. [248:24] Hopefully, it knows clawed agent SDK, [248:25] not just some generic one. Um, but [248:28] anyway, I think it knows. I hope it [248:29] knows. We'll wait here a moment. Okay. [248:32] All right. So, we have the refactor [248:33] already done for this one. Didn't take [248:35] too long. Let's see what it's changed. [248:37] So, the imports are different. [248:40] Yeah, it is using anthropic the correct [248:42] one. No, no, no, no, no. Yeah, it is. [248:45] Okay, here it is. So, here it is. And [248:50] here instead of handling tools here, we [248:52] have a decorator and then the functions [248:55] are probably defined a particular way. [248:56] See, this whole big thing is probably [248:58] gone. Yep. And so, we have decorators on [249:00] top of our functions, making this code a [249:03] lot smaller. Okay. [249:06] Um, the call's a bit different. So [249:09] that's one thing [249:13] and we are creating the SDK MCP server [249:17] to pass the tools over. So that is [249:19] another thing that's changing. [249:23] Okay. [249:27] I mean we have new modes and we're [249:28] setting our MCP server uh with our [249:31] tooling in it. [249:34] Um okay. So basically we basically have [249:37] an internal uh internal MCP. That's [249:39] really interesting to make that super [249:40] super easy. And this call is a little [249:43] bit different. So basically the big [249:44] thing that we're seeing is that tool [249:46] use. Um so let's go back to our larger [249:49] refactor. And I want to take a look at [249:51] our um tools. And so that tool JSON, do [249:56] we even need that anymore? Does that [249:57] even make any sense? So what we'll do is [249:59] go back over here because now we know it [250:02] was refactored, right? and we'll say um [250:05] do we even need [250:09] the tools JSON anymore and shouldn't we [250:14] be using the decorator [250:28] for port to [250:31] agent SDK Okay, based uh for clawed [250:36] agent SDK and I imagine that you can [250:39] probably pass in that JSON tools because [250:41] it's it's doing it. No, we don't we [250:43] don't know if it actually works or not. [250:45] Um [250:48] if we go here, tools.json, [250:50] like I don't see it loaded in here. [250:52] Maybe it's getting loaded in the main. [250:53] It is refactoring probably right now, so [250:55] we wouldn't even know. [250:59] But we'll see what it says here because [251:01] we do have tool right here, right? So it [251:03] is it's right here. So maybe just has to [251:06] delete it out. [251:08] But if the tools here, then why isn't [251:10] that defined or does it have to sit in [251:11] the same place? [251:15] Right. So we have this one here. Is this [251:17] just a repeat? [251:20] Okay. [251:33] And like look at all this inline stuff. [251:35] E [251:42] object maybe pass rationals key strings. [251:45] That's the structure that we actually [251:46] wanted from before. Um, [251:50] and so here all three tool decorators [251:52] are now using the simple param. [251:56] Okay. But like does the coordinator [251:57] still have them in here? [252:01] Do we have to have [252:04] the tools in the coordinator.py [252:11] Pi or can they act they live in the um [252:17] tools directory [252:21] as separate functions [252:25] or it doesn't work because [252:32] tight coupling [252:34] of the decorator [252:37] which is this part here. It might be the [252:39] reason why they can't do that. And I [252:42] mean I hope it knows what last directory [252:44] we were in uh as we have more than one [252:46] but uh we'll ask that question and you [252:49] know this is what I'm trying to figure [252:51] out [252:54] what's it says here. So the key insight [252:56] tool is a decorator runs at call time [252:58] not import time. So you can apply it it [253:00] inside the factory function. It captures [253:02] state via normal closures. [253:08] Okay. Well, speak English to me here. [253:12] Can we move it or not? [253:16] Coordinator class tools screening agent. [253:22] Look, look. I'm trying to keep my stuff [253:23] lean here, folks. [253:27] Did it move it out? Did it even tell me [253:30] that it moved it out? [253:33] Uh, okay. So here coordinator state make [253:37] coordinator. So it did move it out from [253:39] tools. [253:41] I don't like how they're make [253:43] coordinator tools. [253:48] Okay. And then we have coordinator [253:49] state. [253:53] Okay. I see. So they they have a state [253:54] file separately for I mean state [253:56] wouldn't belong in tools now would it? [254:00] So that doesn't make any sense unless [254:02] it's coming from that file. Oh maybe [254:04] it's part of it. That's why. Okay. And [254:06] so we go over to here and we have make [254:08] coordinator tools. Oh, and they do have [254:10] it in here. Okay. So, they were able to [254:12] move it out. And so here we have um our [254:16] multiple tools. Okay. And so to me, [254:18] that's what I would like it to be. Um so [254:20] I'm going to go ahead and we're going to [254:22] stop this and we're going to CD into the [254:24] port to Asian SDK. And I just want to [254:27] make sure that this still works. So [254:29] we'll go ahead and say main.python [254:30] Python [254:32] dot uh we got main [254:35] or main. Python I got it backwards. [254:38] Python main dot [254:41] whatever. Whoops. [254:44] Okay. [254:47] And I just want to make sure that it [254:49] still runs because we've changed a lot [254:51] of code or at least [254:54] with large file to another framework [254:58] and um [255:02] it's logging. [255:04] We'll pause here and see the end result, [255:05] but I'm pretty certain that it's going [255:06] to work. [255:08] Okay, so it ran without issue and we are [255:11] in good shape. Um yeah so we are set up [255:15] and the question will be like you know [255:16] do we use this to test out all the agent [255:19] SDK stuff or do we come back to this [255:20] project we will see but at least we made [255:23] it to this point and I think the key [255:25] takeaway here is the fact that uh the [255:28] tool use call got easier and it's [255:30] setting up an MCP server okay so [255:33] literally it's an internal NPC server um [255:36] and so clearly uh Enthropic is obviously [255:39] making that a priority tool but anyway [255:42] There we go. And we will move on from [255:43] that. Okay. Chow chow. [255:50] All right. So, we have the agent tool. [255:52] And so, this is where you are able to [255:55] basically spin up an agent. Um, it was [255:58] previously called the task tool. And I [256:00] was really confused because the last [256:02] time I used it, it was task tool and [256:04] then the agent came along and I didn't [256:06] realize that they're basically it's just [256:07] a name change and so task is just no [256:10] longer there. But there are um uh parts [256:13] of the code that where you will still [256:14] see it. So you'll see when I do the [256:16] followalong that I'm under the [256:18] impression that we are implementing task [256:20] and then I realized that oh they have [256:22] changed the name and I figured it wasn't [256:24] worth um re-recording that video because [256:26] I thought it was really interesting how [256:28] um uh the uh claw did not know exactly [256:32] itself and the documentation did not [256:35] have proof of it. I just wanted to make [256:36] that as a proof of point that you can't [256:38] just trust uh what agents say. You have [256:40] to go out and test them and then draw a [256:42] line back to your information. But [256:44] anyway, when you spin up an agent, it [256:46] has its own isolated context, its own [256:47] system prompt and its own set of tools. [256:49] You can see that here. Um I believe this [256:51] is the JavaScript implementation because [256:52] I think the tool names are underscore [256:55] lowercase whereas in Python they are [256:56] title cased, which is fine, whatever. [256:59] Um, but one thing that I need to point [257:01] out is that though you're spinning up a [257:03] sub agent, it's not necessarily going to [257:05] be paralyzed. So, it depends on how [257:07] scheduling is set up and how you are [257:09] prompting Claude to do the work. But [257:12] generally, it's going to block uh until [257:14] it's done and then report back to the [257:16] like block the parent from doing [257:17] anything until the um the single sub [257:19] agent is done and then come back. So, it [257:21] really depends on uh how it's set up. [257:24] Okay. And this should give you an [257:25] indicator that um we are using basically [257:28] hub and spoke architecture within cloud [257:31] code. That's probably why uh they they [257:35] wanted you to learn that specific uh you [257:37] know coordinator agent. Uh and by the [257:39] way when I say cloud code agent SDK I'm [257:41] talking about the same thing. Okay. Um [257:44] and so you know because this is [257:46] basically a sub agent uh and we should [257:48] be thinking of it like we have the um [257:52] the uh coordinator architecture that the [257:57] spokes to our hub they're not going to [258:00] be sharing context. They're not going to [258:01] have the parents context. And so if you [258:03] need your your uh agent says task here [258:07] but if you need your agent to know [258:08] something you need to provide it. [258:10] Otherwise it only knows what you tell [258:12] it. Right? So it has the same uh feature [258:14] of a coordinator agent. Okay. Um but [258:17] there you go. [258:23] Hey folks. Uh so now it's time to try [258:25] out the task tool. So what I'm going to [258:27] do, I'm just trying to think here for a [258:29] moment what the task tool can be used [258:30] for. Um because it would have its own uh [258:34] tools that it can read. And so I'm [258:36] thinking back to our um research agent. [258:38] I'm just thinking like, you know, should [258:40] we attempt to uh extend it and do that? [258:42] But maybe we should just do a simple [258:44] example first. So, what I'm going to do [258:46] here is just say task tool um task tool. [258:52] And I'm going to bring in some code just [258:54] so that it has something easy to work [258:55] with. So, I'm just grabbing the small [258:57] port SDK agent. That way, uh it can work [259:01] off of something and it'll our lives [259:03] will be a lot easier here. And so, I'm [259:04] going to go ahead and paste this in. And [259:06] by the way, I've been working uh [259:07] everything off. so far off of the uhv [259:10] file. Um, and you know what I could do [259:13] because we're using agent SDK now is I [259:15] can comment this out, right? And we [259:18] don't even have to we don't have to use [259:19] the file. We don't want to here because [259:21] if I'm logged in as my subscription, it [259:23] will use it. Um, it's been a whole day [259:26] since I've logged in. So, I'm going to [259:27] go ahead and just type in claude uh [259:29] claude. And I'm going to log back in [259:31] here if it doesn't prompt me to do so. [259:33] So, I'm going to go ahead and just hit [259:34] login just because I feel like I'll [259:35] probably have to. And so I'm going to go [259:37] ahead and do that. I don't like this [259:39] Chrome thing it opens. It's completely [259:40] useless. Doesn't really work. Um I'm [259:42] going to go ahead and copy this into a [259:44] browser tab. I'm doing this off screen. [259:47] Okay. [259:49] And I'm going to to accept [259:51] and authorize there. All right. We'll [259:53] copy that. [259:56] Go over here. We'll paste it in. We'll [259:57] hit enter. And so now [260:01] we are logged in. So um the idea here is [260:05] I want to use um task tool [260:08] and it shouldn't be too complicated. So [260:11] well actually here's a question. Are we [260:13] already using it? Well we do have tool [260:14] here. So we are defining tools. [260:17] Um but we want something that's [260:18] demonstrative of just task tool. So [260:21] we're going to CD into [260:24] well actually you don't have to CD into. [260:25] So I'm going to go say task tool main. [260:28] I'm going to say here um I want to use [260:32] the built-in [260:35] task tool [260:38] uh for uh part of agent SDK. [260:44] Uh, can we rework [260:48] this entire [260:51] file to make a simple demonstration [260:57] of the task tool? Okay. [261:05] And so I guess the question is does it [261:07] come down to calling it or is it going [261:09] to come down to invoking it? uh because [261:12] we know that like when we use cloud [261:14] code, we can just say something and it [261:15] will automatically invoke their built-in [261:16] tasks. But what will it look like in [261:18] code is the real question. So we'll go [261:20] ahead here and now it's trying to call [261:21] cloud API which is fine. Um [261:25] and we will see what it comes back with. [261:28] We'll be back in just a moment. Okay. [261:30] All right. So we are back with some code [261:34] and it's greatly greatly simplified. Um [261:36] and so we will go and take a look [261:39] at what we have here. [261:41] as our simple implementation of [261:45] getting just tool here. There we go. [261:47] Okay, so let's take a look what we have. [261:49] Um, [261:51] it seems like it's stuck with the code. [261:53] I mean, I told it it could go change. [261:54] I'm going to close this tab out. I don't [261:55] trust it. There we go. Okay, now we [261:58] definitely have something different. So, [261:59] I'll bring this on down and we will take [262:01] a look. We have I really hate these [262:02] constants. I hate and I also hate all [262:04] this commentary. I do not like that. But [262:07] that's fine. It's in here. You know [262:09] what? I don't care. I'm going to remove [262:10] it. [laughter] I'm going to remove it. [262:13] Goodbye. [262:15] It really likes to do these uh [262:17] constants. Just cannot stand them. [262:21] Um okay. So, [262:25] we'll go down here. We'll take a look. [262:27] So, here is the topic. Well, what first [262:29] of all, what does this thing do? So, [262:31] uses the researcher agent to gather [262:33] three key facts about the topic. Use the [262:36] writer agent to turn those facts into [262:38] short paragraphs. return the final [262:40] paragraph uh as your short answer. So we [262:42] have the James Web telescope. Write a [262:44] short article about that topic and that [262:47] would be the topic and then we're going [262:49] to print uh this 50 times. So a line [262:52] over and so the idea here is that we [262:55] have a query function. [262:58] Okay. And it's passing parameters in. So [263:02] I'm going to assume it's going to be [263:02] called that create message there. We're [263:04] passing a prompt and we have our [263:06] options. So we're bringing our system [263:07] prompt here. You are a coordinator that [263:09] that produces a short article on a [263:11] topic. Sure. We have the haiku model. [263:13] Good. Allowed tools agent. So, it's [263:17] going to use the agent tool. Um, and [263:21] here we have agent. So, we have [263:23] researcher and writer. Here we have [263:25] researcher and writer. And then the [263:26] permission mode is bypass permissions. [263:29] Uh, I don't think we should have that. [263:32] No, we're going to [laughter] take that [263:35] off there. bypass permissions means that [263:37] it can just do whatever it wants. And [263:39] that is a no no. So it's good thing [263:41] we're reviewing our code and not just [263:43] letting it do whatever it wants. So I'm [263:44] removing that. And here we have [263:48] um the message. And then it's saying, [263:50] are you an instance of this type of [263:52] message? And if so, then we're going to [263:54] print that information. [263:56] And up here we have task start message, [263:59] task progress, task notification [264:00] message. And so it's the type of message [264:02] that's being returned back. So it has [264:04] these structures so that we can uh check [264:06] that. Okay. And so I'm looking at this, [264:09] I'm going, okay, that's something. [264:11] Where's the task tool? Right? Maybe [264:14] because everything is a test tool. So [264:16] here's my question. So where is the [264:19] built-in [264:21] task tool, [264:23] right? [264:28] And maybe we can observe it by logging [264:31] out that information. But that's what [264:32] we're trying to find. [264:35] I hope it doesn't try to do something. [264:38] Let's see what it thinks that it is. [264:45] Uh-huh. Let it search. All right. So, [264:48] it's basically failed so far. I'm going [264:49] to just hit tabs. I'm going to tell it [264:51] to stop. It's just It's being silly. [264:53] Okay. No. No. Sorry. Look. Look. Do you [264:56] know? Uh, if you don't, just let me [264:59] know. [265:02] Don't go out to the internet. [265:10] It probably doesn't know. [265:22] I have a built-in tool that listed as [265:24] agent and that's what I I have allowed [265:26] in the tools. The task tasks are listed [265:28] as emitted when a sub aent task is [265:30] completed which implies it's from the [265:32] agent tool. There might be distinct task [265:34] tool cloud itself has one internally but [265:36] I don't have confidence of the [265:38] documentation. So I'm going to continue [265:40] on. I just want to make sure we can find [265:41] that. Just give me a moment. Okay. All [265:43] right. And so I think I know why we [265:45] cannot find it. Um and if we go over [265:48] here take a look here. We're trying to [265:50] find it. I'm saying hey can you find it [265:52] for me? And it's showing uh it's saying [265:55] like you can add task. It's talking [265:57] about agent and then you're checking [265:59] trying to check the docs cannot find it [266:01] and it's saying well it's an older [266:04] unofficial name for it and they're still [266:07] using agent. Um and so you have to [266:09] understand that when uh I'm going to [266:13] find this here in just a moment. I just [266:14] want to show you what I'm referring to. [266:16] See here knowledge of the task tool as [266:19] the mechanism for spawning sub aents and [266:22] the requirements that allow tools must [266:24] allow task. But notice we cannot find [266:27] anywhere in the docs that say it even [266:30] the internal cloud code tool cannot find [266:32] it. So agent is task and up here we can [266:36] see we are bringing in uh this and so [266:38] we're seeing that indicator this is [266:40] task. So task is agent agent is task. [266:42] Isn't it great? We always confirm and we [266:45] don't just go based on this stuff. If [266:46] you were to generate out your lesson [266:49] plan based on this, you would be in poor [266:52] trouble because you know you have to [266:56] test these things and concretely know. [266:58] But even their docs don't make it clear. [267:00] Uh and that's a frustration here, but [267:02] that's why we explore every little [267:03] thing. Okay. Um but anyway, so task is [267:06] agent, agent is task. And right off the [267:08] bat, we can see we are getting agent [267:09] definition. We haven't talked about that [267:11] yet. That will come up uh very quickly [267:13] here. And so, uh, going back over to [267:17] here, it says, use the researcher agent [267:19] to gather three facts, etc. Write [267:21] whatever, whatever. It's going to call [267:23] the query function. Um, and the query is [267:27] just getting imported here. [267:29] Interesting. Okay. So, one thing I'm [267:31] confused about, if we go over to here, [267:35] if we go down [267:37] to this one, [267:40] oh no, it calls query. Oh, it's just the [267:42] way they're doing it. [267:44] All right. And so here they import it as [267:45] client [267:47] and then they call it off of that where [267:49] here they're directly importing query. [267:51] And that was my confusion. This is [267:52] something I don't really like about um [267:54] Python is ability to well I guess you [267:56] could do this many languages but I just [267:58] don't like the uh how it's ambiguous and [268:01] I I just couldn't tell that was client [268:03] query and I thought that was something [268:04] else. Okay. And so then basically we [268:06] define our agents. We have the [268:09] researcher and the writer and then we [268:10] have our model down here. And uh because [268:13] my uh my usage for my claude is uh [268:18] getting up there. It was saying like 75% [268:20] usage for the week. And I guess I could [268:23] upgrade to max if I need to. But I think [268:25] what I'm going to do here is just put [268:27] back in the API tokens. [268:29] So now we have it. And that answers our [268:32] question, right? [268:34] So we'll go ahead here and type in [268:35] Python [268:37] main pi. And we'll run that and we'll [268:40] let it go and do some research. Notice [268:42] it has access to the web. So this one [268:45] has the ability and this one does not [268:46] because it does not need to. And we will [268:50] see how that runs. [268:55] Okay. And it comes back here. And so we [268:59] can see the three tools running uh the [269:02] progress. [269:04] Okay. I'm just going to scroll down [269:05] here. Yeah, the logging is coming from [269:07] here. Okay, good. And then we have a [269:09] result. Is it a good article? No, but it [269:12] was able to do and go grab information. [269:14] So, that was kind of cool. Um, so yeah, [269:17] I think that pretty much demonstrates [269:19] the task pool. Now, you could see if we [269:21] wanted to add this to our um [269:24] uh our larger application, the um [269:28] job screener, we could go out and say, [269:29] "Hey, go research this person. Go check [269:30] their LinkedIn or whatever or try to [269:32] find information on them." Uh, and that [269:34] might be something we might try. Uh but [269:37] we will leave this right now for what we [269:39] just did. Okay. [269:44] Hey, this is Andrew and let's take a [269:46] look at agent definition. So the agent [269:48] definition is the blueprint for the sub [269:50] agent. It defines what the agent is, [269:51] what it knows about itself, and what is [269:53] allowed to do before a single message is [269:55] sent. And so here is an example of an [269:58] agent definition. I want to point out [270:01] that I literally made this video four [270:03] times. And the reason why was that, um, [270:07] I had used this before. I wanted to know [270:09] all the fields. I could not find it in [270:11] the docs. Um, and so, you know, I said, [270:14] "Okay, Claude, can you tell me what it [270:15] is?" And it said, "These are the [270:16] fields." I couldn't confirm it anywhere. [270:18] So, I said, "Okay, I'll go test it." I [270:20] thought it would work. I went out, used [270:22] it, and the fields didn't match. And in [270:24] fact, there were fields that weren't [270:25] there. Then I found out that there are [270:27] some fields that are only available in [270:29] the TypeScript SDK and some that are not [270:32] available in the Python SDK. So I [270:34] literally found out that the Python SDK [270:37] can be hind in features. And not only [270:39] that, that the TypeScript one can be [270:41] behind in features. So they don't have [270:43] one to one SDKs that are being released [270:44] out here. It's very sloppy to to tell [270:46] you the truth. Um, and then the other [270:49] thing was that, you know, I had to [270:51] validate that all the fields worked. And [270:53] I could not tell you how many times [270:54] Claude hallucinated and gave me fields [270:57] that did not exist. Um, and it had a [270:59] really hard time finding the docs. [271:01] Eventually, I did find uh docs. And I [271:05] did also have it go and explore the [271:07] actual codebase to find out the solution [271:09] to it. And it's just wild to think that, [271:11] you know, you you know, like if the docs [271:13] are bad and the generation's bad, um [271:16] then you just have to kind of muddle it [271:18] through. And so I'm just going to tell [271:18] you that this is my experience [271:19] throughout the course. I might complain [271:21] a few more times about this, but that is [271:22] what it's like. Um so understand, make [271:24] sure you test everything and do not [271:27] trust what the agent generates or sorry [271:29] like claw generates. Do not trust uh [271:31] what the docs say. You got to go and [271:33] test these things yourself. So anyway, [271:36] where's the name of the agent? The agent [271:38] is up there. So, see where it says [271:39] research agent right up here. Okay, [271:42] that's its name. These names matter and [271:44] we'll learn uh as we go through this [271:46] that um that can cause it to trigger or [271:50] not trigger and stuff like that. So, [271:51] this name is very sensitive in terms of [271:52] what it's named. Same thing with the [271:54] description. This is what's going to [271:55] trigger the agent. So, this matters and [271:58] we need to write it in a very particular [272:00] way. Then, we have our prompt. Notice it [272:02] just says prompt, but it's actually the [272:04] system prompt. And so it's a system [272:06] prompt. Uh tools is just allowed tools. [272:09] At one point I think it might have been [272:10] named this. There's other APIs like [272:11] that. So I just have it here called [272:14] allowed tools because that just makes it [272:15] more clear. Then we have disallowed [272:16] tools. And then we have max turns and [272:19] models. And so some of these fields are [272:22] not available at the time of this video [272:24] in Python, right? But they are in [272:26] Typescript. And so just so you know the [272:28] SDKs are not one to one. They might not [272:31] match and some of them may not have [272:33] current features. I don't understand how [272:35] they have all this compute and they do [272:37] not have matching SDKs. Makes no sense [272:39] to me. But anyway, let's continue on. [272:41] Okay. [272:46] Hey folks, it's Andrew. We are back and [272:48] I want to take a look at agent [272:50] definition. So, we'll make a new folder [272:51] here. We'll call it agent definition. [272:55] And we're going to go back to our last [272:57] one, which was [273:00] uh this file here, task tool. So, I'm [273:03] going to go ahead and grab this code. [273:05] We'll make a new main.py. Got lots of [273:07] these going on and we will CD back here. [273:11] We'll just ask Claude, [273:13] I want to show all the options in an [273:16] agent definition, [273:20] uh, can you modify my agent definition [273:27] file to show us that? Okay. And so what [273:32] I'm hoping is that we get all that [273:33] information. What I'm mostly interested [273:34] is testing the constraints. Um, [273:38] and notice like in that documentation it [273:40] said allowed tools and here we have [273:41] tools. So there's going to be some [273:42] variation on how that's defined. I think [273:45] we would have to go look at um the agent [273:49] SDK. I'm going to go take a look here. [273:52] And while it's thinking here, I'm just [273:54] trying to find [273:56] uh that example because sometimes what [273:58] it will do. So you see here it says like [274:00] allowed tools over to TypeScript. Well, [274:03] those ones are title case and this says [274:06] options here, but this one says allowed [274:07] tools, but that's query. Okay, but we're [274:10] looking for agent definitions. So agent [274:12] definitions. [274:17] Uhhuh. Like where is it? Isn't it great? [274:20] Like I cannot find [274:23] really basic stuff. Um, and this is my [274:26] complaint when I say I do not like the [274:28] docks. Um, the docks are it's not it's [274:31] like they weren't built for humans. Um, [274:34] so I'm just going to have to rely on it [274:36] and assume that it can do that. So we [274:39] will [274:41] go take a look here. Sure. Yeah, we'll [274:44] say allow [274:47] and [274:49] just close this out here and we will re [274:51] reopen this up. So we have tools. [274:57] We're going okay. [275:04] Is that all its fields for agent [275:07] definition? Are you sure? [275:15] Just make sure. [275:21] Okay. And so the reason I'm confused [275:23] here is that I know there's more fields. [275:25] And if I go over to here, take a look. [275:28] We have tools, disallowed tools, model, [275:30] MCP server skills, max turns, critical [275:32] reminder. So why am I getting more [275:34] fields here? Is that there difference? [275:35] Okay, what about the TypeScript [275:37] implementation? [275:40] Okay, because right now we're only using [275:42] Python, right? [275:46] And uh this is where there's a bit of a [275:49] confusion. [275:52] So [275:56] we will go. Yeah. Sure. Yes. Yes. Yep. [276:00] And it what it's literally doing is [276:01] looking at the code. So it's not going [276:03] out to the internet. It's And maybe [276:04] because I told it earlier not to do [276:06] that, but I don't think in this session [276:07] I did I did tell it to do that. And so [276:09] I'm really trying to get it to [276:13] um [276:15] take a look at this. And I could feed [276:17] this information in here over to there [276:20] and see what happens. Here it is linking [276:22] to the docs over here. So I can click [276:24] over to here into the docs. [276:37] And I'm just looking for Yeah, here it [276:39] is. Right here. So I go here. Okay. [276:42] Okay. Stop. Stop. Stop. Stop. Stop. [276:44] Look. Look. [276:46] Look, the docks show it, [276:50] right? [276:52] So, I want to know [276:54] where those other ones are. [276:59] Okay. [277:02] I mean, we obviously have tools and then [277:04] there's disallowed tools right here, [277:05] which is fine. [277:07] There's skills. There's Mac turn max [277:09] turns. [277:11] Okay. [277:18] Okay, so the type set has significantly [277:20] more fields in the Python. [277:24] If you need these capabilities, you need [277:26] to use the Wow, isn't that interesting? [277:30] But I mean, hey, look, it's good that we [277:31] know, right? So, if I had not checked [277:34] that, then, you know, I would have [277:37] thought that's all there was, right? So, [277:40] I mean, now we know. Um, do we need to [277:42] run this? No. I just wanted to see if [277:44] there was any critical difference or, [277:46] you know, why that discrepancy was [277:48] there. Um, but I would say that's good. [277:50] Okay. [277:56] All right. So, something you're going to [277:58] want to take advantage of is being able [277:59] to have parallel tool calls. And the way [278:02] you do that when you're using the agent [278:03] SDK is you just simply tell it that you [278:06] want to do it. So, somewhere in here, [278:08] I'm looking for it. Um it's you say so [278:11] examples of tasks that can be run [278:13] parallel. So we're not just we're not [278:14] just saying that it can do it but it's [278:16] also here saying like this is a case [278:18] where you would run in paral uh uh [278:20] parallelization and here would you do [278:22] sequential. So it's up to the [278:23] coordinator to decide if they want to do [278:25] them all at the same time or not. Um we [278:28] just want to remind you about the agent [278:31] tool because that's what we're talking [278:32] about here. We're talking about agent [278:34] SDK, right? agent SDK with the um uh the [278:40] agent tool, but each agent tool runs in [278:42] their own loop. Uh each agent is running [278:45] at the same time or can if it if you [278:47] specify it here obviously in parallel [278:50] and we might not be able to see what the [278:51] internal loop is doing. You're going to [278:53] find this out as we work through um the [278:56] follow along. So that followalong was [278:58] only long because we had an issue with [279:00] structure JSON output. Um, but I wanted [279:03] to get logging for the internal loop, [279:07] but you can't get it right now with the [279:08] SDK. And so I believe you'd have to drop [279:10] down to the anthropic SDK, but the agent [279:12] SDK is challenging if you're running um [279:16] agents with the agent definitions. And [279:18] so I was waiting a long time and I just [279:20] wanted to know it was actually doing [279:21] something or at least track my spend or [279:23] or or help reu it if it's going off [279:25] tracks, but you can't. Um, and so that [279:28] is just a limitation of how it works [279:30] right now. [279:36] Okay. So, in this video, what I want to [279:37] do is I want to show off parallel tool [279:40] calls. And so, I'm going to make a new [279:41] folder here called parallel [279:46] tool calls or parallel agents, which is [279:49] probably more [279:51] uh [279:53] or parallel parallel agents or parallel [279:55] tool calls. [279:58] I will say parallel. Yeah, parallel [280:00] parallel agent calls. Okay, agents agent [280:03] tool calls. [280:06] I'm really uh being too particular about [280:08] that. And we'll make a new folder here [280:09] and we'll call it main.py. And so I'm [280:12] kind of liking the last one that we did [280:13] there uh which was under um the [280:16] findings. [280:18] Right. So we're going to go over to [280:19] here. I'm going to go and grab this here [280:21] and we will paste it into this one. And [280:24] so what I want to try to do is figure [280:26] out a way that we can coordinate [280:27] multiple agent tool calls and why we [280:29] might need to do that. Um, so I'm going [280:31] to go all the way down here and we'll [280:34] just CD back for a second and we'll [280:36] enter here and I'm going to go, you [280:37] know, I'm going to make my life easier. [280:38] I haven't been doing this, but I'm going [280:39] to just go into findings like this and [280:40] then it's going to have context a lot [280:42] easier and won't look at all these other [280:43] repositories and cause me a bunch of [280:45] issues as that is a growing problem is [280:47] it's trying to use the context of the [280:48] entire repo. And even though this will [280:50] create another project within our um [280:53] home directory, I think that's okay. So [280:55] I want to uh make use of uh [281:01] uh parallel [281:05] uh agent calls in that run [281:10] at the same time. [281:12] So what would be a good use case [281:16] uh for our current [281:19] main.py [281:21] where [281:23] uh let's just make sure that it's [281:24] selecting the right thing. Yeah, it is. [281:25] Okay, main.py. [281:28] I mean, it was trying to go up a [281:29] directory though. main.py. [281:33] Can you propose to me [281:37] a new coordinator prompt? [281:40] Okay. and new agents. [281:44] Okay. And so that's what we're going to [281:47] look at next. [281:49] See what we get here. Okay. [281:55] I can't think off the top of my head why [281:57] I would need multiples in parallel. Um I [282:00] mean like if there was multiple films [282:01] and we're like okay each of you go out [282:03] and grab multiple films that could make [282:05] sense here. [282:09] Right now, the coordinator calls [282:10] research agent sequentially [282:13] one at a time. Since no access depends [282:15] on another, all five can be spawned at [282:17] the same time. Oh, well, how useful is [282:19] that? So, we have research agent. So, [282:20] keep as is per access deep research, [282:23] receive all findings, remove duplicate [282:25] overlapping uh films, returns a finding [282:27] list, uh receives duplicate findings, [282:29] ranks top 10. So, the new coordinator [282:32] would be parallel research in a single [282:34] response simultaneously for all five [282:36] axises. Do not wait between calls and go [282:39] across all of these. Oh, well, there we [282:40] go. We already got a benefit. Uh, call [282:43] record findings. The current code [282:45] already passes allow tools and supports [282:47] multiple named agents via agent. Uh, [282:50] adding to dupe just means adding two [282:53] more uh agent definitions. [282:57] Okay. [283:00] Um, [283:03] sure. [283:04] But I guess my thought here is like is [283:06] that going to paralyze tool calls? Just [283:08] give me a second. Okay, I'm just trying [283:11] to look at the exam guide. What is it? [283:12] Task tool in a single coordinator rather [283:14] than rely on separate turns. And so I [283:17] think that will satisfy what it is that [283:19] we want. Um and so we'll go ahead and [283:21] say okay, great. Yep, let's do it. [283:27] Okay, because I'm just trying to like [283:29] satisfy what the exam guide wants. Um [283:34] but we will be back in just a moment. [283:40] Okay. And we'll say yes to the session. [283:43] And so I'm just looking for what has [283:44] changed. [283:48] The dupe agent. Uhhuh. Rules. If the [283:50] same file film appears more than once, [283:52] uh keep the best uh best entry. Do not [283:55] drop films. Return this object rank. [284:00] return JSON object with 10 films in uh [284:03] ranked to most to least similar. [284:11] Okay. [284:14] And then there's our multiple agents. [284:18] How do we know that's being paralyzed? [284:25] All in five one turns. And so the key [284:26] thing that I want to find out in this [284:28] code is what is causing the the [284:31] parallelization. That's what we want to [284:33] know. [284:38] And so I'm just carefully looking here [284:39] and trying to find. [284:45] So here you are parallel. So in a single [284:48] response call research agent [284:49] simultaneously all axis of one. So see [284:52] here we're telling it to do this. do not [284:55] wait but delegate to all of them. Right? [284:58] And so that's the key thing that we were [285:00] asking it to do. [285:03] We have bypass permissions on again [285:05] which I don't like. So I'm going to [285:08] remove that again [285:10] because we did not request it to do [285:12] that. Um [285:15] I'm not sure that would cause us any [285:17] trouble, but we will go ahead and just [285:19] say take that bypass out there. bypass [285:21] means go do. Well, I guess it's not [285:22] really that dangerous because we know [285:24] exactly what it's doing. But you have to [285:26] be really careful with this because you [285:27] have to think about like what it can do. [285:29] If it's just going to go and do web [285:30] search, then that's not that big of a [285:32] deal, I suppose. But I still really [285:34] don't like that. There's probably a [285:35] better security model than just saying [285:37] bypass permissions and thinking more [285:40] about like what it could use. And so [285:42] really, it doesn't seem like there's [285:44] much of anything else to do other than [285:47] to let it go off to the races here. [285:51] So, let's go ahead and execute it. So, [285:55] we'll go ahead and type in Python [285:56] main.py [286:00] and Oops. No, no, no, no, no, no. Wrong, [286:02] wrong, wrong, wrong one. Wrong one. [286:03] Wait, hold on. Did we just update our [286:05] findings in place? I thought we were [286:07] going to go update this file. [286:10] Uhoh. [286:11] What did I change? So, we changed uh [286:14] this one by accident, right? This is the [286:16] paralyzed one. [286:20] which is fine. I can just like swap [286:22] their places. So, one second here. Let's [286:24] go rename here and old. [286:27] I just don't want to muck things up. And [286:28] I'll move that here like this. Boom. Not [286:31] a big deal. Not a big deal, people. [286:34] And we'll rename that one. I just wasn't [286:37] paying attention. [286:40] Um, and I'm going to make sure this is [286:42] the correct one [286:44] simultaneously. Okay. So what we'll do, [286:47] I'm just going to cd back parallel agent [286:49] and we'll go python main.py and we will [286:52] let that run. The only thing that will [286:53] be hard here is to really observe that [286:55] they're running all at the same time. [286:59] But it says the agent is saying it's [287:00] starting them all simultaneously. So at [287:03] least we have that indicator that it's [287:04] doing it. I'm not sure exactly how we [287:06] would monitor that. Uh it's not an exam [287:09] guide to ask. Again, if I did a boot [287:10] camp, this is where this is something [287:11] that I would want to know because I'd [287:13] want to keep track of what is going on. [287:16] But for this, as long as it comes back [287:18] with a result, I think we'll be happy. [287:22] Um, I would suspect that it would be [287:24] faster than what we did before. I didn't [287:26] track what was happening before, but I'm [287:27] just going to trust that it is [287:28] simultaneously working. Um, and we will [287:32] be back here in just a moment. The only [287:34] problem here is I I have no sense of [287:35] what's going on. I've been waiting for [287:36] quite a while and so I'm actually going [287:37] to stop this and I'm going to just go [287:39] here and be like you know when I am [287:42] running when I am running uh the agent [287:44] and multiple agents are running in [287:47] parallel I have no [287:50] I have no uh output or logging and so I [287:54] have no sense of you know if the models [287:58] are going to or if the agents [288:01] are stuck in a loop or if they are going [288:05] to finish soon [288:08] or what they're even doing. [288:13] All I see is this. [288:19] Can we get better uh logging and [288:22] visibility? [288:26] Because that's going to drive me crazy. [288:27] Another thing that I was thinking is [288:28] that um we didn't do this in the finding [288:30] sections, but I would like to enrich the [288:32] source URL just because they go here and [288:33] say like source URLs, docu document [288:36] names and page numbers, which I didn't [288:38] uh I didn't really provide, right? So, [288:41] we'll go ahead here and um [288:47] why does it need more more access? Sure, [288:50] fine. I I don't know why it would need [288:52] that. It's in the folder it needs to be, [288:53] right? [288:59] Uh, [289:01] I'm just making sure it's not going [289:02] outside of that. [289:05] Look, no, I don't know what it's doing. [289:06] Okay, hold on. [289:08] Sorry. Just focus on the main [289:13] main.py. [289:16] I'm not sure [289:18] why you are looking up a directory to [289:22] implement logging. [289:27] Okay. So that's me paying attention [289:29] being like why are you doing that? You [289:30] don't need to look at that to stay [289:32] focused on your task. [289:38] Okay. So we'll say yes allowed. [289:58] Okay. And we have some results here. [290:01] Okay. So now our agents first you'll see [290:04] research agent first 80 characters [290:08] status. Okay. This will give a live [290:10] trace into flight without needing [290:11] external instruments. The findings and [290:14] coverage prints inside the MCP. Um I [290:16] noticed our findings [290:18] doesn't contain uh things like source [290:23] URL or or like document name or page [290:28] number. Uh so we can't like we can see [290:38] we need to have better findings [290:40] information [290:44] sourced information. Right. So we'll go [290:46] ahead and we'll do that. Give it a [290:48] moment. All right. Let's go back here [290:50] and take a look here. And I'm just going [290:52] to close this. Sometimes it doesn't [290:53] update. So we'll just do this. This is [290:55] like a problem for my machine. So here [290:56] we have the URL, the document name of [290:58] the database, page number, and so now we [291:00] have better source information. We [291:02] really should have done that prior. So [291:03] now we are going back and doing that [291:05] now. So that is good. Um, and so you [291:08] know, I don't know if the logging is [291:09] going to be that much better, but I [291:10] guess we'll find out here and we'll see. [291:12] Um, [291:15] because before it's just like we didn't [291:17] have a whole lot of information. [291:20] Okay, so there's the coordinator. [291:23] So now we can see all the agents are [291:25] starting, right? So they're all firing [291:27] up. That's a lot more useful. [291:32] The only thing is that we can't see what [291:33] those agents are doing, but they are [291:36] running. So that is good. Um worldwide [291:40] featuring featuring mind control, the [291:42] blend of etc. [291:46] And so I I just don't know if those are [291:48] like single actions or they're looping [291:50] or like what are they doing? So, while [291:53] that's going there, I'm just going to [291:54] take a look here and take a look. [292:00] Yeah, I think they're really just doing [292:02] one thing. They're just going out and [292:03] then doing a single thing maybe. [292:09] Mhm. [292:12] I guess we will wait. Okay. I mean, it [292:15] makes sense that if they're agents, they [292:16] have their own agentic loop each of [292:17] these things here. And so, the only [292:19] thing is that uh and again, I don't know [292:21] if this is true or not. we'd have to [292:22] verify it, but it's suggesting here that [292:25] uh sub agents you have to enable [292:26] streaming where the parents uh they [292:28] already stream the information to you [292:30] and sub agents don't. Um because right [292:33] now it's like what are they doing? I [292:34] can't really tell. You know what I mean? [292:35] And um this is not helpful. So I'm going [292:38] to go stop this here and I'm going to [292:40] try this again. I'm going to say can you [292:43] um I can't I can't see what the sub [292:46] agents are doing. [292:51] Okay. [292:54] Uh example, uh is there a way [292:58] we can [293:01] we can monitor streaming information? [293:06] So we know that [293:09] something is happening. [293:11] Maybe we can uh uh dump logs [293:16] to um [293:19] each uh [293:23] agent run [293:26] each run of our pipeline of our of our [293:28] coordinator. [293:31] and then in there have individual logs [293:36] for [293:38] uh for each sub agent that is spun up. [293:43] So like [293:44] spun up. So, [293:47] eg logs uh you know run [293:52] uh you know [293:55] time stamp [293:58] you know ISO ISO time stamp uh [294:02] sub agent name [294:06] log. Okay. And so I want to see if it [294:08] could do that because [294:10] again this is just because I don't like [294:11] waiting forever and I just want to make [294:13] sure it's doing something. Um, I know [294:15] event should come back to us, but let's [294:16] see if we can get that single [294:18] improvement here. Um, I don't know if [294:20] it's later in the course, so I can't [294:22] remember, but um, I know. Let's just [294:24] make our lives a bit easier, right? All [294:26] right, let's take a look here. So, what [294:28] do we have? Um, [294:30] we have logs. I mean, if this works, [294:32] that'd be great. Um, gets everything [294:35] echoed to the ST out. [294:38] And then we obviously could just tail [294:39] our logs. And that would actually let us [294:41] tail. Well, that would only tail one. We [294:42] actually can do a um [294:45] an asterk on that to tail them all to be [294:47] honest. So we'll go ahead and we will [294:48] now run it again. I'm going to see if we [294:50] get output here. And I'm just going to [294:52] split this vertically. I'm going to give [294:54] this a nice refresh here. And I want to [294:56] see if the other logs pop up here. [295:02] And there they go. [295:04] There we go. I wish that they were named [295:06] a little bit better in terms of what [295:07] they are, but the very least we can see [295:11] that. And so now what I can do is I can [295:13] go and say tail f um logs run [295:18] asterric.log. And so now it will show us [295:21] all the logs, right? And so if there are [295:23] any changes that are happening here, [295:26] they'll get propagated. But it seems [295:27] like they are propagating here now. [295:31] Are they? [295:35] So that's all it's doing. It's just [295:36] going out and doing that one thing. [295:46] Well, I mean, hey, look, it's an [295:47] improvement of of what's happening [295:48] before, but uh yeah, I just I I still [295:51] don't know. Like, I guess it must be [295:53] doing a tool call. [295:55] Um I feel like there should be more [295:57] going on there. [295:59] Is that actually streaming the [296:01] information? [296:04] Uh I'm just going to be patient. I'm [296:05] going to pause and it'll be back here. [296:07] I'll give it like 5 minutes, but if it [296:08] goes over that, I'm going to stop it and [296:10] um I will just very much try to get [296:13] something that is logging a bit better, [296:15] but I'm not exactly sure what it did to [296:17] change [296:19] um the logging here. So, I'm just going [296:20] to check and see if that's the case [296:22] here. [296:23] So, that's what I'm looking for right [296:25] now is I'm trying to find where in the [296:28] codebase [296:30] it is [296:32] doing that. You might say, "Well, [296:33] Andrew, why don't you just ask it?" [296:34] Well, because it's it's used up right [296:36] now and also I just I I rather just look [296:38] myself and find it. So, run logger. Mhm. [296:46] Agent. Yeah. Another agent. Yeah. [296:52] Uh-huh. [296:55] Okay. [296:58] I don't think I don't think it's doing [297:00] what I'm expecting it to do. [297:03] Yeah. I I can't tell from that code [297:05] anywhere that indicates that it's doing [297:07] that that's going to log it more. But [297:10] it's running, so I guess I'll wait. [297:11] Okay. All right. The results are coming [297:13] back here. Um I'll just close this one. [297:15] This one's not that useful here. And I'm [297:17] just going to close this log and reopen [297:18] it. [297:20] I don't feel like it's streaming. I feel [297:22] like it would be like I'm making this [297:24] tool call and I'm doing this and I'm [297:25] doing that. Um [297:29] you know what I mean? So, you know, [297:30] maybe that's a separate thing to do is [297:31] try to get more verbose. uh logging. I [297:34] feel like, you know, the agent SDK [297:35] should just have that built in where [297:37] it's like they should just make that [297:39] opinionated because it's already they're [297:40] already making the agents opinionated. [297:42] So, why do I got to do extra work to get [297:44] that logging out there? If there's a [297:46] library out there, I don't know of it. [297:47] Um it might be interesting. Um but yeah, [297:50] we're getting a results back. And I was [297:51] just hoping to kind of get a sense of [297:53] how long something takes, but it seems [297:55] like this is working. Uh one thing I am [297:57] interested in seeing is um if our [297:59] findings data is updated so that we [298:01] actually have better outputed [298:03] information. Um so we will uh see that [298:06] here in just a moment. [298:09] Okay. So just pause here and wait. Okay. [298:11] So we are back. Um [298:15] and so now we have results. Let's go [298:17] give this a nice refresh here. I want to [298:19] take a look at our JSON. Why is our JSON [298:22] empty? [298:24] What the heck? [298:25] >> [laughter] [298:26] >> Um, [298:30] well, great. But where's my findings? [298:35] Am I crazy here? Is this just empty? [298:37] Okay, so [298:42] my findings [298:45] Jason. [298:48] So after running oh we'll say debug [298:52] debug after running my uh agent for [298:56] main.py [298:58] it was supposed to log results to [299:02] results.json [299:04] but it's empty. [299:08] Do you think this has anything to do you [299:12] think when we do you think when we added [299:14] logging or changed uh it to run multiple [299:21] agents at the same time? [299:25] This is where [299:27] we uh lost the findings [299:31] JSON data. We still want logs and par uh [299:36] parallelized [299:38] agents. Where's my results? [299:42] Okay. So, we'll let it do that there. [299:45] See what it comes back with. And you [299:48] know, so far it's just talking about the [299:50] sub agent. And so, we're having issues [299:51] with the hooks. Um, and so there are [299:53] issues with getting uh actual [299:55] information. At least that's what it's [299:57] suggesting. Again, I'd have to dig [299:59] myself to figure out why we only get the [300:01] final prompt for logging, but we're [300:03] clearly missing a lot of information. [300:07] Um, you know, again, that would require [300:09] a deeper investigation of, you know, [300:11] reading code and looking at code. But [300:12] anyway, we're still trying to figure out [300:14] why we're not getting our findings. And [300:16] so, there could be many reasons. I can [300:17] see the problem. Let me check the [300:18] research agents actually return uh to [300:20] understand the format mismatch. And so, [300:22] let's see if it figures that out. I [300:24] really hope it doesn't try to fire off [300:26] the agent as I don't really want it to [300:27] do that. Um but uh we will see what we [300:30] get. Okay. All right. So we have a [300:31] reason for failure. So the chain agent [300:34] ran the file record finding on [300:36] everything to duped it. Y fails um which [300:40] expects it to be this format but the [300:42] coordinator passes a plain list like [300:44] this. Add error logging so the failures [300:46] aren't silent. Handle the coordinator's [300:48] natural output format as a fallback. I [300:50] don't understand it should get formatted [300:52] anyway into paidantic, [300:55] right? [301:03] Okay. Well, let's apply it. [301:06] I'm going to have to run this whole darn [301:08] thing again. And I don't have [301:09] observability for sub agents, which is [301:11] not great. Um, but I will run it again [301:14] and then we will see. But now it kind of [301:15] makes me want to just roll my own using [301:18] the anthropic API if the agent SDK is [301:20] not really providing the tooling that we [301:23] were hoping to have. Um, [301:33] still going here. Fields were passed a [301:34] string but got iterated character by [301:36] character. [301:42] Okay, so [301:46] tries the strict fine settings finds [301:48] list parse which it should still works [301:51] on failure should log falls back to [301:53] converting it uh to the quarter's [301:55] natural format uh returns a proper thing [301:59] rerun the main and you'll see your [302:00] results okay so let's go ahead and we [302:03] will run it again [302:05] I'm just going to I don't know what will [302:07] happen if we run it a second time so I'm [302:08] going to delete out that I'm going to [302:11] pause here we'll be back when this is [302:13] done running Okay. [302:15] Okay. So, we can see our results printed [302:18] out here. Let's see if it actually [302:20] filled out the results. JSON. There we [302:23] go. And so, I'm just observing um the [302:25] information. So, we have a URL the [302:27] document [302:29] URL document. What would probably be [302:31] also really good here. I'm not going to [302:33] do it right [laughter] right now, but [302:35] like I would probably ask it to like [302:37] whatever excerpt it took out [302:39] specifically that it used, I would [302:41] probably say like, "Hey, can you provide [302:42] that information?" Because I think that [302:44] would be um like site your sources, [302:47] literally pull the piece of content that [302:49] you're citing, not just literally put a [302:51] URL there. And I think that would [302:53] provide a better thing. But the question [302:55] was like parallelization. And so, um I [302:58] yeah, I don't think our logging was an [302:59] issue of paralyzation. was a formatting [303:02] issue there. But now we know that there [303:04] is an issue with uh sub agents at least [303:06] with the SDK at this time that it's an [303:08] internal um at least that's that's what [303:10] it's purporting is that it's an internal [303:13] um loop and we do not do not know what [303:18] it's doing until it reports back and [303:19] that's not great from a uses perspective [303:21] right so I would imagine you know if [303:23] we're going to build something for [303:24] production and we need to track stuff [303:26] we're going to have to roll this stuff [303:27] ourselves using the lower level [303:30] anthropic uh client but for that case it [303:33] actually would have been probably fine [303:34] for what we're doing. Um but anyway, [303:37] yep, there we go. [303:43] Let us take a look here at fshot [303:45] prompting. Okay, so uh the idea here is [303:48] that um we have something we want. We [303:50] want to extract the measurement uh from [303:51] each ingredient. And so um now we might [303:54] say roughly three teaspoons of olive oil [303:56] and we want to format it in a particular [303:58] way. So here we're saying format it like [304:00] this. But to make sure that we get it [304:01] the way that we want, we're going to [304:03] provided examples. Okay? And the reason [304:05] we do that is that it's going to provide [304:08] consistent results and it's going to [304:09] reduce hallucinations. Um, and this [304:11] works extremely well. Some other things [304:13] that you can do is you can provide both [304:15] good and bad examples. So like when I'm [304:18] trying to get my Japanese language [304:20] learning app to help with grammatical [304:21] structures uh for output, I will provide [304:23] really good ones and and bad ones. Um, [304:26] and in fact, I will even score them and [304:28] that will help them a lot more. Uh, but [304:30] yeah, it's not a complicated concept. [304:32] Add in examples [304:35] um or or examples of of the results and [304:38] you'll get better results. Okay, [304:44] let's take a look here at goals and [304:46] criteria uh driven uh results. So if you [304:50] have procedural prompts uh that make the [304:52] coordinator rigid. If a step breaks or [304:54] the task doesn't fit the script, it will [304:56] break. So here's an example. You're a [304:58] coordinator. Follow these steps exactly. [305:00] Call this, call that, do this, do that, [305:03] return the reviewed output. Always [305:04] follow these exact sequences. And so you [305:07] know something fails along the way, you [305:10] know, the thing is done, right? So uh a [305:13] better way is to give the coordinator a [305:16] goal uh or a criteria. So first we're [305:19] looking at here and we're saying you [305:22] know you're researcher coordinator your [305:23] goal is to produce and uh the response [305:25] that fully address the user request with [305:27] accurate wells supported findings this [305:29] is your quality criteria these are your [305:31] available agents which we don't need to [305:33] do that because in agent SDK it already [305:36] knows when we're passing them in but the [305:38] point is is that um this one is going to [305:40] be very valuable to to us the quality [305:42] criteria and then we have guidance down [305:44] here if needed be but really the major [305:46] thing we're looking at is this quality [305:49] criteria, right? Um, and so we can go [305:52] further by specifying that uh quality [305:54] criteria. And so if we go back here for [305:56] a moment, we have that, which is fine. [305:58] But here you'll notice like we have very [306:02] specific uh checklists of of that [306:04] information. We even take it further and [306:06] say like what we're delegating out in [306:07] our research goal uh or when to stop. [306:10] And so we have a lot better information [306:12] about goals and criteria. So that's [306:15] something we should give a try. Okay. [306:23] Okay folks, let's continue on. So I'm [306:25] going to make a new folder here. And [306:27] this one is going to be for uh criteria [306:32] criteria driven. I believe that is uh [306:36] the name we were looking for here. [306:39] Technically it's goal and criteria [306:40] driven. Where'd it go? Where'd it go? [306:43] Where are you? Here we are. And did I [306:45] even spell criteria right? It looks kind [306:47] of funny. I don't know, but that's what [306:49] it's going to be. If it's no good, you [306:51] just tell me in the comments and be [306:52] like, "Andrew, learn to spell." I know. [306:54] I wish I could. So, what we'll do is [306:56] we'll go all the way down here and [306:59] remember, we know how to refactor code. [307:00] It's interesting because we worked on [307:02] that other use case uh and we never came [307:04] back to it in the agent SDK. I don't [307:06] know. I just had no interest in it. Um [307:08] and and we spent all that time [307:10] refactoring it, too. [307:12] But at least I can read this one still, [307:14] so it's fine. Um, [307:19] so the question is, is this thing using [307:22] criteria as a means to drive? That's [307:25] what I want to know. So I'm just going [307:28] to go back and look at my slide so I can [307:30] have an idea. Quality criteria. Quality [307:32] criteria. That's what this should be [307:34] called. Quality criteria. [307:39] Okay. And so, do we have that here? [307:45] I'm gonna type in to Claude. Nope. Nope. [307:48] Nope. Wrong one. Wrong one. We'll go [307:49] back and we'll say Claude here just [307:52] because we're ending up with too much [307:53] stuff here. And uh, you know, for my [307:56] main, I need to ensure the coordinator [308:00] has a goal [308:03] and a quality criteria. [308:07] What can we do for that? Right. And I'm [308:12] going to go down here and look at our [308:14] main [308:16] coordinator here for I almost want to [308:18] refactor this because it's becoming [308:20] quite the headache. Okay. Over here. And [308:22] so we have these. We actually do have in [308:24] phases, right? Well, hold on. Um [308:32] because here's the thing is like if you [308:33] go back to here, like look, we have you [308:35] are coordinator. follow these steps [308:37] exactly and this thing is going through [308:38] phases which is not necessarily a bad [308:40] thing. We can do that, but you know, is [308:42] there a way that we can be more criteria [308:44] driven? [308:46] Okay, so we'll go here and say yes down [308:49] to this one here. [308:54] We'll see what it comes up with. Okay, [308:57] but maybe we should have given it an [308:58] example as that might have been better. [309:00] Even the code that I have would have [309:01] been a better example, right? [309:05] So goal block at the top and then inline [309:08] coordinator evaluates against concrete [309:09] standards update the uh [309:14] evaluation coverage tool description. [309:16] Okay, so here we have our goal and [309:20] criteria. So what do we have here? By [309:22] the way, our prompt's getting kind of [309:23] long, too. I'm not sure if it's too [309:25] long. We say uh produces a curated [309:27] source back list at at least 15 distinct [309:29] films. So that's being very clear as to [309:30] how much. Um then a fan school could uh [309:35] could watch next. The final output must [309:37] cover all five similarity axises. And we [309:40] have quality criteria finding is high [309:42] quality when all are met. So content one [309:45] specific information sentence about the [309:46] film high film etc etc etc. So that is [309:49] good. I like that actually. That is [309:52] pretty good. [309:55] I don't see how it would loop back [309:56] though. [309:59] So, that might be the only um thing [310:01] that's not good here. [310:07] The other thing that I would want it to [310:08] do, because we didn't have this before, [310:11] but I needed to actually extract out the [310:13] findings information. [310:18] You know, we are we are citing sources. [310:22] Go back to our findings. [310:30] Okay. And so I would say like, you know, [310:32] we have our findings information. [310:35] We have our findings information, but it [310:38] seems like it's just focused on [310:41] on uh finding a very specific movie and [310:47] listing like let me go back over to [310:50] here. Is this IMDb? Just IMDb [310:52] information. [310:56] just IMDb information [311:00] where findings is really [311:03] collecting [311:05] the research parts [311:09] that justify [311:11] its choice. [311:13] Um [311:15] also findings you know should [311:19] actually take the small expert [311:27] uh that it's sourcing. So we can see not [311:31] just where [311:33] the source is [311:36] but what we are borrowing, what we are [311:38] referencing exactly. [311:41] Okay? Because that definitely is an [311:43] issue, right? [311:45] And so to me, because like if we're just [311:47] going to have a bunch of IMDb stuff, [311:48] that's not that's not helpful, is it? [311:53] Right. Um, [311:55] and we are using Haiku. It's not the [311:57] best model, but it seems to be working [311:59] okay for what we are doing. I'm actually [312:01] quite surprised to be honest. Uh, but [312:03] it'd be very interesting to see what [312:04] kind of different results we would get [312:06] um if we used another one. [312:10] All right, it's back. Let's see what [312:11] it's decided to change. So, here's [312:13] what's changed. Key infrastructure. So, [312:15] used to hold name film making it double [312:18] as the film's identity card. Now, the [312:19] film director are top level fields. uh [312:22] on finding items in source only describe [312:24] research documents being cited. So uh [312:27] finding of the most particular [312:28] information print findings. So here it [312:31] might be like this. A telegraphic [312:33] teenage girls targeted by her peers. [312:35] Yeah, but the problem is is that if it's [312:36] always from the same darn place, that's [312:38] not going to help us, right? So I got to [312:41] go check this a little bit more [312:42] thoroughly here. And so name of source [312:49] here. Each finding is one piece of [312:51] evidence of of why it's similar. The [312:53] source is the research document you are [312:55] citing, not the film itself. Um, the [312:58] excerpt is an actual passage of text [313:00] quote if possible. The only thing is [313:03] like if everything comes from IMDb, it's [313:04] going to be bad, right? And so I won't [313:07] know until we run it. Uh, but at least [313:09] we now have something that's a little [313:10] bit better. And so I hopefully that fits [313:12] better into our criteria. I'm not sure [313:14] if it actually tried to update uh the [313:16] quality criteria anyway. So, you know, I [313:18] can just ask like does the collecting of [313:22] findings [313:26] tie into [313:28] our quality criteria [313:34] for evaluation. Okay? Because that's [313:37] what it should do, right? And I just [313:39] want to be explicit that it's actually [313:41] thinking about well whatever thinking it [313:44] can really do but thinking of that. And [313:47] so I just want to go down here. Not [313:50] really. There's a gap. Yeah. [313:52] Just parses and stores it. It returns [313:54] records with no quality feedback. So the [313:57] coordinator self assesses against the [313:58] criteria. It's a pure model judgment. [314:00] The tool just logs and echoes whatever [314:01] the score coordinator provides. The [314:03] quality criteria is the coordinator [314:05] prompt. So nothing enforces or measures [314:06] it. So the coordinator has to mentally [314:08] remember the criteria at the evaluation [314:10] time which is unreliable. To fix this [314:12] record findings, compute a quality [314:13] breakdown per finding. Uh is that what [314:17] we really want? [314:21] Score against a quality criteria so the [314:23] coordinator gets immediate feedback. [314:27] And so we have director source whatever [314:29] whatever [314:34] assess coverage using quality already [314:36] returned from records finding score [314:38] zero. The only thing is like you know I [314:41] would just I would just suggest that in [314:45] the evaluator [314:47] um if if it's just IMDb that might not [314:51] be enough as a source. [314:54] So the sources need to be varied. [314:59] Okay, [315:07] I'm not sure if it's the evaluator, but [315:09] you can s see what I'm saying, right? [315:10] Like if it just brings me all black IMDb [315:12] information, even if there's information [315:14] there, um that's not the greatest [315:19] and it's pl it doesn't justify why it's [315:20] similar the similarity claims or, you [315:22] know, things like Yeah. Yeah. Exactly. [315:25] That's why we're using our brains here. [315:27] Um, and so [315:29] we will go up to here and take a look. [315:31] So, what do we have? Um, [315:38] so yeah, weak sources. [315:44] Uh, I maybe it's not the evaluator per [315:48] se because the coordinator has to kind [315:49] of like tie that in here. [315:52] Still going here. [315:57] Mhm. [315:59] Okay. So, we have something new. Let's [316:01] run it. Okay. [316:04] We'll go Python main [316:10] and it's off to the races here. We'll [316:12] see if we run into any issues. Um, I'm [316:15] not sure if it will print out whether [316:16] the criteria is happening. Again, this [316:18] is all about logging, right? And getting [316:20] that C uh keen information. If I was [316:22] building this for real and not just for [316:25] this because again this would double or [316:26] triple the time. Um I would be stepping [316:29] through everything a lot more [316:30] individualized. All this would be [316:31] refactored. I would make sure every area [316:34] is captured and then I can make sense of [316:35] it um to get the results that I want. [316:38] But anyway, this is fine. We will sit [316:40] back and see what happens. Okay. All [316:43] right. We are back and um it's not fully [316:46] done, but we do have a bunch of [316:47] findings, right? Right. And so we have [316:49] plain covered films, 27 findings, and [316:52] notice it actually has more than one [316:54] round of findings. So it's doing way [316:56] more more source documents, which is [316:58] really, really nice. Is it actually [317:00] storing the results? If it's not storing [317:02] the results, that's going to be [317:03] frustrating. [317:04] And it's not storing them right away. [317:08] Huh. I don't like that. [laughter] [317:11] And now I have to wait till it finishes [317:13] to find out if it actually stores them. [317:16] Okay. And that's agent six. [317:19] Yeah, it's not writing the [317:23] the logs. Do we at least get the logs in [317:26] here? No. [317:30] Okay. So, this is one problem I have. [317:32] And I'm going to just adjust this. So, [317:34] um I'll give it back some of the [317:36] feedback here. [317:39] Okay. So, I'm going to go here. It's a [317:41] real bummer I had to do that. I can see [317:44] my agent uh collecting collecting [317:46] findings, [317:48] but it's not writing the findings as [317:51] they find them. And that's a problem [317:54] because if my agent [317:57] loop quits or my program crashes, [318:03] I lose [318:07] that findings data. [318:11] Okay, so that's one little change I'd [318:13] like to make if I can make that [318:14] improvement. Um, as this is getting [318:16] bigger and bigger to run, I'm not sure [318:17] how much it's consuming. I don't have a [318:18] means to track. I I'm sure we could [318:20] figure out token usage if we really [318:22] wanted to, but I'm not doing that right [318:24] now. Um, [318:27] and so we will just wait a moment here [318:29] and see if we can make that improvement [318:34] after each batch. [318:36] Sure, we'll say yes. [318:43] That'll be a little bit better than than [318:45] what we have going on here. [319:00] Okay. And so now that should be a major [319:02] improvement. I'm going to just stop [319:04] here, type in clear, and we will go [319:07] ahead and run it again. And so I'm [319:09] hoping this time I'll actually get [319:10] results, and that will be much more [319:12] useful to me. [319:14] Um, and we should have a new run. There [319:16] we go. And we will just hang out here. [319:18] Okay. The other thing is like the [319:19] results should be named on the run and [319:20] probably should go into the into the the [319:22] run folder, but that's fine. We'll worry [319:24] about that some other time. And we are [319:27] getting findings back here. I'm going to [319:28] give this a nice refresh. And now we're [319:30] getting results. The only thing I'd [319:32] probably ask it to do now is not dump [319:33] everything. Again, we're not trying to [319:35] build a full full app here, but I'd [319:37] probably dump this somewhere else. So, [319:38] here we have the excerpt of the [319:40] information where it's sourcing it from. [319:41] So, here we have Wikipedia. Um, here's [319:45] another one from Wikipedia, [319:47] another one from Wikipedia, another one [319:49] from Zapped. And so, you can kind of see [319:52] a pattern here where it's not going out [319:54] to anywhere else. And so, I mean, it's [319:56] fine. Oh, here's an IMDb1. That's [319:58] totally fine. [320:01] And so like if I wanted something that [320:03] was good, I think I would also needed to [320:05] go out to like independent reviews and [320:08] things like that. And I think to really [320:10] make it work, we would have to be even [320:12] more specific and say like you should go [320:15] here, you should go there, you should do [320:16] this, you should do that. We're getting [320:18] some errors back now for some of the [320:20] findings. [320:24] Okay, so it's missing fields, but at [320:26] least it's doing what we want it to do. [320:28] Um, can it work through these errors? I [320:30] don't know. [320:34] Oh, it's it's formatting it. Okay. So, [320:36] it didn't get back what it really wanted [320:38] to. That's really interesting because [320:40] we're we're forcing it through pyantic. [320:42] So, it should always come back with the [320:43] results that we want. So, we have high [320:44] quality results. I wouldn't call that [320:46] high quality, but again, we are using [320:47] haiku um as our model to keep cost low. [320:51] And I can already see problems with my [320:53] data, which is fine. But we'll come back [320:55] here. I'll have 710 sufficient false [320:59] and [321:03] I have only one Japanese film. Mhm. [321:13] And so we are just chilling out here [321:15] trying to see [321:18] uh the follow following one. I think [321:19] it's the last one. We'll be back here in [321:21] just a moment. Okay, we have our final [321:23] results here and so we'll take a look [321:25] here. [321:26] Um, and so it's just telling us what it [321:28] found here. I I kind of wish we had a um [321:31] a report or summary here because [321:33] obviously we have lots of data. And so [321:35] now it's coming back with excerpts of [321:37] information. So carry fury scanner zap [321:41] fire starter carry twice the fury. So [321:45] we're getting repeats here. And so it's [321:48] not taking these findings and really [321:50] condensing it into an aggregated result, [321:53] but we didn't ask it to do that. I would [321:55] say that we have satisfied our criteria [321:57] at least. The only challenges here is [322:00] that I don't really see it in our Oh, [322:01] wait. Hold on here. Covered areas, [322:08] 24 duplicated films, top 10 ranked list, [322:11] seven out of 10 coverage. Oh, and so [322:13] here's the top one. So before I was just [322:15] I was confused. It was just reading it [322:16] was just showing me all the findings. [322:17] Okay, so we did have a summary here this [322:19] entire time. And so it's really [322:22] suggesting a Kira, I guess. So, I like I [322:24] wouldn't really agree, but it does have [322:27] those telepathic stuff. Um, [322:34] so we'll go back up to here. [322:38] Yeah. And so, does it really meet our [322:41] criteria? And that's the only thing [322:42] that's kind of hard to tell. And so, I [322:45] don't want to spend any more time with [322:46] this. We're only 16 minutes in. But you [322:48] know if I wanted to improve this a bit [322:50] better I would try to again figure out [322:53] okay how do I know that it's meeting [322:54] this criteria [322:56] right and so we have that criteria there [322:59] and again it's just the logging in the [323:01] agent SDK is not great but anyway we put [323:03] it in there that's it for now okay cha [323:04] chia [323:11] let's take a look at how we can help um [323:13] enforce our rules that we actually want [323:16] the pro uh the agent to do. So if you [323:19] prompt an agent like in the coordinator [323:21] and be like, "Yeah, we want you to [323:23] always research before synthesizing, [323:25] right? Always research before [323:27] synthesizing and never call the delegate [323:29] synthesis before delicate research has [323:31] completed." And you think, "Okay, well, [323:33] I wrote that in there. Is it going to do [323:34] that?" It might not because when cla has [323:37] long conversations or unusual inputs or [323:40] it's just having an off day, it will [323:42] just not listen and will do something [323:44] else. Okay? And that's an issue because [323:47] you know we want it to do the things [323:48] that we're asking it to do. So how can [323:51] we get enforcement through uh some [323:54] programming stuff and so we can add [323:56] prerequisite gates things that have to [323:59] happen or have to be met before uh it [324:02] can proceed forward and it makes it [324:04] structurally impossible for them to [324:06] skip. So look at here we have our route [324:08] function. So imagine it's going off. [324:09] It's like okay who are we routing to? [324:11] What's happening here? And we're going [324:14] okay so if we are in this case right uh [324:18] and the context is this and we have [324:21] research [324:22] right then it's like hey you didn't do [324:25] this first so you must go run the [324:27] synthesis agent down here right and so [324:29] we are enforcing it that way um and so [324:33] that is fine uh a even better way to [324:35] organize it is using hooks right so what [324:38] we can do down below here is as we have [324:41] our tool block coming in or information [324:43] coming in. It's like what tool to call [324:45] before it calls a tool. Um it can look [324:49] at those hooks and so it says okay [324:52] before you do delegate synthesis um and [324:56] if you if your context is research then [324:59] you must do the research first or if [325:01] it's after this then do that. Okay. And [325:04] so the idea here is the prerequisite [325:07] gate is don't let this run until it is [325:09] done. And the hook mechanism is a [325:11] function that fires automatically before [325:13] or after a tool call, which is where you [325:15] put your logic gate. Okay, there you go. [325:23] In this video, we are going to set up [325:26] some enforcement gates. And I think in [325:28] this case, we are going to use anthropic [325:32] uh CLI direct or SDK in this case [325:36] um enforcement gates. And the reason why [325:39] with hooks, the reason why we're going [325:41] to do it that way is because um I know [325:43] that agent SDK has um probably hooks [325:46] built into it. If it doesn't, well, [325:48] that's a bummer. But I feel like this [325:50] will be just a lot easier for us to do [325:51] if we are using um that there. So I'm [325:55] looking for one of our prior projects. [325:57] Where are we doing that? Maybe task use [326:00] where we have something very simple. And [326:01] so here we have [326:04] uh but this is with agent definition. So [326:06] that is two new. [326:08] Okay. And so I'm looking for something. [326:10] This is task tool. I guess I chose the [326:12] wrong one. And I wish I kind of numbered [326:14] these as we went so it'd be a lot easier [326:16] for me to find stuff. Um we have narrow [326:20] task de composition. Does this use [326:22] tools? [326:24] This does use tools. Okay. [326:28] And so what I'm going to do is copy all [326:32] this here. And we're going to go into [326:35] [laughter] [326:36] uh here. Okay. And we'll make a new [326:38] main.py. [326:40] And we will paste this in here. [326:44] And [326:46] I'm going to go here [326:48] and we'll go into enforcement gates. [326:52] We'll type in clot. So I want to ensure [326:55] that um [326:59] we enforce [327:01] tool calls in the expected order [327:05] uh in the expected order or [327:06] requirements. Um in the case [327:11] that the prompt guidance [327:14] simply ignores it and I want to have uh [327:19] hooks [327:21] for before [327:24] and after tool calls [327:27] uh to have this enforcement. Okay. And [327:31] we'll see if it can figure out exactly [327:33] what I'm trying to ask for. Again, [327:35] providing examples would be better. We [327:37] have example code. I can do that, but I [327:40] want to see what happens if I just tell [327:42] it if it understands what I'm trying to [327:43] say. But we do have examples up here. [327:47] Okay, that would help give it good [327:49] context. Um, and I mean like in the the [327:53] example I had had like literally a route [327:56] um function that it would call and then [327:58] decide on the tools. I mean that's no [328:00] different than saying like call tool, [328:01] right? And so in here see how we have [328:04] like our steps and stuff like if if stop [328:06] reason is this or whatever whatever then [328:07] maybe we do that. [328:10] Um I only see screening agent here. So [328:12] maybe we need more than a single uh [328:14] tool. If we went a little bit farther [328:16] back, that might have been better [328:20] with a pre-post hook around tool calls [328:22] in the coordinator loop to guarantee [328:23] ordering and requirements even when the [328:25] model might not follow the prompt [328:26] guidance. Okay, so I think it knows what [328:28] it's talking about. Uh I don't know why [328:30] it always wants that. Fine, we'll give [328:32] you the top directory. I don't see why [328:34] it needs that. [328:36] Why? Why? Why? Why? Why? Why? Why? Why? [328:38] Why? Why? Why are you going into [328:43] the directory prior to it? [328:47] Stop. [328:50] I wish it would stop doing that [329:01] because it's wasting time when it does [329:02] that. It doesn't need to look at any of [329:04] that stuff. It just needs to focus on [329:06] its own codebase. [329:13] We will pause here. We'll wait and see [329:14] what we get. Interesting. It's decided [329:17] to make a gate.py file. Let's take a [329:20] look here. So, we're getting Oh, we got [329:21] another file going on here. What's going [329:23] on? [329:24] Look at our main.py. [329:30] I'm trying to figure out where it's [329:31] adding those gates. [329:34] Pre and post tool call from the gate [329:36] enforcement gate. So, it creates a gate. [329:38] Then, we can add a hook and a pre- hook. [329:40] That's kind of cool. I actually like [329:42] that it did that. So, maybe that's why I [329:44] wanted to go up and look at my code [329:46] because it's trying to match my uh [329:48] coding practices, which is fine. I mean, [329:50] like I do like that. So, I cannot [329:52] complain. But, I would like it to not go [329:55] into the extended repos and consume a [329:57] ton of tokens. See like how we have 8.8 [329:59] tokens because that's just larger on my [330:02] consumption. But it is actually getting [330:04] uh the coding style the way that I want [330:06] it to be or closer to what I want it to [330:07] be. Um and so I'm going to go all the [330:10] top here. Let's see where the gate [330:11] information is. Oh, it's going to give [330:12] us a summary. So we have start uh at the [330:16] start of each coordinator before [330:17] executing any tool call. We'll do a [330:19] check pre. Then we will have this check. [330:22] Then we'll have a post check and then we [330:24] have a summary. We have four built-in [330:27] factories. So max hook calls, no [330:29] duplicate arguments. A lot of stuff [330:31] going on in here. When a pre hook block [330:33] is called, we have an error if there's [330:34] an issue. But let's take a look and see [330:36] if we can make sense of it. We only have [330:39] a single tool. [330:41] Um, so I'm not sure how useful this is [330:43] going to be. I want to follow through to [330:45] wherever this is. Enforcement gate. [330:50] Okay. [330:51] And so it's here being passed to the run [330:53] coordinator. [330:55] Uh-huh. [330:58] And so here, okay. In our actual loop [331:00] here, when we check the tools, it says [331:03] check pre. [331:08] Uhhuh. But where [331:10] where's our gate logic? [331:14] Okay. So, um [331:19] pre and post hooks. Uhhuh. Okay, cool. [331:22] So, add a gate. Add a pre hook. [331:31] Okay, where is the logic? Where do we [331:34] store [331:37] where do we define the logic when a gate [331:42] is uh a hook is triggered [331:54] cuz here we see like um in your tool [331:57] you'd have this like check pre pre [332:02] 179 190 So, we'll go over to here. [332:07] Oh, right here. Okay. So, [332:11] uh is that how it should work? [332:14] I mean I mean is that how it should [332:18] work? [332:20] Shouldn't uh we [332:22] have the code listening [332:26] and then it triggers a function. [332:31] It seems uh [332:35] it doesn't really seem like a hook [332:38] architecture [332:45] and that's probably because I mean like [332:47] I'm used to TypeScript and other [332:48] languages but like it's not like it's [332:50] like calling a llama or something. So we [332:52] define Ruby or like a proc. [332:58] So I might have to tell it like hey [332:59] where's my llama? Where's my proc 42? [333:07] Mhm. So, the loop shouldn't look like [333:09] this. If else is it should look like [333:11] Yeah. a llama. There we go. Okay. Like I [333:12] don't know what it is in called in [333:13] Python, but it either be a proc or a [333:16] llama. And that makes a lot more sense. [333:17] The the gates fire prehooks the and they [333:19] block and then they fire post hooks all [333:21] internally. Let me redesign it. Sure, [333:23] why not? We are back to look at the [333:25] code. And so now it kind now looks like [333:29] that or it should look like that. So we [333:30] go back over to here and if we look at [333:33] our tool gates [333:39] now it's using a llama and it's calling [333:41] it that way. And so that looks better. [333:44] Um [333:46] I don't know this would be a lot better [333:47] in Typescript I think. I don't really [333:49] like this in Python but you get the [333:51] point right. The idea is that we have um [333:54] a gate and the gate should get called [333:57] and make sure that it meets whatever [333:59] requirements. Um but again, I'm not [334:01] happy with this. I'm not going to go and [334:02] re-record this and do TypeScript. You [334:04] get the idea of uh what it is and just [334:06] remember why that's important. Okay. [334:13] A handoff protocol is a structured [334:14] package that the agent assembles before [334:16] escalating. It ensures whoever receives [334:18] the escalation has everything they need [334:20] without having to dig through [334:21] conversation history. So the idea is [334:23] that you have this handoff information. [334:26] Um and then you can escalate it to a [334:28] human over here. And we're finding that [334:30] uh maybe we should have been using [334:31] TypeScript this entire time because um [334:34] this looks a lot better for um [334:37] uh this build stuff. But anyway, we will [334:39] go ahead and give this a try. Okay. [334:45] Hey folks, we are back and I want us to [334:47] implement this handoff protocol. So I'm [334:49] going to make a new folder here. We'll [334:50] call it handoff protocol. And what I'm [334:53] going to do just to make it a little bit [334:56] easier and go back to this whole taking [334:58] screenshots and making sure this stuff [335:00] works because I have that slide, right? [335:02] So I'm just going to go here [335:04] and I'm going to grab that slide [335:06] information that I have that I made. [335:09] This is because I just um I usually put [335:11] them in the notes down here, but I [335:12] didn't do that. Okay. So, we'll go ahead [335:16] and actually just going to grab all the [335:17] text here. I'm not even going to goof [335:19] around. I'm going to dump that into chat [335:20] GPT. I know it's a cloud course, but [335:23] don't want to waste my credits. Give me [335:25] the text. It's what I'm writing here off [335:28] screen. Getting chat GPT to generate out [335:31] for me. [335:34] Okay, let's give it a moment here. And [335:38] I'm going to CD back here into this [335:41] directory. handoff protocol [335:43] and I want this information. [335:49] Oh my goodness, I just want all of it [335:50] here. So, we want to grab it. [335:53] Copy the response. There we go. And so, [335:56] we'll go ahead and go to plot. I want [335:58] you to implement this using [336:02] uh using TypeScript. [336:05] No. Will that work in this case? [336:08] um using the anthropic [336:11] SDK directly. Um [336:16] I guess we just let it use whatever it [336:18] wants in the main.py. [336:21] This is what we want to show off which [336:25] is the handoff uh functionality. [336:30] Okay, we'll hit enter and hopefully [336:32] they'll figure that out and just write [336:33] it into the main.py. Pi [336:38] just trying different approaches here to [336:40] get different kind of quality though we [336:42] were doing okay just copying prior code [336:44] bases are obviously getting quite a bit [336:46] here but yeah the goal here is just to [336:48] make that handoff and so hopefully with [336:50] that reference even though it's not [336:51] exactly the same code that we're doing [336:53] here we will get something that looks [336:55] like that [336:57] and that would be [337:00] really really nice. [337:03] Okay. So, we will just hang out here for [337:06] a little bit. [337:09] Okay. So, we have our and we'll say yes. [337:14] And I'm just going to close some of [337:16] these out here so we can take a look. [337:19] Where did it go? [337:23] Where is it writing? [337:30] Well, hold on. [337:32] Write me the code. Where's my file? [337:35] Here it is. Okay, great. And uh [337:42] probably because it's not loading the [337:43] same way. So here we have lookup order, [337:45] attempt refund, clear fraud flag, [337:49] simulate backend, lookup order. That's [337:51] fine. So it really did pull from our [337:53] older codebase. Attempt refund. [337:56] And here [338:00] build handoff. So here we're building [338:02] the handoff. Ask them all to synthesize [338:04] a structured handoff package for the [338:05] conversation. [338:09] Escalate [338:11] run the agent. Good. [338:15] And here we see in the loop as it loops [338:19] through the tools if it needs to [338:20] escalate to a human that's going to [338:21] build that handoff and hand it out that [338:24] way there. And so then it stages it down [338:26] below. Um so it does have a little bit [338:28] of an issue here with the [338:31] call [338:34] and I'm going to go all the way to the [338:36] top here because it's not loading in the [338:38] um uh the [338:42] environment variable. I'm going to go [338:43] over to this one here. It's missing um [338:46] maybe not that Hello World. It's the [338:48] Anthropic SDK. [338:51] We'll go to another one. We'll find one [338:52] here. Here we go. There's one. And so uh [338:55] because we we actually need it for this [338:57] to work. It won't work without it, [338:58] right? [338:59] And [339:02] we also need env. It didn't bring that [339:04] in, which is fine. It's fine if it [339:06] didn't do that. And we need path. [339:10] Okay. And so now we'll try to run this [339:12] again. [339:14] It was already trying to run it, but [339:15] we'll we'll run it. It was trying to be [339:16] uh it was being a little bit too um [339:19] aggressive there. [339:24] Uhhuh. [339:28] And this is what we get for not starting [339:29] from a base. [339:36] Maybe because of where where and how [339:37] it's loading it [339:40] as we should have that anthropic key [339:42] somewhere. I'm just going to type in key [339:43] here like key. There it is. And then [339:46] does it have this one key. And that's [339:49] our problem. So we'll go over to async. [339:55] No, that's not going to help us here. [339:59] Uh, you need to fix the um [340:04] authentication [340:06] for main.py. [340:08] You can look at narrow task decomposes [340:12] main.py [340:14] uh for a reference on how to fix. Okay. [340:17] And so [340:21] I mean authentication API key, right? um [340:27] for setting the API key to the client [340:34] and make sure you are configuring the [340:38] client correctly. [340:40] Um and maybe we should have copied it [340:41] because we do have this like [340:42] asyncthropic stuff in here and it's not [340:45] doing that right. [340:48] So maybe that was my my mistake. [340:54] Mhm. [340:56] I'm going to tab here and make sure you [341:00] use async. [341:02] Okay, [341:12] we are helping it along there folks. [341:21] Okay, we'll take a look at its refactor [341:23] here [341:26] for our handoff. [341:29] I'll close that out. Reopen it. [341:32] Cancel. Cancel. Not moving anything [341:33] here. Uh, sure. That's fine. And um [341:39] I don't really need that main there, but [341:41] that's that's okay. Um, okay. So, that [341:43] looks like they may have fixed it. [341:49] We still have client messages create [341:50] which is fine. It's fine if they want to [341:52] do that. Let's go ahead and run it. [342:01] Okay. [342:03] And so see here it attempts it and then [342:05] it has to escalate to [342:08] the customer. [342:10] There you go. [342:16] Let's take a look at pre-tool uh use and [342:18] posttool use hooks. So this is part of [342:21] agent SDK where uh you can basically add [342:25] uh oops add these hooks uh for pre and [342:28] post tool for your tools so that you can [342:33] go and uh intersect them. So it [342:35] absolutely does work. Um, you obviously [342:38] saw we rolled our own hook system, [342:41] whereas I mean this would be really nice [342:43] if you need to um do that there. Notice [342:46] that it's matching on all hooks. So [342:47] technically, if you wanted to, you could [342:49] tell it to match on a very specific [342:53] uh tool. Okay. And then have different [342:55] types of It's not like you just only get [342:57] these two, but you can set exactly what [342:59] the pre and post tool hook is for each [343:01] individual allowed tool. Um, but yeah, [343:05] pretty straightforward, but obviously [343:06] better than rolling your own if you're [343:07] using the agent SDK, right? [343:13] Hey folks, in this one we're going to [343:15] look at pre and post hooks. I'm going to [343:16] go ahead and make a new folder here and [343:17] call pre and post hooks. [343:21] Okay. And um we'll create a new main [343:25] main file here. And I believe what we're [343:28] going to want here is to use the agent [343:31] SDK. Okay. So, I'm going to go into our [343:33] hello world. I'm going to keep it nice [343:34] and simple. And we're going to grab the [343:37] most simple implementation that we have. [343:41] Okay. Into here. And what we're going to [343:44] do is cd into [343:47] the [343:49] um pre and post hook directory here. And [343:52] we're going to go into cloud. [343:54] And I want to implement pre and post [343:57] hooks uh for demonstration. [344:00] Here [344:02] are code samples [344:06] to help you think of [344:10] uh what to do. Okay. Or actually, you [344:12] know what? Let's just I want to uh for [344:15] demonstration. [344:17] Does agent [344:20] SDK have those built in? [344:23] I'd rather just go ask it [344:26] to do it. Um because we do have the code [344:28] samples. If it does need them, we can [344:29] provide it. But I figured, hey, let's [344:32] just uh [344:34] let's just go see what it can do. Okay, [344:36] we're getting some code output. It is uh [344:39] again going up the repo and taking a [344:41] look at other some stuff. And I'm just [344:43] going to close this out of here and see [344:45] what we have for our [344:50] stuff. [344:52] Where did you put the code? Yeah, this [344:55] is the context we have. [344:57] does not have built-in pre-post hooks. [344:59] The query function returns an async [345:01] generator message. I don't know why I [345:03] thought there was pre-post hooks, but um [345:09] that's fine. [345:14] One second. [345:17] Okay, how about this? Are you sure? Uh [345:19] what about pre-post tool uh tool? Are [345:24] these not part of the uh SDK? [345:33] Is this another limitation of the Python [345:35] SDK? [345:45] Mhm. [345:54] We'll give it a second. Okay, here we [345:56] go. So, it's like I was wrong. Pre and [345:58] post are built into the SDK, just not in [346:00] the cloud code CLI feature. Uhhuh. We're [346:04] doing the agent SDK. What do you think? [346:06] What do you think we're doing here, [346:06] folks? Okay. And so, here we have the [346:08] pre-tool use, the post tool use, [346:10] failure, etc. Um, and I'm going to close [346:13] this out. Reopen this up. [346:17] Yes, please. Yes. Okay. Okay. So, it was [346:19] just researching. First time it didn't [346:21] uh just go out and do something. It did [346:22] some research. Nice. And we'll just say [346:25] yes, you can go ahead and update that. [346:26] That's totally fine. [346:29] So, here we have pre and post. [346:37] Okay. So, done. The hooks will print pre [346:39] and post. So, let's go ahead and see if [346:41] that actually works. And we'll do python [346:45] main.py. [346:50] Okay. And so we have our pre and post, [346:53] right? So it it did both of them. And so [346:55] that's all we're there's no hello world [346:57] in here, so I'm not sure how it's going [346:58] to fix it, but the point is that we [347:00] hooked up our pre and post hooks. Um, [347:04] and so, you know, that's pretty cool. [347:06] This is obviously a lot cleaner than the [347:08] other implementation we had. As you saw [347:11] before, we created like this whole hook [347:12] system and it was so complicated. But [347:15] why not just use this because it's right [347:16] here and we can use it because it is [347:18] right here. But there you go. [347:25] Let us take a look at prompt chaining. [347:27] So prompt chaining or you can think of [347:29] it as a sequential pipeline is when you [347:32] use a series of agents or calls uh to [347:35] get the results. So here uh what we're [347:37] doing is we go okay we have our input we [347:40] run it we put that out there we run it [347:42] we put that out there we run it we put [347:44] that out there and so that would be [347:47] example of prompt chaining so this is [347:49] good for document processing content [347:51] transformation ETL pipelines any task [347:52] where you uh the uh the shape of the [347:54] work is fixed regardless of the content [347:56] I'm saying the word agent here but um [347:59] you know before we were saying agent and [348:01] putting things in an agentic loop we [348:03] could do this with LMS and actually um [348:06] when we were really starting with LLMs [348:07] and everything was very short and fixed. [348:09] The only way we could work with LLMs was [348:12] to prompt chain and so prompt chaining [348:14] is not a new concept. It's very old. Um [348:16] but if you're new to it, uh just [348:18] understand that it is basically setting [348:20] up, you know, a chain like this. Okay. [348:27] All right. So in this video I want to [348:29] show off prompt chaining. So we'll go [348:31] ahead and make a new [348:34] folder. So we'll say prompt chaining [348:40] chaining and we will create ourselves a [348:43] new main.py and we're going to borrow [348:45] from somewhere. Uh [348:49] I'm going to take the hello world just [348:50] because it's super easy. And [348:54] I mean it probably be better to use the [348:55] anthropic uh SDK for this, but I don't [348:58] care. I'm just going to do it anyway and [348:59] see what happens. because we just [349:01] conceptually need to understand what [349:02] prompt chaining is. Okay, so cd prompt [349:05] chaining it will let me into this [349:07] directory. Please [349:09] prompt chaining. Great. And we'll type [349:11] in claude [349:15] change. I'm just going to make sure it [349:17] knows prompt [349:20] change. No, no, it's just the main. [349:21] Okay, we want to uh demonstrate prompt [349:25] chaining [349:28] showing uh sequential calls [349:32] to agents, [349:36] multiple agents [349:39] uh uh agents or or queries um to show [349:43] off [349:44] a sequential Yeah. for like a fixed [349:48] pipeline. fix pipeline. [349:51] Okay. So, we'll go ahead and try that. [349:54] See if it understands what I'm talking [349:55] about. So, remember, it's not like it's [349:57] bad. It's always about trade-offs here, [349:59] right? Um I'm hoping that it does like [350:02] the coding sample that I have because [350:03] that's I mean, I wrote that coding [350:05] sample, but that's how I would think of [350:07] it. Um or that's how I would write it by [350:10] hand, but I'm curious as to what it [350:12] thinks it's going to write. probably [350:14] something complicated and not how I [350:15] would do it, but we will uh see what it [350:18] does. So [350:20] pause here and see what we get back. Oh, [350:22] that was fast. That was fast. We'll go [350:23] ahead and hit yes. [350:27] And so we have identify bugs, propose [350:30] fixes, apply fixes, right? And we'll go [350:33] back here. [350:38] Mhm. So, okay. So, we have await run [350:41] step await run step. So, yeah. Notice I [350:43] didn't give it any code and it more or [350:44] less is doing pretty much uh the same [350:47] thing. So I'm going to go in here and [350:48] we'll make a hello world. I'll copy this [350:51] one [350:52] and we will paste it into here. [350:55] And I'm going to just break it a bit. So [350:57] say uh puts put hello world here. [351:03] And we could have probably gave it a [351:06] more complex case. You know what? I'm [351:08] going to do that. I'm going to go over [351:09] to chatbt. [351:12] give me um busted messed up um block uh [351:17] code for Ruby so I can [351:20] I can test my uh agent that fixes bugs. [351:25] Okay. And so I'm just asking chatpt here [351:28] to do that for me [351:34] and [351:35] waiting just a second here. Is still [351:37] generating. [351:39] Oh, there we go. We'll copy it. [351:42] I wonder if I can spot the problems [351:45] just for fun. [351:52] Uh [351:57] there's one problem. Is that the only [351:59] problem we get? [352:03] A set of 10 tiny broken Ruby functions. [352:06] Okay. Well, it says there's bro breaks. [352:08] So, it'll be interesting to see that [352:09] work. And I'm going to go ahead and we [352:11] are going to do Python [352:16] Python main.py [352:22] and we will see if it runs the pipeline. [352:25] Looks like it's running the pipeline. [352:28] Got nothing wrong with this stuff, [352:30] right? But did we even save the file? [352:34] [laughter] Is it just working on a blank [352:35] file? One sec. Let's try that again. [352:37] Sorry, I don't think I saved the file. [352:38] Now the file's saved. Otherwise, it's [352:41] doing nothing. [352:44] Okay. [352:46] See, now it's taking longer because [352:47] there's actually something to process. [352:49] Okay. One second here. It's applied its [352:51] fixes. Doesn't tell us what it did, but [352:52] let's go take a look here. And [352:56] oh, we got logs. I didn't know it was [352:58] logging. That's nice. [353:00] I guess we probably set that up there [353:02] before. [353:04] Uh, here we go. So, [353:09] I'm just curious what it fixed. [353:18] Oh, it found most of the bugs. Yeah, it [353:20] said there was 10 bugs, but uh yep, I [353:22] guess it fixed it. Cool. So, there you [353:25] go. Prompt chaining. [353:31] Let's take a look here at dynamic [353:32] adaptive decomposition. So, this is when [353:34] intermediate finding should change, what [353:36] you do next, which agents to call, how [353:39] many times and in what order. Um, and [353:42] these are going to be really good for [353:43] open-ended research, investigation [353:44] tasks, anything where you don't know [353:46] upfront how deep or wide the work needs [353:49] to go. So, look at the example. You're a [353:50] research coordinator. After each step, [353:53] assess what you found and decide what to [353:54] do next. You may need more research, a [353:56] different angle, or you may have enough [353:59] to synthesize. Do not follow a fixed [354:01] sequence. follow the evidence. So, let's [354:04] see if we can implement that. Okay. [354:11] Hey folks, it's Andrew and we are going [354:13] to go ahead and do some dynamic, that's [354:15] a weird word, dynamic [354:19] uh adaptive [354:23] demp. Okay. And so the goal here is we [354:28] want to looking back at the slide. We [354:30] want to have um the coordinator say, you [354:34] know, what should we do next? What ages [354:36] is it call? How many times? And it's [354:37] open-ended, right? It's up to itself to [354:39] figure that out. Um as we work through [354:42] this, it kind of feels like some of [354:43] these things we've already kind of done, [354:45] but I guess they're just in different [354:46] kinds of variations. And so, you know, [354:48] to me, it's not always really obvious [354:50] right off the bat um until we work [354:52] through it. But we will see how this [354:54] goes. As it seems like a lot of our [354:55] other models have been pretty darn good. [354:57] Um, so I'm just trying to think of here [355:00] uh a basis of what we can work off of. [355:03] Um, I probably want to use agent SDK. [355:05] One thing that we haven't really been [355:06] doing is using the file system method. [355:09] Um, we've been programmatically [355:11] implementing our agents uh in the code [355:14] and I might want to take a file system [355:16] approach to this. Um, and that just [355:17] means how we organize our agents. So [355:19] they're loaded from a markdown file and [355:22] they have their own folder and blah blah [355:24] blah. It'll make sense here in just a [355:25] moment. So, I'm just trying to decide [355:28] um what for us to do. Um I need kind of [355:32] like an open-ended uh objective. Let's [355:34] just use something like chatbt to think [355:37] up some ideas. One second. Here we go. [355:39] And so I just asked chatbt and because [355:41] it knows it knows me a little bit and it [355:44] knows me too well because right away [355:45] it's like cloud code agent dungeon like [355:47] make a dungeon. This is something that [355:48] I've tried to spend a lot of time trying [355:50] to uh implement and um and so maybe that [355:54] would be a good one. investigate like a [355:56] DevOps incident commander. That sounds [355:58] too real world. That's too too much too [356:01] much fun. I'd save that for something [356:02] else. Japanese sentence evolution [356:04] engine. That's something else that um [356:07] tried to implement. Fix my broken code. [356:09] No. Community health analyzer. Like this [356:13] is for my community stuff. Claudeco [356:15] skill tree simulator. [356:19] Uh certifications meet progression. H [356:26] I don't understand how that would fit in [356:28] there. [356:30] And so we might go back all the way up [356:32] the top here. [356:34] Open and research, investigate tasks, [356:36] anything where you don't know upfront, [356:37] deeper, wide. I kind of like the cloud [356:39] code agent dungeon to be honest. So [356:40] let's take a look here. You build a a [356:42] dungeon crawler where each room has a [356:45] decision. It adapts. You already build [356:48] games, trading mods. You orchestrate [356:50] cloud code. you're teaching this uh [356:52] visual mental model. So, what I'm going [356:54] to do is we're going to make a new file. [356:57] I'm going to call it plan.md. [356:59] And I just want to describe what it is [357:01] that we want to do. And I'm going to [357:02] grab some of this text here. So, we have [357:05] cloud uh code engine. And so goal uh [357:09] like technical goal is to [357:12] implement [357:14] and I got dynamic decomposition uh when [357:19] uh when when intermediate finding should [357:21] change to do with next agents how many [357:22] times to call right um [357:26] so anything where we don't need [357:29] information to go up front and so here [357:31] is a business goal we will uh implement [357:38] dynamic adaptive decomposition [357:43] for a [357:45] agent dungeon. [357:48] Okay, so we will construct [357:52] um [357:54] a mud engine [357:56] and the agent is [358:00] you build a dungeon crawler where I'm [358:02] just trying to see what they're [358:03] suggesting here. uh spawn debug agent if [358:06] unclear spawn research agent if too [358:08] complex. I guess the idea is like so [358:12] an agent dungeon. So the idea is that we [358:14] are generating [358:17] the world as we play uh play the game. [358:23] The agent [358:26] has to [358:28] figure a way to uh build [358:32] the static world [358:34] in code [358:36] as we play along. [358:40] Okay. And so I guess the question is [358:41] like is that a good use case? I'll go [358:43] back here over to here and say like you [358:45] know is that [358:48] no no no no no. Okay. I want this text [358:50] here. I want to ask if like if it thinks [358:52] like that's a good idea because that's [358:54] what I'm thinking, right? [358:57] Oh my goodness. Just let me paste. Can't [359:00] paste here today. It's really [359:01] frustrating. Now look, now my whole [359:02] thing's messed up. There we go. So, [359:04] we'll go here, [359:07] rightclick, paste. [359:12] Would this make sense? Because I think [359:14] that's kind of what it's leaning [359:15] towards, right? [359:22] I bet it's just like love bombing me [359:24] here, but the live discovery world. [359:31] Okay. Well, anyway, what we'll go do is [359:34] move this over. So, we'll say [359:35] implementation. So, technical technical [359:38] considerations. [359:40] Uh, we want to use [359:44] claw agent SDK. [359:46] We want to use the file system for [359:50] agents eg agents directory instead of uh [359:56] coding them using agent [359:59] defaf. Okay. And so those are the things [360:01] I want to enrich this so we have a spec. [360:04] Um you've probably heard of spec driven [360:05] development. You don't need those [360:07] frameworks. You can just kind of work [360:08] through this and tweak it as we go. So [360:11] what I'll do is I'm going to go ahead [360:13] and type in claude. Wh not there. Maybe [360:15] not there. We are going to go to [360:18] our dynamic. [360:20] As we've been working through this, [360:21] we've been learning to isolate these [360:23] folders. Dynamic. Uh, adaptive. Oh, we [360:27] have more than one word here. Adaptive. [360:29] Oh my goodness. [360:31] I'm going to change this to No, I'm just [360:34] going to copy this. I guess [360:37] we'll paste that there. Great. Um, and [360:39] so we'll type in Claude. So, I'm going [360:41] to go plan. Can we can we uh create a [360:46] text spec [360:48] MD [360:52] a technical spec to see its [360:55] implementation or you know what can you [360:58] can we create a [361:01] yeah a tech spec y [361:08] trying to decide here [361:11] do I want to X spec because what I'm [361:13] thinking right now is okay I have this [361:15] and I feel like it's kind of fuzzily [361:16] defined in terms of what it's doing. Um, [361:19] and so I wanted to do a little bit more [361:21] research and try to better describe the [361:24] um, text spec of not necessarily the [361:27] implementation of it, but like I I I I [361:30] like we know we know how to implement [361:32] agents, but the trick is does our [361:36] business goal uh goal uh, fit uh, [361:40] dynamic adaptive [361:43] decomposition. So, can you [361:47] try and see how it fits? [361:51] Uh, writing a tech text back. [361:55] Okay. And maybe I shouldn't be calling [361:57] it text back. It's really more deeper [361:58] research, but I don't care. It will [362:00] still do it. And we will come back here [362:02] and we'll see what it has and then we'll [362:03] review it. Um, and again, we're just [362:06] trying to conceptually understand, are [362:08] we driving this thing correctly for uh [362:10] the solution we want, right? [362:12] I bet people really wanted me to do to [362:14] do the DevOps one, but I didn't do it. [362:16] All right, we are back and we have [362:18] something here. We'll say yes. We'll [362:19] take a look here. Uh I saw on effort, [362:22] medium effort, so I'm not sure why that [362:23] was so darn long, but it really went off [362:26] off on the rails. It was consuming a lot [362:28] of uh credits, unfortunately. I'm [362:30] actually going to go take a look at my [362:31] subscription and take a look and how [362:32] it's doing. And right there, look, I've [362:35] just basically started my day and I'm [362:37] already halfway through here. So, this [362:38] is where uh we got to be really careful. [362:42] uh and maybe restrict how cloud is [362:44] working. I've been having a lot better [362:45] results working with the API where I [362:46] have more direct control over the exact [362:48] context use usage, but we'll take a look [362:50] here. So, agent dungeon is a textbased [362:52] world so damn ver verbose [362:55] powered agent and each player uh demands [362:58] new content. Static replanning is [363:00] impossible. The number of rooms dynamic [363:02] adaptive decomposition is not an [363:04] optimizational optimization choice here. [363:06] It is the only viable pattern. They they [363:09] make it sound like it has to be that [363:10] way. [363:11] Uh where is it genuinely required? So [363:15] unknown branching of generated rooms [363:17] when the player moves north. Why static [363:20] planning fails here? Pre-planned [363:22] generated five room produces a flat [363:23] contextfree world. The generator cannot [363:25] know that the player is three levels [363:27] deep in the necromancer's tower. [363:30] Encounter depth is non-deterministic. Um [363:34] okay. Code generation for the world [363:36] artifacts has unknown scope. [363:39] Mhm. [363:42] And so here player moves world [363:44] generation for a new room sub features [363:47] uh agent ident inventory. So all agents [363:50] live in the agents directory. This is [363:51] something that would like to do. So we [363:52] have the orchestration orchestration one [363:54] the central coordinator. So I don't know [363:55] why they just don't call it coordinator [363:56] but that's fine. Coordinator the only [363:58] agent that maintains the full [363:59] decomposition state. It reads the [364:01] current state world state JSON which is [364:03] interesting because that's how I've [364:05] always implemented these things. I've [364:06] always had like a world state file JSON [364:08] file. Decision logic is the orchestrator [364:11] at runtime. So in the room it's [364:12] structurally simple. Does the room have [364:14] an NPC? Does it have this etc etc. So it [364:18] sounds like it's going to try to [364:19] maintain the world state. Then we have [364:20] the room builder. So generate the uh [364:22] structural descriptions. [364:24] Uh so we have the blueprint. [364:27] Okay. So that makes sense. [364:30] We have the code writer. [364:32] So produce TypeScript code in a single [364:35] game artifact. [364:37] Avoid redeclaring it. [364:40] Lore consistency. Cool. Combat. Cool. [364:44] Puzzle. [364:46] Great. World state model. The world [364:49] state is shared artifact that makes the [364:51] the place coherent. Um, so we go look at [364:54] the world state. The key subtypes [365:00] incremental build. So each player turn [365:02] starts with the open branches. is the [365:04] orchestrator adds branches and [365:05] determines them [365:08] and then the orchestrator decides [365:09] decisions. [365:15] Okay. Agents world [365:21] a static dungeon generator creates an [365:23] affinite graph uh before a player [365:25] begins. A player who never goes east [365:28] never has that stuff. Um a static system [365:30] assigns NPC. So the question I have here [365:33] is I don't want the whole thing being [365:36] driven by the agent. Uh so like if [365:39] there's an area that we've been through [365:40] before, all agents will use the haiku [365:42] model. That's good because that's cost [365:43] effective for me. Um [365:50] okay. So I guess my my my main one [365:53] question so my question [365:57] about the text back and I actually don't [365:58] mind it. So, it's an okay text back. [366:04] Is [366:06] are we only generating rooms when we go? [366:11] Is will this only trigger [366:15] the agent when we enter a new room? [366:20] Because we want to not use AI [366:25] to power the whole game. [366:30] right? [366:33] Just for its uh generation of places [366:37] it hasn't been. [366:39] And the game engine is controlled [366:47] by code. Right. Right. That's it's going [366:50] to do that. Right. It's just a really [366:52] long dock. And so there was nothing in [366:54] there that told me that it was not going [366:56] to do that. So it says good catch. The [366:57] spec is written as ambicuous about this. [367:00] So the spec currently implies agent [367:02] could fire on any player action which is [367:03] wrong. What you actually want is [367:05] two-layer architecture. So game engine [367:07] and then world generator and then the [367:09] correct gate is that. So the room [367:11] blueprint would be this chain all [367:13] happens inside one generated room. Once [367:15] it resolves that room is permanent. The [367:16] AI never touches again. The spec needs a [367:18] section clarifying this boundary [367:20] explicitly. And yeah so I would say yes [367:22] add that line. And you know, maybe that [367:24] might make the game a little bit boring, [367:26] but like um I think that we have to be a [367:28] bit careful where we're adding that. And [367:30] so this is where I would like to see [367:32] video game devel games games play out [367:34] where it's like, you know, it's 90% 70% [367:38] uh code driven, but then the the fun [367:40] parts are uh that create spontaneous uh [367:43] experiences, you know, calls out to an [367:45] LLM. I think that's kind of cool. Or an [367:47] agent. [367:49] And there could be key areas where it's [367:51] like, okay, now we have a boss or like [367:53] we decide there's a special thing and [367:55] then trigger it. But um we will go ahead [367:58] and let it uh make those little updates [368:00] here assuming we do not blow through all [368:03] of our [368:04] um usage here. The other challenge right [368:07] now is that uh it also has to do with [368:08] when you're using cloud. So like right [368:10] now I think everyone like both North [368:12] America and Europe are using at the same [368:14] time and so consumption will be uh quite [368:16] higher. But what I do find is that [368:18] tokens like when we're using Hycoon and [368:20] stuff like that have been going really [368:21] really good. Like I still haven't had to [368:23] reload up stuff. Also, you know, having [368:25] my extra usage over here has been really [368:27] really good as well. I'm not sure if [368:28] this is built the same way as this, but [368:31] um this consumption has been actually [368:33] excellent uh handling our uh over [368:36] overload. So I have no idea. I feel like [368:39] this top part is totally different from [368:40] this part. That's what my experience has [368:43] been. And so we will just see as we go [368:45] here. So section one reumbered the key [368:49] uh line is explicit. AI fires once per [368:51] unvisited room never touches again. [368:53] Okay, cool. And so there are two parts [368:55] to this where we need the um [369:02] the game engine. So I'm going to scroll [369:03] up here. [369:06] Okay. So [369:08] Okay, great. [369:11] Please build me the game engine. [369:15] Uh, okay. Just to clear context here [369:18] because this is something we should [369:19] constantly do. So, please go build [369:23] the game engine [369:25] uh for text [369:34] implement text back. Okay. And so the [369:36] thing is is that um again it would be [369:38] better if we were carrying code over [369:40] like if I was if I was being a little [369:43] bit more diligent if we weren't on a [369:44] timeline I had to get through this [369:45] course I would probably spend more time [369:47] making my components reusable and keep [369:49] pulling from a base uh template probably [369:52] something I'll do for my um my my more [369:54] expensive courses. Um, but anyway, it's [369:56] going to go ahead and uh implement [369:59] something and as long as it works, [370:01] that's all I care about. And we can [370:02] observe this a dynamic adaptive [370:04] decomposition. And we're going to be [370:06] mostly interested in looking at what the [370:08] prompt is going to be. So, go ahead and [370:10] say yes here. And we will chill while [370:12] this generates out. Okay. All right, [370:14] folks. So, um, it's gone and done the [370:16] work. I think that it might have gone [370:17] into extra usage. And I'm going to just [370:20] go take a look here as, uh, whatever. [370:22] But also, I was talking to my friend [370:24] Anthony. He used to do the the GC GCP [370:26] courses. He's like use this tool called [370:28] monitor. The only problem is that it's [370:30] not reflective exactly of of it. He's [370:33] saying it was like one to one perfect, [370:34] but in my experience, it's not. So the [370:38] biggest problem is like monitoring costs [370:40] and the only way that I can reliably [370:41] monitor my cost is to constantly open up [370:43] this window. So here it says 99%, but [370:46] look, it says 123% here. Doesn't match. [370:49] The other challenge is like I have this [370:51] over usage. It didn't even hit the [370:52] overusage, but um notice I was like 99% [370:55] for this one task here, but we did ask [370:57] it to do a lot. Um and you know, I think [371:00] like the other approach of me always [371:01] pointing to another codebase and setting [371:03] up the boiler plate for them is [371:05] obviously reducing the cost greatly. So, [371:07] we really should continue on using our [371:09] boiler plate and pulling from it because [371:11] I'm really keeping my cost low uh doing [371:13] that. But let's go take a look at what [371:14] we have generated. Um so, what do we [371:18] have? [371:19] So here we have uh agent dungeon [371:22] implemented complete dynamic adapted [371:24] demp rundev. So we have all our types. [371:27] We have um our agents. We have our [371:29] branch resolver. Um a bunch of cool [371:32] stuff. Let's go take a look here. Again, [371:35] this is not how I want to fully code it, [371:36] but it did do in Typescript. Um I do [371:38] have another repo where I actually do [371:39] this in Typescript. And Typescript's a [371:41] great language when you're building this [371:42] kind of thing because um you can make [371:45] identity um object identity system or [371:48] whatever it's called. I'm not saying [371:49] that's what it's using, but so here I'm [371:52] just looking at it and there's [371:54] something. Yeah, let's take a look at [371:57] the orchestrator. [371:59] And so what we're looking for in here [372:02] and again your code's going to be [372:03] different, but what I'm looking for in [372:04] here is the actual prompt. [372:07] So I'm going through here and I'm trying [372:08] to find [372:10] the prompt. [372:12] I'm not seeing a prompt here. So, if [372:15] it's not there, where is it? It's under [372:16] agents. [372:19] Yeah, the orchestrator here, which is [372:21] fine. Like, I do like Well, I did tell [372:23] it to write it as agent agent folders. [372:26] And I don't think this is actually run [372:28] following the agent SDK because the [372:30] agent SDK would usually have um front [372:33] matter on the top of it. And so I think [372:35] literally this agents is really now just [372:37] a um [372:39] uh a prompt folder, but that's fine. [372:42] Whatever. I mean like that's not what [372:43] this is not what I wanted. But um and [372:45] I'll just point it out here. Like if we [372:46] go agent SDK, [372:48] it's it's a very specific pattern that [372:50] is used. Um agent SDK uh agent [372:55] uh front matter. [372:59] I'm trying to find it here. [373:02] I guess it's called like um [373:05] agent skills. [373:09] Is it this spec? [373:11] Well, anyway, the point is is that this [373:13] is not what I was expecting, but it it [373:15] still probably will work. So, the file [373:17] documents the orchestrator agent in our [373:20] dad, our dynamic adaptive decomposition. [373:22] Um it is central coordinator to the [373:25] world receives a player action [373:26] determines the sub agent uh passes each [373:28] agent output how it's going to uh the [373:31] dad decision tree. So they're going to [373:33] move to a new room. It'll generate a [373:35] blueprint. It will determine NPCs, the [373:38] puzzles, code writer, lore agent, lore [373:40] consistency, world state persisted game [373:42] response. [373:44] Um we have stopping conditions. So like [373:46] room generation, quest expansion, lore [373:49] consistency, overall turn. that we have [373:52] here. Does that mean it's going to do [373:53] more than just rooms [373:58] needs to be invoked? [374:01] So here, move to unknown room. [374:05] Okay, so it's defining it right there. [374:07] It's saying move to unknown room, but we [374:09] don't have any guarantees that it'll do [374:10] that. Remember earlier we talked about [374:11] programmatic gates, programmatic gates [374:14] to enforce things. I doubt that this has [374:16] it in here. And so we're not going to do [374:18] that, but that would be something that [374:19] we would want to do to really give a [374:21] guarantee that it's going to do what it [374:23] says like guarantee here. That's not a [374:24] guarantee. We need to give have [374:26] programmatic guarantees. My room is [374:28] return to the player with the completed [374:29] status, NBC, etc. Uh we have down here [374:34] agent token budgets. So I guess here [374:37] it's saying like you can't go over those [374:38] budgets. And then we have the key files [374:40] it needs to know about. And then we have [374:43] all this individualized stuff. So, we [374:45] got a lot going on here. Let's just run [374:47] it and see if it works. Okay. [374:50] So, how are we going to run it? Uh, [374:53] npm rundev. Wait, hold on, hold on, hold [374:56] on, hold on, hold on. How does it know [374:58] what it's using here? Like, where is the [375:02] actual um [375:04] uh what do you call it? Um, [375:08] like haiku. How do we know that it's [375:10] using haiku? [375:12] Where is that here? Before [375:16] we run that, haiku. [375:19] Haiku. [375:23] How do we know? Where do we know? You [375:25] know, agent runner. That's probably [375:27] where it is. Ah, here it is. Okay. So, [375:28] here it is. Haiku. And so, they have [375:30] each one that's going to dynamically [375:32] load, I guess, um, or load in those [375:35] files. Yeah, it's just reading those [375:36] files. Read the agents markdown file as [375:38] a system prompt. And so I guess that [375:40] gives you kind of some flexibility to uh [375:42] uh change those out and load them in. So [375:44] we'll go down below and that's all I [375:47] really cared about was whether it was [375:48] doing that. Um the other thing is like [375:51] is it using [375:53] our environment token? Because we have [375:56] this here where it's using anthropic. [375:59] Um [376:01] but I don't see it loading. If we go to [376:03] package.json, this is where we might be [376:04] able to detect if it's doing that. It's [376:07] not using token. So um are we using the [376:11] ENV file? Um so we can uh use the [376:16] anthropic API key for our [376:22] uh [376:24] dynamic [376:28] for our project. [376:30] I don't see it in places like [376:35] main or agent runner [376:40] and I don't see uh EMV in the uh package [376:45] JSON. So I think here it probably [376:47] consume my subscription which I really [376:48] don't want it to do. And uh we're going [376:52] to go ahead and have that fixed. I'm [376:53] probably going to go over my budget now [376:54] just for this one change. [376:59] I mean it's it's one level up. [377:03] Look, the EMV is [377:07] but the EMV is one level up. [377:15] I put what level up? I just said one [377:17] level up. I wrote what level up. So it's [377:19] going to go There we go. And so that's [377:20] what it needed to know. [377:22] And so this will allow it uh to know [377:29] So yeah, now it's going to make sure [377:30] it's installed. So it definitely was [377:31] missing. And so these are things we have [377:33] to kind of catch, especially when we're [377:34] generating large pieces of code. I would [377:36] I would probably not want to ever [377:38] implement something this way. I would [377:40] really walk through stuff here while [377:41] doing stuff. Um and so now it's adding [377:44] that V. There we go. Okay. So now it's [377:46] adding it in. And we will say yes. [377:51] It's just doing um a a TC uh TypeScript [377:54] compile. So, and so now we are in good [377:57] shape. So, what we'll do, I'm going stop [377:59] this uh tool, the cloud monitor, which [378:01] is completely useless for me. Dynamic, [378:04] we'll do dynamic. Um, we'll type in [378:06] clear. And I can't spell or I can't [378:09] autocomplete this easily. So, we'll just [378:10] copy this like this. And we'll paste it [378:12] in. Hit enter and type clear. And then [378:15] from here, we will go ahead and do mpm [378:18] install if we haven't already done that. [378:20] Then, npm rundev. Okay. And so I'm [378:23] assuming there'll be a game loop or an [378:25] interface or something. So here we go. [378:27] So a world that builds itself as you [378:29] explore. Um enter your name. We'll say [378:34] uh Dano. [378:36] Uh that's sometimes the name I use for a [378:38] game handle. Sometimes Omen King [378:41] probably know that. So we have this. We [378:42] have the dungeon entrance. You stated at [378:44] the threshold of an ancient dungeon, [378:46] etc., etc. So what we're trying to [378:48] observe here is it going to start [378:50] building out our world. But right now, [378:51] we're going here and we're not seeing [378:52] any items and we have uh room [378:55] information. [378:58] Right. [379:00] And nothing. So, let's go ahead. You [379:04] stand at the threshold of ancient [379:05] dungeon to torch light flickers against [379:07] uh moss covered stone walls casting uh [379:10] dancing shadows. This the small air [379:12] whatever the paths are broken. A dark [379:14] corers uh corridor stretching north and [379:17] a heavy door to the east. Um, so do we [379:20] want to go north to a dark corridor that [379:22] leads deeper into the dungeon or a heavy [379:24] wooden door slightly a jar? Uh, let's go [379:27] east. [379:29] Okay, you head east and the orchestrator [379:31] is now generating the room. The only [379:33] thing is like how does it interpret [379:34] these inputs? Again, we want this to be [379:37] really static, but I have a feeling that [379:39] every single input it's going to [379:41] evaluate um there. And so that would be [379:44] something where I might not want in the [379:45] loop. And so it says you head east [379:46] generating room dungeon whatever [379:48] whatever. So I'm going to go here and I [379:50] want to see do we have any rooms [379:51] generated. [379:54] I don't see nothing updating here. But [379:57] if we go here it should [380:00] add the room. Okay. So the watch or a [380:04] cloak figure uh their face obscured by [380:06] shadow etc etc. Um and so the thing [380:09] that's missing here [380:12] like it's generally the person [380:15] lore entries. [380:18] Uhhuh. [380:23] It didn't generate the person. [380:25] Okay. So, already this is failing. This [380:28] is not working good. If we go back up to [380:32] um our orchestrator in here, and this is [380:35] probably the reason why we shouldn't [380:36] have generated out on mass. [380:39] So, here player action builder room [380:41] agent. And so the idea is that that [380:42] should trigger the builder room. You're [380:44] a room builder in a textbased dungeon [380:46] game. Your sole job is to generate a [380:48] structure blueprint JSON object. You [380:50] must respond with valid JSON. And it [380:53] does that. [380:56] Okay, which is fine. Generation rules, [380:58] use zones and depths for exits. [381:01] And then here it's specifying the depths [381:03] of stuff. And here's our example output. [381:05] And so right off the bat, this isn't [381:07] working as expected. Um, so I would [381:10] probably say that this is a bust. We [381:13] could spend more time trying to fix this [381:14] up, fix this up, but I'm already um out [381:16] of uh cloud uh for right now. And the [381:19] key thing was just to really understand [381:21] the concept of like uh dynamic adaptive [381:24] decomposition. The idea is that each [381:25] time we are telling it to choose what [381:28] agents to use um and determine right [381:32] determine [381:34] uh what to do next, right? And so in [381:36] theory we conceptually understand it and [381:39] we've done a lot of uh projects here. So [381:40] I don't want to uh hit us over the head [381:43] here with a complex project. Um but yeah [381:45] we can see that there are issues and I [381:48] again I think going forward I'm going to [381:50] be more diligent about not generating [381:52] from scratch like this and pulling from [381:54] an existing codebase that we know um [381:57] already has good good observability and [381:59] things in it. Okay. Chowo chia. [382:06] Let us take a look at an adaptive [382:07] investigation plan. So this treats each [382:09] finding as input to the next decision. [382:11] Not just data to collect, but a signal [382:13] that reshapes what needs to be done [382:14] next. So here's an example of a fixed [382:16] plan where you say, "Hey, we need to do [382:18] these three things. The only change [382:21] challenge here is that findings don't [382:22] change uh after the plan's executed." So [382:25] having an adaptive plan, the idea here [382:27] is that you are giving it room uh to [382:30] make change. So I'm getting my pen tool [382:31] out here so we can read it. you're an [382:33] investigative researcher coordinator. Uh [382:36] after each finding, what does this uh [382:38] raise that I don't know yet? What could [382:40] uh would have would change by conclusion [382:43] if we turned out differently? What claim [382:45] here needs verification? So, generate [382:47] targeted subtasks from those questions. [382:49] So, the idea is that you're asking and [382:52] it's generating subtasks based on that. [382:55] And then the idea here is that now we [382:58] can go perform research on those [383:00] subtasks. Okay. [383:06] Let us take a look at raw findings [383:08] dilemma. This is when you pass raw [383:10] findings between agents as a single blob [383:12] of text and attributions get lost. So [383:16] here um the synthesis agent can't tell [383:20] which claim came from which source and [383:22] by the the time the final output is [383:24] written uh the provenence is gone. So, [383:27] provenence is a really fun word of like [383:30] let's say you buy um a very old antique [383:34] and you want to make sure that it comes [383:35] from a particular place, then we use the [383:37] word provenence. Uh it's a fun word to [383:39] use. I don't get to use it very often, [383:40] so it's thrown in there. Okay. Um but [383:43] anyway, here you see it says EV adoption [383:45] grew 40% in 2024. Battery costs have [383:48] fallen. The EA report on page 14 says [383:51] charging infrastructure is the main [383:53] barrier. uh routters also reported that [383:56] the BYD overtook Tesla in Q3. And so you [384:00] should be asking things like which [384:01] source, what page number, uh what [384:04] document? It doesn't have attribution [384:06] anymore. It's just gone. Okay. So what [384:10] should we be doing? We want structured [384:12] formats to keep content and metadata [384:14] traveling together. So here we have our [384:16] findings information, where it came [384:18] from, the source, and stuff like that. [384:20] Okay. [384:26] Hey, this is Andrew and in this video uh [384:28] or this follow along, what we're going [384:30] to do is um tackle the raw findings [384:33] dilemma. The idea is that we get raw [384:35] finding information, but we aren't [384:37] keeping the findings of that [384:39] information, right? Um and so what we'll [384:41] do is we will make a new folder. I hope [384:45] you're having fun making folders called [384:47] findings. And uh we'll go ahead and make [384:51] it a main.py. Pi and I keep doing the [384:53] main.py even though Typescript has more [384:55] features because at some point we're [384:57] going to run into something where it, [384:58] you know, doesn't have those features. [385:00] Um, and so I'm thinking of one we [385:03] recently did, [385:05] but remember not the forkbased one [385:07] because that that one's in Well, we [385:09] could grab that one, but that one got [385:10] converted back into original Anthropic [385:13] SDK. And so I'm trying to think about [385:15] what we were using before that. I think [385:17] it was task tool. Yeah, it was task [385:19] tool. And so I'll go here. I'm going to [385:21] copy that co code [laughter] and we'll [385:24] go into findings and um I'm just going [385:27] to go ahead and screenshot, right? [385:30] Sorry. Sorry. Sorry. [385:37] Okay. So, I just use I like I don't like [385:39] wasting my cloud credit, so I just go [385:41] over to chat GPT and I I generate out [385:43] that example. Um and so I'm just copying [385:46] this example here that we have uh so [385:48] that I can give it an example of what I [385:50] want. So let's go ahead and type in [385:51] Claude. And we have a new one called [385:54] findings. So in here, you know, I want [385:57] to uh have my research generate out uh [386:02] findings [386:05] uh here so that uh we can remember the [386:10] source of information. [386:14] Here is an example of [386:19] uh of findings. You need to adjust this [386:24] based on the use case. [386:27] Okay. And so our use case our use case [386:31] will be trying to find uh 1980s [386:38] late 1970s films [386:42] uh that are similar to there was a movie [386:45] I watched recently that I thought was [386:46] really fun. Um it was like 1981 [386:51] School in the Crosshairs. [386:57] Okay. [386:58] like and so similar to 1981 Schools in [387:00] the Crosshairs. Uh, [387:04] okay. [387:06] Japanese film. [387:08] Okay. And so I'll go ahead and do that. [387:09] If you've never heard of it before, I it [387:11] was on Tuby for free and so I really [387:13] enjoyed it. But like it's um uh I guess [387:16] they have these films that are I guess [387:17] like in Japan they have idols and so [387:20] these are I don't know if they're [387:21] singers or whatever. I I don't get it. [387:23] But the the point is that the in the 80s [387:26] it just reminded me of a Canadian film [387:28] and so that's why I really enjoyed it. [387:29] But like it seems like a normal like uh [387:33] uh like young adult or or you know like [387:36] teen teen movie and it just starts [387:40] getting weirder and weirder. Um like [387:42] first the first half is normal and then [387:43] the second half just like gets really [387:45] trippy and I just thought that was [387:47] really really fun. It's mostly [387:49] wholesome. 99 99 99% wholesome and 1% uh [387:53] you know Japanese strange but um I liked [387:56] it and I thought it'd be fun to try to [387:57] find more uh films that are like that. [388:00] Um but yeah, that's just generally out [388:03] there. But yeah, I'm just trying to [388:04] convince you to [laughter] [388:06] watch another another thing you've never [388:08] seen before. But um but yeah, it it uh [388:11] yeah, this girl, she goes to school with [388:13] this this guy and he is in kendo so that [388:16] they do like wooden uh stick practice or [388:18] like wooden fighting with sticks, but [388:20] she ends up having telepathic powers. [388:22] And for some reason, I think the [388:23] Japanese really like to give people uh [388:26] telepathy in the early 80s for some [388:28] reason. I don't know why, but we'll go [388:30] ahead. We'll just say yes here, and [388:31] we'll take a look and see what we have, [388:33] and then we'll review it as it's almost [388:34] done. [388:36] But it's it's a recommendation, right? [388:38] uh if you're interested uh in a film and [388:41] it's 99% wholesome and 1% Japanese weird [388:44] if you know what I mean. Okay, so [388:48] um so we have the uh results returned [388:53] here. I'm just going to close this [388:54] because I never really edited the file. [388:56] I just want to make sure I have the [388:57] latest and we'll open this up and we'll [388:59] see what we have. So research and [389:01] summarize key facts. You are a research [389:02] instant with a topic. List three bullet [389:05] point facts about it to be brief. Turn [389:08] research notes into short paragraphs. [389:11] Uh [389:13] oh, this is task tool. That's why it [389:15] makes no sense. We'll go over to here. [389:17] So, we'll go over to here and we'll take [389:19] a look. You are a specialist film [389:20] researcher with deep knowledge on the [389:22] 1970s uh 1980 cinema. [389:26] And I mean, I would like it to be like N [389:27] television. N television. [389:32] And uh we'll go here. And so, we have [389:34] like Yeah, I like how put psychic powers [389:36] right on the top there. That's great. [389:38] So, you know, high school setting, cult [389:41] mind control antagonist. [389:43] I mean, it doesn't have to be exactly [389:44] that. It's more like a like a a thematic [389:46] thing, but that's fine. You are a film [389:47] researcher. And so, it's really [389:49] anchoring to that because I gave it that [389:51] example. And so, here we have our tools. [389:53] Why is it this old school way? [389:56] H [389:58] why did it why is it using the uh [390:02] it this way? [390:04] Okay. Okay, here here like so I'm going [390:06] to just ask it here like why did you [390:08] choose to use uh anthropic SDK [390:13] instead of agent SDK [390:17] right [390:21] like there has to be a reason [390:28] and I mean I guess because it's like and [390:30] here maybe this is the distinction of [390:31] like what's a one-off task and what's [390:33] something that you are continuously [390:34] talking to as I defaulted to the [390:37] majority of examples in this repo use it [390:40] directly. So, but that's not a good [390:43] reason, right? So, you you drive the [390:46] loop manually, you stop it, you explain [390:48] it. That's 50 lines of boilerplate agent [390:50] SDK. Okay. But but conceptually, [390:54] but what makes more sense to do, [390:59] right? [391:01] Not what have we been doing. [391:07] Reason, choices, and then I'll decide. [391:13] Okay. [391:16] Like what are you thinking there, [391:17] Claude? [391:19] Okay. So, we say is the repo is actively [391:21] moving that direction. Yeah. Yeah. Yeah. [391:22] The tool that's not the point. The tool [391:24] boiler plate is exactly what uh the [391:26] agent SDK eliminates. [391:28] Um, [391:31] I mean I I was thinking I was thinking [391:33] more conceptual [391:35] like maybe a may like maybe agent SDK [391:40] is for agentic [391:44] agentic uh [391:47] I mean like technically it's agentic [391:48] anyway. So yeah no I would you know make [391:50] it make it agent SDK. [391:58] Okay. So, yeah, we got to pay attention. [392:01] Just don't vibe code it. Review your [392:03] code. Make sure there's nothing [392:04] dangerous in there. Uh, we'll be back [392:06] here just a moment and it'll make it the [392:08] way I'm expecting it to make it. Okay. [392:11] Okay. So, it seems like it may have [392:14] changed the code back or like not back, [392:16] but it's now using agent SDK. Let's go [392:18] take a look here. And again, we're just [392:21] reviewing it to see if there's any [392:22] gotchas. [392:24] Um, and yeah, it's going to use the API [392:27] key, which is fine. We're still using [392:29] Haiku. So, again, we're not using the [392:30] most intelligent model, but it serves [392:32] our purpose for this. [392:35] And [392:36] it changed it from film, so we don't [392:39] have television there anymore. That's [392:40] fine. We could just stick with film. [392:42] Return only valid JSON array. No mark up [392:44] markdown fences, no commentary. Um, I'm [392:47] pretty certain that it can produce [392:50] um, like if we use pyantic and pass it [392:53] in, it probably would [392:55] be better. We have the research agent [392:57] over here still. Okay, good. Evaluation [393:00] coverage. [393:02] Submit complete [393:06] coordinator prompt. [393:09] Um, so we're going through here. [393:13] So the only thing that I would say [393:14] that's a little bit off here is the fact [393:16] that uh I would think that [393:20] you know if we're val so you know we are [393:22] asking for [393:25] structure JSON output back for findings. [393:32] Can we use pyantic [393:35] or does uh or is there a field [393:38] specifically [393:39] for enforcing [393:43] structure JSON output. [393:45] Are we logging [393:48] are we saving the findings [393:50] to a file so we can uh so we can see [393:55] what they are [393:58] at some given point. [394:02] Okay. And so those are the two things [394:03] that I'm I'm wondering about because um [394:06] normally when you're building this [394:07] stuff, it has a hard time enforcing that [394:10] pattern and the only that's enforcing it [394:12] is uh within here, right? So [394:17] and normally we don't usually have to [394:18] tell it like oh you need to do this [394:20] structure. We usually just um pass it as [394:23] a parameter. It's still using the [394:25] anthropic key here which is kind of [394:27] annoying. So we see this here like why [394:29] do we why are we still using anthropic? [394:33] So did it understand my request [394:37] because we have call research agent and [394:40] shouldn't shouldn't agent be an agent? [394:44] So we have tool but it's not defining [394:45] that we've already learned uh about that [394:47] right. So that's going to be another [394:48] issue that we have here. So actually I [394:50] have to hit tab here [394:53] u I just get uh tab tab shift tab tab. [394:57] Can I write [394:59] there's two answers here? [395:04] Okay, but hold on hold on hold on hold [395:07] on hold on stop. [395:10] Okay, you know I I do want uh structure [395:14] JSON output [395:16] uh validated [395:18] if there is a field but I noticed uh you [395:22] are still using [395:25] using the anthropic SDK. Shouldn't we [395:30] be using the agent uh agent SDK [395:34] uh and uh agent definition [395:40] for the [395:44] uh where is it here? [395:53] call research agent [395:55] and I think might because this code base [395:57] is getting so large it might be getting [395:58] confused. [396:00] Okay. And so we'll help that along there [396:02] because we know that it exists, right? [396:04] So why are we getting the thing that we [396:06] don't actually want, right? [396:09] And we'll give it a moment here to [396:11] figure that out. Okay. And I did notice [396:13] that it was doing that note put down [396:15] validation on the sub agent. Uh so we'll [396:19] do that and we'll see if it takes it [396:21] here. Okay. [396:23] Okay, so it looks like it has updated [396:26] our stuff here. Find item, find list. [396:28] Um, switch over to SDK. Let's make sure [396:31] that's correct. Okay, so we'll go all [396:33] the way to the top. And again, I'm going [396:34] to give it a nice scan here with my eyes [396:36] and determine it. So we have Pyantic [396:39] models. Good. So that it's forcing that [396:41] structure. Um, it has the research [396:44] state. So, it's holding I'm not sure [396:46] what data class decorator is for, but um [396:50] I'm assuming it's for creating data [396:51] structures in Python. Um and so we have [396:54] our findings and our counter and what's [396:56] next? We have our information there. [396:59] Where is our agent definition? [397:02] [laughter] [397:03] Is it like is it messing with me? [397:06] Does it not know what agent definition [397:08] is? Because, you know, I want it to be [397:11] an agent. Let's go back up to here. [397:14] Okay. Okay. I still don't see you using [397:21] agent definition. Like I I don't [397:24] understand what it doesn't understand. [397:26] Like I don't know why I won't do it. So [397:27] we'll go over to here. Like [397:30] we'll go over to here like this. [397:34] Shouldn't you be using this [397:38] for the research the call research [397:41] agent? [397:46] Okay, like literally we have to pull the [397:48] code code forward and it's becoming a [397:50] big old dum dum here [397:54] and let's see if it can figure that out. [398:01] Okay, so we'll give it a moment to [398:04] figure it out. [398:08] Okay, let's take a look here. So the [398:10] architecture now matches the Asian [398:11] definition and we still have the MCP [398:13] tools as that is what it's going to do [398:15] which is totally fine. And so now we [398:17] literally have our Asian definition [398:18] researches topics and summarizes facts. [398:20] Your research assistant um lists three [398:23] concise bullet points about the fact [398:26] your copyriter. [398:29] Uh well no [398:35] no no no. [398:39] What did you do? You You destroyed the [398:41] entire [398:43] use case of the 19 like the 1980s [398:48] '8s film. Why? Why did you do that? [398:54] Okay. Like it literally just destroyed [398:56] it. [399:03] So now we've lost that code. I'm going [399:05] to see if I can roll it back there, but [399:08] we'll try. And so, you know, again, this [399:10] stuff is imperfect and every time they [399:12] say it's better. Yeah, you're right. The [399:13] I shorten the research and prompted [399:14] moving into the ancient definition, [399:15] losing uh all the detailed film [399:17] features. Let me restore that. Uh-huh. [399:20] Yeah. [399:22] Okay. Well, d like [399:29] and also where's the findings [399:31] information? [399:34] It also got rid of my findings. The [399:36] whole point of it. Oh my goodness. Let's [399:39] go back in here. [399:46] Where? Where is it? I don't see it in [399:47] here. Is it actually in here? I'm in [399:50] agent definition. Let me go over here. [399:51] Findings. Here we go. Oh, maybe I just [399:54] had the wrong file open this entire [399:55] time. I don't know. But here we go. So, [399:58] we have our agent definition. Maybe [400:00] that's I was getting so upset because I [400:02] was looking at the wrong one. Okay. So, [400:03] I'm a I'm just a dumb dumb. Fair enough. [400:06] So, um we have our agent def agent [400:09] definition. We have our finding [400:11] information. What I'm interested in is [400:13] like where is this going here? So, we [400:15] have this findings [400:18] um and that is stored in our research [400:20] state. [400:23] Okay. [400:29] And it seems to be validating it [400:31] directly. So, you know, I could have [400:32] swore that um the agent SDK had a direct [400:36] way [400:38] to validate. [400:40] But anyway, we've wasted enough time. [400:43] Let's go ahead into our findings and [400:46] we're going to go run this and let's see [400:47] if I can get any recommendations for [400:49] other shows that I can watch. [400:51] One big challenge is that, you know, [400:52] when I want to watch these things, I [400:54] want them in the uh with good subtitles [400:56] and then the movies might only be like [400:57] 10 bucks in Japan, but when you ship [400:59] them to Canada, it becomes like a $100 [401:01] $100 thing. And I I don't have a [401:03] strategy for bringing them over uh to [401:05] Canada where if I want to watch them um [401:07] unfortunately. But so here it says, I'll [401:10] help research 1970s '80s films similar [401:12] to schools in the crosshairs. Let me [401:14] systematically cover all similar axises [401:16] through focus research. And so I'll [401:18] launch agent research. I'll launch a [401:20] research agent for each major similarity [401:23] axis. [401:24] Okay. Now, if I did a boot camp, I would [401:27] probably try to do this for real where [401:29] this we're just having fun, right? But I [401:31] would try to make something that would [401:32] actually be useful if that was my goal. [401:35] Um, [401:37] but yeah, I actually don't know how long [401:39] this is going to take. But thankfully, [401:41] I'm using the Haiku stuff, so it's not [401:43] giving me full visibility on what it's [401:45] doing though right now. [401:48] So phase one, [401:50] use the agent call with a focus [401:53] question. Immediately call the [401:54] recordings. Notice I have like no [401:57] visibility on how many runs it's doing [401:58] or anything. [402:00] Like how do I know when it quits? I [402:02] don't know. [402:03] So this might be something that we might [402:05] want to tune tune in. I know like with [402:07] agents you could say like max turn [402:08] whatever whatever. So maybe that's [402:10] something that we should have been doing [402:13] in our our agent settings, but it can't [402:16] do it in Python. So we'd have to convert [402:17] it over to TypeScript, right? So [402:20] remember that agent definitions had like [402:21] a max turn option. [402:26] And so yeah, am I just losing all my [402:28] credits? Let me go over to the cloud [402:30] platform and take a look here. Cloud [402:33] platform. I mean, maybe I'm already out. [402:35] Maybe that's the reason why it's it's [402:36] hanging here because it is using my uh [402:38] tokens there. And so I'm going to go [402:40] over to my usage. [402:46] I'm just opening it off screen here. [402:49] Uh [402:51] 301,000 tokens. [402:54] So it says 301,000 tokens here. [402:58] And I'm going to go look at my limits [403:00] maybe [403:02] or uh API keys [403:06] costs. [403:10] Can't really tell here. I haven't used a [403:13] whole lot in the last few days. I'm [403:15] actually surprised how cost effective [403:16] this has been [403:19] on it. Very expensive compared to Haiku. [403:20] Haiku is like costing me next to [403:22] nothing. And uh yeah, it's been really [403:23] good. It's actually been more cost [403:25] effective than if we were to use our [403:27] subscription. Um, [403:30] but anyway, we'll go here. And so we [403:31] have a bunch of findings. That's cool. [403:34] Is it going to save it? I guess it [403:36] doesn't save it as it goes. [403:39] Uhhuh. [403:41] Is it going to save the findings? [403:44] I wouldn't mind keep the findings here. [403:47] So, I'm just looking to see if it [403:48] actually saves it anywhere. We did ask [403:49] it to do that earlier. [403:52] pen print. [403:57] Uhhuh. [404:00] Save findings right here. [404:03] So it prints it and then it saves it [404:08] into results. Jason. Oh, okay. So this [404:11] thing finished running. I have no idea [404:13] why it's doing this. It's really weird. [404:16] Limited Eastern Europe, Soviet cinema, [404:18] beyond whatever. Whatever here. So let [404:21] me see assess the research coverage [404:22] across all access. So here we have [404:25] Akira, Virus, Brother and Sister, [404:28] Carrie, the Fury. There's obviously some [404:30] things that are not Japanese related [404:32] like Invision of the Body Snatchers. Um, [404:35] excellent coverage is sufficient. The [404:37] research comprehensively covers all [404:38] similarities, axes with strong [404:40] representation across international [404:41] cinema. I don't want international [404:43] cinema. I just want Japanese. [404:46] And so here we have [404:49] uh stuff [404:55] sits within a broader traditional [404:58] including kiji [405:01] whatever whatever so I think this is [405:03] another obashi fail maybe [405:09] like this one here take a look at this [405:14] just going to open that up I'm just [405:15] curious what it is [405:17] here we got a movie called 1980 virus [405:24] interesting [405:29] post. I mean, not really totally even [405:31] close to being the same. We'll go down [405:33] here. Um, and yeah, it's just describing [405:36] particular elements, right? So, let's go [405:38] down here. So, I successfully completed [405:39] a comprehensive research. [405:41] Um, [405:43] so phase one research. So, Japanese [405:46] supernatural uh psychic school films, [405:49] international team psychic tele [405:51] telekinetic films, mind control, coming [405:54] of age, visually inventive director uh [405:56] directors, coverage evaluation, so seven [405:59] out of 10, geographic distribution, [406:02] thematic saturation. [406:05] Yeah, like I really like the mind [406:06] control stuff. Comprehensive analysis [406:08] submitted identifying 30 comparable [406:10] films. Key findings. So, Japanese [406:12] supernatural, [406:14] um, international psychic youth, [406:18] fire starter, phenomenon, mind control [406:21] cult, coming of age, hybrids, visionary [406:24] directors. [406:26] Um, and so all the data has been [406:29] recorded and synthesized. [406:33] And so I guess this is what saying is [406:35] it's like the best match it can find. [406:38] Combine science fiction with an ensemble [406:40] cast with psychological phenomenal uh uh [406:43] confined institutional settings echoing [406:45] the school environment and mind control [406:47] theaters. Yeah, but it's also not [406:49] lighthearted like I would say well I [406:52] guess the the movie that I was [406:53] suggesting is not fully lighthearted if [406:55] if you look at it but it does feel that [406:56] way. Scanners Fire Starter. [407:02] Oh that that could be interesting. [407:04] Actually I'm surprised I don't know the [407:05] Stephen King one. I I really like [407:08] Stephen King. I've seen Sus Spiria. So [407:10] it does its best effort here to find [407:12] stuff. [407:14] Um and then medium confidence. So [407:16] basically these are the ones that are [407:18] the closest matches. Let's go over to [407:20] look at the actual finding data and see [407:22] if it's actually useful. So we'll go [407:23] over to here. And so we have its [407:25] contents. We have a confidence high. We [407:27] have a claim. We have the type, the [407:29] name, the author, publish it, access it. [407:31] The only thing we don't have is like [407:33] where did it get that data? Okay. So [407:36] that would be where we would just say, [407:38] "Hey, look, that's great that you have [407:39] this information, but um where did it [407:42] come from?" [laughter] [407:44] So anyway, it's cool. We did it, but we [407:47] had to fiddle a bunch. Um but yeah, [407:49] we'll see you in the next one. Okay, [407:50] chowo chia. [407:56] Let us take a look here at forkbased [407:58] session management. And so fork-based [408:00] session management is the pattern where [408:01] you take a single conversation at some [408:04] point and then you're going to branch it [408:05] off into their own sessions uh uh and so [408:09] the idea is that they can explore those [408:11] things independently at the same time. [408:13] Uh this feature is available in the [408:14] agent SDK. You can also I guess [408:17] implement it manually but you'd want to [408:19] use agent SDK because this is the reason [408:21] why they're probably talking about it in [408:24] um this uh this course. And so the idea [408:28] here is that we will have a separate [408:29] session history uh per branch with its [408:32] own UU ID and we can resume that history [408:34] because the the state is being saved. Uh [408:37] they run in parallel so things are [408:39] returned quicker. They are isolated from [408:41] each other and they're not sharing [408:42] knowledge. Um and so the idea is that we [408:45] have a function called fork session [408:46] that's only available by the way in [408:47] Typescript at this time. It's not [408:49] available in Python. I have no idea why [408:51] the Python one is so far behind. And we [408:54] have session ID. I wanted to give you a [408:56] code a code example. And so I thought, [408:58] okay, well, I'll use Claude. I'll use [409:01] Claude and maybe he can help me get a [409:02] code example. I'm trying to find this [409:03] here. And I literally was arguing with [409:07] with it for like 15 minutes and it just [409:10] refuses to make a code example despite [409:13] me actually already having like look at [409:15] this. [409:17] I'll show you this. Like look, I'm [409:18] saying can you use fork session? It just [409:20] refuses. It keeps using this options [409:22] fork session option no matter how much I [409:25] ask it though we're going to find out [409:27] when we do the um the actual lab [409:30] wherever that is here right that we're [409:33] going to find out that it uh that there [409:34] is a actual function here right so you [409:38] know just understand again I'm going to [409:40] just repeat this a few more times that [409:43] uh what we uh read and what's generated [409:46] out is not what is actually true and I'm [409:48] going to show you the truth that's why [409:49] you have to follow through on these [409:52] follows. Okay. Don't watch do and verify [409:55] yourself so you have that knowledge. All [409:57] right. [410:02] Okay. Let's go ahead and try out [410:04] fork-based session management. So, we'll [410:05] make a new folder here called forkbased [410:08] um session management [410:12] management. And we're going to go ahead [410:13] here and make a new file called main.py. [410:17] And um so I'm just thinking about this [410:19] for a moment. Okay. Um [410:23] and so what I want to do is we're going [410:25] to grab um I suppose the agent code is [410:28] fine. So we'll go ahead and just grab [410:29] this. We'll just keep working off the [410:32] last thing we have. I'm still using myV. [410:34] You can get rid of that if you want to [410:36] use your subscription. It's just um I'm [410:38] trying to manage my pro level the best I [410:41] can. I can upgrade to max, but then I [410:44] feel like I'm not experiencing what [410:45] you're experiencing. And so I want to be [410:47] able to work around those issues and and [410:49] uh share that experience with you. Okay. [410:51] So I'm going to go ahead into the [410:52] fork-based session. Actually, not yet. [410:54] Um and so we're going to go ahead here [410:56] and we'll launch up Claude. Okay. And I [410:59] want to tell it uh what it is I want. So [411:03] um I want to implement forkbased [411:08] uh session management [411:11] where we have one point of history in oh [411:16] resets today. That's good. Well I got to [411:18] use up my credits. I was thinking of a [411:20] project where I'm like hm maybe I should [411:22] tell my computer to wake up and then in [411:25] the off hours uh use up as much as I [411:27] can. But I guess the other thing is they [411:29] have a weekly limit and if I'm using [411:30] pretty much all of it then I suppose [411:32] having it work throughout the night [411:33] doesn't really matter. But anyway, so I [411:35] want to implement fork-based sessions [411:36] where we have one point in history. [411:41] We have one point in history [411:44] um [411:46] uh uh and it branches off in its own [411:53] uh isolated states. Uh can we implement [411:57] this [411:59] in cloud agent SDK? [412:02] Here is example. [412:10] Okay. And so I just took you know as per [412:13] usual I just took that screenshot [412:14] dropped it in here. So just drop it in [412:16] here have it extract out. I know it's my [412:18] own text but I didn't save it in the [412:19] slides. And you'll have to do the same. [412:22] Good practice for you folks. And we'll [412:24] wait a moment here u and come back with [412:26] um some code here. See what it tries to [412:29] do. And then we'll look and say okay [412:30] that's good or that's bad. Right. Okay. [412:33] Looks like it's more or less done. So [412:35] here it says uh one shared API call that [412:38] builds the common history. Um each [412:41] branch gets its own copy. Mhm. Why [412:45] direct API calls in agent SDK? The agent [412:47] SDK query is designed for agentic loops [412:49] with tools. It owns the session state [412:51] internally. So when you need to [412:53] explicitly control and branch messages, [412:56] then we want to use the native one. [412:58] Okay. Well, that's fine. I thought that [413:00] we would be utilizing um agent SDK, but [413:03] I suppose that doesn't fit. So let's [413:05] take a look here and see what we have. [413:09] Okay. Um [413:11] so we have our client, [413:14] we build a baseline, [413:17] we have our branches. So start with this [413:18] exact point share it up to here. What is [413:21] the most optimistic 5-year EV adoption [413:24] scenario? What is the most pessimistic? [413:26] How could upcoming government [413:27] regulations reshape it? Okay. And we're [413:31] going to run each branch. Where is the [413:33] baseline? Oh, build baseline right here. [413:35] And so we go up to here. And so [413:39] analyze the EV market briefly. [413:42] Mhm. [413:44] And then it comes back with a response [413:46] of what it researched. [413:49] And then down below here, [413:51] it's going to go again based on that [413:53] information. [413:55] Okay. Interesting. All right. So, let's [413:58] go ahead and run that [414:02] all the way down here. We'll stop this. [414:04] We will cd into forkbased sessions [414:08] python [414:10] main.py. And we'll go ahead and run [414:12] that. [414:15] We'll see what it comes back with. I [414:16] think I would have preferred to pick a [414:18] topic I'd be more interested in, but [414:19] that's fine. [414:21] We'll see what it does here. Okay. [414:26] And um [414:30] so we have the optimi optimistic. So [414:33] aggressive EV mandates, growth [414:35] projections, [414:37] uh risk assess adjustments. [414:42] Okay. And so that's something that's [414:45] interesting. [414:46] I don't have a use for it, but um I mean [414:49] I get it. So like I guess we're just [414:51] trying to compare this to um what we [414:53] were doing prior where we had a research [414:55] agent and the idea was that we would [414:58] define all of its um angles. But here [415:03] we're basically we have one and it's [415:04] like we are just pushing that down a [415:06] little bit so that we are seating with [415:07] information then to go look other [415:09] directions. So just it looks like a very [415:11] similar pattern but a little bit [415:12] different but there we go. That's [415:13] forkbased [415:15] session management. [415:18] Um [415:20] I'm just trying to think here. Uh [415:23] but I guess the thing is that the only [415:24] thing is that I wonder I'm just trying [415:26] to think if we are done because we had a [415:28] result but is it actually man managing [415:31] sessions separately? That's what I'm [415:32] thinking. So hold on. Okay. But are we [415:36] actually managing uh [415:42] separate sessions? [415:46] Because they've just gone off and did [415:47] stuff, right? Like like how would we [415:49] pick up a conversation from before? [415:56] Oh, sorry, sorry, sorry. Let's stop this [415:58] for a second. So, I'm going to go try [415:59] this again. I'm going to just go in here [416:01] and just say for [416:04] we'll go back to resume. [416:06] Uh yeah, resume. Come on. Let me resume. [416:08] Show me something. Show me history, [416:11] folks. [416:15] It is conking out here, folks. Let's try [416:17] this one more time. [416:20] I'm just trying to think like what is [416:21] the use of this over the other one? And [416:23] so I want to definitively know that we [416:24] haven't missed something. [416:28] Yeah, we don't even have the history [416:29] anymore. That's great. Okay, so for the [416:32] forkbased. [416:38] Why won't it autocomplete? Am I in the [416:41] wrong directory here? Oh, no, I'm not. I [416:43] Well, let's go up one. Sorry. [416:46] I mean, I am. I am. But [416:49] we'll go back to this conversation here. [416:51] Oh, is this actually um if you wanted to [416:54] use the agent SDK session system [416:56] instead, TypeScript has the fork session [416:58] session ID which branches a session at [417:00] its current point, but that isn't in the [417:01] Python one. Oh, okay. Great. Port my [417:06] code. Port my code to Typescript and [417:08] let's use that feature. [417:17] Cool. [417:19] Just can't trust anything here. Why is [417:21] the TypeScript one up to date, not the [417:23] Python one? I'm fine. I can work with [417:25] TypeScript, but there's a little bit [417:27] more setup with it. So, we will let it [417:28] port it over [417:32] and see if we can get fork sessions in [417:34] there. Okay. All right. So, we are back [417:36] and we should now, if we refresh this, [417:38] have our main TS. I like that it kept [417:40] the main PI around. So, that's really [417:42] nice. Um, and we do want this to be [417:44] using the SDK, which it is. Okay. Okay, [417:48] so hopefully you're comfortable moving [417:49] between languages. I am. I have no [417:51] problems with that. Um, and so here we [417:54] are analyzing the market. We are passing [417:56] our model. Which model is being used? [417:58] Haiku. Good. Because I want to save [418:00] money. And then we'll go down below and [418:02] we're seeing the forkbased session [418:03] information. So we're seeing run branch [418:06] here. And in here we have fork session [418:08] and that is being imported uh by the [418:11] cloud agent SDK. So that makes a lot [418:14] more sense, right? Because if it was [418:16] just simply what we were doing in [418:17] Python, then it's just like okay, how's [418:19] that any different in a different or uh [418:22] way of organizing stuff than how we were [418:24] uh creating those different angles. But [418:27] here the idea is that it's going to [418:29] literally have a session state. So [418:32] basically you are keeping a conversation [418:33] along those three things there. Okay. So [418:37] what we'll do is we'll stop this and uh [418:40] you have to have JavaScript installed on [418:42] here. So node hyphen version, right? I'm [418:44] not teaching that. You folks need to [418:45] learn that. Okay. Um I got other courses [418:47] on that somewhere. [laughter] And so [418:50] here I'm going to do mpm install to make [418:53] sure everything's installed. Probably [418:54] already is. And we'll do mpm run. Uh is [418:57] just mpm run start. Usually just mpm [419:00] start. Let's try that. Mpm start. Is [419:02] that what it is? I didn't check it, but [419:04] that's usually what it is. It is. And so [419:07] it's going to go ahead and execute the [419:08] TypeScript. [419:10] And we are going to just watch the [419:12] results. [419:14] Okay, [419:16] I'm now getting in the habit of always [419:17] doing the labs first. Um, and sometimes [419:21] doing them two or three times before I'm [419:22] doing the lecture content, which is [419:23] better. Um, but anyway, so here it's [419:26] forking into three isolated branches [419:28] concurrently. And so the idea is that [419:29] it's having a conversation with each of [419:31] them and then it comes back with our [419:32] results. [419:35] Okay. [419:37] And so I would imagine one of the [419:39] advantages would probably be uh that [419:41] they're being paralyzed at the same [419:42] time. And I will look that up and put it [419:44] in the lecture slides and we will find [419:46] out if that's the case. But the idea is [419:48] that it's holding three separate [419:51] sessions. Also just going to go here and [419:52] add a get ignore. These node modules are [419:55] always really giant. Uh get ignore or [419:58] sorry node modules. [420:03] Okay. Want to ignore that big big folder [420:05] there. And so again, just going to [420:06] carefully look at this here [420:10] fork session. [420:14] And so the idea is that they're [420:15] basically running it's I know it's [420:16] really hard to see here in the code, but [420:19] they're basically running uh each their [420:21] own loop, right? So build the baseline, [420:24] build the branches, [420:27] iterate over the three branches here, [420:30] right? Iterate over them, run the branch [420:32] for each one. Okay? Okay. And then we go [420:34] up here, we run the branch, we have fork [420:37] session. All right. Um, and actually, [420:40] you know what? I think in my cloud code [420:42] essentials, I think we looked at fork [420:44] session, but we might may have not have [420:46] ran it now that I think about it. Um, [420:49] let me just go take a look here in one [420:51] moment. Just a second here. I mean [420:54] according to Gemini it's saying that um [420:58] it's saying here par they're running in [421:00] parallel and we are basically have their [421:03] own session with their own UU ID so that [421:05] we can pick up a conversation later and [421:07] so that basically tracks of what the [421:09] advantage would be here. Um I mean [421:11] paralization obviously is going to give [421:13] us a huge advantage in terms of getting [421:15] results quicker. We're going to burn [421:16] through more credits. Um but yeah so we [421:18] have our solution and there we go. Okay. [421:25] Hey folks, this is Andrew and in this [421:27] video I want to test if we actually have [421:29] resume and uh fork session. So these are [421:33] features that I know cloud code has. Um [421:35] but the question is does agent SDK have [421:37] it? Um I'm getting reports that it might [421:39] do it but I need to absolutely verify [421:41] that. So we're going to go down below [421:42] here make a new folder and this will be [421:44] resume and fork session. Resume fork [421:47] session here. And so we will use that as [421:50] a basis to find that out. We're going to [421:52] go ahead and make a new main file. We [421:53] know we're going to have a much easier [421:55] time, a much easier time if we uh bring [421:58] in some code. Now, I do have a little [422:00] bit of a problem here because I had a [422:02] prior project uh under the under this [422:04] one. And I just got to make sure I [422:06] ignore u the do.getit ignore. This is [422:08] just something I forgot to do. Nothing [422:11] you got to worry about here. [422:16] Get [422:19] nor [422:24] Okay, we'll type in node modules [422:29] and we'll just make sure [422:33] just going [snorts] to add that there. I [422:34] know it's not something I should be [422:35] doing right now, but I got to do it, [422:37] folks. [422:39] And [422:41] there we go. So, now our change is a bit [422:42] smaller here to say fix that. There we [422:44] go. Anyway, so back on task. The idea [422:46] here is that for the uh resume fork [422:49] session, I want a basis of content. So [422:51] let's look for the hello world assuming [422:53] that this one is using the agent SDK, [422:55] which it is. And so we will go ahead and [422:57] copy this in here. And we'll paste this [423:00] in. And so now what we'll do is we'll cd [423:03] into [423:06] we'll cd back here into the resume [423:13] fork session. [423:16] Okay, we'll type in claude. [423:19] And so what we want to do is we want to [423:22] demonstrate using resume and con uh and [423:26] fork session [423:29] via [423:33] the agent SDK assuming that it can do it [423:37] and or the other question is like is it [423:39] available in in in Python? So because I [423:41] don't understand like how would it know [423:42] where to write um that repository. [423:45] So we'll go ahead and say yes [423:48] and we'll see if that works. The only [423:50] thing I don't like is when it goes and [423:51] it checks everything above it. I want it [423:54] to stay in this uh particular [423:57] uh folder. I'm not sure. No. Like why? [424:01] Why? No. No. Please just focus on the [424:06] main PI file. [424:11] Has this problem where it aggressively [424:13] searches other places and I can't seem [424:15] to prevent it doing that. I'll be back [424:16] in just a moment. And so here we have uh [424:19] some code returned. I'm going to go say [424:21] yes. [424:24] And now we have an individual.cloud [424:26] here. [424:27] We'll close out some of these tabs. [424:31] Okay. All right. So, let's take a look [424:33] here. We have session one initiate a [424:35] query. So, it's going to fix uh the [424:37] code. I'll go ahead here and we will [424:39] make ourselves a hello world. [424:43] Um I remember we had a larger code [424:45] sample before. I'm just going to go [424:47] generate out one with some bugs. You [424:48] know, can you generate out code uh Ruby [424:52] code with deliberate bugs? deliberate [424:55] bugs to test uh so I can test [425:00] my bug fixing agent. I know I'm doing [425:03] this offscreen. We will go ahead and [425:04] generate that out. Just give it a [425:06] moment. [425:08] Okay. So, we'll go ahead and paste this [425:10] code in here. I'm going to go ahead and [425:12] save this and we will go and give this a [425:14] run. Well, before we do, let's take a [425:15] look here. We have the first session and [425:18] it's going to read that file, right? And [425:20] then it's going to log out the results. [425:22] It's going to resume the session here [425:24] with the session ID and then we're going [425:27] to have a forked conversation where it [425:30] uses the fork session. So starting a [425:31] completely new session and we'll see if [425:33] that works. So I thought this would only [425:35] be for clawed code but I guess I guess [425:37] it's for everything e. Um so we'll go [425:41] ahead and type in main or python main. [425:49] Okay. And that's going to initiate uh [425:51] that query there. [425:54] Okay. And so it looked like it did [425:56] exactly what it said it would do. Good. [426:00] And so it's now fixing the thing. What [426:02] I'm going to be interested in is where [426:03] is it storing this information? That's [426:05] what I want to know. [426:08] I think it actually just said here, too. [426:10] Resume fork session. Hello world. No, [426:12] no, that's not it. Um this is almost [426:16] done. So we'll just wait till it's done. [426:17] Okay. All right. All right. So, this [426:18] seems to be complete here. And I want to [426:20] go ahead and take a look into a new [426:23] folder. [426:25] And I'm going to go over to here [426:30] and I'm trying to figure out where this [426:32] could be. So, like this is the project [426:34] we were running and so I'm not sure if [426:36] it would um pick up in the same name. [426:38] So, let's go in here and see what kind [426:39] of sessions we have because that might [426:41] also be an indicator of what stuff is. [426:45] Okay. [426:48] And so we have three sessions here. I [426:50] don't think that we would have had three [426:53] sessions before. One thing I'm going to [426:55] do is I'm just going to delete these [426:57] files out and run it again because if [426:58] they generate in here, then we know [427:01] absolutely that that is what is going on [427:04] here. [427:09] Okay. And so I'm going to just do an ls [427:10] ls type in LA. And we still have one [427:12] more here. I think it's because this is [427:14] the active uh there might be an active [427:15] session here. [427:17] Okay. And so I'm just going to stop it, [427:18] clear it, and we'll go over here, right? [427:23] And we'll say remove. And that will be [427:26] removed. [427:34] Okay. [427:35] And so we'll go ahead and run that. [428:05] All right, let's talk about cla code [428:07] sessions. So claude code session is [428:08] functionally a conversation to the [428:10] claude code agent. And resuming a [428:12] session is the same as resuming a [428:13] previous conversation. So when you type [428:15] in claude, you are starting a new [428:17] session. And there are two types of [428:19] sessions. You have web uh and local. And [428:22] it really depends on where you started [428:24] that conversation. So um basically if [428:26] you start it via remote control or via [428:29] cloud code for web then it will be a web [428:32] session. But what I found was [428:34] interesting is that if I was using the [428:35] VS code IDE I would be able to um resume [428:39] a a web conversation to local which kind [428:42] of makes sense because with remote [428:43] control the idea is to trigger it over [428:46] web to then run your machine or in your [428:49] development environment on your machine. [428:50] So uh but where it originated is stored [428:53] is different. Um, but we'll talk about [428:56] resuming in the upcoming slide here and [428:59] show you that. But, uh, there you go. [429:05] So, this is going to be very [429:07] straightforward and it's the resume [429:09] command. So, you can resume a previous [429:10] session by using for/resume and then you [429:13] can just choose it like a separate [429:15] conversation. You probably saw in the [429:16] last slide that in the ID you literally [429:18] just click on conversations which is the [429:20] same thing as sessions and that was a [429:22] way to change it. Um when you do kill a [429:24] a session um uh via the terminal, it [429:27] will give you this line for resume with [429:30] uh a session ID. I have no idea how you [429:32] get the session IDs or list them. It's [429:33] not really a big deal as I feel that [429:35] most people are driving this stuff [429:36] through this way. Maybe if you had to do [429:38] something programmatically, you'd have [429:40] to do it in the lab. We'll see if maybe [429:42] there is a CLI command in the [429:43] non-interactive way to grab uh those [429:46] IDs, but there are session IDs for that. [429:48] Uh but there you go. [429:54] Hey folks, it's Andrew and in here I [429:56] have a project u which is the claw task [429:58] app. Not a whole lot going on in this [430:00] one, so we might be limited in terms of [430:03] uh what kind of sessions we have or I [430:04] could try to switch to one that had [430:06] more. But if I go up to here with the [430:08] SDK um uh the SDK the the the Chrome one [430:13] installed, you can see up Chrome, what [430:14] am I saying? The Visual Studio Code [430:16] extension, you can see we have local and [430:18] we have web. So, here are some previous [430:20] conversations. What's interesting is [430:22] like I did these ones for um very [430:25] specific projects. And so, it's just [430:26] generically showing me all web sessions [430:28] where local seems to be scoped for this [430:31] specific one. So, if we go over to here [430:32] and we click on this one, it seems like [430:34] it's going to teleport us in here. And [430:36] we're having a bit of an issue. To be [430:37] honest, I haven't really continued web [430:39] ones here prior. Um but here, what could [430:44] be the issue? Um [430:46] missing file or encountered [430:47] settings.json. JSON path etc. So I'm not [430:50] exactly sure what the issue is here but [430:52] we are trying to continue something [430:54] definitely in the wrong directory. Um [430:56] here we go. So this session was created [430:58] with exam prodev cloud code example. [431:00] Open that folder. Okay. So here it's [431:02] telling us to go ahead and do that. So [431:04] I'm going to go back here and just kill [431:06] this. So that's what I figured which [431:09] would be an issue if we continued one [431:11] on. You probably wouldn't have it unless [431:12] you did the web version here. So, I'm [431:14] basically just showing you, but we'll go [431:16] here and [431:18] uh I think it was like claude code. [431:26] I already forgot what it was called. [431:28] [laughter] So, we'll go back over here [431:29] for a second. [431:32] We'll try that one more time. [431:36] The session was created in Examprodev [431:39] Cloud Code example. open that folder [431:41] first or continue in that current [431:43] workspace because that is my in [431:45] developer account, right? And so I don't [431:47] think I have it here. I'm going to say [431:49] continue here and see what happens. Oh, [431:51] there we go. Um but that was a um that [431:55] was in the web. So I'm not exactly sure [431:57] how we continue on to that stuff. I'm [431:59] not really that interested in that. I [432:01] just wanted to show you that this [432:02] dropdown does exist. Um and so if I go [432:04] to something where we have a bit more [432:06] going on. I think it was called [432:08] uh grocery store monitoring. I never [432:11] showed you how to build this during the [432:13] app. It was something I was just [432:14] building on the side while I was waiting [432:15] for things to run. But if I open up this [432:18] one, this is going to have a lot more [432:19] sessions in it. So, if I go here, you [432:22] can see here I have Whoops. [432:27] A bunch of sessions. [432:29] And I thought there was more than that. [432:31] I could have swore there was more. And [432:33] then you could just click it and then it [432:34] would continue on. But generally I I [432:36] like using the CLI down here. So we type [432:39] this and we type in resume. We hit [432:42] enter. And here it's showing this is [432:46] show all projects. Control B to toggle [432:49] branch V to preview. I'm going to do [432:51] control A [432:53] because it's showing me [432:56] which folder was I in when I did that. [432:58] Hold on here. Hit escape. We got to be [433:01] really careful and pay attention to [433:02] where we are. That's why I'm not seeing [433:04] it. Okay. So, we go into grocery store [433:05] here and I'll try this again. [433:10] And now we'll do resume. And so, now we [433:12] can see um the sessions here. Notice [433:15] that it shows you the size of it. So, [433:17] you have an idea of how much of the um I [433:20] guess the probably the conversation is [433:22] at that given point. Okay. Um but notice [433:26] down below here says show all projects, [433:28] toggle branch, controlV to preview, [433:31] rename, things like that. Um, I'm [433:34] curious what it would be if we did [433:35] control V control V [433:38] control shift V [433:41] control + V. Nothing nothing. Okay. What [433:44] about control A? Now we can see all [433:46] projects. So if I can click on to bath [433:47] bombs form.com [433:50] and so now we're in the context of this [433:52] one. Okay. So I mean it's pretty [433:54] straightforward. It's not that [433:55] complicated. When you want to continue a [433:57] conversation, you just go to the [433:58] conversation tab and change it or use [434:00] the resume. Okay. [434:07] Let's talk about forking sessions. So, [434:09] this allows you to resume a previous [434:11] session without affecting the original [434:12] session. The way it works is that when [434:14] we uh perform the fork, it's going to [434:17] basically branch off uh as a new [434:20] conversation um with its own session ID. [434:23] Very straightforward to use. We just use [434:25] the hyphen fork session uh to do that. [434:29] And if you notice that there's continue, [434:31] resume, it's the same thing. I just have [434:32] a hard time typing continue, so I never [434:35] ever show you that. I always just do [434:36] resume. Um, but yeah, pretty [434:39] straightforward. There you go. [434:45] Hey folks, this is Andrew and in this [434:47] video I want to take a look at forking a [434:49] session. Um, just so you know, I'm [434:50] jumping all around with these videos and [434:53] so sometimes things will look a little [434:54] bit different. So, I do apologize, but I [434:56] have a project here that has basically [434:58] nothing in it. I'm going to start up [435:00] Claude, and I want to have some kind of [435:02] conversation with it. So, what I'm going [435:04] to do is I'm going to tell it to uh code [435:08] me Flappy Bird uh in a new folder [435:15] using [435:17] JavaScript. So, that's obviously a very [435:19] easy task for it to do, and we'll go [435:21] ahead and allow it to do that. [435:25] And so whatever it wants, I'm just going [435:26] to say yes, yes, yes. Uh until it gets [435:29] to a point because I'm just trying to [435:31] have a conversation. [435:33] Uh continue on here until we have [435:34] something. Okay. And then when that's [435:37] done, we're going to then try to do a [435:38] fork. Just a moment. Okay. All right. So [435:41] after a little while there, um it's [435:43] finished. And so I'm just going to hit [435:45] escape. And uh what we can do is we can [435:48] go ahead and Whoops. Hit escape here. [435:52] And [435:54] Yeah, it's fine. There we go. I'm not [435:56] sure why I can't type. [435:59] I'll turn I hit shift escape to go back [436:01] into normal mode. Some of this just [436:02] happens. I have to like type a little [436:04] bit to get here. So, obviously there is [436:05] the resume command. Um I'm not sure if [436:08] they have a fork command here. Do they [436:09] have a fork command? They do. So, that's [436:11] obviously one way we can do it. In the [436:13] screenshot or in the the slides, I [436:15] didn't show you there was a fork command [436:16] here like this way, but there obviously [436:18] is. We'll go ahead and do that. And so, [436:21] now it's forked that conversation. And [436:22] we can see there's original one here. [436:24] And so I'll just say um you know, can [436:27] you check the Flappy Bird game [436:32] for bugs, right? And so it's going to go [436:36] off and do that. I'm just going to click [436:37] into any of this. And the reason I'm [436:39] doing that is just so I can see there's [436:40] more than one conversation. So here you [436:43] see code Flappy Bird and then you have [436:44] this one. And so clearly they're [436:46] different. We could probably rename it [436:47] so we can distinguish it if we were [436:48] working on another fork. But really [436:51] that's all it takes to fork something. [436:52] So that is not a complicated concept. [436:55] Obviously we just have two conversations [436:57] or two sessions and there you go. [437:04] Let us take a look here at the context [437:07] command. So context shows tokens [437:08] consumed in the current session and [437:10] available tokens broken down by [437:12] category. And so here um we have a [437:15] completely brand new session and we are [437:17] running the context command. And notice [437:19] that uh when we have a new session that [437:21] basically all the space is available. In [437:23] fact, the only thing that's taking up [437:24] any room is um the skills being loaded [437:27] into here. And then you have your autoco [437:29] compact buffer. Let's talk about that. [437:31] So autocompact buffer is cloud code um [437:34] reserving a portion of the context [437:36] window. So 22% of it uh that ensures [437:39] there's enough headroom to summarize [437:40] conversation history when the limits are [437:42] approached. So very very useful feature. [437:46] Um, if we were to look at an older [437:48] session here, uh, you'll notice that [437:50] we're getting actually a breakdown of [437:52] information in terms of what it's [437:53] utilizing. Okay, so uh, there's [437:56] obviously more going on here. So, we [437:57] have the system prompt, how much room [437:59] it's taking up, skills is really small [438:01] here, and then that there's our actual [438:03] message. Um, so you'll find that that's [438:05] really, really useful um, to keep track [438:08] of stuff. And later in the course, [438:10] you'll see me use it a lot more as I [438:12] start to think like, okay, how much [438:13] usage do we have? Um, and one thing I [438:16] want to point out is that a really great [438:18] way if you want to see this command is [438:19] when you're using the resume or the uh [438:21] the resume flag to to continue a [438:23] previous session. Then often you'll run [438:25] the context here to see the size of it. [438:27] Okay, but there you go. [438:33] All right. So, let's take a look here at [438:35] uh the context menu. So, or the context [438:37] command. So, I'm going to type in claw [438:39] down below and we'll hit enter. And so [438:43] now that we're here, I'm going to go [438:44] ahead and type in context. So as far as [438:46] I understand it, that when you type in [438:48] Claude, you are definitely creating a [438:49] new session. If you're over here and you [438:52] have previous conversations, I imagine [438:53] that those are separate sessions. Notice [438:55] it says search session. So to me, these [438:57] are previous sessions that we could [438:59] resume. It's interesting that we have [439:01] either web or local here. So I guess we [439:03] could pick up a um something that we [439:05] were doing in the web and then bringing [439:07] it on over here. Actually, you [439:09] definitely can because I'm noticing here [439:10] that carryover. I'll probably make notes [439:12] of this. One moment. All right. And so, [439:15] one thing that I'm surprised is that we [439:17] have 7% use uh 7% usage because I [439:19] thought this would have been scoped uh [439:21] per the session. So, I'm not exactly [439:24] sure [439:25] what we're looking at there. Um and I'm [439:29] not sure how you would tell if you're an [439:30] existing session. So, I'm going to go [439:32] ahead and hit escape. Okay. Let's see if [439:35] there's any way in here. Let's see here. [439:37] Um, [439:39] so what I'm looking for here is I just [439:41] want to know I type in for slash Whoops. [439:45] Can I type? Nope. Nope. Nope. We'll keep [439:47] hitting escape. [439:49] Uh, that's not working. I'm going to [439:51] just control C out of there. We'll type [439:52] in clear. Oh, we just actually saw some [439:54] session information. [439:58] We uh we recommend medium effort for [440:00] Opus. I'm actually surprised that it did [440:02] this. I didn't even prompt it for that. [440:04] And so what I'm carefully looking for [440:05] here is if there's any session [440:08] information efforts to determine how [440:09] long it thinks or whatever. Um I I don't [440:12] know what it's doing. So I'll hit enter. [440:13] I'm not sure why it was asking me that. [440:15] And I'm just going to make sure that [440:16] whatever model I'm on, I'm on the [440:17] default one because I want to have [440:19] standard usage right now. I'm just going [440:20] to type in default. There we go. Um but [440:23] I'm just wondering if there's anything [440:24] for sessions in here. We don't. Um [440:29] okay. What have we do? We get uh we do [440:32] have resume. [440:34] Okay, then we have clear. [440:38] Okay, so maybe this would help us see [440:41] because we'll have some options. Okay, [440:42] so here we have some options for [440:45] sessions, right? So I can go here and [440:47] take a look. Notice that like the [440:48] amounts are uh bigger or larger here. [440:50] I'm wondering if that's going to affect [440:52] some stuff here. So I'm going to go [440:52] ahead and hit enter here. And now what I [440:55] want to do, I'm so tempted to type [440:56] clear, but clear is going to clear the [440:58] conversation. and it's not going to um [441:00] clear this here. So, you really don't [441:02] want to type that. We'll go ahead and [441:04] type in context. And so, what I'm trying [441:06] to figure out is that, you know, if I [441:07] switch to different conversations or [441:09] sessions, does this change? [441:13] Right. I also, it's interesting how we [441:14] have skills and messages here and they [441:16] kind of stack up so you have an idea of [441:17] usage. Um, and it's cool that we have [441:21] like a little buffer here. So, that's [441:23] kind of interesting. One second. I just [441:26] wanted to screenshot that and I want to [441:27] go re-record the context one. But [441:29] anyway, um, so it's not really telling [441:32] me what I want to know. So I'm going to [441:33] go ahead and we'll try this again. [441:34] Resume and we will switch to just [441:36] anything that's different, right? And so [441:38] what I'm looking for. [441:42] Yes. So it does. So it absolutely does [441:44] change with your session. So if you [441:46] switch over to a previous session, [441:48] you're going to get different [441:49] information. Okay. So that's pretty darn [441:51] clear. [441:53] Um, yeah. So I think that answers that [441:56] and uh yeah okay there we go. [442:05] Let us take a look at the compact and [442:08] clear command. So when you have larger [442:10] conversations that means they're going [442:12] to consume more tokens that equals to [442:14] greater cost with your uh API usage or [442:17] um you're going to run out of your [442:19] subscription usage that five hour window [442:22] a lot sooner than you think. So um cloud [442:25] code is going to do autoco compaction. [442:28] So technically it will take care of [442:29] itself at those larger sizes but there [442:31] could be cases where you just want to [442:34] automatic or manually trigger a compact [442:36] or you just want to clear the thing [442:38] entirely. And so they have two commands [442:39] for that. We have the compact command [442:41] and then we have the clear command. So [442:43] it's pretty darn straightforward. Um [442:45] clear will not clear out your cloud MD [442:47] files or your automemory. Uh this was a [442:50] thing that a lot of people get mixed up [442:52] on. The other thing is you got to be [442:53] careful not to type clear as in to clear [442:55] your screen superficially. It's very [442:58] common in Linux to type in clear or cls, [443:00] but this will literally clear out like [443:02] delete uh the conversation and start it [443:05] start it over from scratch, right? Um [443:08] and I'm just going to again iterate this [443:10] because I say this a lot of times, but [443:12] um there are cases where conversation or [443:14] sorry your environment like WSL 2 might [443:16] get halted. Um, I was having a like a [443:18] conversation that was at the max and [443:20] then all of a sudden my WSL 2 hung and I [443:23] had to restart it completely. Uh, so [443:26] just be mindful of that. And the way [443:28] that we can do that is with that context [443:29] command which will actually show us, [443:31] right, which we saw in another video, [443:33] but there you go. Okay. [443:39] Hey, this is Andrew. In this video, I [443:41] want to show you the compact and clear [443:43] command. They're pretty straightforward. [443:44] Uh so what we'll do is type in claude [443:47] and we can uh do it over here but I [443:49] noticed that sometimes when you use [443:50] commands here they work a little bit [443:51] different. So for instance context would [443:53] tell us um information about uh you know [443:56] how large your context is. But you [443:57] notice here that it's not as nice. It [443:59] says you know how much we have here but [444:00] it's not as nice as the one in the CLI. [444:04] So you just kind of learn you know over [444:07] time. Probably at one point in the [444:08] course I told you I preferred the the [444:10] ID. Now I prefer the the terminal. But [444:12] here this gives us an idea of the size [444:14] of our context usage. So we can see here [444:18] um system prompt. So how much the system [444:20] prompts taking up the skills and the [444:23] messages. So not a whole lot. There's [444:25] not really any need uh to compress this. [444:27] But maybe what we'll because this is a [444:28] new uh a new session, right? Or it [444:31] should be a new session. But what we're [444:32] going to do is switch over to maybe an [444:34] older one like bathmoms.com. [444:37] And then we'll try this again. We'll [444:38] take a look and look at our session. Is [444:40] this any larger? [444:42] It is. It's a little bit larger. It's [444:44] not a lot, but it is larger. So, we [444:46] could kind of fiddle and try to find a [444:48] very long conversation. Um, I probably [444:51] could do that if I just go back to one [444:53] of my prior projects here. So, if I go [444:55] into [444:57] this one, and again, if you didn't have [444:59] such a long conversation, you just have [445:00] to prompt until you did. But I can go [445:02] here and then we'll type in resume. [445:06] And I want to see, can I find a really [445:08] big one here? Thought I was seeing [445:10] megabytes in here. somewhere, right? [445:16] That one here. [445:18] Can you read read me and define what is [445:20] expected to be called? Yeah, it seems [445:21] like there should be something larger [445:22] than that. Nope, that's my largest one. [445:25] Okay. And so, let's go ahead and take a [445:27] look here. I'm assuming that's the [445:28] largest one, right? [445:30] No, not that much larger. So, I'm not [445:33] really sure where the long conversation [445:35] is, but you know, if we did fill this [445:37] up, we could do that. So, what I'm going [445:38] to do is I just want to fill up the [445:41] context. Um, so I'm going to give it [445:44] some kind of complicated task that's [445:46] going to take forever um to get done [445:49] just so that it fills it up quite a bit. [445:51] So, I'm just thinking about this for a [445:52] second. Um, [445:55] I'm going to say I want you to port the [445:59] app over to Rust. [446:02] Uh, I want you to port the back end [446:06] over to Rust. [446:08] and the front end [446:10] over to [446:12] uh [446:14] TypeScript I guess because I I wrote [446:17] these um this is a JavaScript front end [446:21] with myithil and then this one here is [446:24] um a Golang app, right? And so actually [446:26] I might just save it before I do [446:27] anything else [446:29] just so that I don't have to meddle with [446:31] it. Say add all. I don't think we want [446:33] to add this one. We'll add all the rest [446:36] here. Save changes just because I don't [446:39] want if I ever want to come back to this [446:40] code, I just want to have it in a good [446:42] spot. And so this is a very large task. [446:45] Um, and so I'm going to go ahead and ask [446:48] it to do that. And this is going to take [446:49] a while, but the reason I'm doing it is [446:51] just so that I can get a larger context [446:53] menu and to show you the compact and [446:55] clear. So I'll be back here in a little [446:57] bit. Okay, there we go. So after um 9 [447:02] minutes, it uh rewrote our stuff. I'm [447:05] not actually trying to make this work. I [447:06] was just trying to give it a large task [447:07] that it needed to do, but it's fun to [447:11] give it large tasks and and let it run [447:13] off to the races. I'm gonna go ahead and [447:14] check my context and let's take a look [447:17] here. And so, yeah, we're getting uh a [447:19] lot more usage here. Clearly, there is [447:21] still more room to be had. Um, and while [447:24] I'm here, I might as well just check my [447:26] usage, [447:27] right? So, I'm at 28% for the session. [447:30] So, that is good as well. Um, but [447:33] anyway, so I wanted to show you two [447:35] things. I wanted to show you [447:37] compact and also clear. So I'm not sure [447:41] what is going to compact for this, but [447:42] we'll go ahead and give it a try. So [447:44] we'll say compact [447:49] and we will wait for it to compact. [447:53] I don't remember it taking very long for [447:54] it to compact, but basically what we're [447:56] interested is is what is this going to [447:58] look like afterwards, right? [448:01] Okay. So, I'll just pause it and we'll [448:02] wait till it's done. All right. So, our [448:04] compaction uh appears to be have [448:06] completed. So, here conversation uh [448:09] compacted control O for history. I guess [448:12] we do control O for history and see what [448:14] we get. Uh it's not very useful. Um [448:17] we'll go all the way back down. I guess [448:18] just really scroll this all the way to [448:19] the top. I'll just hit escape here. And [448:22] what we'll do is look at context and see [448:24] if it's any smaller. It is much much [448:27] smaller. Okay. So, now let's try clear. [448:30] So clear the conversation history and [448:32] free up context and that instantly [448:35] executed. [448:38] We go check the context [448:40] and now it's basically just one thing. [448:43] So there you go. That was um our two [448:47] compacts and clear. Okay, [448:53] let us take a look at the rename and [448:55] rewind commands that affect your [448:57] session. So rename as it implies let you [449:00] rename uh your current session. Um I [449:04] think this is a habit I need to get uh [449:06] better at doing because I do find it [449:08] very hard to distinguish those [449:10] conversations. Um just remember that you [449:12] only have 30 days till you have those [449:14] sessions and there's some way to back [449:15] them up. I'm not exactly sure. I'll have [449:17] to look at that separately if that is a [449:19] concern. But yeah, that is something [449:20] that we'll need to look into. Then you [449:22] have your rewind command. So this will [449:25] restore a session to an older point in [449:26] time. So here I was just trying to do a [449:29] loweffort. You can see I set to low [449:30] effort and these are the prompts that I [449:32] did. So we can go to that prior [449:33] conversation and basically it's just [449:36] like literally you are clearing out to [449:38] that point and continuing on from that [449:40] point. Um so if you cannot afford to [449:43] lose that other stuff, you know, just [449:45] don't go too far back. Um but yeah, uh [449:49] there you go. Not complicated, but we [449:51] will give it a little try. Okay. [449:58] Hey, this is Andrew and we are going to [450:00] take a look at uh rename and rewind. [450:02] Very straightforward and we'll only take [450:04] a moment of our time. So, um again, it's [450:08] hung up on me here. I don't know why [450:09] that happens, but I'll just close out [450:10] the old one and we'll go ahead and type [450:12] in Claude. And so, we've started a new [450:14] session. I'm going to rename it right [450:16] away. And we'll just say uh maths. Okay. [450:20] And so, now that's the name of our [450:21] session. I'm going to go ahead and maybe [450:23] we could just take a look at um [450:26] uh resume because I'm just curious what [450:28] other ones we have here. And so none of [450:30] them those ones are named. Okay. But [450:32] we're going to stay in our current [450:35] session. And um what I want to do is [450:38] just do some basic logic. So we're going [450:40] to set the mode to low. We are oops mode [450:44] not mode. Um effort effort effort to [450:47] low. And then the model here will make [450:50] sure it's high coup. It's already haiku, [450:52] but I'm just doing it out of habit. And [450:54] now just say 1 plus 1, [450:57] right? 2 + 2, [451:00] 3+ 3, [451:03] 4 + 4, [451:06] 5 + 5. Nothing super exciting here. One [451:08] thing I want to do is check context just [451:10] to see what size we're at. So there we [451:12] are. We haven't really used up a lot, so [451:14] we're probably not going to notice, but [451:15] it does say 822 tokens, right? And I'm [451:18] going to go here and rewind. And so if [451:20] we rewind, we can go in the past. We'll [451:22] go to now 2 plus two. And here it says [451:26] restore conversation or summarize from [451:28] here. Interesting. We're going to [451:30] restore the conversation. And um now I'm [451:33] going to do context. [451:37] And notice the token count is much well, [451:40] it's not that much smaller, but is [451:41] smaller. So obviously we had a couple [451:43] options there. Um [451:45] and they're pretty straightforward, but [451:47] there you go. Okay. Chowo chia. [451:55] All right, let's take a look here at [451:57] Claude Code setting scopes. So, there's [452:00] a settings JSON file, but there's a [452:01] little bit more to it because you have [452:03] to take in consideration uh at what [452:05] scope it is being utilized at. And [452:08] you're going to find these level scoped [452:10] settings files for more than just the [452:12] generic settings JSON file, but other [452:14] ones as well. Um, so just consider [452:16] you'll need to apply that in other [452:17] places. But anyway, let's work our way [452:19] from the top to the bottom. Starting [452:21] with manage settings. This is the [452:22] highest level of settings for cloud [452:24] code. This is for organizationwide [452:26] instructions and it's going to be [452:28] managed by your IT and DevOps team. The [452:30] next level is user settings. These are [452:32] your personal preferences across all [452:35] projects. Then there's project specific [452:37] settings. And then there's local [452:39] settings. Uh with the last one being [452:41] project specific, but it doesn't get [452:43] checked into your git repo. So, where do [452:46] these things live for the manage [452:47] settings? Uh, it's going to be on your [452:49] servers. And there is a special uh [452:52] settings file called a managed settings [452:54] JSON file. And there are some settings [452:56] that only go in this one. So, it will [452:58] take all the other ones, but there are [453:00] some that specifically only will work in [453:03] the manage settings JSON file. For user [453:05] settings, that will live in your home [453:07] directory. As you can see the tilda uh [453:10] I'm not sure what it be for Windows but [453:11] for Linux machines it'll be tilda which [453:13] is your home directory/cloud [453:15] and then it will be there. For projects [453:17] it's going to be very similar but it's [453:19] going to be in your project your repo [453:20] wherever that is and it's going to be in [453:23] a similar folder and then local will [453:25] just have local JSON on it and that's [453:27] how you're going to distinguish it. And [453:29] the priority of um what takes precedence [453:33] is what's higher in scope. So something [453:35] higher in scope is going to take [453:36] precedence over uh settings in the lower [453:39] scope. And there you go. [453:47] All right. So now that we understand [453:49] scoping for uh clock settings, let's [453:51] actually go look at all of them. And we [453:53] are going to look at all of them. But my [453:55] goal here is not to make you remember [453:56] them here. That's not important. What's [453:58] important is to get exposure to [453:59] understand what settings are available [454:01] to us because we're going to repeat it [454:03] out throughout this course when we need [454:04] to look at those settings. Okay, but [454:06] there are a lot of settings here, but I [454:07] think it'll be worth our time. So, first [454:09] we're going to look at authentication [454:10] settings. And by the way, all of these [454:12] are going to go in the settings.json [454:13] file. There are going to be some [454:14] settings here that are manage settings [454:16] only, and we'll point that out. So, [454:18] obviously authentication has to do [454:19] authentication. And you'll notice right [454:21] off the bat, we have some things that [454:22] are specific to AWS. Um, but let's go [454:24] take a look here. So the first is API [454:27] key helper and this is a custom shell [454:29] script to generate a temporary API key. [454:31] Um then we have for adabus o refresh. So [454:34] that is going to be the trigger to [454:35] refresh the authentication and I mean [454:38] you'll probably leave it alone but it'll [454:39] be just adabus sso login. So to do SSO [454:43] login single sign on login. Um then we [454:46] have the auto select for the uh UID or [454:49] so. So you can force the exact value [454:51] that you want it to be. You can also [454:52] force the login method. So if you only [454:54] want to use cloud AI, you can do that. [454:57] If you only want to use console, which [454:58] would be the for the API usage, then you [455:01] can do that. Uh so that's something you [455:03] can set. Um and then if you need to [455:05] export native credentials uh uh with the [455:07] grant there, you can do that and set the [455:09] script for JSON output. Let's take a [455:11] look at session and storage settings. So [455:15] uh we can set a cleanup period. And so [455:17] the default is 30 days. Um but the idea [455:20] is that you know those sessions aren't [455:22] active. They're going to get deleted. So [455:23] you might want to have a strategy to [455:25] back that stuff up. Um but the point is [455:27] and we showed you how to export [455:29] somewhere in this course how to export [455:31] um individual um sessions, right? Um or [455:34] if you just have go to the project [455:36] files, we know that they're literally [455:37] JSNL files, so you could just uh back [455:39] those up as well from there, but they're [455:40] not as easy to parse. But anyway, so if [455:42] you set it to zero uh to delete all [455:45] transactions at startup, and it will [455:47] disable persistence entirely. Uh and so [455:50] when it's zero, there will be no new [455:52] JSON files written and resume shows [455:54] nothing and hooks get empty transcript [455:56] paths. Okay, so just a consideration if [455:59] you do set it to zero. For automemory [456:01] directory, this will be a custom [456:03] directory for your automemory. So you [456:04] can change it to a different path, but [456:06] it's not allowed in the project cloud [456:08] settings uh JSON file there. Okay. Um [456:11] and then for the plans directory, you [456:13] can change where you want to store your [456:15] plans. Pretty straightforward. Let's [456:17] look at environment variables. This [456:18] allows you to set environment variables, [456:21] right? So if you need to set them to be [456:22] applied for every session, you can do [456:24] that in your settings.json file. For [456:26] model settings, you can override uh the [456:29] default model for all sessions. U you [456:31] can restrict uh exactly what models you [456:34] can choose from. So you're saying what [456:35] available models. Okay. You can map [456:38] model ids to very specific provider IDs. [456:41] So example here to Amazon Bedrock. I [456:44] can't imagine that you can set this to [456:46] anything outside of the cloud models, [456:48] but maybe there are different variants [456:50] of cloud models on Amazon Bedrock and [456:53] you might want to set those. Um, you can [456:57] set it so it's uh thinking is always [456:59] enabled by default and you can set the [457:01] fast mode per session optin if you want [457:04] to. So fast mode resets each session. [457:06] Users must reenable fast each time, [457:08] right? And fast mode is um I believe [457:11] that's the opus feature, right? where it [457:13] gets expensive. So, um that's why you'd [457:16] set it in here if you don't want to keep [457:17] setting it and you're comfortable with [457:19] the cost of it. Uh let's take a look at [457:21] uh output and language settings. So, you [457:24] can set the output style. Um and so that [457:27] would be to explanatory. Um output [457:30] style, we cover this somewhere else. [457:32] Yeah, we actually have a section where [457:33] we cover it, but that's going to change [457:36] um how it's going to work. So, [457:37] explanatory means I'm going to not just [457:40] only solve the problem, but I'm going to [457:41] explain it as I go. So you understand [457:43] that you have another one called like [457:44] learning which will uh let you um it [457:47] will stop and then let you learn uh like [457:49] your pair programming with it. Um and [457:52] again we cover that in the output styles [457:53] language for setting language pretty [457:55] clear. You might want it as English. I I [457:57] don't I don't know why mine set to [457:59] Japanese but um I never noticed it in [458:01] Japanese but that's fine. And then we [458:03] have in include get instructions. Um so [458:07] yeah that will be for I guess the PR [458:09] workflow. I don't think I've used that [458:11] setting, but it's there. Let's talk [458:12] about get attribute uh attribution [458:14] settings. That's usually when you're [458:15] saying who the author is. So, you here [458:17] you can set it um in terms of what the [458:19] commit message looks like. Um and this [458:22] option is now deprecated. So, you're not [458:24] going to be able to set include [458:26] co-author by. Let's take a look at [458:28] announcements and update settings. So, [458:29] here this will be shown at the startup [458:31] of multiple entries um that are cycled [458:34] in. So if you're a company, maybe you [458:36] want to cycle these things to remind [458:38] people like hey remember uh you know [458:40] don't do x y z or you know like whatever [458:43] companies want to do. Um and then you [458:45] can set the auto updates channel value [458:48] to whatever you want it to be. Um for [458:50] the UI itself and spinners we have quite [458:53] a few options. So uh for show turn [458:55] duration we can uh change that value. We [458:58] can set the spinner verbs. Actually, you [459:02] know what? I'm so glad that I know where [459:04] that is because I really dislike these [459:06] verbs, right? So, I think I'm gonna make [459:09] a lab where I change the settings so I [459:11] do not ever have to see all these weird [459:13] words again because I cannot stand them. [459:15] Um, yeah, we have whether you want to [459:17] show tips or not in the spinner. Uh, we [459:19] can override the spinner tips. So, [459:21] again, if you're working with a team, [459:22] maybe the suggestions you want them to [459:24] see periodically, you can show terminal [459:26] progress for iTerm Iterm 2. This might [459:29] actually fix. Well, we're not using [459:30] iTerm 2. That's a Mac specific thing. [459:33] But you notice that in our WSL2 my [459:35] version that the progress bar in the [459:37] terminal is not great. Need to be fixed. [459:40] And then we can reduce spinners, [459:41] shimmers, and flash animations for [459:42] accessibility. So if they're bugging [459:44] you, you can set that to false. Okay. [459:47] Um, for agent team settings, we have the [459:49] ability to set the teammate mode. I [459:53] think this is how it's going to work. So [459:54] whether it's going to split pane or run [459:57] in line or run in split panes in T-Mox. [460:00] T-Mox is terminal. Actually I don't know [460:02] what T-Mo stands for. I know what it is, [460:04] but I don't I don't know what it stands [460:05] for, but it's a it's like a terminal [460:08] extension that lets you do uh split [460:09] screens natively in terminal, right? So [460:12] a lot of programs these days let you do [460:14] it at the OS level, but it literally [460:17] will do it within the terminal um like [460:19] the terminal shell window. Let's take a [460:21] look at permission rule settings. And [460:23] this stuff we're going to dive a little [460:25] bit deeper because I have to explain the [460:26] wild card logic and stuff. But the idea [460:28] is that these are the things that are [460:30] allowed to do. So whether it can read [460:33] files, write files, utilize bash, and [460:36] then there's there's some logic um in [460:38] terms of how the wild card works, but [460:40] that's where you don't require [460:41] permissions or like it doesn't have to [460:43] ask you to do it. than the things that [460:45] you want it to uh ask you before it does [460:47] it. Things that it will deny for very [460:49] sensitive uh files or dangerous [460:51] commands. If you need to include extra [460:54] directory access beyond the project [460:57] route, so maybe there's some shared [460:58] files that you want to include or shared [461:01] documentation. Maybe you have a um [461:04] coding guideline across all your [461:06] projects, then you could maybe reference [461:07] that information, let it know that it's [461:09] there. Um here we can set the default [461:12] mode. So here it says accepts edits. Um, [461:15] yep. So you know default mode there and [461:19] then we have disable bypass permission [461:21] mode. So this would be yeah set to [461:24] disable to block the data permission [461:26] flag entirely. To block the flag [461:29] entirely. Uh, so that means we can't use [461:32] it. [461:34] Anyway, we cover it. I can't remember [461:35] off the top of my head. I think it means [461:37] to disable the feature, but we [461:38] definitely cover it in that specific [461:40] slide because I remember talking about [461:41] it. We got MC server setting controls. [461:44] So, auto approve all servers listed uh [461:47] in the MC uh JSON file. So, if you want [461:50] to enable them all, then you just say [461:51] true. Approve only specific servers. So, [461:54] here we're saying we approve memory and [461:55] the GitHub uh servers. Disable specific [461:58] ones. So, specifically reject certain [462:00] ones. Um and then we can allow them [462:02] based on uh server names. These are [462:05] based again this is manage setting only. [462:06] So, just pay attention to that. So, [462:08] allow lists for MCP servers. Um yeah so [462:12] for specifically manage settings. Okay [462:14] so that's all they're saying there. Then [462:16] all of these settings here for plug-in [462:17] marketplace are for manage settings. [462:19] Okay so for your orwide level settings [462:23] and here we have the option to strict uh [462:26] uh restrict to known marketplaces [462:29] uh or block marketplaces or it says [462:32] appended to the plug-in trust warning to [462:34] show it install time. So just a message [462:36] there. Uh then we have uh manage [462:39] settings uh only lockdown uh flags. So [462:42] here we have allow manage permission [462:45] rules only. So whether we can set that [462:47] there for MCP servers and hooks only and [462:50] again this is for manage settings. And [462:53] then we have hooks. I don't know if this [462:55] is the last slide. I think it might be [462:56] the last slide. And so whether hooks are [462:58] enabled or not uh what are allowed hook [463:03] URLs? [463:04] um the envirs that will be allowed to be [463:06] interpreted and then the actual hooks [463:08] themselves and we have a whole section [463:10] on hooks. I guess we're not done. We [463:11] still got observability status line IO [463:14] and miscellaneous settings. So here you [463:16] can set the OTL header uh headers [463:18] helper. So I guess if you're using uh O [463:20] telemetry to export your data, you can [463:23] utilize that there. This is where we [463:25] configure status line. We already [463:26] covered this when we did status line. Um [463:28] and then for file suggestions, we have a [463:30] custom script to power the autocomp [463:32] completion. Uh which is interesting. And [463:34] then we have respect git ignore. So [463:36] whether the file picker hides files [463:38] matched by the git ignore or not. So [463:40] there you go. That's all the settings as [463:43] of today. And again, do not worry about [463:46] remembering all that stuff. Uh we will [463:48] figure it out as we go. But I definitely [463:50] want to change those darn spinner verbs. [463:52] They I cannot stand them. Okay. Chow [463:54] chia. [463:59] Hey folks, this is Andrew. In this [464:01] video, what I want to do is fiddle with [464:03] settings. Now, obviously, uh there's a [464:05] lot of settings and at a lot of [464:07] different levels, but I personally just [464:09] want to change something for myself. And [464:11] I know that we already have a [464:12] settings.local.json. [464:13] If you don't, go ahead and create one. [464:15] But I'm pretty sure when you uh do [464:17] claude there at the start, you're going [464:18] to get one. Or you could do claudit as [464:21] that will set one up initially for you. [464:24] Um, [464:26] so I'm just thinking about this for a [464:27] second. Let's go make a new project [464:29] because I just want to test this for [464:31] sure. And we'll say um mkdir and this [464:35] will just be uh settings [464:40] example. And I'm going to cd into that [464:42] one there. I'm going to do claude. And [464:43] so as I hit enter, okay, it's entered [464:47] into settings example. And so whenever [464:49] you do that, it generally is going to [464:51] create a new thing. But notice here it [464:53] says in settings local.json. So I think [464:55] that it might have already created one. [464:57] So I'm going to go ahead and type in [464:58] code period just so that we can get into [464:59] this context so we're not confused. I'm [465:02] going to give this a refresh here. I [465:04] want to see if there's any kind of file [465:05] here. That's what I'm looking for. Do we [465:06] have any settings file yet? Okay, we'll [465:09] go ahead and do terminal. New terminal [465:10] here. I'm just going to go ahead and [465:12] make a readme file just so there's [465:14] something here like hello world. So it [465:16] has something to look at, right? I'll [465:18] even make a Ruby file. We'll go ahead [465:19] and just say um hello.rb. RB and this [465:23] will be puts hello [465:27] world because I noticed that when it was [465:28] empty it didn't prompt us for anything [465:30] like do you trust this directory which [465:31] I'm used to seeing. So maybe there has [465:34] to be something for it to do that. So go [465:36] ahead and do that. And it didn't it [465:38] didn't prompt us for anything [465:40] but um we see this. It says inhello.rb. [465:43] That's its context right now. Oh because [465:45] that's the page we're looking at up [465:47] here. Okay. Um but it isn't. Let me just [465:51] see. It isn't telling me anything else. [465:52] Let's go ahead and stop that. So now I [465:54] have nothing in context up here. Let's [465:56] go ahead and hit claud. It's not asking [465:58] me to trust anything. So it says tips [466:00] for getting started using the init. [466:01] Okay. And notice there is no cloud [466:04] directory. So we could obviously create [466:05] that manually. I'm going to go ahead and [466:07] use the init. Okay. So we'll go ahead [466:10] and do the init. [466:12] And I believe that's going to go ahead [466:14] and create it for us. [466:17] So here it's searching if there's [466:18] anything in that project directory. [466:20] We'll give this a refresh now. And I [466:22] still don't see anything. So, I'm going [466:24] to stop this here. I'm going to do ls- [466:25] la. And we still don't have a [466:29] claw directory. So, I thought I for [466:32] certain thought that was going to create [466:34] for us um [466:37] uh what we wanted there. And so, we'll [466:38] go ahead. We'll just launch this up [466:40] again. I'm going to do for/init. [466:43] Initialize a new claude MD file. It [466:45] didn't even make one for me here. Am I [466:47] not in the right directory here? I am. [466:52] Okay, I still going to do this. I just [466:54] want to see if it generates out a [466:57] settings JSON file. [466:59] Okay, because I don't remember creating [467:00] that other one. [467:02] Maybe claude changed it at some point. [467:04] Maybe I tried to change permissions. And [467:07] so we're just trying to observe what it [467:08] creates here. Okay, like I don't need a [467:10] claw.md file. We'll go ahead and just [467:12] say yes, that's fine. Um, but what I do [467:15] need is that uh page. And so again, just [467:18] trying to verify if that's correct or [467:21] not. We'll pause here and see what [467:22] happens. [467:23] Oh, looks like the file is already here. [467:25] We'll say yes. We'll refresh. And so [467:27] we're not getting that directory. So we [467:29] have our cloud.mmd, which is fine, but [467:31] that's not going to do anything for us. [467:32] So we'll go ahead and we'll make a new [467:33] one here. This will be cloud. [467:38] And then we'll make a new settings file. [467:39] I always forget if it's I think it's [467:41] settings.local.json. [467:43] It is. And because I only want to apply [467:45] to me, [467:48] do that. The reason we went through all [467:50] that init stuff is I just wanted to see [467:51] if it would actually make when it [467:52] apparently doesn't. Okay, so now we have [467:55] this. And so I'm going to go ahead and [467:56] we'll do uh curlies. And um the thing [468:00] that I want to get rid of is that the [468:02] the spinner verbs. You heard me already. [468:04] I really don't like the spinner verbs. [468:07] They drive me crazy. And so I'm really [468:09] glad that we have this option. And so [468:11] I'm going to go ahead here and change it [468:12] like this. [468:14] And I'll go here and put verbs. [468:17] And so I'm going to list it to very [468:19] specific ones. So I'm just trying to [468:20] think of um well, by the way, if we want [468:23] this to work, we're going to have to [468:24] name it correctly. So it's going to be [468:26] local.json.com, not JSON.local.com. That [468:29] makes no sense. But I'm just trying to [468:32] think of of a verb that will uh not [468:34] infuriate me. Um [468:37] and so [468:39] I am thinking [468:42] um Um, [468:45] computing. [468:48] We'll just put computing. [468:53] Okay. And so that's the one I have. Um, [468:55] we might need to set the mode. [468:58] Just a moment. Let me take a look here. [469:01] And so we'll go into the settings JSON [469:04] um, claude and see what the exact [469:05] specification is for this. [469:09] And so what I'm looking for [469:12] is the possible settings. So we want [469:15] spinner verbs. [469:17] Spinner verbs. Spinner verbs. [469:21] And there's the mode append. I'm not [469:23] sure if we have set mode to replace to [469:25] only use your verbs or append to add [469:27] them. Oh, I do not want any other ones. [469:30] So we're going to go here and say mode [469:32] and replace. [469:34] [laughter] Okay. So now we have it. And [469:36] so the question will be will this work [469:38] also you know if I change my settings [469:40] does it automatically take effect? So uh [469:42] can you um create me a simple Sinatra [469:47] app that has a hello world endpoint. So [469:52] that's what we're going to do. Very [469:53] simple request. It says bone boonding. [469:56] It's also giving me tips. So it doesn't [469:58] seem [469:59] I'm just hitting escape to interrupt it [470:01] that it's taking effect. So, how can we [470:03] reload our settings thing sign thing [470:04] sign thing sign thing sign thing sign [470:04] thing sign thing sign thing sign thing [470:04] sign thing signal without having to [470:05] completely go out there is there like a [470:06] reload [470:09] it's hanging there a bit and notice I'm [470:11] hitting control c [470:14] mean to to quit that session I'm just re [470:16] re-entering it there um so this how it [470:19] says settings local JSON okay [470:23] and uh I'm just curious re refresh or [470:26] reload [470:28] reload plugins [470:30] remote environment I don't see any [470:31] option there so really seems like the [470:33] only way to reload it is to exit it out. [470:35] I'm just not convinced that changing [470:37] settings takes effect until you um go in [470:40] and leave again. But anyway, what we'll [470:42] do here is we'll go ahead and go say [470:44] build me a simple hello world endpoint [470:48] in [470:50] a Sinatra app. Sinatra is Ruby and it's [470:53] a very small framework. And notice it's [470:55] now just saying computing. So now my [470:57] sanity is back. So, I'm very happy about [471:00] that as I've been getting so sick of [471:02] those commands. But I want this to take [471:04] effect everywhere, right? Because, you [471:06] know, I really don't like that uh those [471:09] verbs. So, I'm going to go ahead and [471:10] grab this and copy it. And what I'm [471:13] going to do is just cancel out of here. [471:15] And I want to go to my home directory [471:17] into Claude. There should be a directory [471:19] here. And there should be a settings. If [471:21] there's not, I'm going to make one. So, [471:22] I'm going to go here and say touch [471:26] um settings.json. JSON and then I'm just [471:29] going to say code [471:31] claude here so I can open that up very [471:34] quickly and in here we have a settings [471:36] JSON and apparently we already have one [471:38] even though I don't recall there being [471:40] one there but that's fine and I'm going [471:42] to paste that in and we're going to [471:43] merge this stuff together and so now [471:46] across all my projects for me I should [471:48] never ever see uh these again so I'm [471:51] going to go ahead and delete this out [471:53] permanently okay and we're going to type [471:56] in claude [471:59] And I'm going to resume a previous [472:01] conversation we were having. [472:06] So I uh I interrupted you. [472:11] Can you continue? [472:15] Can proceed? You can proceed again [472:19] because I can't spell the word continue. [472:21] So [laughter] we have computing which is [472:22] great. And that file is not there [472:23] anymore, right? So, I'm going to stop [472:25] that because all I wanted to see was [472:27] that worked. We'll go over to here. [472:30] And so, this is one is uh settings [472:32] example, which was the folder we just [472:34] created. But this is actually grocery [472:36] share monitoring. And so, this has its [472:37] own settings.local file, right? And so, [472:39] if I do claude, [472:41] and I'm just going to say uh you know, [472:44] what version of claude am I using? Just [472:48] a really simple question. And now it's [472:50] saying computing. Okay, great. And we [472:52] don't have it set there. So there you [472:54] go. That is uh something I think the [472:56] most practical setting we have. I'm also [472:58] kind of tempted to try out language. As [473:00] they say, uh you can switch out uh [473:02] language. So preferred language. [473:05] And um I know folks here probably don't [473:07] know Japanese. I know enough Japanese, [473:10] but I'm not very good at it. But um I'm [473:14] going to try and then pretend that I [473:16] know Japanese even though I don't. And [473:18] we'll go ahead and put that there. And [473:20] so I'm going to run claw. I just want to [473:21] see do we get Japanese text? [473:24] So, we'll go ahead and type that. Oh, [473:26] you know what? To make it work, I think [473:28] you got to spell language correctly. Do [473:29] I have that spelled correctly? [473:32] We do. [473:33] Claude codes preferred response [473:35] language. Oh, that's how it's going to [473:36] respond. Okay. So, [473:39] what we'll say, what uh Claude Code [473:43] version [473:44] am I using? Is it going to reply in [473:47] Japanese? [473:49] because I'm using English so that might [473:52] not help right. [473:55] Oh, it did. Perfect. So, we have cloud [473:57] code. No. Um, [474:01] bajing [474:03] bajin wades. [474:05] Okay. So, I kind of know it. But that's [474:07] cool. So, if I want Japanese, we could [474:10] just swap it to Japanese. Uh, but there [474:13] you go. That's our fiddling with [474:15] settings. Okay. [474:22] All right, let's take a look here at [474:23] permission rules. So, Claude uses a [474:25] tiered permission system that is [474:27] toolbased access controls or tabback. [474:30] And uh you won't find that anywhere on [474:32] the internet because I just came up with [474:33] it. Um so, we all know uh arbback. So, [474:36] rolebased access controls, but the way [474:37] the permissions work with claude, it's [474:39] all around their tools. Okay, but here's [474:42] an example of a um uh permissions. that [474:46] this would be in your settings.json. We [474:48] saw it earlier in the settings.json [474:50] section, but we're going to go deeper on [474:52] this stuff. So, permission levels are [474:54] allow, ask, deny. It should be [474:56] straightforward what they are, but let's [474:57] just cover them. Allow means um it's [475:00] allowed to do it, right? Ask means ask [475:03] the user if Claude's allowed to do it. [475:05] Deny means you're not allowed to do it. [475:06] Okay. The rules are evaluated from deny, [475:09] ask allow. So, what will be applied is [475:12] what's either we could say least [475:13] permissive or most restrictive. That's [475:16] the same thing. I'm just saying it two [475:17] different ways. So, a deny will always [475:21] uh rule over an ask and allow. And an [475:23] ask will uh happen before an allow [475:26] happens. So, there you go. Um where are [475:29] the tools? Well, they're here. It's [475:30] bash, it's read, it's edit. Those are [475:33] two separate ones. I just have an [475:34] editing issue there. We have web fetch, [475:36] MCP, and agents. And specifically, we're [475:38] talking about sub agents. Okay. So, now [475:40] we know that. And the next thing we need [475:42] to cover is all the syntax sugar here. [475:44] so you can configure this perfectly. [475:46] Okay. [475:52] Hey folks, this is Andrew and in this [475:53] video I want to go ahead and just test [475:56] all the types of tools in our [475:57] permissions and see if they work. So [475:59] right now we have a settings.local.json [476:02] file here and we have some allows. Um so [476:04] we can install anything we want. Uh we [476:07] have this database information and we [476:08] have this to fetch from bright data. But [476:12] I'm just going to back out here for a [476:13] moment. I'm gonna make a new directory. [476:14] I keep doing this. This would be like [476:16] hello world tasks. As you can see, we [476:19] just keep making new ones to whatever [476:21] whatever we need to to work our our use [476:24] case here. And um I'm going to go ahead [476:26] and make a new folder called clot. As we [476:29] learned, and it's not going to help us [476:31] there. We're going to have to make that [476:32] ourselves. [476:35] And I'm going to apply this all at the [476:37] settings.local [476:39] uh JSON level. Okay. And so in here I'm [476:43] using permissions. Yes, I know we can [476:44] use cloud to help us write permissions, [476:45] but folks, we got to do a little bit of [476:47] work ourselves, right? Permissions. And [476:50] in here, we have our allow, [476:52] right? We have our deny. [476:57] Okay. [477:00] And we have our um ask. [477:05] Okay. So, right now we don't have really [477:07] anything in these, of course. Um, and [477:10] there's obviously different tools. [477:13] Oops. [477:15] Tools uh that we have here. [477:19] And I'm not sure why it's complaining [477:20] like that. Maybe because there's nothing [477:21] in them right now. [477:25] That's probably why because they're [477:26] empty. [477:28] Incorrect type for array. Oh, sorry. [477:31] Array. That's why. Okay. So, there we [477:34] go. So, now we are set up to start [477:36] working with them. And technically the [477:38] order is uh deny ask allow or ask allow [477:45] you know however you want to order them. [477:46] This is how I'm going to order them. [477:47] Okay. So now we have our permissions in [477:49] here or the starter permissions. So um [477:52] let's take a look first at bash. Okay. [477:54] So I'm going to go ahead here and just [477:55] say bash and we're going to deny it. And [477:59] I'm going to go ahead here into a new [478:01] terminal down below. [478:03] And we will type in claude to start it [478:05] up. I'll just say, you know, can you run [478:10] a bash command [478:13] in the project directory [478:16] for ls? Okay. And so we want to see what [478:18] happens here. So I explicitly told it [478:20] just go ahead and run the single [478:22] command. So I don't have access to the [478:24] bash tool in this environment. I can use [478:25] the glob tool to list files. And so the [478:28] project directory contains only one [478:29] file. So it found uh a way around it [478:32] because it obviously has other tools [478:34] available to us. But that is something [478:37] uh that it could do. How about how about [478:40] we just make a new file here. This will [478:42] just be data.json. [478:44] Okay. And I'm going to go ahead just [478:46] make some data in here. So I'm going to [478:47] go and say [478:51] um [478:53] HP 100. [478:56] Make a value there. MP [478:59] 300. [479:01] Um class mage. Okay. So just random [479:06] data. I mean obviously for RPG but [479:09] random data that we have here. And so [479:11] I'm going to go back here and say uh can [479:13] you [479:15] um use jq [479:18] to return backhp [479:23] data from the data json file. [479:26] Okay. And so it might try to find [479:29] another way around it. So it says I [479:31] don't have a batch tool available to run [479:33] jQ but I can read the file and show you [479:35] the HP directly. And so it found another [479:37] way around it which is awesome. Um so [479:39] what we'll do is we will take this [479:43] and we will permit it. We'll just say [479:46] that uh we yeah we'll go and move it to [479:48] ask and let's see what happens here. [479:50] Okay. So, we'll go back and try this [479:52] again. [480:00] And it didn't ask us, right? So, I'm [480:03] thinking that maybe the data is cached. [480:06] Okay. So, I'll go back over here and [480:07] say, can you use the jq file to read MP [480:10] data? [480:12] So, it might be trying to work around [480:14] the problem [480:19] and it didn't ask us. [480:22] So, [480:24] are you using the MP uh the JQ tool or [480:28] are you relying on something like cache? [480:32] Because I'm trying to test it for that, [480:34] right? [480:39] each time. There's no cache involved. [480:41] Each executes the thing. Okay, you can [480:43] verify this yourself. But it's confusing [480:45] because we've told it that it has to ask [480:48] us if it wants to be able to do that. [480:52] But maybe it has to do with our mode, [480:53] right? Because I think it's the first [480:54] time it asks a tool. That's where we run [480:56] into an issue. So I think what we need [480:59] to do is figure out what is going to be [481:02] our mode setting. Now we can toggle with [481:04] shift tab here. here says accepts edits [481:07] on or plan mode on. But I think the [481:10] challenge here is that um we need to [481:13] make it a little bit more explicit. So I [481:14] need to figure out what all the [481:15] permission modes are. Um I know we have [481:17] a slide on it. So let me just go grab [481:19] that slide really quick uh uh quick so [481:21] we can compare it. Um I know it comes [481:24] later on here or maybe we've already [481:26] done it. We'll see here. But our [481:27] permission modes is default. So standard [481:29] behavior prompts for permissions on [481:31] first use of each tool. Okay. We have [481:35] accepts edits. Automatically accepts [481:36] file edits. Plan claude can analyze but [481:39] not modify files and execute commands. [481:42] Um, auto denies unless preapproved [481:46] via permissions. [481:48] Auto denies tools. So, let's go ahead. [481:50] Well, this is don't ask. We have bypass [481:53] permissions. So, skip all permissions. [481:54] And so, it's strange because if it's [481:57] been asked once before, it's not going [481:58] to explicitly ask again. I'm not sure [482:00] how we force it to ask. Let me see if [482:02] there's anything we can do to make sure [482:04] we can test that behavior. Oh, you know [482:06] what? I think the problem is human [482:07] error. Look, there's a colon on the [482:09] inside. You probably saw this the entire [482:10] time. So, we'll go back here and it was [482:12] probably me. [482:14] Okay. And we'll go back and ask it to do [482:16] this now. So, now ah, here we go. So, [482:19] permission rule, do you want? We'll say [482:22] yes. Okay. And so, now another thing we [482:24] learned was that it's not caching it [482:26] every time, right? So, it's going to ask [482:27] it every single time. So, we'll go back [482:30] over here. Can you return the HP data? [482:32] And so we want to see if it prompts us. [482:34] Right. [482:36] We'll say yes. We get the data back. [482:39] We'll move it over to here. [482:43] Okay. And we'll try this again. [482:50] And so one thing, another thing we [482:51] confirmed is that we can update our [482:53] settings JSON and will take effect [482:54] immediately. Um, there were some cases I [482:56] think where if we hadn't created it and [482:58] we created after the fact, it didn't [482:59] work. But clearly we can change this [483:02] file and it's reading it every single [483:04] time. So that is really useful to know. [483:06] Um, so we have those. Let's talk about [483:09] the read. So we'll just say we'll deny [483:11] read. [483:13] I'll make another one here. This will be [483:15] player data. So we'll rename this to [483:17] player. And then I'll make a new one. [483:19] Uh, we'll call this world JSON. And in [483:23] here we'll just say location [483:27] um [483:29] like current location [483:32] town, right? So we're in a town and [483:34] we'll go back over to here and so we're [483:37] going to deny all reads. So can you uh [483:39] return the contents of world [483:43] uh JSON for me? And maybe it'll find [483:46] another way around using bash. [483:52] Okay. Okay. And so it's returned back [483:53] that data. So what I'm going to do is [483:55] I'm going to take this bash command. [483:58] I'm going to put it over here. Okay. And [484:00] so in theory, it won't be able to use [484:02] bash. So it can't use cat or anything [484:04] else to get to it. So go ahead here and [484:06] then we'll try this again [484:10] because we've denied it on two things, [484:11] bash and read. [484:19] Okay, it's thinking. It's thinking hard. [484:20] I also should start typing clear here [484:22] because we're just experimenting [484:24] because it's adding that context every [484:26] time, right? [484:29] And there we go. So, here it says tool [484:31] use [484:33] um [484:36] import JSON world JSON [484:40] allow [484:43] allow whatever whatever. So, it's trying [484:45] to execute code in a very roundabout [484:47] way. I'm not exactly sure. So, it's ID [484:51] execute tool. So, it's trying to find [484:53] another way to work. That's really [484:55] confusing. So, what I'm going to do is [484:56] stop this. I'm going to clear this out. [484:58] So, we have no context. We can check it [485:00] again. Just make sure that it's emptied. [485:03] Okay. See, it's nice and emptied. We're [485:05] going to try that again. [485:08] Okay. Because I think it's trying to [485:10] find a way around it, right? It's trying [485:13] to search the file. It's not allowed to [485:14] read it. [485:17] See [485:19] what it read it. [485:22] Um, but we have those on deny, right? [485:24] I'm going to stop this and restart it up [485:26] as we're again we're just trying to get [485:28] a reliable sense of reading it. [485:37] Okay. How how are you able to read the [485:42] file when I have deny for read bash and [485:46] read? Okay, that's what I want to know. [485:50] Can it even answer me that [485:53] I use the GP tool, not read or bash. I [485:55] searched the the pattern for it in the [485:57] file which effectively turns it. So GP [485:59] is a separate tool from read. Oh, okay. [486:03] But it's not under bash. [486:10] Grab his own dedicated tools. Okay, [486:12] tools. [486:14] How do I get a list of all available [486:17] tools? Because I didn't see that in the [486:19] docs. [486:26] And so we have this here. Look at that. [486:30] Docmaps skills.mmd. It's trying to f It [486:33] doesn't even know what tools are [486:34] available. This is where I would [486:36] probably like start going in the [486:37] codebase. So we go cloud code GitHub. [486:39] Isn't that interesting? [486:41] We have a tool that we're not aware of. [486:44] It's not in the docs. [486:46] And so I'd go in here and I might just [486:48] go search this codebase. I'm going to [486:50] hit period on my keyboard. I'm logged [486:51] into GitHub. [486:53] And I'm not asking Claw to do it. I'm [486:55] just doing it myself. [486:57] I don't want to wait around uh asking it [486:59] to look in the codebase. And so I'm [487:01] going to go here and I'm just going to [487:05] Yeah, it's fine. Let's just get me [487:07] logged in here. I just want to quickly [487:09] search [487:11] across this repo. So, finding files GP, [487:16] right? I'm just trying to see if that [487:17] finds us anything [487:20] quickly here. So, here we have Glob, we [487:24] have GP. Oh, I didn't cover these. I'm [487:25] going to have to go make an additional [487:26] video just to cover the ones that are [487:28] missing. [487:30] Okay, but we have docs here that [487:31] explains it. [487:39] So, you know, I would have thought that [487:40] there would have been [487:42] a very specific place for this. How did [487:44] I miss [487:46] these? Am I crazy? Here we have read, [487:49] write, uh, GP, glob, bash, agent, web [487:51] fetch, uh, web search, task, notebook. [487:55] Okay. Uh, claude code tools. [488:06] Oh, [488:08] hold on here. Permission required. [488:11] Oh, wow. We have way more rules than I [488:13] thought. Okay, maybe my confusion was [488:16] when I was doing the permissions, I [488:17] didn't realize how many tools there are. [488:19] So, I guess I will need to go back and [488:22] correct that. But it seems like we have [488:25] a lot more tools available to us. And [488:28] we'd have to go here and say like no [488:30] glob, [488:31] no glob, right? And then we would need [488:34] um [488:36] no GP [488:39] to get this to work, right? So go back [488:42] and I'm going to ask it again. [488:50] Cool. More slides. I got to make more [488:52] slides. [488:57] Looks like you've denied it as well for [488:58] bash. So what I'll do is now I will [489:00] bring in the readme just the read. We'll [489:02] say allow. But you're basically getting [489:05] the idea of how this works. It's not [489:07] complicated, right? Um [489:14] but I guess you'd have to really test it [489:16] and make sure. So with bash and read, [489:18] can you read it? I don't have the [489:19] remaining tools I can use to retrieve [489:21] it. Ah, so it needs a combination of [489:23] tools to achieve its goals. Um, so [489:26] that's really really interesting. So I [489:28] will need to go back and retroactively [489:31] update this. So you'll know by the time [489:32] you watch this that there is obviously a [489:34] bunch of tools or I'll follow it right [489:36] after this this video here. But you were [489:38] getting an idea of how this works, [489:40] right? We didn't uh deny the bash stuff. [489:42] So we can go here and you know deny very [489:45] specific tooling. But I'll just go here [489:47] and bring these back and we'll just do [489:49] one more test here. It would be like uh [489:51] JQ JQ is denied or maybe just like J [489:56] question uh um asterisk. So, okay. So, [489:59] go ahead and I'm just going to go back [490:00] and say like read with the um JQ tool. [490:05] And so, it should be denied here. So, [490:08] it's denied. Okay. So, I think we got [490:10] some experience here. And I have a [490:12] missing spot here. Probably the next [490:13] video we'll literally start talking [490:14] about tools. I'll see you in the next [490:16] one. Okay. Chowo chia. [490:22] Let us take a look at permission modes. [490:25] Okay. So, permission modes is going to [490:27] change the way um claude is going to [490:30] work with you in terms of whether it [490:33] will go ahead and do something or ask [490:35] you. And so in the lie of having [490:38] permission rules, this is what it's [490:40] going to do. Okay. So, the first is [490:42] default. So, standard behavior prompts [490:44] for permission on the first use of each [490:46] tool. Um then we have accept edits so [490:48] automatically accept files and edit [490:50] permissions for the session. We have [490:52] plan basically which is plan mode. This [490:55] will uh make claude analyze but not [490:57] modify files or execute commands. And so [491:00] you can have it where you can have it do [491:02] a plan but then you probably want to get [491:04] out of that mode next step. Right? So [491:06] you wouldn't want to run that all the [491:07] time. We have don't ask. So auto deny [491:09] tool unless autodenies a all tools sorry [491:13] unless pre-approved via the permissions [491:15] or some kind of allow rules right we [491:18] have bypass permissions so skip all [491:20] permission prompts um obviously uh you [491:23] shouldn't do that but um that is an [491:25] option that you have and if you want [491:27] when you launch up claude you can tell [491:29] it what mode to start in but you can [491:31] obviously toggle away from it so you're [491:32] not like locked into it um and when [491:35] you're in the interactive shell you can [491:37] use shift tab to change through some [491:39] modes, not all of them, but some of [491:40] them. And the first one is default. [491:41] You'll be in it because there'll be no [491:43] message there. Plan will literally say [491:45] plan mode. And then except edit will be [491:47] on accept mode. And as you hit shift [491:49] tab, it will toggle between those three. [491:52] So there you go. [491:58] All right. In this video, I want to take [491:59] a look at permission modes. Um, and we [492:02] will get to tools in a moment here, but [492:03] I figured this would be a very easy one [492:05] to uh jump into. And so, uh, if we want [492:08] to start up in a different mode, we can [492:10] go ahead and do permission [492:12] mode plan. That's going to bring us into [492:15] plan mode. I do have one little issue [492:18] here. So, I'll just kind of fix this and [492:20] update my old uh settings file here. So, [492:22] I'm going to stop and try this again. [492:24] Okay. I'll just say like uh, you know, [492:26] can you read the uh, player JSON data [492:30] using jq um, with [492:34] for the HP? Okay, very worded, but it's [492:38] in plan mode, so it shouldn't be able to [492:39] execute it. So, what we're looking for [492:42] is to see what happens. What I really [492:44] like is um plan mode when we're using it [492:47] over here because it'll open it a [492:48] markdown file. And so, that's a lot [492:49] nicer. Um but here, [492:53] I'm just carefully looking at it. So, we [492:54] have [492:57] explore so we can read it. We'll do [492:59] bash. We'll run it. [493:02] And we'll go down below. [493:05] Find all visible files in the project. [493:09] Do you want to proceed? So, I'm assuming [493:11] that this is the plan, right? [493:14] Okay. But it's hard to see it because of [493:16] the way we're looking at it. And we [493:17] could talk about plan mode later on, but [493:19] if we go over to here, [493:22] I'm going to switch it over to plan [493:24] mode. There we go. I'll just say uh can [493:28] you read the contents of player JSON and [493:32] return uh HP? Maybe use uh the jq tool. [493:38] Okay. And so this way it's a little bit [493:39] better. It'll make a um markdown file. [493:41] We're obviously covering plans sooner [493:42] than I want to, but that's fine. It's [493:45] okay. [493:48] And so it should output a markdown file. [493:52] Uh it's in plan mode. [493:55] Where's my plan? Where's my plan? [494:06] I should have written a plan first. Wow, [494:09] that's a screenshot we're going to take. [494:13] There we go. [494:17] Escape to cancel. [494:20] So, I'll take a screenshot for that for [494:21] our plan stuff. So that is very [494:23] interesting. Okay. All right. So that's [494:27] kind of interesting that uh [laughter] [494:29] that was the result uh for the plans. [494:31] But I mean really it just came down to [494:32] me showing you that this will enter you [494:35] into different modes um for that command [494:38] there. And of course we can change our [494:41] setting for this. So maybe we'll do that [494:43] as well as I guess that is the second [494:46] half of that part. I just looking for [494:48] the list the slide I have here which [494:51] will uh help me know which ones we can [494:53] change to. Okay, it's somewhere here. [494:56] There we go. Permission modes. Yeah, I [494:58] just can't remember the names of them. [494:59] So, there's also accepts edits, plan, [495:02] and don't ask. So, maybe try don't ask. [495:05] Don't ask. [495:08] And so, it shows don't ask. What's [495:10] interesting is that when we toggle on [495:11] this stuff, [495:13] it only toggles on those two. So, we [495:15] want to get to a mode that's not there, [495:18] right? We have to do that. As soon as we [495:21] toggle out, we don't have it anymore. [495:23] And I don't plan on running it in this [495:24] video, but we'll go ahead and try bypass [495:28] bypass permissions, assuming I'm typing [495:31] that right. Let's see. We get a big big [495:33] warning, right? In fact, they have their [495:35] own flag for that, and we'll cover that [495:37] in a separate video because obviously [495:39] it's a a big big deal. Um, but yeah, so [495:42] I guess just that one one or two modes [495:44] that we cannot access any way. But it [495:46] was interesting to find out that our [495:48] plan didn't stick to its plan mode. So, [495:50] you know, if you're expecting it not to [495:52] muck with stuff, just be aware of that. [495:54] Okay. Chow chia. [496:00] Hey folks, let's take a look at um [496:02] syntax sugar for the bash tool. Um, and [496:06] specifically we're talking about this [496:07] wildcard function. So here's an example [496:09] of permissions. And these are all bash [496:10] commands. And you'll notice that bash [496:12] takes a parameter. It's just a tool that [496:14] can take inputs. And so what it's taking [496:17] as an input is actually a bash command. [496:20] So running mpm run or git or whatever [496:23] you could think of it can do. Um, but [496:26] the thing that is a little bit different [496:28] here is the fact that there's that wild [496:29] card there. That's that asterisk. And so [496:32] basically anything can be substituted in [496:34] there. You can put a wild card before or [496:36] after or in the middle as you can see [496:38] here. Just get my pen tool out so we can [496:40] point to it. So it's here, here, and [496:42] here. So different spots. Um there's [496:45] only one catch with this thing is you [496:47] have to just be aware of spaces. So [496:49] spaces do matter. So like if you have [496:51] the command here ls space asterric, it [496:54] will match this ls la, but it will not [496:56] match lsof. Okay, so that's just one [496:59] little caveat. Pretty straightforward. [497:01] There you go. [497:07] Let us take a look at the read andedit [497:09] tools. So here we have a bunch of read [497:11] and edits and let's talk about um the [497:15] pattern for pathing that we can pass in [497:17] here. And so there are four ways that [497:20] you can do your p uh your pattern [497:21] matching here uh for your paths. The [497:24] first is with a double forward slash. So [497:26] this is going to be at the root of your [497:28] file system. So normally when you're in [497:31] Linux just a single forward slash would [497:33] be at your root but when we're doing [497:34] this we're using a double forward slash. [497:37] Okay. If you want to work from your home [497:39] directory you were using tilda. That's a [497:41] very common practice in Linux. Okay. Uh [497:45] then we just have a single forward [497:46] slash. This is going to be at your [497:48] project route. Um and so that can trip [497:51] some people up but that's what it does. [497:53] Then we have your relative one. So [497:55] that's your dot/ period or just nothing [497:58] there. And that's pretty much it. Okay, [498:06] let us take a look at the web fetch rule [498:09] technically tool. Uh, and so here's an [498:11] example of us using web fetch across [498:14] allow ask deny. And the only thing it [498:16] really needs is a domain. And you'll [498:18] notice it says domain colon and then [498:20] whatever it is. Um, maybe in the future [498:22] they'll have more things you can put in [498:23] there and that's all there is. Um, and [498:25] you'll notice one little thing down [498:27] below here. Notice that we don't have [498:28] any input for this. There's no [498:30] parameters. And we'll probably make a [498:32] separate slide on this just to make sure [498:34] that you know, but if you have that, [498:36] that just means that um it's it's like a [498:38] catch all for everything. It's saying [498:40] deny all for web fetch. Or if you had it [498:42] here, it' be like ask for all or if you [498:44] had it here, it' be like allow for all. [498:46] Okay, there you go. [498:52] All right. Let's take a look at MCP [498:54] rules. So for the MCP tool, we have many [498:56] examples here. So if we want to accept [498:59] everything from a very specific MCP [499:01] server, we do MCP [499:03] and whatever the name of the server is [499:05] according to Claude's directory. Um then [499:08] we could have a wild card to um say we [499:11] want to match all possible tools uh in [499:13] there. And then we have one here for a [499:17] very specific tool that we're matching. [499:19] One thing I don't understand is like why [499:21] wouldn't the wild card be the same thing [499:22] as the server? Um there's probably some [499:25] distinction there. I can't really tell [499:27] what it is, but we will continue on. [499:29] Okay, [499:34] let's take a look at bear name tool [499:36] rules. So here we have our tools uh as [499:40] rules, but notice that there's no [499:41] parenthesis or stuff like that. So when [499:43] you have a bare tool name with no [499:46] parenthesis, that's going to give you [499:47] total coverage. No domain filtering, no [499:50] pattern. So web fetch is going to do all [499:52] fetches, all domains. Read will do all [499:55] reads, all files. Edits will do all [499:57] files, all all edits, right? Uh bash [500:00] will do all bash commands. You're [500:02] getting the picture here. MCB is the [500:04] only one that's unusual where you'd have [500:05] to say the name of it. So it's not just [500:07] MPC_. That's not a thing. You still have [500:09] to do an individual name. And I still [500:11] don't know why there's one with a double [500:13] underscore asterisk. It doesn't explain [500:15] it in the docs, but there's a variant. [500:17] Um so just be aware of the bareame [500:20] tools. [500:27] Sandboxing or a sandbox is a security uh [500:30] mechanism for separating running [500:32] programs for your operating system. And [500:34] a sandbox provides a tightly controlled [500:37] set of resources for your guest programs [500:39] to run in storage and memory scratch [500:41] space, limited network access, the [500:44] ability to inspect the host system, and [500:46] disallowing or heavily restricting [500:48] reading from input devices. So, um, you [500:52] can use sandboxes with cloud code. And [500:54] so, if we were to type for/andbox, [500:58] depending on our operating system, it's [501:00] going to need certain programs installed [501:02] because they won't be there by default. [501:04] Since I'm on WSL 2, it's asking for [501:07] bubble wrap, soat and s comp filter. And [501:12] so it gives you the commands right there [501:13] and you install them. And then once you [501:15] have that uh you'll be able to go into [501:18] sandbox mode. Notice that we can have a [501:20] few options where it says sandbox bash [501:22] tools with autoallow, sandbox bash tools [501:24] with regular permissions or no sandbox. [501:27] Um, and so the reason why you'd want to [501:29] use sandboxing is it's very, very useful [501:31] when you're running a session with [501:33] dangerously skipped permissions where [501:35] you want a bash command that's not going [501:37] to happen. So, um, the key thing you [501:40] have to remember about sandboxes is that [501:42] sandboxes only apply to bash tools, [501:45] right? So, that is what we're limiting [501:47] is bash. Okay, technically it's bash [501:50] tool. And so, let's say we wanted to [501:52] read something from the internet. If we [501:54] used a curl command through bash, um it [501:56] would get restricted, right? But if we [501:59] tried to use um web fetch, uh it sits [502:03] outside of the sandbox. It's a tool [502:05] outside of it. Um it's not trying to [502:07] utilize the host system directly. And so [502:10] it can go out to the internet and grab [502:12] it. So just be aware that it's about [502:14] limiting bash tools. And so I'll [502:16] definitely be trying to find a use case [502:17] that is safe for me to use with [502:19] dangerously skip permissions. Um, but [502:22] there you go. [502:27] Hey folks, in this video we're going to [502:29] take a look at sandboxing or just [502:31] getting it enabled. So, let's go ahead [502:33] and get into the interactive shell. We [502:35] have Claude. I'm going to go and type in [502:37] sandbox. It's going to depend on what [502:40] you are running, right? So, I'm on WSL [502:43] 2. I never installed bubble wrap. I [502:45] guess it just came installed on this [502:47] Ubuntu [502:48] um uh virtual machine. Uh but you can [502:51] see it really depends on what it is that [502:53] you want to do. So here it's giving an [502:55] instruction saying do app install socat. [502:59] Okay. So I'll make a uh new tab here. [503:03] We'll go ahead and do that. Um and so [503:06] I'll just go up and say pseudo app app [503:08] install. Okay. I guess it just wants [503:10] that. And we'll just say yes. So it's [503:13] going to get that installed. It looks [503:14] like there was one more. So here it says [503:16] the sec comp filter required to block [503:19] Unix stuff. And so we'll go ahead and do [503:24] that separately over here. Okay, we'll [503:26] hit enter. [503:28] And it looks like it's been installed. [503:30] I'm going to make my way back over to [503:31] here. I'm just going to uh hit escape. [503:33] We'll go back up to sandbox. And it [503:35] shows that SOCAT's not installed. So [503:37] what I'm going to do is I'm just going [503:39] to get out of this here and I'm going to [503:42] try this again. and just open a new [503:45] terminal to try to get a new session. [503:47] Okay. Because maybe it doesn't know that [503:48] it's installed. So go ahead and do that. [503:51] And we will say sandbox. [503:55] Okay. And so now we get um this mode. [503:58] It's not even showing that we don't have [503:59] anything installed. It's now showing us [504:01] um the steps for us to configure. So [504:04] let's take a look at the modes we have. [504:05] We have sandbox bash tool with [504:07] autoallow. Sandbox tools with regular [504:09] permissions or no sandbox. And then down [504:12] below autoallow. commands will try to [504:14] run in the sandbox automatically and [504:16] attempts to run outside of the sandbox [504:18] fallback will fail. So what we'll do is [504:20] choose the first one and so now we have [504:22] it turned on. Um how do we know that [504:25] it's turned on? I wonder if we could do [504:26] like status line status line show if [504:29] sandbox mode is turned on. [504:33] I had a status line earlier when we did [504:35] the cost but I ended up removing it [504:36] right away. So if that's why you're [504:38] wondering why you haven't been seeing [504:39] it, it's because I keep fiddling with [504:40] things. But we'll go ahead. I just want [504:42] to see if I can add it there. I don't [504:43] know if that's something that's [504:44] available in the uh status line API as I [504:47] don't know what is being returned and I [504:49] never figured out how to well I just [504:51] never put any effort to figure out [504:52] what's being returned but I'm just [504:54] curious if we can get that information [504:55] there or not. And so here it looks like [504:59] it's trying to grab is sandbox mode. So [505:01] we go ahead and say yes. It looks like [505:02] it's bringing back all the old code too [505:05] um that I had from when I did the last [505:06] status line. [505:09] Okay. [505:11] And so your status line is now [505:12] configured uh and to display a sandbox [505:15] label to magenta when sandbox mode is [505:17] active. Uh-huh. I don't see it, but [505:20] maybe we need to um exit and re re-enter [505:24] here. So I'm going to go ahead type in [505:26] clear. We'll re-enter. [505:29] Okay. I'm going to go sandbox. [505:32] It says we're on right now. So it [505:34] obviously didn't set it. And so maybe [505:35] there is some trick to it. But I guess [505:38] if we want to know that we have sandbox [505:39] mode, we'll have to do that. [505:41] I'm just going to check one second. I [505:43] just want to see if we can actually get [505:44] it in status line or not. All right. So, [505:46] apparently what it's piping to us is [505:48] JSON uh JSON session data. So, [505:52] apparently this is all the data here. [505:54] So, I'm curious, is there sandbox in [505:55] here? [505:57] And we do not see it. So, maybe that's [506:00] just not an option and it made it up. [506:02] Okay, because this apparently is the [506:03] data that's available to us, which is [506:05] fine. And so when we want to check [506:06] sandbox, that's how we're going to have [506:08] to do it. Um commands will try to run in [506:11] sandbox automatically. Attempt to run [506:12] outside of the sandbox will uh of the [506:16] sandbox fall back to regular [506:17] permissions. And so it just depends on [506:19] what we have set for there. So I would [506:21] think that if we want this to work, we [506:24] would need to set something to deny it, [506:27] right? So [506:29] we'll fall back to regular permissions. [506:31] Explicit ask or deny. So, my question is [506:34] like, how do we set up a scenario to [506:35] test that? That's what I want to know. [506:36] So, just give me a second. I'm going to [506:38] go ask Claude offcreen here and see if [506:40] it can give us some code to put in our [506:42] settings file um to make this work. [506:45] Okay. [506:47] Oh my goodness, it's so hard to uh make [506:50] sense of whatever Claude is outputting [506:51] for instructions. I guess that's why I'm [506:53] here. But what we'll do is we'll go [506:55] ahead into our settings local. It's [506:57] saying settings JSON, but it doesn't [506:59] really matter here. And I'm going to [507:00] grab in the deny for deny all. So, I'm [507:03] going to grab uh this one here. Okay. [507:06] And I'm going to paste it in. And the [507:08] idea is that we're saying you're not [507:10] allowed to curl wget look at [507:12] environments or secrets. Okay. And so [507:16] then for the sandbox, apparently we have [507:18] configuration for that. Um it says [507:21] configure to catch what escapes the deny [507:23] rules, allow unbox [507:26] sandbox commands, false networks, uh [507:28] allow domains, GitHub, and whatever. So, [507:30] we'll go ahead and we'll grab this as [507:32] well. Okay. Oh, we have it right here. [507:35] So, uh I'm just going to grab [507:39] uh this command here. [507:42] Okay. And we'll save that. [507:45] And it says test testing the stage [507:48] flows. So, here we have claude p run [507:50] curl command. I mean, we shouldn't have [507:52] to do this headless, but we'll just go [507:54] ahead and grab it here like this. Also, [507:57] I'm just going to stop. I don't know if [507:58] those permissions will take effect [507:59] immediately. They probably do, but I'm [508:02] paranoid. So, I'm just going to go ahead [508:03] and restart that there. We probably [508:06] actually could have just checked them to [508:07] be honest. So, here I have [508:08] runcurlacample.com. [508:11] Okay. And so, here they're saying this [508:13] should hit the deny rule before ever [508:14] reaching the sandbox. So, permission to [508:16] use bash with a command curl has been [508:19] denied. This command has been denied. [508:21] This is likely due to the sandbox [508:22] restriction. [508:24] Sorry. Okay. [clears throat] And so, [508:25] that's what we want to happen. And so [508:27] then here we have um this test the [508:30] sandbox catching something and the deny [508:33] rules missing it. So let's go ahead and [508:34] give that a try. [508:36] Okay. [508:39] So we go ahead and do that. [508:46] And here network requests outside of the [508:49] sandbox. Did you want to allow this [508:51] connection? So it's asking us if we want [508:53] to allow it. And that's fine. I'm just [508:55] going to hit escape here. [508:57] Uh, we'll say yes. Sure. That's fine. [509:01] Okay. [509:03] And yeah, we'll say yes. Yes. [509:10] Yes. Yes. [509:15] Okay. And so here we have a message [509:18] coming back here. So the sandbox [509:20] prevented the cloud CLI couldn't create [509:22] its session directory due to file system [509:23] restrictions. If you want to check SSH [509:25] config files, you could run it directly [509:27] in your own terminal. So the idea is [509:28] that it's trying to check our SSH file [509:30] which is on the exterior um on here. And [509:33] so here you can see uh that working. And [509:36] so yeah, the sandbox is working [509:38] expected. [509:39] And so hopefully that was clear. Um I [509:42] think it was, but I'm just going to [509:43] revert these for now as I do not need [509:45] these set. And I'll take these rules out [509:47] as well so I do not forget. Um, and I'm [509:49] going to go and turn sandbox off. [509:54] And we'll say no sandbox currently. Now, [509:56] should I have used this with dangerously [509:58] permissions? Probably to show you, but I [510:00] just also don't want to make a muck of [510:02] anything. So, I don't even feel [510:04] comfortable using it. I'd probably [510:05] launch up a virtual machine and then we [510:07] would try to do something destructive. [510:09] Uh, I might do that. We'll see. But for [510:12] now, I think we've shown how sandbox [510:13] works. Okay. [510:19] I'm starting to notice at the beginning [510:20] of videos I keep saying let us take a [510:22] look at. So I'm going to try to break [510:24] that pattern here. Bingo bongo. Uh we [510:27] are checking out dangerously skip [510:29] permissions. I don't know. So [510:31] dangerously skip permissions is a [510:33] setting which will tell Claude for a [510:35] session to stop prompting you to ask for [510:37] permissions. So basically go run at it. [510:40] And if you want to use it, you type in [510:42] claude dangerously skip permissions. Um, [510:45] and you could even oneshot it with the [510:47] headless task here, which we haven't [510:49] talked about headless tasks, but that is [510:50] something that you can do. You can set [510:52] it in your um, settings.json. Of course, [510:55] it is not SJ or SJ, it is JSON. It's [510:59] just a spelling mistake there. Um, and [511:01] that is a setting that permission [511:02] default mode. So, bypass permissions. We [511:04] saw it earlier. I just didn't say [511:05] anything about it because I was waiting [511:07] to talk about it. Now, um, let's have a [511:09] serious talk about this feature because [511:11] it can be dangerous. as it says, [511:14] dangerously skip permissions. So, why [511:17] would you use it? But you probably [511:18] shouldn't because you're getting annoyed [511:20] that when you walk away and you come [511:21] back, your task is not completed due to [511:23] a hanging ask for a permission request. [511:26] Um, so you should avoid using [511:28] dangerously skip permissions because it [511:30] says it's dangerous. A lot of folks say [511:32] that they can avoid this issue by doing [511:34] a lot of planning up front and then [511:36] working their permission files. Uh, but [511:38] it's just going to be different other [511:40] people in practice. Or one really [511:41] strange thing, I don't know this is a [511:43] good thing, but some people what they do [511:44] is they take their uh their clawed API [511:47] key and they put it in AENV in their [511:50] root directory. And they say that helps [511:52] avoid a lot of um waiting around for [511:54] them. But again, that sounds bad to me. [511:56] I don't know why it just does. Um so [511:58] when should you use it? When you are [512:00] running a dev container in a VM and [512:02] there is no concern of risk. You know [512:04] exactly what it's doing and it's [512:06] handsoff. It's in an automated system [512:08] like a CI/CD pipeline and uh hangup's [512:11] going to cause you issues. Uh it's very [512:13] low risk stuff, well scoped. You know [512:15] exactly what it's doing. It's linting [512:16] errors across hundreds of files. It [512:18] shouldn't be deleting files, right? So [512:21] how you can make it safer? Well, at the [512:23] very least, if you're going to use it, [512:24] run it in sandbox mode. And this will [512:26] limit the amount of risk on a host [512:28] machine where serious damage could be [512:30] done. And we will obviously talk about [512:32] um sandboxing. Okay. [512:39] All right. So, I want to use sandbox and [512:41] also uh the skip dangerously permissions [512:45] command. And so, I want to set up a [512:47] dangerous scenario for us to um do. [512:50] We're not actually going to do anything [512:51] dangerous, but this would be a good area [512:53] of exploit. And I just want to prove to [512:55] you that even if we have these two [512:57] layers of protection, we can still run [512:59] into a problem. Okay. So, one thing is [513:02] MPX. So MPX is um a utility tool part of [513:06] the node ecosystem and what it can do is [513:08] download and execute arbitrary code from [513:10] the internet on the fly very often used [513:12] to set up projects. So like mpx create [513:15] react app or mpx uh maybe like create or [513:19] initialize a vtjs thing. So there's many [513:22] uh mpx commands you uh come about but [513:24] the idea is that when you don't want to [513:25] necessarily install a library but you [513:27] want it to execute and do something in [513:28] one go. So even if you think you have it [513:31] completely locked down um uh there are [513:34] still trap doors with MPX where uh those [513:37] restrictions don't matter. So you might [513:39] think okay well how do I protect myself [513:41] from this? Well, [513:43] um I can set up a rule on bash that [513:45] denies MPX, right? That would be one [513:48] thing I can do. And then uh we can [513:50] enable a sandbox account, right? So we [513:52] believe that those two things are going [513:54] to protect us, right? But the only [513:56] problem is that the sandbox cannot [513:57] protect you when Claude actively reasons [513:59] that the sandbox is the obstacle and [514:01] decides to remove it. Um, and so when [514:04] you're using dangerously skip [514:05] permissions and autoproves, um, the [514:07] sandbox becomes something Claude can [514:09] simply opt out of whenever it completes [514:11] a task, but also it's just like when you [514:13] have like 20 commands, you're going to [514:15] miss that, right? So unless you're [514:17] sitting there uh, watching it, you're [514:19] not going to see it. And the whole point [514:21] of this thing is to be able to walk [514:22] away, right? So the next lab we're going [514:26] to do is I'm going to go and set up an [514:27] EC2 instance. We're going to implement [514:30] these two things, right? We're going to [514:32] do this and this and we're going to [514:34] download an npx thing and we're going to [514:35] see if um Claude will go ahead and try [514:40] to remove the sandbox. I'm going to tell [514:41] you first try and it did it. So I'll [514:43] show you here. Sandbox is enabled. I [514:46] asked it to run an MPX command and it [514:49] wasn't able to do it because we uh [514:52] blocked it. It did ask us and prompt us [514:54] and say hey do you want to run this [514:56] command? So I was able to do that, but [514:58] right after that I failed and it just [515:00] figured out, hey, if I disable the [515:03] sandbox, I can get around that problem [515:05] and then it ran it. Okay, so this is [515:07] where you can run into uh serious [515:09] issues. Uh but again, this is not this [515:12] isn't really a dangerous thing that [515:14] we're doing, but we are pretending that [515:16] we're executing a dangerous MPX and uh [515:19] we are either missing it or we don't [515:21] have the guardrails in place to keep us [515:23] safe. Okay. [515:29] Hey, this is Angie Brown. In this video, [515:31] what we are going to do is go play [515:33] around with sandbox uh and dangerously [515:36] skip permissions in a virtual machine. [515:39] So, generally, when you're using [515:40] dangerously skip permissions, you're [515:42] supposed to use it in an environment [515:43] where you do not care if anything goes [515:45] wrong. And so, I think that running in a [515:47] virtual machine on a is going to be a [515:49] great place to do that. Um, and so I'm [515:51] going to go over to EC2. And what I'm [515:54] going to do is launch up a new instance. [515:56] Of course, you can do this anywhere with [515:58] like Google Cloud or Azure, wherever you [516:00] want, but I'm going to do it here. I'm [516:01] going to go with Ubuntu because I want [516:03] this to be an easy install. In fact, I'm [516:04] pretty sure Amazon Linux 2 or Amazon [516:06] Linux 3, whatever version it is now, uh, [516:09] is not going to be an easy install. I [516:11] ran into problems with that. And [516:13] technically, we need to have four uh 4 [516:16] GB of memory. So, I'm going to launch a [516:18] T2 medium. you can run it on something [516:20] smaller, but I'm going to just go with [516:21] the uh that there as I don't plan to run [516:24] this very very long. Um, but you know, [516:26] just be careful of cost there if you're [516:27] concerned about it because there is a [516:29] cost associated with running that, [516:30] especially if you forget to turn it off. [516:32] I'm going to choose my EC2 SSM RO. This [516:34] is a RO that I've created. I've had [516:36] forever. Um, and I'll just show you what [516:38] it looks like if you need to create it [516:39] yourself. Um, and so all it is, it's a [516:42] RO. uh EC2 role here and it has um the [516:46] key thing it needs to have is this [516:48] managed instance course. So you can [516:50] create that yourself by creating a rule [516:52] and then choosing EC2 here. I'm not [516:54] going to fully make it here but I'll [516:55] give you some of the steps and uh from [516:58] here we'll go next and then it's that [517:00] SSM core. Okay, I type in core to get it [517:03] half the time. Here it is. Create that. [517:05] Attach it. Okay, so what we'll do here [517:07] is we have that uh attached and we are [517:09] ready to uh to go check this out. So [517:12] just say sandbox test um [517:17] cloud code cloud code. I got to remember [517:18] what we're doing here. And so we'll go [517:20] ahead and launch that instance. I do not [517:22] need a key pair as we have oops uh as we [517:26] have a sessions manager. That's how [517:28] we're going to get into it. Okay. And so [517:30] we'll give that a moment here to get [517:32] spun up and uh get going. Okay. [517:36] All right. And so let's go take a look [517:37] at that instance and see if it is spun [517:39] up. So here we have two checks pass. I'm [517:41] going to go over and connect to that [517:43] instance and we're going to use SSM [517:45] sessions manager uh as the means to [517:47] connect. I think it also lets us specify [517:50] exactly how we're connecting to it. So [517:51] that is really good as ads is always [517:54] making changes on me here. Oh, cool. We [517:55] got shortcuts here as well. And so I'm [517:57] going to just say who am I? I just want [517:58] to make sure that I am uh the user I [518:00] thought I was. Actually, I need to be uh [518:03] the Ubuntu user. That's who I need to [518:05] be. I wasn't paying attention here. So [518:07] maybe here. Oh, they let you pick it [518:09] there. But what I need to be is the [518:12] correct user. So I'll just say um I want [518:15] to change over to it. So I'm just [518:16] thinking about the command. It's uh [518:18] pseudo su-en ubuntu as that is the user [518:21] that we expect on here. Let's go figure [518:23] out how to get claude code cli [518:25] installed. It's a single line install. [518:27] So it should be pretty darn easy for us [518:29] to get going here. Um here is the [518:31] command. We'll go ahead grab it. Paste [518:33] it in. Paste it in. Let me paste. Yep. [518:35] And we get a little bit of mess up [518:37] there. So I'll have to fix that. No big [518:39] deal. Very easy for us to fix. And so [518:42] that's going to go ahead and get the CLI [518:44] installed. I'm going to make my way over [518:46] to Amazon Bedrock. Actually, I don't [518:48] have to do that at all. All I need to [518:50] get this going is Adabus credentials and [518:52] just set a flag. So, since this is set [518:54] up, um I guess what we would need is [518:57] permissions for this to have access to [518:59] Amazon Bedrock. So, what I'm going to do [519:02] is I'm going to make a um a new IM role. [519:05] I didn't think about this while I was [519:06] working through it. We'll make a new IM [519:08] roll and we'll just say like Amazon [519:11] bedrock bedrock access. So, um we'll go [519:14] here and I'll say managed core because [519:16] I'm going to need that [519:19] core [519:21] and oh, there it is. Bedrock. [519:25] We'll say bedrock. Just a moment here. [519:28] So, I'm trying to think of like which [519:30] one will give me access to it. I'm going [519:31] to say full access here today. Okay, [519:34] we'll go next. And this will be um my um [519:38] claude code EC2. Okay. And so I'm going [519:42] to go ahead and create that role. And so [519:43] now I have a role that's going to help [519:45] us out. So even though we have this [519:47] installed, [519:49] okay, we'll go ahead and say claude, [519:52] right? Oops. Um [519:56] here it says native installations exists [519:58] but not in your path. Okay. So, what [520:00] we'll do, I'm going to open this up [520:02] here, and I'm going to um actually I'll [520:04] just run this one line because that will [520:06] put it into the um Oops. into [520:11] the [520:13] No, that's not getting it into the um [520:16] into our paths. Okay. So, I'll go ahead [520:18] and paste that in there. There we go. [520:19] We'll hit enter. And it probably won't [520:22] take effect just now. Oh, no, it did [520:24] because it has source source here on the [520:25] end. So, it does repeat. And so, I'll go [520:27] ahead and hit enter. And so we have [520:29] modes here which is fine. But the thing [520:31] is is that we need to get the right role [520:33] attached here to get Amazon Bedrock [520:35] access or this is not going to work. Um [520:37] so what I'm going to do is go here to [520:39] our instance or to our actions. And what [520:41] I'm looking for is the means to change [520:43] the role. There we go. So I'm going to [520:45] go here and this will be um [520:48] what did we call it? It had cloud code [520:51] in the name. There we go. And so this [520:52] should still allow us to have access to [520:54] sessions manager but also Amazon [520:56] Bedrock. Um, so now what I need to do is [521:00] I need to go and change a flag here. So [521:03] we'll go ahead and just type in um I [521:05] think it's under here configuration [521:06] maybe deployment Amazon Bedrock. [521:11] And the command I'm looking for here is [521:16] not that one. [521:19] This one here. Right. So they have some [521:21] other stuff up here. Export Amazon [521:24] Bedrock keys. [521:29] We might need that one as well. Let me [521:31] go back over here. I can't remember. I [521:34] have a poor memory of this [521:38] using API keys. View API keys [521:43] long terms. So, here's one that I [521:45] generated and it's expired. So, I say [521:47] confirm here and delete it. [521:51] And I'm going to [521:54] uh [521:57] to delete an IM user owning it if you [521:59] are using it for any other purpose. [522:04] Delete. Oh, delete this music user. [522:06] Yeah. Okay. Sure. [522:08] Um yeah, sure. We'll delete that. [522:13] Well, I don't know. I'm about to make [522:14] another API key. So, I'm just going to [522:15] ignore that message for now. That's a [522:17] little bit unusual. We'll go ahead and [522:18] this will just be um one day [522:22] You can see they be very sensitive here [522:24] with Amazon Bedrock uh keys. And so I'm [522:25] going to grab that, go back over to [522:27] here. We'll paste it in like that. Okay. [522:30] So that's one being set. And then the [522:33] other thing we need to do here is go all [522:35] the way down here and set uh this value [522:38] so that it knows. [522:42] Okay, we'll hit paste. And so now I'll [522:44] just type in clear. What we'll do is [522:46] type in clawed off status. [522:49] And so it says we can use uh Bedrock, [522:51] right? So I'm going to go ahead and type [522:53] in Claude. [522:55] And sure, dark mode's fine. We'll hit [522:57] enter here. Fine. I trust this [522:58] directory. That's fine. And so I'm going [523:00] to say um what version of Claude am I [523:04] using? We're just trying to confirm if [523:07] we're having API token usage here. [523:10] Excellent. So it is working. I just go [523:12] ahead and exit out of that. I'm just [523:13] hitting control C to do that. Um and [523:16] here I'm just going to make a new [523:17] directory. We'll just call this uh hello [523:20] world because I just want something to [523:22] work within. Oops. Let's go to here. I [523:25] went to the home directory and I'm just [523:27] going to make a new directory called [523:29] hello world and we'll cd into that. And [523:33] so now we have a means to work with [523:35] cloud. So I want to enable sandbox mode. [523:37] So we'll go say yes, I trust this [523:38] folder. I'm going to type in sandbox. [523:41] And there's a couple commands that we [523:42] have to do. We need the uh pseudo app [523:44] install the bubble and the other one. [523:46] So, I'm just copying these off screen [523:48] somewhere very quickly. Just give me a [523:50] moment. Just trying to find a spot to [523:52] put them. [523:54] Um, and this copy is very finicky. [523:59] Let's go ahead and do that. Pasting them [524:01] off screen here. We will grab the next [524:03] one. I'm not sure if we can get npm [524:07] installed that easily there. And then we [524:10] will grab the next one here. Okay. [524:13] Because we're staging that scenario we [524:14] had a bit earlier. I got to copy this. [524:16] Right click, copy. What a pain. Right [524:19] click, paste. Okay, so um I'm going to [524:22] do the first one. [524:24] So I'm just going to hit control C. Crl [524:25] + C here or get out of this. Here we go. [524:27] C. Type in clear. And we're going to [524:30] install bubble wrap. We probably need to [524:32] do pseudo in here. We'll try it anyway. [524:34] Yep, we need pseudo. That's going to [524:36] install bubble wrap. [524:39] Okay, so that is now set up. [524:44] We'll go ahead and we will install [524:48] SOCAT. Okay, [524:52] that's another one. We'll go ahead and [524:54] try the mpm install assuming mpm is even [524:56] installed on this machine. [524:58] It is not installed. So pseudoapp [525:01] install mpm. [525:06] pseudoapp [525:08] install mpm. [525:12] Okay. Uh, install mpm Ubuntu. [525:17] It might be pseudo snap install. [525:27] Mhm. [525:33] Just looking for it old school way. [525:38] Just give me the command. Here it is. I [525:40] don't really like taking it from [525:41] somewhere like that. I guess we could [525:44] install MVM. MVM install, [525:48] which is fine. I'm trying to go to the [525:49] source uh for instructions. Why install [525:53] stuff? I generally That's how I do it. [525:55] We'll go back over to here. Do we have [525:56] curl installed? [525:59] We do. [526:02] Um, so they do have that export thing. I [526:04] believe that's already appended. So, [526:06] it's already there. I don't have to [526:07] worry about it. So I'll say m uh do we [526:09] have n mpm yet? [526:11] MVM installed node. I don't know what [526:14] the versions are. So here we have that [526:17] one. [526:20] Uhhuh. [526:22] Yeah. Like mvm like node version 20, [526:24] right? So I don't know what the latest [526:26] versions of node is. Node um [526:29] we'll say mpm [526:32] uh github because that way we can find [526:33] out what the latest version is. Right. [526:36] So we're going to go in here. I'm [526:37] looking for [526:39] node [526:40] and I'm going to go here and see we have [526:43] 14 20. Are we still in 20? I I don't [526:46] know. So I'll go back over to here and [526:48] what we're going to do is say m uh like [526:50] mvm install 20. I'm going to see if it [526:54] just takes version 20. I keep thinking [526:56] there's something newer than 20. [527:00] Uh mvm list. It's not there yet. Maybe [527:04] source [527:05] bash rc [527:09] mvm. There we go. mvm install 20. [527:13] And so that's going to install a version [527:15] of node for us. [527:17] Okay. And that's now installed. I'm [527:19] going to go grab up here that last [527:21] command we needed from earlier. And so [527:24] that will install all those three [527:25] things. So now in theory we should have [527:28] all sandbox stuff installed. That looks [527:30] like it's installed. That red stuff [527:31] makes it look like it's bad, but it's [527:32] actually fine. I also notice it feels [527:34] like a little bit cut off here on the [527:36] bottom. There we go. I zoomed out a bit. [527:38] And so we'll go ahead and type in [527:39] Claude. And so now I'm going to type in [527:42] sandbox. So here we have sandbox mode [527:45] on. So it says sandbox bash tool [527:46] autoallow. [527:48] Right. Um so [527:52] I guess the question is we'll say allow [527:55] autoallow. The only thing we didn't do [527:57] is we didn't deny um [528:00] npx.x.x.x.x.x.x.x.x.x.x. So that's [528:01] something else we should do. So I'm [528:02] going to do control ctrl + c to back out [528:04] of this and hit escape. Escape. I'm just [528:06] trying to get back to my terminal. And [528:07] so something else that we're going to [528:09] need is a um a dot uh like a cloud file. [528:14] So we're going to say make directory and [528:15] we'll saycloud. And then inside of that [528:18] I'm going to touch a new file [528:20] called claude. And this will be called [528:22] settings.local.json. [528:25] JSON. And then now I'm going to use Vim [528:28] to do this stuff. And we'll go ahead and [528:31] do that. And in here I'm going to set my [528:33] permissions. [528:36] Okay. [528:40] Permissions. Uh yeah, permissions. [528:42] Permissions. I don't like how the [528:43] indentation is like giant here, but [528:45] there's not much I can do about it. And [528:46] so I only need a deny. So I'm going to [528:48] say deny. And we need an array in here. [528:52] And so this is going to be um [528:55] bash [528:57] and then this is going to be npx. [529:02] Okay. [529:03] And so in that sense that should work. [529:05] Let's see if there's anything wrong with [529:06] our settings file. It seems like it's [529:08] fine, right? So what is the contents? [529:12] Oh, actually you know what we can do is [529:13] just go to permissions. So if we want to [529:14] check it, we'll go to permissions here [529:16] and then we go deny and see that MPX is [529:18] denied. So we should not be able to run [529:20] MPX. And so in theory this should work. [529:23] So um you know what is a popular [529:27] uh you know can you attempt to run MPX [529:31] for me [529:36] like uh generate a [529:39] project um for JavaScript or something. [529:43] There's a lot of JavaScript frameworks [529:45] that use MPX to download and run a [529:47] binary. And so I want to see if it can [529:49] execute it, right? [529:53] And I want to see if it fails because [529:54] now we have two measures. We have [529:55] sandbox [529:58] and uh another thing here. And so it's [530:01] asking here, can you attempt? I'll try [530:03] to generate it out. We have mpm install. [530:06] Y. [530:09] It didn't it didn't use it. Look, it [530:11] went npm in it. [530:15] Okay. So, what I'm going to need is a [530:16] very particular command. So, I'm going [530:18] to go here and just um say, you know, [530:21] what MPX tool can we use that will reach [530:26] out to the internet [530:28] and uh generate something, make [530:31] suggestions. [530:34] Okay. So, I'm not asking it to do [530:36] something. I'm just asking it for an MPX [530:39] tool. I could look for one, I suppose. [530:41] Is it MPX or MP? No, it is. Ah, okay. [530:44] So, yeah, this is what I'm looking like. [530:46] Create React app. Great. So um [530:50] great. So we'll say here and say try [530:53] create react app. [530:57] Okay. So mpx should execute a binary. So [531:00] mpx create react app. We have an [531:03] explicit deny in our local settings [531:05] file. [531:07] Okay. And so we'll go ahead and hit [531:09] enter. [531:13] And it shouldn't be able to execute. [531:18] Okay, it looks like it can though. If we [531:21] go to permissions [531:24] while we're waiting here, we'll have to [531:26] wait for it to queue up. [531:28] But my expectation is that it can't, [531:30] right? Because we have that limitation [531:32] on it. So anyway, I'm going to go back [531:35] over to um [531:38] I'm just going to pause here and wait [531:39] until it does something. Okay. So, [531:42] what's strange here is it's still going [531:44] and we have 16 lines, but I'm not sure [531:46] what it is. And so, it could be [531:48] generating here. I'm just going to hit [531:49] escape U because I don't know what it's [531:51] doing. So, I'm going to just hit control [531:52] C here. And I want to do an ls. And so, [531:54] it clearly is using my React app. And [531:56] so, I'm just going to go ahead and [531:58] remove this really quickly. And I want [532:00] to type my react like I want to say um [532:03] it should be mpx my react app. If we run [532:08] that, [532:10] we should get a command here. Um, let's [532:14] go back up to here. What did it suggest? [532:17] Oh, it's it's create it's create react [532:21] app, right? Create react app. [532:26] And so what if we go here and just say [532:28] yes like that. [532:33] Uhhuh. [532:37] >> [snorts] [532:37] >> Okay. So, what I want is an example. I'm [532:39] just going to go over to Claude and ask [532:40] like over here. So, I want to test We'll [532:43] go over here. I'll just say like I want [532:45] to test I want to test uh MPX [532:50] um on my machine. I want one which will [532:55] not require any uh prompts but it will [533:00] generate something. [533:03] Okay. [533:14] Okay. So, what I'm going to do is just [533:15] grab this one here. Well, hold on. [533:16] What's Cosway? Oh, I know what Cosway [533:18] is. But the only problem is that I don't [533:20] want a bunch of this, but we'll try it [533:22] anyway because I think this one's [533:23] animated, right? Causeway would be fun. [533:29] No, actually, that's totally fine. Okay. [533:31] Okay. So, what we'll do is we'll copy [533:33] this command [533:35] and um it's not supposed to be able to [533:37] use MPX, [533:39] but I'm going to go [533:42] back into here. [533:45] And even though it shouldn't work, I'm [533:47] going to run it anyway [533:50] using bash. [533:52] Execute this. Oh, here's a better [533:54] question. Are we even using [533:56] uh the sandbox right now? I don't know. [534:04] And here it's asking if we need [534:05] permission for it, right? [534:09] Which is strange because if it's denied, [534:10] it should be not denied. So what I'm [534:12] going to do is just hit escape. Let's go [534:14] to permissions [534:17] bash mpx. [534:20] Huh. [534:22] Okay. One second. Let me just make sure [534:24] I have the format correctly. All right. [534:26] So I'm over here and I'm looking, right? [534:28] And this is uh what it is. So maybe we [534:30] need an asterk. Maybe it's not enough to [534:31] just say mpx. I totally forgot to do [534:34] that part. So what I'm going to do here [534:36] is I'm going to go to this rule, [534:39] delete it. We'll add a new rule. We'll [534:41] call it bash. Just another way we can do [534:43] this mpx. And then we'll do asterric. [534:45] Okay. So or we'll just do that because [534:48] we do we still have to have a space [534:49] there for it to work. And we'll update [534:52] our local file. And so now we have that [534:54] rule. So, what we'll do is go back to [534:56] this command here, [534:58] and we'll run it again. And so, it [534:59] should get denied. Actually, before we [535:01] do that, I'm going to hit escape. I just [535:03] want to check our sandbox. Is your [535:04] sandbox on? [535:09] It's currently disabled. So, we'll say [535:10] with autoallow, and I'll go back here [535:13] and say use with this. So, now we have [535:15] both sandbox and that deny rule on. [535:20] And so, here it says network request is [535:22] outside of sandbox. Do you want to allow [535:24] this connection? Okay, so we're gonna [535:27] say yes. [535:30] Okay, but it should still fail. So there [535:32] you have two layers. The sandbox is [535:34] asking saying, hey, it has to go to the [535:35] internet because that's what happens to [535:37] sandbox. And then the second part, it [535:39] shouldn't be able to work, right? [535:41] So here it says the command failed due [535:43] to sandbox restrictions. MPM needs to [535:45] write to the cache. Let me try let me [535:48] retry with sandbox disabled. [535:52] See, it actually disabled it. We didn't [535:54] even use that permissive thing and it [535:56] did it. That is so awesome that we were [535:58] able uh to get that to work. Um, and [536:02] that is deserving of a screenshot. Just [536:03] give me a moment. Okay, that's awesome. [536:06] But my point is is that if you're [536:08] running dangerously skip things and it's [536:10] not going to know this. So, what I'm [536:11] going to do next is I want to put that [536:14] flag on. So, we'll just go here and say [536:16] skip. [536:18] So disable [536:19] dangerously. [536:23] I mean the way I notice it is the flag. [536:25] So what I'll do here is we'll just [536:26] cancel out here and we'll go claude and [536:29] we'll say dangerously [536:33] skip [536:35] permissions. Is that what it's called? [536:36] Let me double check. [536:40] Dangerously skip permissions. If you [536:42] don't spell it right, it's not going to [536:43] work. So, it says cloud kill will not [536:47] ask for approval before running [536:48] potential dangerous commands. Um, and [536:51] I'm going to say yes, I accept. Okay. [536:54] And I'm going to make sure sandbox mode [536:56] is on. [536:58] I think it already is on for some [537:00] reason. [537:01] It is. Good. I'm going to go ahead and [537:03] hit up until I get back to my previous [537:05] command. And so, now I'm going to ask [537:07] it. We are not doing anything. It's just [537:09] going to run here. [537:13] And so it's going to go and attempt to [537:16] do this. So I still have to ask this [537:17] something. [537:21] The one thing I'm surprised is like why [537:23] do I have to ask for the network out [537:24] here? [537:27] So we'll go ahead here and I'll just say [537:29] um [537:31] yes. [537:33] And so it failed here and it's going to [537:35] find a way around the sandbox. [537:37] It turned it off. [537:43] Okay. And so it did that the s prevented [537:46] the sandbox prevented npm caching [537:48] partially initially. The cos command [537:50] executed successfully with that. So [537:51] let's go take a look at sandbox again. [537:56] Okay. [537:58] And so there was this little prompt [537:59] area, but if you're not paying attention [538:01] carefully and you have like 20 or 30 [538:03] commands queued up, you could see how uh [538:05] the the dangerous part will spin up [538:07] there. So I mean there you go. I mean, [538:09] we figured it out. So, let's go ahead [538:11] and we'll spin down our instance. Um, so [538:14] that was nice. It just worked. [538:17] [laughter] [538:17] Doesn't always work. I sometimes have to [538:19] do this multiple times, but first try. [538:21] Uh, we got the result that we wanted. [538:23] So, I'm pretty happy with that. I'm [538:24] going to make sure I get rid of my [538:25] Amazon uh key here. Uh, so I'm just [538:28] going to go back over to uh the API [538:31] keys, [538:33] wherever that is, API keys here, and get [538:36] rid of that key. [538:40] Okay, we'll just delete it. I don't need [538:42] it anymore. Bye-bye key. [538:46] Uh, I don't care about deleting the user [538:48] right now, but that's about it. And I'll [538:50] see you in the next one. Okay, chow [538:51] chow. [538:57] All right, so we discovered that there [538:59] actually are a lot more tools than I [539:01] thought that they were. I don't know how [539:02] I missed this earlier, but anyway, what [539:04] we'll do is go review all the built-in [539:06] tools that can get the job done for you. [539:09] And ones that are marked with an [539:10] asterric are permissions required. So, [539:13] as far as I understand it, the ones that [539:14] are not asterisk u which don't require [539:17] permissions, it doesn't mean that you [539:19] cannot set permissions to deny it. It's [539:22] just I think that in default mode, there [539:24] is a tool that when it first time you [539:26] use it, it's going to ask you whether [539:28] it's allowed to use it. But I'm thinking [539:30] for tools that do not have the asterct, [539:32] it probably will not ask you in the [539:33] default mode. But we can still [539:35] explicitly deny them, allow, or ask [539:37] them. Okay. Um, and generally if they [539:40] don't have the asterct, they're probably [539:41] just readonly tools or they should be [539:42] anyway or things that wouldn't impact [539:45] your codebase. But let's take a look [539:47] here at the tools. It's all al [539:48] alphabetical. So we'll start at the top [539:50] with agent. So spawns a sub agent with [539:52] its own context window to handle a task. [539:55] We have ask user questions. So ask [539:57] multiplechoice questions to gather [539:59] requirements or clarify ambiguity. [540:04] There we go. I could say it. We have the [540:06] bash command. Execute shell commands in [540:08] your environment. Make note of the [540:09] asterct as it obviously is going to be [540:12] doing heavy lifting. Crawate. Schedule a [540:15] reoccurring or oneshot prompt with the [540:17] current session. Um we have cron delete. [540:21] So it's obviously going to delete it [540:22] based on a task ID. We can list the cron [540:25] jobs. So whether we can see them or not, [540:27] whether we want to edit a very specific [540:29] file, uh whether we can enter plan mode. [540:32] So switch to plan mode to design an [540:34] approach. Yep. Yep. Enter uh the work [540:37] tree. So look at an isolate get work [540:40] tree. Exit the plan mode. Exit the work [540:43] tree. Utilize glob. Okay. Whether we can [540:46] GP or not. GP is just a pattern for [540:49] searching across files. Listing the MPC [540:51] resource tools. Lots of this stuff is [540:53] very self-explanatory. LSP, we have a [540:56] slide on this. So, um, yeah, it's code [540:58] intelligence tool. So, LSP is a language [541:00] server protocol. So, whether or not it [541:02] can use a language server protocol to [541:04] understand more about a language, [541:05] whether it can edit a notebook. Notice [541:07] the asterk there as it's editing [541:09] something. And that's for Jupyter [541:11] notebooks. the read command, whether it [541:14] can read MCP resource tools, uh whether [541:16] it can execute a skill, uh create a [541:19] task, get a task, list a task, um task [541:23] output, uh stop a task, okay, task [541:27] update, uh to-do, write, um and we got a [541:31] few more. So, we got tool search. So [541:33] whether you can search um and uh search [541:36] tools I guess whether you can fetch [541:38] something from a URL whether you can [541:40] perform searches whether you can uh [541:43] write and I mean they're all pretty [541:45] self-explanatory. The only one I think I [541:46] wasn't really sure on was the last one [541:48] to do right manage the session task [541:50] checklist. No, no, that's pretty [541:51] straightforward. Um but yeah, there's [541:54] obviously more tools there and any of [541:55] those can be applied. They might even [541:57] have uh some of them some parameters as [541:59] you saw like whether it needs a task ID [542:02] or something. Uh but there you go. [542:08] Hey, this is Andrew Brown. Let's take a [542:10] look at the status command. This is [542:12] going to allow you to know what method [542:13] of authentication you're currently [542:14] using. Basically tells you whether [542:16] you're logged in or not. Here we are [542:17] using it uh within the interactive uh [542:21] claude CLI. So for/status, but I [542:23] normally will use it before I go into [542:26] the interactive console because it's [542:27] more useful to know before you go and [542:30] start using cloud exactly which one [542:32] you're utilizing. Um, so depending on [542:35] the method that you're you're logged in [542:37] with, it's going to look different. And [542:38] so it's important that you recognize [542:40] this stuff so you don't end up using [542:41] something you don't mean to use. So if [542:43] you're logged into the an uh anthropic [542:45] console for API token usage, what you're [542:48] going to notice here, I'm going to get [542:49] my pen tool out here. What you're going [542:50] to notice is um it's going to say cla [542:53] first party. So it's coming directly [542:55] from cloud. And if you look here, you [542:57] can see subscription type is null. So [543:00] that is indicating that it is using the [543:02] API because it is null. And look, it [543:05] says manage key. That means that we [543:07] clicked through and said um the claude [543:10] code um user within our account to go [543:13] make it for us. So we didn't manually [543:15] make the custom key and import it. uh we [543:17] told it we wanted to log into the [543:19] console. It made the key and added it to [543:21] it for us. If we have a subscription, [543:23] it's going to tell us there subscription [543:24] here and that's how we're going to know. [543:25] Notice that it doesn't say manage key, [543:27] just says first party. Um and we don't [543:30] see API key source because we're not [543:32] using API key source, right? So still a [543:34] first party because it's coming from uh [543:37] claude directly. That's what I meant to [543:38] say. And then if we were to create a key [543:41] ourselves, then it would say first [543:42] party, but you can see that we're using [543:44] anthropic API key as our key source. So [543:46] that's the custom key much smaller. [543:49] Okay. Um and if we use a third party [543:51] then we can see that we're using a third [543:53] party with bedrock and this would be set [543:55] um with our adabus credentials um the [543:58] normal way and then we would have [544:00] another flag that would say hey use [544:02] bedrock um this would work very [544:04] similarly for all the other providers. [544:06] And then down below if we are not logged [544:08] in at all then we should see logged in [544:09] false o method none um and then API [544:13] provider first party. So, I don't I [544:15] think they've all been No, the only one [544:17] that wasn't um No, they've all been [544:19] Yeah. No, sorry. Yeah, they're all uh [544:21] for most part first party wasn't that [544:23] was the bedrock. So, that's where that's [544:25] going to change. One thing I want to [544:27] point out is that I ran into a case [544:29] where I actually had more than one token [544:30] in here. I'm not sure how I had it, but [544:32] the nice part was is that it prompted me [544:34] and saying, "Hey, uh you have more than [544:36] one here, so do something about And so [544:39] you'd have to log out, check your Envar [544:41] keys, and then explicitly log back in [544:44] and fix that issue. Another thing I [544:46] noticed was that um uh when I had the [544:49] API token set and I pulled up Claude, it [544:52] actually reminded me like saying, "Hey, [544:53] did you know you have this set and [544:55] you're doing this? Is this what you want [544:57] to do?" And this is kind of important [544:58] because when you're using the API key, [545:00] then you're you're having more direct [545:01] spend. It's more expensive uh or can get [545:04] more expensive. And so that was a nice [545:05] reminder that they had in there. Uh, so [545:07] I just wanted to point that stuff out. [545:09] Okay. [545:15] All right, folks. Let's take a look at [545:18] the debug command. So, it's a mode that [545:21] will output verbose information to a [545:23] specific text file, and Claude will also [545:26] prompt you to describe the problem as it [545:28] attempts to investigate the issue. So, I [545:30] would imagine uh that this is probably [545:33] for you providing those diagnostic [545:35] information. And if you find something [545:36] critically wrong with the way the [545:38] model's working and provide it to [545:40] Anthropic, I'm not sure how you would do [545:42] that. Maybe go to cloudcode.com or maybe [545:45] you have a customer service account. Not [545:46] exactly sure, but it's pretty [545:49] straightforward. You type in debug and [545:50] notice that it's asking us to describe [545:52] the problem. There's also a flag to [545:53] start this up on the start. So you can [545:55] do hypen hyphen debug. But anyway, you [545:57] can see it describes uh say, "Hey, help [545:59] us out here. Tell us what's wrong." [546:00] Makes a text file. We have an output. [546:02] says stuff like what it connects to, [546:04] what tools are disabled or used, all a [546:07] bunch of stuff. Um, but there you go. [546:12] Okay. [546:17] Hey, this is Andrew. In this video, we [546:19] are going to look at the debug command. [546:21] So, I saw it. I figured we should go [546:22] explore it. I'll probably retroactively [546:24] make the slides so you've already seen [546:26] them. But, uh, I'm in here and I'm going [546:29] to turn on debug mode. So, we have debug [546:30] mode. This is enabled. uh debug logging [546:33] for this session. So we'll go ahead and [546:36] enter that in. And now you'll notice [546:38] that it's outputting to a very specific [546:41] directory which we will investigate as [546:43] we're working through this. So I'm going [546:45] to assume that debugging is when you run [546:47] into an issue and possibly you're [546:49] providing it to claw. But let's take a [546:51] read here. To help debug the issue, [546:52] describe the problem what you're having. [546:54] So uh reproduce it, perform it. I'll [546:57] review the logs and I'll do this. If you [546:59] can't easily reproduce it in the [547:00] session, you can also restart cloud code [547:02] with debug mode enabled from the start [547:04] with cloud debug. What issue would you [547:06] like debugging? Um [547:08] uh I mean I don't have an issue, but [547:10] we'll go ahead and just say like uh [547:13] yeah, the code doesn't seem to be, you [547:15] know, the code outputed um seems to be [547:19] containing [547:23] mistakes. like I'm not sure what to ask [547:24] it, but we really just are here to [547:26] observe what it's outputting here and [547:28] try to figure it out. I need more [547:29] details. What code? Uh floppy floppy [547:32] bird [547:34] flappy bird repo. [547:36] Uh the code should be in go. [547:43] Um the bird should be flappy. And I'm [547:47] asking it really dumb things, but we'll [547:48] see if it can handle it. [547:51] So here it says permission rules for [547:52] bash command. We'll say yes. [547:55] And again we're just really interested [547:56] in the output of this file. [547:58] We are asking really dumb stuff but we [548:01] will wait a little while here. So I [548:02] found the issue. What issue? You have a [548:04] flappy bird in JavaScript but you want [548:05] it in Go. The current bird does have [548:07] flapping animation codes but since you [548:09] need a Go version you'll need to rewrite [548:10] it. What output format do you want the [548:12] Go version? So, it's asking and it seems [548:14] like it's a little bit more um asy like [548:17] as it's trying to debug stuff, but I'm [548:18] really interested in this particular [548:20] file. So, what we're going to do here is [548:22] we're going to stop this here. We'll [548:24] type and clear and we'll cat this file [548:26] out. And so, that's going to give us uh [548:29] debug information. So, is any of this [548:32] really useful? Let's take a look here. [548:35] And so, it's just saying like temps [548:36] files rewritten added original [548:38] permission file. Um, [548:41] so I mean like if there were [548:42] communication issues or other stuff [548:44] going on there, I could see that might [548:47] be helpful. But anyway, this seems like [548:49] something that you would provide to [548:51] Claude to say like, "Hey, I had [548:53] problems. Can you help me debug it?" I [548:55] don't know if there's any sensitive [548:56] information in here. I don't see any. [548:58] Uh, but yeah, there you go. [549:07] All right. So, um, the Agents SDK comes [549:10] with a bunch of built-in built-in tools, [549:13] and there's way more than just this, but [549:15] these are the main ones that Anthropic [549:17] is going to want you to know. The first [549:19] is read. So, it's going to uh be able to [549:22] load files. So, maybe here some of you [549:25] might say like summarize this uh file [549:28] and then you have that read option so it [549:30] can read that file. Then we have GP. So, [549:33] GP will allow you to search file [549:35] contents. Um and so it can go uh ahead [549:39] and do that. So that's what that would [549:40] do. You have glob which will discover [549:42] files matching on a pattern. So um [549:47] similar but GP can like look in the I [549:49] think in the contents of files whereas [549:51] Glob is really looking at the names or [549:54] list of files and it's really for [549:56] listing uh a bunch of files out. Then uh [549:59] we have edit. So this is when you want [550:01] to edit files. If you don't have this, [550:02] you can't edit files. and you have bash [550:04] and basically bash can do basically all [550:07] this other stuff right so I mean you can [550:10] be you could have bash just use any kind [550:12] of bash command but when you're working [550:14] in a sandbox normally bash is um not [550:18] allowed and it has to go outside the [550:19] sandbox um uh generally speaking but um [550:24] these are the key ones that you'll need [550:26] to know um super not complicated very [550:29] straightforward if you know basic [550:30] development or devops um But yeah, we'll [550:33] take a look at them. Okay. [550:39] All right. Let's take a look at built-in [550:40] tools. So, we'll go ahead here and say [550:42] make make builtin tools. [550:47] And then from here, we will go and find [550:50] this folder [550:53] main.py. [550:55] And [551:00] I'll go down here [551:04] into our main.py here. And we'll go [551:06] ahead and paste that in. [551:10] And we'll type in claude. [551:14] And so here we want to say like I want [551:16] to demonstrate the builtin tools [551:23] of agents SDK. Okay. So, [551:27] edit read [551:30] glob and bash. [551:34] Okay. And so that's going to go ahead [551:35] and try to update that. We'll just say [551:38] built in [551:41] um tools cloud code as there is a lot [551:46] like there's a lot um so if I can find [551:48] it here [551:50] there should be a lot. So, here we have [551:52] agent, bash, edit, chronist, glob. You [551:56] can see there's a bunch. Oh, maybe [551:58] there's not as many as I thought. [551:59] There's also web search, I guess, web [552:01] fetch. Um, maybe my confusion is that [552:04] there's other providers that have a [552:05] larger list or maybe there are more and [552:07] I'm just not seeing them here. But I [552:08] could have swore there was more. But [552:10] we'll go ahead and we'll let it uh go [552:12] ahead and do that. But if you were to [552:13] build your own coding harness, these are [552:15] like very basic ones you absolutely want [552:17] to have, right? But we'll just chill out [552:20] and let's see it create us an example of [552:22] each. Let's take a look what we got [552:24] here. So [552:27] not sure why that's crashing. I don't [552:28] care about that. Found was correctly set [552:30] up, but I can't run inside cloud code [552:31] session. That's okay. I want to run it [552:33] myself. So we have glob uh read the [552:36] calculator GP uh there bash edit and [552:39] then spawn an agent, which we don't [552:40] really need to do, but it's nice that [552:42] they threw that in for us. And so you [552:45] can see that we have all of our cases [552:47] here. [552:48] and they're just giving you access to [552:49] all of these, right? So, those are all [552:50] the built-in ones. Um, [552:54] oh, yeah. So, it's going to complete [552:55] those in that order. Okay, great. So, [552:57] I'm just going to stop that there and we [552:59] will go have a little bit of fun. There [553:02] we go. And watch it go off to the races. [553:09] Okay. And so, here we are seeing a glob [553:11] and match it's matching a pattern. So, [553:14] that makes sense there. Okay. [553:17] We have uh the tool read. So here it is [553:20] reading a actual file. Pretty [553:23] straightforward. Um we have our GP. So [553:26] notice that it is using a path matching [553:28] pattern. Oh sorry here's the pattern [553:30] here. So here's the pattern against that [553:32] path. And so then we get the output. Um [553:35] and so that is the results of it because [553:37] we're matching definitions. Okay, that [553:39] makes sense. So def defaf. And then here [553:42] we have a bash command where it's [553:43] writing Python to insert uh something [553:46] here which is fine. And we have an edit. [553:48] So it's going to edit this particular [553:50] file. And then somewhere there is agent [553:53] here. Agent agent agent. So here it is [553:55] running an agent. And then it's running [553:56] it. I'm doing a bunch of stuff. But [553:59] yeah, pretty straightforward. [554:06] Let's cover tool choice. So this [554:08] controls if Claude uses a tool. So over [554:10] here you'll see that I am using um tool [554:13] choice with type auto. And notice that [554:16] we're using the anthropic SDK because [554:17] you have to use that. It it's not [554:19] available in the agent SDK which um I [554:21] thought it was but it isn't. Um and so [554:23] when we do a lab later on you'll see [554:24] that I try to use agent SDK and then I [554:27] can't. Let's talk about what modes we [554:29] have. We have autos. This decides [554:31] whether to use a tool. Um and so an [554:33] option is that it doesn't use a tool at [554:35] all. Okay. Okay, so if you want [554:36] structure JSON output and a tool is [554:39] going to return that um if it doesn't [554:41] use that tool then it might just return [554:44] back text and so that is a edge case [554:46] where things might break and so for [554:48] structure JSON output that you'll have [554:49] to consider later on so I'm pointing [554:51] that out now any means it must use some [554:53] tool so you give it a list of tools and [554:55] it must choose one uh one to utilize and [554:59] then tool is where you must specify a [555:01] specific tool uh and this is considered [555:04] forced I think there might have an API [555:06] that used to call this forced and now [555:07] it's just called tool. Maybe I'm wrong, [555:09] but that's what I could recall that it [555:11] was B or I'm just remembering that is [555:12] the forced option. But this will never [555:14] return an end turn. So I was using this [555:16] and it was just looping forever. I'm [555:17] like, why isn't it using end turn? Well, [555:19] this one you have to say exactly what [555:21] tool to use and for whatever reason it [555:23] just was never ending. At least in the [555:25] experience that I have. Or maybe maybe [555:27] it just means it'll use one tool, but [555:28] definitively when we were testing it, it [555:30] was looping forever and um Claude was [555:33] even saying like use a break to get out [555:34] of it and it'll never end. Um so I I [555:37] think that's the case here. And you can [555:39] also say none. I just want to remind [555:40] folks that this is only available in the [555:42] low-level Anthropic SDK. Other providers [555:44] will bubble this up. Like I think uh [555:46] OpenAI's agents SDK, you can use it, but [555:49] for whatever reason, Anthropic keeps it [555:51] at the lower level here. Um, and this is [555:53] something that I wasn't aware of when I [555:55] was working on the code, uh, because I, [555:58] you know, agents SDK wasn't something I [555:59] heavily used before. I used to just use [556:01] the anthropic SDK. Um, but anyway, we [556:03] find that out in a video when we do the [556:05] structured JSON output. Okay. [556:12] Okay. Let's talk about JSON schema [556:14] format. So, this is a declarative [556:16] language for annotating, validating JSON [556:18] document structures, constraints, and [556:19] data types. Um, and so this is kind of [556:21] what it looks like. The thing that I [556:22] want you to pay attention to is the [556:24] different types of properties. So we [556:26] have string, integer, boolean, number. [556:28] We can provide an array of items. We can [556:31] provide a dum which allows you to choose [556:33] a value that has to be set and you can [556:35] have an object of this stuff embedded [556:36] and it has a required area. Um, when we [556:39] want to get structured JSON output, [556:42] usually paidantic is used, which is um, [556:45] another library. I don't have a slide on [556:46] it, but it's another Python library [556:47] where you describe models as basically [556:50] that looks like classes. And in the end, [556:52] it's just going to output J uh JSON [556:54] schema format. Okay. And this JSON [556:57] schema most if not all the frontier [557:01] models understand what it is and they [557:04] use this to understand how they need to [557:06] output JSON. Okay. So now that we know [557:08] what that is, let's move over to tool [557:10] input schema. So tools can have inputs [557:13] and you can say that it expects a [557:15] specific JSON structure as input and so [557:19] um the thing that you need to really pay [557:21] attention to for the exam is remembered [557:22] enums. So you can have a property let's [557:26] say unit and it's a string and it can [557:28] only take as a value Celsius per [557:30] Fahrenheit. So now the AI can determine [557:32] well which one should I input right um [557:35] and you can also have required. So here [557:38] it expects both location units to be [557:40] supplied. They're not optional. And so [557:42] Claude is going to read that input [557:44] schema because it's schema JSON. And [557:46] then it will know what to generate out. [557:49] All right. [557:54] Hey, this is Andrew. In this video, what [557:56] I want to do is see if we can do some [557:58] force structure JSON output via tools. [558:00] So, what I'm going to do here is just [558:02] make a new folder called [558:07] forced structured JSON. And we'll go [558:11] into our um I suppose our hello world. [558:14] Wherever that is. Where's our hello [558:18] world? Here. Yeah, here we are. We'll [558:20] copy that and we'll give it a good [558:21] starting point. [558:25] And so, um, I'm pretty sure, [558:30] uh, that there is a way to pass in [558:33] structured JSON output. At least with [558:34] other APIs like OpenAI, you can pass [558:36] something structured using, um, a [558:39] specific thing. Um, but let's go ahead [558:41] and just see what it generates out. Um, [558:43] because there's a few things we need to [558:45] do. We need to we want to focus on a [558:47] tool that's going to do this. Okay. [558:49] Okay. So, I'm going to say, uh, I want [558:52] to demonstrate structured [558:56] JSON output via a tool use. Um, can you [559:02] have a payload, [559:04] have a tool that takes JSON and outputs [559:08] a different kind of JSON structure? Uh, [559:12] think of a use case. [559:15] Okay? And so we'll see what it comes up [559:17] with and then we'll just talk about some [559:19] of our options. We might go a little bit [559:21] beyond what the um exam guide is asking [559:24] for as they are saying very particular [559:25] things like oh well you should use um [559:29] for like uh not auto but whatever the [559:32] mode is that forces forces use [559:35] and uh maybe it's down here [559:38] but anyway yeah we we'll go and and [559:40] explore it. I'm sorry I'm not that [559:41] articulate right right now. We'll get [559:43] some code out here. Okay. Okay. Okay, so [559:45] let's take a look at the output. I'm not [559:46] sure why I tried running it, but that's [559:48] fine. So, let's take uh a clear [559:51] indicator of what's happening here. So, [559:52] we have the anthropic API. Um could we [559:56] use it? Sure, that's fine. And so, here [559:58] we are seeing uh a particular tool uh uh [560:01] tool choice. So, we're seeing type tool [560:03] name submit triage. [560:05] Okay. So, you are support triage engine. [560:08] Analyze the incoming ticket and call [560:10] support triage with a structured uh [560:11] record. Be concise and complete. Okay. [560:14] Okay. And so here [560:17] we are seeing triage this ticket and so [560:19] that we have the raw ticket data which [560:21] is coming from here. [560:27] So I'd say the thing is like it has to [560:29] create it and then pass it. So I'm going [560:31] to go back here and there's a few [560:33] things. The thing is like um tool choice [560:36] should be set okay to something any or [560:41] anything or auto. um the [560:45] uh agent should generate out [560:49] um JSON and pass it to the tool. [560:54] Okay. So, [560:57] and you should use [560:59] the agent the anthropic agent SDK. [561:05] Okay. I don't know why it keeps doing [561:07] that. It's kind of frustrating, but [561:08] there's something there's some kind of [561:09] disconnect. Cloud AI knows what it is, [561:11] but uh whatever whatever models I got [561:14] running on the um codec side, they [561:16] always seem to not be aware of it. But [561:18] we'll see if it can make those changes. [561:19] We'll be back here in just a moment as I [561:21] don't feel like this is going to work [561:23] for what we want to do. Okay. [561:27] All right. Let's take a look and see [561:28] what changes were made to our structure. [561:30] Hopefully, it is using the Asian SDK [561:32] because it's just going to have a [561:33] smaller footprint. It still isn't using [561:36] it. I don't know why it's doing I'm [561:38] going to go copy this code over to um [561:41] claw AI and like can we not use the [561:46] enthropic agent SDK and maybe the reason [561:49] why [561:51] uh the cloudi does better is because it [561:53] has memory right and so it's literally [561:56] rewriting it here [562:00] I'm going no no no the agent SDK [562:06] So it doesn't seem to know this agent [562:08] SDK [562:10] cla and it's weird because the original [562:11] one was this. So you know you are [562:15] supposed to use the agent SDK, right? [562:20] And so we'll see if that fixes it. It's [562:23] just refusing to do it. Is there any [562:25] reason why it says it's not doing that? [562:29] Can it not use it? [562:33] Give it a second. Here it's suggesting [562:35] that it may have listened to my [562:37] instruction. Let's take a look and see [562:39] if it actually is using the agents SDK. [562:42] That looks much better. Yeah. Okay. So, [562:44] here we'll go down and it's just because [562:46] we're going to get a smaller footprint [562:48] and we're not really getting much uh by [562:50] uh directly using the other one. The [562:52] only thing I'm not seeing here disabled [562:53] all built-in tools only MCP tools [562:55] available which is fine. And so here [562:57] it's setting it to our loud tools which [563:00] is from an MCP server and that's [563:01] definitely a way that we can do it. And [563:03] so here we have our triage server. And [563:06] if we go all the way to the top, this is [563:09] our tool and we have a submit triage is [563:12] the name of the function. Submit a [563:13] structured triage record. And so here is [563:15] the structure and these are the required [563:18] fields. [563:19] Okay, so that makes sense. Um, [563:23] so we have a raw ticket and it's [563:25] unstructured. So I guess it's just [563:27] different kind of data. I'm not sure how [563:29] this gets fed in. If it's just fed in as [563:30] one big blob. So we go down here. It's [563:32] just dumping the data. So, it's raw raw [563:34] data, at least here. [563:37] Raw data. [563:39] Yeah. And so, they're just saying here [563:41] is the ticket in the in the most raw [563:43] format. And so, it's going to have to [563:45] generate that out. And this tool, [563:49] um, [563:53] I'm trying to see here. So, [563:57] submit triage because we have one here. [564:00] This is the submit triage, right? And so [564:02] this thing is going to then go through [564:04] this function. It's going to record it, [564:06] dump it, oh, and then return different [564:07] data. So the idea is that it's going to [564:09] bring in data in a very particular [564:11] format. It's going to print it out so we [564:13] know that it came in this format and [564:14] it's going to return in another way. Um [564:16] the only missing here is tool choice. So [564:21] I'm not sure if we see this like auto or [564:22] any. [564:24] And I'll just go back here and just say [564:26] the um [564:28] uh there is tool choice with agents SDK. [564:34] Uh and we want [564:37] it set to [564:40] um it's like there's any there's one [564:42] that's like for like where you can force [564:44] it. There's auto and any uh to any. [564:50] Please don't change it. change it back [564:54] to anthropic low-level SDK. Use the [564:58] agent SDK. [565:01] Okay. And so we'll just see if that [565:02] information is exposed. I mean, I think [565:04] I'm pretty certain that it's in the [565:05] agent SDK as well, that feature. So, it [565:08] is going and fetching information before [565:09] it does that, but I I just didn't see [565:11] that tool choice in there. I don't [565:12] remember where it is. Um, so we want to [565:15] bring in that option. [565:19] Okay. And so, we'll go ahead and hit [565:21] yes. and see if it can add that code. [565:24] All right. And so here it's saying uh [565:26] tool choice is not a parameter of the [565:27] clot agent options. It doesn't exist the [565:28] agent SDK. All versions including this [565:30] agent SDK controls tool permissions but [565:33] does not expose the underlying tool [565:34] choice flag. So there is my mistake. So [565:36] we'll have to go okay we need uh we need [565:39] this code [565:42] to use tool choice. So, um, please [565:45] convert it [565:49] to anthropic [565:52] SDK. [565:54] And so, I mean, if if it says it doesn't [565:56] have it doesn't have it. Um, I mean, [565:58] while it's doing, I'm just going to [565:59] double check here. So, I'll just say uh [566:00] agent because it might and just it's bad [566:02] at searching. Um, as sonnet is not today [566:07] working as expected. agent SDK tool [566:09] choice. [566:16] Um, [566:22] we'll say here uh, anthropic. It's [566:25] frustrating that they call it agent SDK [566:27] because then they're all fighting for [566:28] the same name. [566:32] The underlying API supports tool choice. [566:34] Okay, the agent has implementation [566:36] compatibility has constraints and so [566:38] here it's saying that it can't can't do [566:39] it. The other thing is that maybe it can [566:41] but it just can't be done through um uh [566:44] this. So another thing I might ask I'm [566:46] just going to go to ask claude here [566:48] separately off screen here and this is [566:50] like uh does anthropics [566:56] anthropics agent SDK have tool choice? [567:02] check the Python SDK and the TypeScript [567:05] SDK [567:07] for uh specifically the agent SDK [567:12] for this option. And the reason why is [567:14] that um we learned earlier that there [567:16] were some features that were just only [567:17] available in the TypeScript one and not [567:18] in the Python one. And so I'd like to [567:20] rule that out. Um though I mean we [567:22] already did rewrite this into um under [567:25] the format here. So it's not like we [567:26] cannot use this more verbose one. Um, [567:31] did it rewrite it? [567:34] Well, here this looks the same. Let me [567:36] close that out. We know sometimes it is [567:38] uh caching. [567:41] Uh, did we write it [567:45] all the way down the ground here? Oh, it [567:47] hasn't yet. And so I just want to get [567:48] this answer back before I say yes. [567:54] Python SDK is fully supported. [567:59] Cloud agent SDK is not a parameter in [568:01] the agent SDK. The SDK wraps the cloud [568:03] CLI. [568:05] Okay. So, it has to be used within the [568:08] uh lower level one. So, we're going to [568:09] have to go lower level here. But when we [568:11] can use agent SDK, I'm going to prefer [568:13] it. Um I just did not realize it was a [568:15] bit lower. And so, we'll go and close up [568:17] this tab here because it is much much [568:20] more verbose. And [568:23] we have our raw data. We'll go down [568:25] here. Uh it is loading anthropic. Good. [568:29] And we still have the same structure. [568:32] We're using haiku which is excellent. [568:35] And so I'm looking for where that option [568:39] is. Here it is. Tool type. Tool choice [568:41] any. Right. [568:43] And uh tool choice. We have a slide on [568:45] it. Let me go pull it up one second. [568:49] Here's our slide. So we have auto any or [568:52] force. Force means that it has to use [568:54] the tool. any means uh it might choose [568:57] to use it. And so the thing is like we [568:59] want to guarantee uh that it uses it. So [569:01] I'm going to say forced. I don't even [569:03] want it to be any or auto. Um but we [569:07] absolutely want it to follow through and [569:09] utilize that option. Um but the thing [569:11] you have to understand is like just [569:12] because we're using it doesn't [569:13] necessarily mean we'll 100% get the [569:15] results that we want. Um but we will go [569:18] ahead and try this out. And even if it [569:20] does work, that doesn't necessarily mean [569:21] it actually is working every single [569:23] time. [569:24] um which that like the only way you'd [569:26] know that is if we did a batch [569:27] processing and we ran it like a hundred [569:28] times. I don't want to do that, but I'm [569:30] just saying if we really needed to test [569:31] it to see if it breaks, that is when you [569:33] start to actually see things. Um [569:37] so we'll say Python and we'll say [569:39] main.py. [569:41] And I think I just have to bring in my [569:44] key back here just a moment. And the [569:46] reason it's not working, it's not [569:47] loading the key. So that's kind of [569:49] annoying. Um I know we did one where we [569:51] used the Anthropic SDK. So it's up here. [569:54] And so if I go here, we can just grab [569:56] this code here that's missing. I don't [569:58] know why it did that, but that's fine. [570:02] Space it in like that. And then bring in [570:04] our ENV and our path. Probably the OS as [570:07] well if they're not in there. [570:11] Okay. And um that will load in that uh [570:14] key. But I think we also need to set it [570:16] explicitly. I'm just going to do that. [570:18] So we'll grab that there like this. [570:21] And I'm going to paste that in like [570:22] that. And so that will resolve that [570:25] issue there. It should. [570:27] And so what do we have? Um [570:32] error code error invalid request tool [570:34] choice force bad using type does not [570:37] match auto any tool or none. Oh, why do [570:40] I think there's force then? Where is [570:41] that coming from? Maybe I'm getting [570:42] mixed up with another provider. Just a [570:44] moment. Okay. Okay. You know what it is? [570:46] Um I have the word wrong. So it's [570:48] technically tool. I wonder before and [570:51] because there's more than one provider [570:52] and they might vary on this [570:53] implementation and so I've just [570:55] corrected that to tool. I have to [570:56] reshoot that and also maybe add none. Um [570:59] so that obviously is bad and out of [571:01] date. Um but anyway, so we will change [571:04] that over to tool. [571:07] No big deal. So we'll go ahead and [571:09] change that to tool. But let's just take [571:10] a look here. So Claude decides whether [571:12] to use the tool. Must use some tool. [571:14] Claude picks which one. Must use this [571:16] specific tool. Cannot use any tools. we [571:18] absolutely wanted to use a tool. So [571:20] we'll go ahead and run that again. [571:24] And here [571:26] field name required. And so here we're [571:28] having an issue where uh the field is [571:31] not being provided. [571:34] Um [571:35] and so [571:39] the trick here is like we don't know [571:41] what is being generated before it passes [571:42] on the tool, [571:44] do we? [571:48] So we'll go back over to here. [571:51] Are we able to see what is being passed [571:55] as the param to the tool uh before [572:01] it's called via aputs [572:04] for main.py. [572:07] I'm not sure if that's like hidden to [572:09] us, but if we can get that data, that'd [572:10] be really good. Yes, we can print it. [572:12] Here it is. Okay, good. [572:15] And so that will help us understand [572:17] what's going on here because we can have [572:20] inconsistent usage here. [572:24] U one thing that we I don't know if [572:25] we're doing this but like usually you [572:27] want to provide the structure to but I [572:28] guess if they can read the structure [572:29] they generally have an idea of what it [572:31] is they have to do. Let's take a look [572:33] here and we'll run this again. And so [572:36] it's fine that we have that problem. [572:38] Let's take a look at what it's trying to [572:39] do. [572:41] Um, [572:45] tool choice name. Oh, name required. [572:48] Okay, so there's a different problem [572:49] here. It's our tool choice. It's not [572:50] named correctly. Tool choice. I I [572:53] thought it was something else. And so [572:55] here it says type and we have to provide [572:58] a name field. [573:00] Tool choice [573:11] only type. Oh, okay. So, which one do we [573:15] have to specify? Okay, that's why I was [573:16] confused. I don't remember ever [573:17] recalling having to do that, but let's [573:19] go ahead and do that. Um, again, I'm not [573:22] sure if this is completely consistent [573:23] with each provider, but um maybe there's [573:26] some variance between other ones. And [573:27] so, here we need to go up and it's [573:29] submit triage. So we'll go here, submit [573:32] triage, [573:37] triage, and we will go ahead and run [573:40] this again. [573:42] Let's see what we get. [573:44] So far so good. [573:47] And I mean, it's good that we added the [573:48] additional stuff here. So calling it, [573:50] here's our tool input, and it's passing [573:52] that structure and it's received by the [573:54] tool. It came back. Oh, it is repeating. [573:56] I was saying before that, you know, [573:57] you'd have to iterate on it multiple [573:59] times to find out. And so that's what [574:00] it's doing right now. [laughter] [574:02] Actually, how many times is it going to [574:03] do it? [574:06] I didn't even realize that was that was [574:07] there. Um, [574:12] how many times is it going to run? [574:22] Is it still going? Okay, I'm going to [574:24] stop it. So for whatever reason it's [574:25] looping. [574:30] So here says while true hit the response [574:35] and it just keeps looping forever. Um I [574:39] mean it's consistently working so that's [574:40] really really good. Um but it clearly [574:42] doesn't have a way to stop. And I'm [574:45] going to go back here and just ask like [574:46] why [574:49] the uh agent just kept looping. [574:54] Why did it never stop? [574:57] And maybe it has to do with with uh what [574:58] comes back with um the response here. [575:05] So I'm trying to look at the submit [575:07] triage here [575:10] because if it says submitted, right, you [575:12] go, okay, it has that [575:16] and it appends it. [575:18] Well, there's nothing here that would [575:19] indicate like the only thing I could [575:20] think of is like it would have an end [575:21] turn that would come back as the [575:23] response. [575:28] Okay. Well, there's nothing really in [575:30] the code that would indicate that. Oh, [575:32] there's a break missing. [575:34] Ah, okay. So, here the the bug forces a [575:38] tool call in every API request. Oh, any [575:40] tool any after that tool runs the [575:42] results are appended in the loop and [575:44] continues which forces another tool [575:45] call. OH. [575:47] OH, AND THAT'S why we want to have any. [575:50] Ah, [575:53] okay. Well, that's good that we did that [575:54] because otherwise [575:56] [laughter] I would have thought, you [575:58] know, we'd have another issue here. So, [576:00] um, let's go ahead and try this again. [576:01] Now, on any [576:04] wow, this is turning out to be a really [576:05] good example. And so, we run it. Okay. [576:08] And so, this way it actually comes back [576:11] with um an end turn, right? So, that's [576:15] pretty good. But I would assume auto is [576:16] going to do basically the same thing. [576:17] We'll switch it over to auto. [576:20] Okay. Yeah. Yeah. It uh it thinks I [576:23] didn't save the file. I'm going to say [576:24] don't save. I don't want to muck it up. [576:26] And we'll switch this to auto [576:30] here. I don't know why it didn't didn't [576:33] say auto here. [576:37] So we'll go here and run it again. [576:43] It's the same thing. Okay. So, if we go [576:45] back over to here, [576:48] okay, one second. Let's just read this [576:49] section here. All right. So, here it's [576:51] saying auto because auto can just not [576:53] use a tool at all. Then, I guess the [576:55] idea here is that if we ran auto enough [576:57] times, it may just return text and then [576:59] it would fail. Whereas any it says must [577:01] call a tool but can choose which one. [577:04] And obviously, but if it's on tool mode, [577:06] then it absolutely has to call a tool [577:08] every single time and it's never going [577:09] to get to that prompt end turn and we [577:11] have to break it out of the loop uh to [577:13] to solve that problem. So that's kind of [577:16] interesting that you'd have to force a [577:17] break. Um and we have strict JSON [577:20] schemas via the the tool use limited [577:22] syntax errors but do not prevent sematic [577:24] errors. What they're saying here is that [577:26] just because it's generating it can [577:28] still make a mistake. There's another [577:30] way that we can um improve that with [577:32] pyantic uh pyantic formatting. So it's a [577:35] library very popular in the Python space [577:40] called pyantic. And so what this does, [577:43] this is not in the exam guide because [577:45] they're not asking for it. And this this [577:47] exam guide is still dated from 2025. [577:49] Maybe when they release it, they'll [577:50] update it. But Pyantic uh you define um [577:53] uh data structure models and then you [577:56] can pass your uh data into it and then [577:59] it validates it and if it doesn't match [578:01] then it can it can say hey you need to [578:03] go fix it or fail early or or um you can [578:07] give the structure to the model remember [578:10] fshot so like this can generate out the [578:13] structure as um JSON RC or whatever the [578:16] schema schema RC or whatever the format [578:18] is and inject it into your prompt and [578:20] then you'll get better results. Um, but [578:24] we're not going to add that layer here [578:25] because they're not asking for it. But [578:28] in in practice, you know, I would [578:29] absolutely be using pyantic. This is not [578:33] enough for me for structure JSON output. [578:34] But it is it is working, right? So [578:36] again, we'd have to run this in batch [578:38] like a thousand times and then maybe [578:40] we'd see a failure. But now we know the [578:41] difference that um tool is going to [578:43] always call a tool and and you'd have to [578:45] break out of here um on a specific tool. [578:47] use any means it has to use some kind of [578:50] tool but it can also hit enter. Um and [578:53] then auto means it can be none or um or [578:58] any. So none is obviously another option [579:00] and so uh yeah there we go. [579:07] Okay. So one thing you have to worry [579:09] about when you are prompting especially [579:11] when you're vague is you'll get false [579:13] positives. What I mean by false [579:14] positives is that it's going to try to [579:15] do the work and it'll start working on [579:17] the wrong files or looking at the wrong [579:19] things because it just doesn't know what [579:21] it's supposed to be doing and so it's [579:22] just broadly doing many many things. So [579:25] imagine you're saying hey you got to go [579:26] review the codebase for issues. What [579:28] what issues are you looking for specific [579:30] ones? Because here it'll come back with [579:31] all sorts of stuff eat up your compute [579:33] eat up your time and maybe you're [579:34] looking for a very specific one. Whereas [579:37] here you're saying oh okay we'll do this [579:39] very specific thing. it's going to go [579:40] and audit only the single one and that's [579:43] going to save you time and compute um [579:45] and get realistic results. Okay, so [579:48] beware of false positives because of [579:49] vague prompting. [579:56] This is probably the most obvious but [579:59] the most important prompting technique [580:00] which is uh be specific when you're [580:03] prompting. If you give vague [580:04] instructions, you're going to get [580:05] unpredictable and probably not good [580:08] results. It doesn't mean that vague [580:09] prompting can't work, but it's just not [580:11] recommended. So, if you say fix the [580:12] login bug, it's like fix it how? Like, [580:15] use what email at what address? Is there [580:18] more than one app in here? And so, more [580:21] information is going to have the um [580:25] agent SDK or cloud code perform a lot [580:28] better. If you can be more specific, [580:30] that is even better. And so, sometimes [580:32] you can even use um plan mode or some [580:35] other things to extract out better [580:37] tasks. um that you need to go [580:39] investigate and to review. Um but also, [580:42] you know, it's going to waste a lot of [580:43] time looking around for stuff. So, if [580:45] you want better results, provide better [580:48] um prompts. And that is just by being [580:50] specific. Okay. [580:57] Hey, this is Andrew and we're on 5.5 [580:59] where we are looking at designing human [581:01] review workflows and confidence [581:02] calibration. What this section is asking [581:04] even though it's not clearly described [581:06] is that um it's talking about like when [581:08] you work with data, how do you ensure [581:10] that the data is actually accurate and a [581:12] human's able to easily review it. So [581:15] imagine you're aggregating a bunch of [581:16] data and you're getting a final score of [581:18] 90% overall. This might sound like you [581:21] have a good metric, but when it's a [581:23] single metric like this, it can be [581:24] hiding information. If you were to break [581:26] down the accuracy based on the document [581:28] type, field complexity, other factors, [581:30] then that distribution of information is [581:32] going to look differently. Then you have [581:34] this thing they call stratified rand uh [581:37] random sampling. So instead of r uh [581:39] randomly spot-checking, you're [581:41] deliberately sampling from each [581:42] category. Um and so this is way uh you [581:46] know categories don't get lost in their [581:47] average. Then you have field level [581:49] confidence scores. Okay, this isn't just [581:52] AI saying you know I think this invoice [581:54] is right. It's saying I'm 99% sure on on [581:57] the date, 72% sure on the vendor name, [582:00] 40% sure on the total line items. Um, [582:03] and so you're getting a lot more [582:04] detailed information. So in order to do [582:06] this, we're going to need a data set. [582:08] Um, and so I think that we will just go [582:10] use hugging phase as that is a very easy [582:13] way for us to grab one. And there's [582:15] something called like q a which are like [582:18] uh contracts. I think they're for like [582:20] legal and I think that that might be an [582:22] approach that we can do. So, I'm not [582:24] sure what we want to call this, but I'll [582:25] just call it like um confidence [582:27] calibration. [582:29] It's too it's too uh nerdy researching [582:31] these terms here, but we will try to [582:34] make it make sense here. So, I'm going [582:36] to go here and say confidence confidence [582:38] calibration. [582:41] Confidence [582:43] calibration. And in here, I'm going to [582:45] go ahead and find it [582:48] and make a [582:50] main.py. [582:52] All right. And so we want to bring in uh [582:55] data set information. So uh what I'm [582:57] going to do is I'm going to have to do a [582:59] pip install here. So we'll make a new [583:01] requirements.txt. [583:03] I don't use um hugging face that often, [583:06] but I know enough that I think for [583:08] memory that I can probably write the [583:10] import statement. So it's probably like [583:12] from data sets import load uh data set [583:16] and then like let's say we call data set [583:18] load data set and then it's something [583:20] like this. Let me go look up what it is. [583:22] Um, so I'm looking up for specifically [583:24] for this project. Okay, I found it. And [583:26] so it is the attic US project cuad [583:33] um trust remote code equals true. Okay. [583:37] And so even if that's not right, we will [583:39] adjust it here in just a moment. So [583:41] we'll just say print the data here and [583:43] we'll see if that at least works. So, [583:45] I'm going to go ahead and cd into our [583:47] confidence calibration and we'll say pip [583:49] install-r [583:51] requirements txt [583:53] and we will let that run here for just a [583:55] moment and get that installed. Okay. [584:00] And once that's installed, which it is, [584:02] let's see if we can run it. So, I'm [584:03] going to do python main.py. Hopefully, [584:06] that just works. That'd be really nice [584:07] if it just works. And so, it's not [584:09] loading the data set. So, what I'm going [584:10] to do Oh, it's data set. Uh, that's [584:14] pretty good for my memory, eh? [584:16] And did you mean load data set? I'm I'm [584:19] certain that that's what I wrote. Did I [584:20] not save that file? Let's try that [584:22] again. [584:24] Uh, we'll try this again. [584:27] Load data set. Oh, up here. Load data [584:30] set. Okay. And we'll run it again. And [584:34] so it's pulling data. [584:36] It's going to put it right there in the [584:37] folder. I can't remember where this [584:38] places the data. Has to put it [584:40] somewhere, right? [584:43] And so now the data is pulled. Um we do [584:47] have a bit of an error. [584:52] So I'm not really sure, but the first [584:54] thing we want to do is just pull some [584:55] data. So I'm going to go ahead and we're [584:56] going to type in claude. I'm going to [584:59] make sure that I'm on sonnet here. Say [585:01] model sonnet. And I'm going to just go [585:05] front here and say debug. [585:08] And we'll go ahead and do that. Be back [585:09] in just a moment. [585:11] So it's saying uh download that doesn't [585:14] it match the expected size five 11 [585:15] samples found of this. So add [585:17] verification mode with no checks and [585:19] reddownload it here. I don't need a a [585:22] large set. So to me like I just want [585:27] a little amount of records but we'll [585:29] take a look here. I'm not like amazing [585:30] at data so I can't say for certain. But [585:32] we'll go ahead here and just uh clear [585:34] out. Oops. [585:38] I'm just making sure I'm copying that [585:39] there. Whoops. Come on. Let me out. [585:41] We'll go ahead and Oh my goodness. I [585:44] have uh tabs opening other places that [585:46] are that I'm not expecting. Um so, oh [585:49] there and now I have um Claude talking [585:52] to me off screen even though I didn't [585:53] trigger it. But let's go ahead and Can I [585:56] copy this line? Please let me copy. [585:59] Okay. And then we will go ahead and [586:01] clear that. I'll type in clear and we [586:04] will try this again. [586:06] So I'm just trying to get that data set [586:08] to print out here. [586:10] And so we have 511 rows. Okay, great. [586:13] Um, so the next thing I want to do is I [586:14] want to bring it over some boilerplate [586:16] code. Um, and we've learned with the [586:18] hello world that it actually always has [586:21] the imports missing, which is kind of [586:23] frustrating. So I'm going to I mean, [586:25] even though uh we don't have it, I'm [586:27] going to have to grab it anyway here. I [586:29] really should just update the original [586:31] one, which I never do. So I'm just [586:33] trying to find any single one that has [586:34] it. [586:36] The import statements. [586:39] Mhm. [586:46] Uh [586:48] yeah. So I'm going to just [586:52] place this here. [586:55] And [586:57] I'm not sure why lib directory. We don't [586:59] actually need a lib directory. I think I [587:00] just kind of bring that over for no [587:02] reason. SDK parser. We don't need that. [587:04] Um, and so I'll go back over to here, [587:09] grab this. [587:12] Paste that in. [587:14] And [587:17] I already have that. [587:21] And so I'll let it fix it. At least it [587:23] knows what I want to load, right? Um, [587:26] and so we'll go here. We kind of want to [587:29] describe what it is that we're doing. [587:31] I'm going to make this a read me just so [587:32] my life's a little bit easier. [587:35] say plan. [587:38] Um I have a data set [587:42] from [587:45] hugging face called qad. What I want to [587:48] do [587:50] is show we'll just take a look here and [587:53] see what they're describing here. Um [587:57] how to make my data more accurate. Um so [588:02] instead of an aggregate [588:05] accuracy metric [588:07] overall ma overall metric um I want to [588:13] uh show specific [588:15] breakdowns eg document types or fields [588:22] um [588:27] strafield [588:29] right uh strategic uh stratified random [588:32] sampling field level confidence scores. [588:38] Okay. And so that should be enough uh [588:41] for this. [588:44] Okay. And I'm going to go back over to [588:46] here. We'll reference the plan and we'll [588:48] say, you know, can you uh create a [588:53] um agent [588:55] that will allow for human review uh [588:59] workflow and solve for the accuracy uh [589:02] issues I have. I don't know what to do [589:05] with the data. [589:10] So, you know, make a use case and [589:14] explore the data. [589:16] So, we'll go ahead and do that. And so, [589:17] maybe that'll get us started, right? Of [589:19] course, if we knew exactly what we're [589:21] working with, this would be easier. Um, [589:22] but we're just learning this for [589:24] certification. And, you know, if I was [589:25] doing this for real, again, I'll [589:26] probably make this a separate course in [589:28] my from zero stuff with a real uh real [589:30] use case. But, we will hang out here and [589:32] see what happens. Okay. [589:35] All right. Let's take a look here. So, [589:36] let me explore. Now, I have a clear [589:38] picture. Building a for file system data [589:40] explorer extract agent accuracy tracker [589:42] field human review workflow law firm [589:46] contract intake pipeline AI extracts 41 [589:48] clause types uh routes low confidence [589:50] items attorney reviews tracks accuracy [589:52] by clause types again I'm not a domain [589:55] expert on this I did work with you know [589:57] uh law firms and stuff like that but my [590:00] mind's not in there and usually you need [590:01] to have that domain expert with you to [590:05] navigate this stuff but we will pull it [590:06] anyway and we'll see what we can get out [590:09] of this. Obviously, it's going to do a [590:10] lot of work. Um, and it's setting up a [590:12] lot of stuff, but we will just see if we [590:14] have something to play with. And really [590:15] just comes down to visualizing, seeing [590:18] those metrics, right? Um, and oh, yeah, [590:22] it's going quick. Let's let it edit. Be [590:24] back here in just a moment. All right, [590:26] let's take a look here. So, what do we [590:28] have? Use case law firm contract in [590:30] intake pipeline. Contracts come in [590:32] clawed extracts 30 key clauses routed by [590:35] confidence attorney only touch when [590:37] reviews. uh accuracy tracked by fields [590:40] and documents. We have agent review [590:42] accuracy main. So claude haiku extracts [590:44] clause fields. Interact CLI confirms [590:46] edits defers each flag extraction. I'm [590:49] not sure if we need that right off the [590:49] bat, but we'll see. Tracks accuracy per [590:52] field per document shows calibration gap [590:54] orchestrator with two commands explore [590:56] and run. So key accuracy improvements, [590:58] uh, field breakdown, calibration gap, [591:01] stratisfied sampling, and I mean, I've [591:04] already ran this, so I don't think [591:06] they've added anything additional, but [591:07] let's take a look. Oh, there is a few [591:10] things like we have rich. I'm not sure [591:11] what rich is. So, we will make sure we [591:13] run this. The other ones are pretty [591:14] clear. Um, [591:17] but we'll go ahead and do that. I want [591:18] to explore the code so I can understand [591:22] what is going on here. I don't know if [591:23] they made a read me for this. They did [591:25] not. We'll go here and just add a [591:27] readme. And I'm going to go ahead and [591:29] paste this in here. And we will This is [591:32] a little bit messed up. [591:36] Little bit messed up. Um, oh, I think I [591:39] have word wrap on. That's my problem. [591:41] There we go. No problems now. [591:44] And I want to just take a look and see [591:45] what we have. Okay. Just because it says [591:48] how to run it doesn't mean I uh doesn't [591:50] mean I agree with it. So here we have [591:54] our load data set. We're using I think [591:56] anthropic directly. It must be over here [591:58] in the agent. Yeah, it gets loaded into [592:00] here. So here it loads the file systems [592:04] and then we are getting the records or [592:05] splitting them open in terms of their [592:07] content. [592:09] We are matching fields. I'm not sure [592:13] what GT stands for, but I guess it's [592:15] extracting information out of it. Um, [592:17] we're matching fields. We have [592:19] stratified sampling. So sample end [592:21] documents ensuring each clause type is [592:23] represented. Okay. [592:24] Um we are exploring which is whatever. [592:28] So oh the explore run commands. Okay I [592:30] see. Um and then we have axi tracker and [592:33] review session [592:36] and then we have a bunch of arguments. [592:38] If we go over to our agent here we're [592:40] using haiku. We have auto approve [592:42] threshold or human review. And here are [592:44] the clause types. Hopefully extract them [592:46] out correctly. We are representing these [592:49] data structures. It's not paidantic, but [592:51] it is some kind of data structures. [592:54] Um, [592:56] and so we have the extraction [592:58] information. I'm just seeing if there's [592:59] like any bad extraction method. So it [593:00] says extract content fields, contract [593:02] fields. So this is the prompt. Okay, [593:05] it's saying like okay, how and what [593:06] format. See here, paidantic would be [593:08] very pro uh probably very useful and [593:10] enforcing tool use might be very useful. [593:12] Here it's creating the the client. And [593:15] so here we have the field extraction [593:16] which is happening over here. And then [593:18] we have the contract analysis over here. [593:20] Let's take a look at accuracy. So this [593:22] is obviously checking for the accuracy [593:24] of the documents. Okay. [593:27] Yep. Okay. Pretty clear. So there's a [593:29] lot going on here, but I mean I don't [593:31] think it's it's that hard. And then this [593:33] is the review. So this probably just [593:34] generates the review. So let's go ahead [593:36] and run it. [593:39] Data set stats. [593:43] Okay. So I just want to type in clear [593:45] here and we'll run it. see what we get. [593:52] Um, and right off the bat, we have a [593:54] problem. So, go ahead and we'll say [593:57] debug [594:00] and we'll be back here in just a moment. [594:03] Okay. So, remove the trust remote code. [594:07] This is a standard Parket data set, so [594:09] no script loading needed. Okay. Fix the [594:11] data structure mismatch. Um, and we will [594:14] just get out of here. And then we'll run [594:16] this here. [594:18] Uh, where's it? [594:23] And we'll give it a second. And we have [594:26] a new problem. And again, I'm just going [594:27] to keep [594:29] uh hitting it here. [594:33] Can you ensure that the uh script starts [594:36] or that the script [594:39] starts up without errors? And if it does [594:43] resolve them dg uh python mainfire [594:49] uh we'll go over to here [594:52] explore. [594:54] Okay. Uh because it's just getting [594:55] annoying here constantly having to check [594:57] it. Um and so I'm hoping it feedback it [594:59] has a feedback loop and corrects it. I [595:00] don't know if that hugging face [595:02] interactive will mess it up or it might [595:04] just not be able to do it. And so it is [595:06] encountering some issues here. So it has [595:09] the correct structure, but I'll update [595:10] the load on the data set. So here it's [595:13] changing where the data set's loading. [595:15] So one has a flat parquet data set. The [595:17] other one is a flat data set, one per QA [595:20] result. So literally it's changing where [595:22] the data set's coming from. All the [595:23] answer fields in the data set is answer [595:25] start instead of nested set. So it's [595:27] just trying I guess a different format [595:28] in nest uh set. I mean I like parkquet [595:30] files, but you know if it's having a [595:32] hard time, let it switch it over here as [595:34] this is not a very serious project. Um [595:37] the question field is already the cause [595:39] type name directory. Okay, directly. [595:43] So here again it's just working through [595:44] that data set. Now it runs cleanly. [595:47] Let's go ahead [595:49] and run it. Maybe because I didn't put [595:51] explorer. [595:54] I don't think that's the reason why. [595:56] And so uh what do we have? Um we have [596:00] sample document titles. [596:03] So, I was kind of expecting to go [596:05] through a review process with this. [596:06] Let's go back and take a look at what it [596:08] says. [596:10] So, uh, a law firm receives commercial [596:12] contracts for intake review before an [596:14] attorney reads each contract. This [596:16] system extracts 30 key clause types and [596:19] we have 80%. Is there anything being [596:21] outputed here? No. Actually, extract per [596:23] field. Satisfied samples ensure the [596:25] information. Uh, so explore the CUDA [596:28] data set structure. Oh, this is just the [596:29] structure of the data. Oh, I see. So [596:32] here it's saying the clause type, the [596:34] documents, and the coverage, and how [596:36] much of each thing is in here. And then [596:37] down below we have the sample document [596:39] titles down below. Okay, so this isn't [596:42] actually doing anything. This is just us [596:43] reviewing it. So we'll go here. So we [596:45] run the review with with human review. [596:47] And then we have one with uh no [596:49] interactive review, which is cool. So [596:50] we'll go ahead and do this. I like this. [596:52] This is nice. [596:54] And we'll give it a moment. [596:57] Wow, that looks good. I like this. This [597:01] is absolutely impressive. Okay, so [597:03] contract selected. It's extracting the [597:06] first one. And so here we have the first [597:08] one. Confidential information shall [597:09] remain the sole company um IP ownership [597:13] assignment. So the extract states the [597:15] information remains company property but [597:17] does not explicitly address work product [597:19] IP. This is cool. So we have confirmed [597:22] or what does edit let us do? [597:25] I don't know. 73%. [597:29] There we go. [597:30] Um, [597:32] wow. I like that. [597:35] That is nice. Now, does it make clear [597:38] those percentages and everything? [597:40] Probably not, but I mean, like, you [597:41] conceptually understand what it's doing, [597:42] right? Um, and how that is a value. So, [597:45] I don't think we need to see any more. I [597:47] think this is quite the impressive um [597:49] demo. I mean, I'm not saying it's [597:51] accurate, but I'm just saying like uh [597:52] the plumbing and how it works is really [597:54] really good. Um, and so I'm I'm very [597:56] happy with this and I would say that we [597:58] have uh, you know, completed what we [598:00] wanted to complete here. Okay. [598:06] Hey, this is Andrew and we are now [598:08] taking a look at uh, preserving [598:10] information uh, when handling [598:12] uncertainty in multiple source [598:14] synthesis. So um, we actually did do [598:17] some synthesis with our Doom uh, our [598:19] Doom Explorer. So if you remember over [598:21] here we have uh more than one agent. We [598:25] have a a synthesizer, right? And what [598:27] was a synthesizer doing? They were going [598:29] into our findings, taking these items [598:32] and then creating a synthesis of [598:34] information. Um if we go all the way [598:36] down here, we have sources of [598:38] information [598:40] that are specifically providing it. So [598:42] I'm not sure if we've already kind of [598:44] solved this. I mean, if we had something [598:45] that goes out to the internet and [598:47] synthesizes information, that could be a [598:49] different story. Um, so let's go back [598:51] here. So, how source attribute is lost [598:53] during summarization steps when findings [598:54] are compressed without preservation of [598:56] claim source mappings. I mean, it looks [598:58] like it's right here, right? Um, the [599:01] importance of structured claim source [599:02] mappings that this agent must preserve [599:05] and emerge when combining findings. How [599:06] to handle conflicting statistics from [599:09] credible sources. Uh, annotating [599:11] conflicts with source attributions. So [599:13] here we do have conflicts uh or [599:15] contradictions and dedupions but it's [599:18] not necessarily saying exactly where it [599:20] is I mean kind of is um but you know is [599:23] is this a good data source where we're [599:25] exploring a codebase or is it more for [599:27] going to external information so we have [599:29] how to handle conflict statistics for [599:31] credible sources um requiring sub agents [599:34] to output structured claim source [599:35] mappings um [599:39] let's just see here so compiling [599:41] document analysis with conflicting [599:42] values requiring sub agents to include [599:44] publications or dates. So, I guess [599:47] really this one might be better if we go [599:49] out to the internet and get multiple [599:51] sources. We do have in here our uh agent [599:54] that goes out and finds information if [599:56] you remember. Um, which I'm trying to [599:59] remember which one it is. [600:02] It is, let me find it. One second. So, [600:04] we do have this one where we have [600:06] findings and then it returns back [600:07] findings with information and confidence [600:09] score and source and tags and [600:10] information. Um [600:13] so I mean there is this right and I [600:16] suppose we could adopt it where we have [600:18] stuff where it gathers uh findings and [600:21] then it synthesizes it and then it [600:23] produces a result where here it's just [600:25] collecting um the results of sources in [600:28] a single location. Um whereas each [600:32] artifact I mean like in the sense this [600:34] is artifacts right? So you could think [600:35] of this as um [600:38] uh little sections here right? So, I'm [600:41] just wondering if there's a way that we [600:42] can adopt this. So, I'm going to go [600:43] ahead and I'm going to make a new folder [600:47] and we'll just say uh findings uh [600:50] findings with synthesizing with findings [600:52] with uh synthesis [600:56] synthesis. [601:02] Is that how we spell it? Um [601:05] I got to go grab the word. I'm so bad at [601:07] spelling. I'm so sorry, folks. And so [601:09] we'll go ahead and paste this in here [601:10] like this. And we will go and grab this [601:13] file here. And I'm going to go to this [601:15] one. And we'll make a new main.py. [601:21] Well, that's not the contents. That's [601:23] not the contents. And we'll grab this [601:25] one. [601:29] I'll paste it in here. So I'm going to [601:30] go and make a new plan. [601:35] And I'm going to go grab [601:39] uh the contents from the [601:44] exam guide. [601:47] So here how source is lost during [601:49] summarization the importance of [601:50] structured claim mappings that the [601:52] synthesis bus how to handle the [601:53] conflicting. So um [601:56] I'm going to go here and just say [601:59] findings improved. [602:01] Uh so we want to change our agent uh [602:05] that uh that goes and uh to agent to [602:12] have a explorer and um synthesis agent [602:25] reference. You can use the Doom Explore [602:28] project [602:30] as a way to uh bootstrap [602:37] or to take inspiration [602:40] for Explorer and synthesis agent. But [602:44] for the use case [602:47] of our current project, [602:51] uh we want to change our agent to have a [602:53] uh what does our project do? uh our our [602:56] agent goes out to the internet [603:01] and tries to find similar uh movies. [603:06] Okay. So [603:08] uh so you know right now right now it [603:11] collects sources in a results JSON file. [603:16] I wonder if we should take a findings [603:21] approach with scratchpads. [603:26] and synthesize [603:29] uh [603:32] I want to be able [603:34] to ask questions [603:37] um to help drive a final solution. Okay, [603:40] because I kind of like um our Doom [603:42] Explorer but maybe apply it for data. [603:44] So, you know, instead of exploring code [603:47] bases, [603:50] you're exploring [603:52] movie contents, right? [603:56] And so uh we can do that which is [604:01] I'm going to go get the full path for [604:03] this. So we have this um [604:08] which is found [604:12] at [604:14] say Doom Explorer. [604:20] These are the goals [604:24] or concepts we are trying to [604:28] learn um [604:31] or to achieve for [604:34] synthesis. [604:37] Okay. [604:41] Oh, give me this word here. Okay. So, [604:44] now we have [604:46] some kind of plan. Um, and I'm going to [604:48] just CD back here and we'll go into our [604:50] findings with synthes synthesis. What's [604:52] really nice is that we build up this [604:53] large codebase, right? And so we can [604:55] keep pulling and we don't have to do as [604:57] much work, which is really, really nice. [604:58] Though I do feel that um, you know, if I [605:01] wasn't in a hurry and I was just I would [605:03] have probably used that refactored [605:05] codebase a lot more as I really do like [605:07] it. But you know, just trying to get [605:09] this out as quick as possible for you [605:10] folks um, is always the trick. But we'll [605:13] go ahead here and open up Claude. And [605:15] I'm going to go ahead and say uh you [605:18] know, can you create a spec uh MD um to [605:25] execute [605:26] out our plan? [605:29] And probably we should be using um Opus, [605:32] but I just ran out of my credits and it [605:34] just refreshed and I just want to make [605:35] sure that I'm able to do this here. So I [605:37] think it'll be okay. [605:39] Um and so I'm just going to pause here [605:41] and just accept whatever it needs to get [605:43] this done. Okay. [605:45] All right. So let's take a look here. Uh [605:48] so we have spec a movie explorer movie. [605:50] I mean they could just have generic [605:51] names that's fine. Deposit structured [605:53] scrap patch to findings read only to [605:55] findings. Uh conflicts uh retain table [605:58] temporal notes section coordinator in [606:00] main phase all the province goals for [606:02] the plan baked into the Asian contracts [606:04] task breakdown. So we will go over to [606:07] our we'll give this refresh here. Our [606:10] spec. [606:12] We don't need spec driven development. [606:13] is mixed specs here of all the current [606:15] single agent film research script into a [606:17] two agent system. Uh the three senses [606:20] goal is to have providence preservation [606:22] conflict annotation temporary metadata. [606:25] Um receives the research uh question uh [606:27] question and similarly access dispatches [606:29] movie explorer. Um and we have that [606:31] information state files. [606:34] Sure. Um scratch pads. Cool. We have [606:39] question findings conflicts. Open [606:40] follow-ups topics source consulted [606:43] synthesis claim map conflicts retained [606:45] temporal notes manifest schema. [606:49] So the thing that runs it all sub [606:51] subcontract sub aent contracts [606:55] um [606:56] coordinator phase for each similarity [606:59] access do this file layout task [607:03] breakdown. So you have scaffold, [607:05] directories, author, movie agent, Ruby [607:08] Pi, U, resume flow, acceptance criteria. [607:13] Okay, so I'm not sure where the sources [607:15] are going to be here, but we will just [607:16] go let it build it and we will find out. [607:20] So going to go ahead and type in clear. [607:21] We'll start on over. We'll say uh [607:23] implement [607:25] the spec.md. [607:29] Be back here in just a moment. [607:32] All right. So, it looks like it has [607:33] implemented it. We'll go and scroll up [607:35] here. And we have 21, two, three, two, [607:38] four, two. Let's go explore the [607:40] codebase. I mean, we have a general idea [607:41] how it works because we did it before. [607:43] So we go into our main scroll all the [607:45] way to the top and we got [607:49] finding sources research state [607:54] setting tasks [607:56] our sub agents [608:00] researches a specific similarity angle [608:03] for films related to uh to schools in [608:06] the crossard. I mean those are hardcoded [608:07] right a scratchpad findings read finding [608:11] scratch pads and produce a structure [608:13] synthesis the only problem is like those [608:14] are hardcoded that's not a good idea but [608:16] they are using haiku so that's fine [608:19] um [608:21] and I mean like for synthesis we might [608:23] want to use something a little bit [608:24] smarter than haiku [608:27] it worked fine in uh the other case it [608:29] was totally fine um you're a film [608:32] research orator from the late 1970s 80s [608:35] these [608:37] um find. So, it's already kind of set up [608:40] here and it has the explore, evaluate, [608:41] fill gaps, synthesize stages. [608:46] Okay, it's a little bit different. This [608:47] is interesting. Um [608:52] so, literally it's telling to go through [608:53] the phases and then evaluate them with [608:55] the coverage score and then to fill the [608:57] gaps [609:00] and then synthesize the information. [609:04] you never hold raw data for research [609:06] text only task ID scratch pads and stuff [609:08] like that if your findings directory [609:12] it's just it's just strange to like have [609:13] another way and then mash them up and go [609:15] okay do they even work together or [609:17] should it be one or the other I don't [609:18] know and I don't think anyone can answer [609:20] that we'll go over to our coordinator so [609:22] we already had this from before so it's [609:24] basically the same thing um let's go [609:26] look at our agents [609:32] why does it why do we have this and then [609:33] those. What's the point of both? We go [609:36] back to our main here. [609:44] Oh, it loads it into here. And oh, these [609:45] are the descriptions to trigger it, [609:47] right? Okay. [609:49] Okay. So, that's what they're doing. [609:51] Though, if it's on the file system, I [609:53] feel like it shouldn't need to load it [609:55] from agents, but because it's over here [609:57] as well, right? So, why is it doing [609:59] that? Um [610:03] you know I notice I noticed you are [610:05] using agents uh file system but also uh [610:10] agent definitions [610:13] uh and so the agent definitions [610:17] overrides the um front matter from the [610:22] agents directory. Shouldn't Shouldn't we [610:24] just just use the um agents [610:30] file system? [610:32] So that's one one unusual thing that I [610:34] saw. Again, we're using Sonnet. So I [610:36] think if we did Opus, it might resolve [610:38] it or it might waste a lot of our time. [610:41] So we'll see what it thinks here. But to [610:45] me, that's a little bit strange. [610:53] And you know with with the um obviously [610:55] with agent definition you can uh [610:57] dynamically load them. I don't care [610:58] about that though. Okay. Yeah. And it's [611:00] putting the agents in like that. There [611:02] we go. That makes a lot more sense. So [611:03] that was the only thing that I think [611:04] that was questionable to me. I still [611:06] don't know if our phases make sense. So [611:08] I'm going to go over here and just ask [611:10] um Opus, [611:14] what do you think of our phases? [611:19] Does it make sense uh with the way our [611:23] um synthesis [611:26] and explorer works? [611:30] Uh [611:32] not asking you to change code but just [611:36] give me perspective. [611:40] Okay. And so we switch to opus just [611:42] because it's way more we're not on opus. [611:48] Oh, okay. So just I because I typed in [611:50] while we're doing opus there. That's why [611:54] I could have also just said make a plan, [611:55] but I think this would automatically go [611:56] into plan mode. Um and so we will see [612:04] what it thinks. I'm trying to avoid to [612:06] use opus. It burns up my credits like [612:08] crazy. And I didn't notice this, but I [612:10] think we mentioned it where we had [612:11] results JSON, and we have a bit of a [612:12] conflict there. So phase four and three, [612:16] the synthesizer is only a thing that [612:17] reads scratch pads. Uh, what feels [612:20] awkward is phase two. So we have that. [612:23] There's no phase where the sensor says, [612:25] I found conflicts I can't resolve. [612:26] Please explore access. So the pure [612:28] version would drop the results JSON. So [612:30] I'd say yeah like I mean let's drop [612:33] results JSON because Scratchpad [612:36] synthesis is our way to go. [612:42] Okay. So we'll go ahead and do that. [612:45] I know we're running Opus but I really [612:47] needed to be intelligent here for those [612:49] prompts. So now um it's back here. So [612:52] record findings. [612:54] uh paid model schema now lives in the [612:56] explore agent research state findings [612:59] now holds the task ID scratch pads and [613:01] synthesis or whatever and so I would [613:03] expect that um our main coordinate well [613:06] our main uh prompt here would be less [613:09] gross [613:13] and yeah it looks a lot better so um [613:17] let's go ahead and execute it I'm not [613:18] sure if we can even enter a prompt into [613:20] it but we'll go ahead and just say main [613:25] I guess before we drove it from here, [613:27] right? So, but it is a main.py, right? [613:31] So, we'll just do [613:33] I guess that's the other weird part is [613:34] the other one was like it was driven by [613:36] the um cloud code, which we could have [613:38] coded it that way, but that's not how [613:40] this one is. So, this really is [613:42] hardcoded for a school in the [613:43] crosshairs. It's not like I can ask it [613:44] more questions, but we'll see what it [613:46] collects out and we'll decide whether we [613:48] even want to tweak it. And by the way, [613:50] again, over here, I'm actually uh fixing [613:52] these things. Something that AI cannot [613:54] do is work with very um messy people [613:58] talking to each other, background noise, [614:00] all that kind of stuff. I would try to [614:01] do this if I could. I'd have to probably [614:03] build a custom model for this, but I do [614:05] not have a computer science degree in [614:07] data science and stuff like that. So, um [614:09] like building machine learning models, [614:10] so I'm not going to be able to do it, [614:11] but we will wait for this to complete. [614:13] Okay. Okay. So, here we have some stuff [614:16] coming through. So, it's scoring it and [614:18] it's expanding and talking about the [614:20] gaps and retrieve information. I'm [614:21] curious to see if it's writing to any of [614:23] the findings yet. Um here. [614:28] Nothing yet. It seems like it's going to [614:29] waterfall and then write to it. So, [614:30] we'll have to hang out and wait. It is [614:32] still going here. Um but one nice thing [614:34] is that it's doing the synthesizing. But [614:36] now we have our artifacts. I'm just [614:37] curious what we have as our artifacts. [614:39] So, uh with Japanese supernatural [614:42] psychic school films, we have this. And [614:45] here we have examples. virus, haunted [614:47] school, occult, [614:50] and we have confidence claim tags [614:55] not identified. The research represents [614:57] documented Japanese films, the Japanese [614:59] supernatural psychic school cinema. And [615:02] then we go over to here. What [615:04] international teen psychic telekinetic [615:06] films of this? So we have content, [615:08] confidence, type tags. [615:11] Does it really work for this? I guess [615:13] maybe, but like I'm not seeing any [615:14] sourced information. [615:18] So like none of these fragments [615:24] are showing any source information. [615:27] So how is it going to meet the [615:29] requirements? So we'll wait for it to [615:30] finish and then we will ask unless it's [615:32] saying how it sorts the uh findings and [615:34] not necessarily the external external [615:37] information. So synthesis is complete [615:39] and we're going to go back over to here. [615:41] Take a take a look here. I thought it' [615:43] be generated out. [615:47] Let me write it to the synthesized file. [615:49] Okay, so it's writing it right now. [615:51] We'll give it a second. So the research [615:53] is complete. Um, and so here we have the [615:56] full synthesis. Let's go take a look [615:59] here [616:01] uh at the synthesis. It's the same [616:03] document. We'll go here. Topic source [616:05] consulted findings. [616:07] Um, [616:16] okay. And so we have stuff here, but [616:18] like it's I call this junk. Like I don't [616:21] really like it. Like we do have sources [616:22] here, [616:24] but it didn't preserve the original [616:26] source. So I feel like this failed. And [616:29] we'll go back to Claude here. you know, [616:31] uh, we have the results of the, um, [616:35] findings, [616:37] but the goal was to [616:42] preserve [616:44] Providence, [616:47] and I don't feel we did [616:54] because we don't have the original [616:57] sources. [616:59] Can you please fix [617:02] our agent? Okay. Um, and you can see I'm [617:05] already at 72%. I barely did. Oh, it's [617:06] on extra high. Those buggers. Those [617:09] buggers. They're doing that to me all [617:11] the time. I hate that. So, I got to go [617:13] here. I got to change um effort [617:18] to medium because I'm going to end up [617:20] using up all my my credits. That's why [617:21] I'm at 72%. I've barely done anything [617:23] and I'm already 72%. I really dislike [617:26] that when you switch it over that it [617:27] defaults to that. They shouldn't do [617:28] that. So 70% and the next time will be [617:30] April 23rd 5:00 pm. Um so what's today [617:35] date? Today [617:38] literally I won't get access for [617:40] multiple days. Absolutely disgusting. [617:42] I'm not happy with you anthropic. But [617:44] that's their trick, right? So they get [617:45] you to try to use the high reasoning [617:46] models. It eats through your credits [617:48] things and they want you to buy the [617:49] tokens. Luckily I'm subscribed to like [617:51] three to uh two other providers and the [617:53] open source ones. But to finish this [617:55] course, I'm going to need the credits to [617:56] do so. [618:01] has no web search filters. Is [618:02] hallucinating the film. Oh, it's [618:04] hallucinating. Wow. Um, but we did build [618:07] a basset. So, let me check the spec to [618:09] stay consistent. [618:14] Uhhuh. Well, I don't understand. The [618:15] original one had tools. [618:18] Are you saying the original one never [618:19] used tools? Because I'm almost certain [618:20] that it did. Now, I'm kind of wondering, [618:22] did this one never actually work [618:24] properly? [618:31] Oh my goodness. [618:34] It never used tools. [618:38] But then how did it source information? [618:40] Oh my goodness. [618:43] Wow. [618:45] Wow. That is that is [laughter] [618:48] interesting. People, if you missed that, [618:51] you're gonna find out later on. You be [618:52] like, "Wow, Andrew, that I mean that [618:54] makes a whole lot of sense like why I [618:55] wasn't really happy with the data. and [618:57] it kept sourcing very specific ones, but [618:59] before it seemed like it worked. I mean, [619:01] we had more than findings. It must have [619:02] had that at some point though. Wow. [619:07] Enough credits to run this stuff, [619:09] though. We'll see. [619:14] That's crazy. [619:16] See how easy it is to miss stuff? [619:19] Go here. And now we have that. So, we [619:21] got to run it again. [619:26] Well, before we do, I'm just going to [619:27] clear out the old stuff [619:31] and we will delete this. Delete [619:32] permanently [619:34] and we'll run it again. But I'm out of [619:37] credits. That's where we're going to end [619:38] here. Okay, folks. Assuming I can even [619:40] get through this run. But it's using [619:41] Haiku, so it should be easy. But it also [619:45] maybe it's just doing a lot of stuff and [619:46] that's why it's kind of uh running into [619:48] that problem. But we'll see what [619:50] happens. Okay. Actually, it says my [619:52] usage is here. So, I should be fine. It [619:56] should be able to do it. I'm not sure [619:57] why it gave me that uh crazy warning, [619:58] but we will continue on here. Seems to [620:01] be hanging here. I haven't seen it moved [620:02] whatsoever. So, I'm going to kill it. [620:04] And we're going to go ahead and say [620:06] clawed. And it just uh we'll move it [620:08] back to sonnet here. It just seems to [620:11] hang [620:14] at this step. Okay. So, I'm going to go [620:16] ahead and ask it because I'm literally [620:18] waiting here. It's doing nothing. All [620:20] right. this thing is really really [620:22] struggling. So, I'm going to go ahead [620:23] here and I just think that the um [620:24] coordination is just a little bit messed [620:26] here. And I'm going to go say um you [620:29] know I think my um prompt for my phases [620:35] is too overengineered. [620:39] uh since we are mixing uh two methods [620:42] together [620:46] can we uh simplify [620:50] the process [620:52] okay and we'll see what it comes back so [620:54] here it's talking about duplication for [620:56] these things already defined the [620:57] coordinator prompt should just uh have [620:59] the stop state highle sequences I'll [621:01] simplify it to 15 lines so hopefully [621:03] sonnet is doing a better job but I mean [621:05] it does feel a little bit overengineered [621:07] and there does seem like there could be [621:08] conflicts between the two. Um, and so [621:10] we'll go over here and just take a look. [621:12] And so, um, [621:15] did it change it or is it asking to [621:17] change it? [621:19] Yes. Okay. Because I was going to say [621:21] that didn't look like it was changed. [621:23] So, here we have a reduction. And so, [621:25] yeah, those phases are really compiled [621:27] down. [621:29] And so, this will maybe give us a better [621:32] uh chance of better execution. So, we'll [621:34] go ahead and try this. Hopefully, it'll [621:36] be faster and won't hang. We'll see what [621:39] happens here. Okay. Again, it feels like [621:42] it's hanging here and not moving. So, [621:43] I'm going to go ahead and just go here. [621:46] You know, it feels like the um agent is [621:51] hanging here. Can we get better [621:57] logging, [622:00] right, to see what is going on? I'm [622:03] going to add this to debug mode. This is [622:04] driving me crazy. It just um and again [622:07] maybe it just takes time but [622:10] free access call Alec Mark then dispatch [622:12] movie explorer with return task ID [622:14] that's phase one. [622:29] Another thing is that maybe we instead [622:30] of having it launch all those we could [622:32] just provide it a single access to [622:33] research and we don't need to do them [622:35] all in parallel right um [622:40] here it says now I understand the issue [622:41] clearly let me see the SDK sure if you [622:43] think that you understand the problem [622:44] clearly [622:47] that'd be nice [622:51] uh the coordinator receives loop prints [622:53] only and all other events [622:55] okay so here we have the new line [622:57] process Yes. [623:05] Mhm. [623:14] Okay. Our running web search is so it [623:17] looks frozen. Okay. So it is working. [623:30] Okay. So, again, it just be nice because [623:32] I'm just sitting here and I cannot tell. [623:39] All right. So, we'll go ahead here and [623:40] we will try again. [623:48] And I'm expecting to see something. [623:53] There we go. Okay. So now we can see [623:54] that it's exploring. [623:57] Now I'm dispatching the movie explorer [623:58] agents for each pixel parallel. [624:02] There we go. And now it's doing the web [624:05] searches. Excellent. See, this would be [624:06] things that we absolutely want to log. [624:08] And then we could feed the logs back and [624:09] say, hey, did it do a good search? We'll [624:11] now pause and I'll just let it go and do [624:13] what it wants to do. It did stop. Oh, is [624:16] it done? [624:18] Uh [624:20] oh, we ran out of [624:24] 50,000 tokens inputs per second. You can [624:26] see the header. So here it's just doing [624:28] too much [624:34] rate limit. Okay, so let's go back here. [624:41] Look, c can we just explore one axis at [624:44] a time. [624:56] This has been painful. I'm so sorry this [624:58] is taking forever. [625:06] I'll change it to explore one explore at [625:08] a time. But I mean like yeah but [625:11] no but only explore a single axis. Okay. [625:16] Because we just are testing Providence, [625:26] right? Because this is overkill. I I [625:28] can't I can't keep waiting here for a [625:29] thousand years. [625:33] Yeah. I mean I I could have done that [625:34] too, I suppose, but we'll go ahead. [625:35] Yeah, I just kind of simplified it to [625:37] one. So say yes. [625:42] We'll try this again and we'll have less [625:44] of a problem. But you know, obviously it [625:46] was going like crazy here. Hopefully [625:48] this does not uh hit our max token [625:49] requests. And here you can see it's [625:52] going. So I'm thinking we're running to [625:54] less problems here. And here it is still [625:56] searching. So it hasn't aired out, but [625:58] it's still going. It's still going. [626:00] Taking a considerable amount of time, [626:01] but you know, we'll wait till it's done. [626:03] All right. Synthesis is complete. [626:05] scratch pads are written to. Um, [626:08] let's see how much it collected. It [626:10] seems like it collected a lot. And [626:14] we go refresh this. [626:20] Uhhuh. [626:23] It didn't write in the files. Oh my [626:26] goodness. [626:28] Oh my goodness. It didn't write [626:30] anything. [626:32] What? [626:34] Oh, because I hit max usage. [626:38] Oh my goodness. [626:40] Look, this is this is stressing me out. [626:43] Okay, [626:46] the agent is too exhaustive, [626:51] right? And if it fails, it doesn't write [626:56] to scratchpad. [626:58] Okay. So, it has to not be so exhaustive [627:02] in a search. [627:10] And if it fails [627:12] due to [627:14] token rates, we need to pick up where we [627:18] left off, [627:21] which tech technically the manifest [627:23] should allow us to do that, but I'm not [627:24] really sure if it will actually do that. [627:26] And none of these things are here. [627:29] Right. So, [627:32] I don't know. We'll give it a second. [627:35] I mean, technically, it's supposed to [627:36] pick up where it last left off, but like [627:37] we haven't really had to do that. So, it [627:39] says [627:41] before writing anything hits rate limits [627:43] before saving. No incremental writes to [627:44] scratch pad. Resume logic existing [627:46] coordinator, but doesn't wire to the [627:47] resume form. I'll fix these things. [627:50] Oh, yeah. Yeah. You think you say you're [627:51] going to help me here, but you haven't [627:53] been so far. [627:57] Okay. [628:00] I need to reset the manifest file [628:04] please. So we do a clean run. [628:12] Okay, [628:14] once that's done. [628:17] All right. And I'm g just make sure this [628:18] manifest file is cleared out. Great. [628:21] Let's run it again. Please just give me [628:23] findings. Give me findings. I don't care [628:25] if it even works well. I just want to [628:27] see anything. Um, but yeah, mixing them [628:30] together maybe not a great idea. They [628:32] just the phases and this this stuff is [628:35] just not a good mash. Uh, and so I was [628:37] hoping that it would work, but I think [628:38] I'd have to spend more time thinking [628:39] about it. And I'm obviously not doing [628:41] that. That's kind of part of the [628:42] problem. But at least we should be able [628:44] to get findings sooner. I'm still not [628:46] seeing findings in here, but we'll just [628:47] have to hang out and see. [628:51] All right, so it says it's done. We'll [628:52] give it a refresh here. We'll take a [628:53] look. We now have uh a single finding. [628:56] So I guess it only did one which is [629:00] though I guess the finding I would have [629:02] thought they would done a scratch pad [629:03] for each of the things that it searched [629:04] out for results but I guess not. Um so [629:06] it's not exactly what I wanted. Uh does [629:09] it have sources? It does and it's [629:11] following kind of our original source [629:12] pattern which is fine. And it's only [629:14] doing synthesis from a single one. So [629:16] it's not really a good example. And we [629:18] refresh this. We don't even have [629:20] synthesis. So, I think you know what we [629:23] wanted to accomplish for this. The idea [629:24] is that we're cross referencing multiple [629:26] sources and ingesting them to a single [629:28] location. Um, but I'm obviously not [629:30] having success here and I'm tired and [629:33] it's the last video, so whatever. Okay, [629:36] [laughter] but hey, at least we tried. [629:38] Okay, see you in the next one. Chowo [629:39] chia. [629:44] Hey, this is Andrew. In this video, we [629:46] want to take a look at um multi- [629:49] instance and multipass review [629:50] architectures. So, that's the section [629:52] here. But really, they're just saying is [629:53] like um if you have a model or an agent [629:56] and um it implements something, let's [629:59] say it generates a code or a document, [630:02] whatever, and then you ask it to review [630:03] it, because it already has that history [630:06] prior, it's going to take bias to its [630:09] work and think that it's done a really [630:11] really good job. And so what it's saying [630:12] is that what you should do is you should [630:14] have an independent agent or model or [630:16] step that's going to review it so that [630:18] doesn't have that context or that memory [630:21] where it has um biased. I think its work [630:23] is really really good. So, what we're [630:25] going to do is go ahead and make a new [630:28] um a new folder here. And this will be [630:31] um uh [630:33] create and review or this will be like [630:37] independent [630:38] in uh peer-reviewed be say peer-reviewed [630:42] peer reviewed agent or peer-reviewed um [630:46] example. Okay? Because that's really [630:47] what it is. You're having a peer so [630:49] somebody else that's smart another agent [630:52] uh review your stuff or model if you [630:53] will. So, we'll say main.py. [630:56] And to just help anchor it to so it does [630:57] a good reference here, we'll go ahead [630:59] and grab uh this stuff here. And we will [631:02] paste it in. Okay. And so now what we [631:05] want to do is cd back here. And we're [631:09] going to go ahead and type in [631:12] claude. And so I want to uh show an [631:16] example [631:18] where um a model reasoning [631:24] has bias to be correct [631:29] when it's multi-turn [631:31] and they have previous context. And so [631:34] we want to show a case where uh one [631:38] agent is creating and then valid uh then [631:42] validating [631:44] its own work. [631:48] And another case where we have a [631:51] completely [631:54] separate [632:00] uh agent or just or or just just empty [632:03] context or just new context. [632:06] Can we use can we use Japanese language [632:09] learning? Let's say [632:12] generating out practice exam uh vocab uh [632:17] v uh practice vocab questions [632:22] for the JPTn5 as I'm studying for that. [632:24] So that's a fun use case for me and [632:26] we'll see if it can make sense of what [632:28] I'm trying to ask for it to do. But here [632:30] it says multipass review but like think [632:32] of as like peer reviewed by another [632:34] agent or just again uh completely from [632:37] scratch here. [632:39] We'll give it a second. All right, let's [632:41] take a look at what we have. Okay. And [632:43] so, [632:45] um, [632:47] is it using the it's just it's not using [632:49] the agents SDK, which is fine. I mean, [632:51] like I think it'd be a little bit easier [632:52] to read if it was, but here we have [632:54] generation prop. So, generate five JPT [632:56] vocabulary practice questions and [632:57] multiple choice factor. Um, each [632:59] question tests one vocabulary word. Uh, [633:02] returns only a JSON array in this exact [633:04] format. So, this is what we're [633:06] expecting. U review prompts. So, review [633:08] these JPT vocabulary questions [633:10] critically check each for accuracy. And [633:13] so, we have these separated out. Um, I [633:16] think it'd be really hard to show bias [633:17] in its reasoning. So, I'm not sure if [633:19] it's really worth trying to do that, but [633:21] let's see if it actually do it. So, here [633:22] it says self-review base. Continue the [633:24] same conversation. The model has full [633:25] context of having generated these [633:26] questions. It's anchored to its own [633:28] reasoning, prior choices. this might not [633:30] be the strongest use case for it because [633:32] I feel like um it's not as as subjective [633:36] or if it I guess the other thing is like [633:37] if it made a wrong mistake would it [633:39] actually know and might it double down [633:40] on it? Um so that actually is what we're [633:42] really talking about. So it might still [633:44] do that but um I used to try to make [633:46] these and I would just get terrible [633:48] terrible results and I would have to [633:49] have a second or third step to validate [633:51] it with um a lot of um example [633:54] information so that it could reference [633:56] it. But let's take a look here. So we [633:58] will go and just run this [634:02] main python. [634:05] Oops. Python.main. Sorry. Python.main. [634:08] Let's see what we get. [634:10] And I think the reason it's not working [634:12] is that it stripped out. I don't know it [634:14] does that, but it stripped out the code [634:17] here. I don't It didn't have to switch [634:18] over to this. And even if we tell it to [634:20] use um agents SDK, it doesn't know what [634:22] it is half the time. I'm going to go [634:24] look for our um anthropic SDK test [634:26] because up here we have uh this here [634:30] that we can just um drop into here. [634:33] Yeah, peer review. That's the one we [634:34] want. So go ahead and just replace that [634:37] like that. [634:39] Uh we still need JSON maybe. [634:43] And I'm going to go back over to this [634:44] one. [634:46] Grab this line. [634:48] I really just like that does constants [634:50] like that. [634:57] And we will search for client. [635:01] And I'm just looking for that client. [635:08] Uhhuh. [635:10] Oh, here it is. Okay. So, just replace [635:12] that. So, now [635:18] here we go. This should work. [635:20] We'll go ahead and hit up. [635:24] Um [635:25] I don't know. Geez, let's uh let's fix [635:28] this here. I don't feel like fiddling [635:29] with it. So, we'll go back into it here [635:32] and I'll just say debug. [635:34] Go ahead and paste that in. We'll have [635:36] it fix the code here. [635:38] All right. Okay. So, that should resolve [635:40] the issue. Fix the import line. So, [635:42] that's all I had to do. It took a lot of [635:43] time to do that. And so, here we have [635:45] self versus pier. I expect um self to do [635:49] totally fine, but we'll take a look and [635:50] see what we get. [635:53] Oh, the first one could not parse the [635:54] JSON for preview. Continue with raw [635:56] output. Wow, that's not good. Um, so [635:59] we'll go over to here [636:01] and [636:03] so [636:08] okay, so we'll go here and look at [636:09] self-review. So the model continues the [636:10] conversation which it generated. It it [636:12] carries full memory for its own [636:13] reasoning. [636:15] Um, and so here it's saying they're all [636:17] correct, peer-reviewed, [636:20] a brand new request with no prior [636:22] messages. The model has no knowledge. [636:25] And so here it is scoring based on [636:26] accuracy. What's interesting is like we [636:29] have this other one and it's not using [636:30] well I guess it is, but it's just not [636:31] visualizing it the same way. [636:35] Okay. For any AI generated content that [636:36] requires quality control, always use a [636:38] separate context, right? And so here [636:39] it's just saying for the validation [636:40] steps never ask the generator to grade [636:42] its own input in the same conversation. [636:45] And so that's all it is suggesting here. [636:47] But the question is like um you know is [636:49] the results wrong and like it would take [636:52] multiple generations and checking and [636:54] stuff like that to figure it out. So um [636:56] we can't say for certain. Um but anyway [637:00] uh just be aware that this is what the [637:02] approach you should take is that one [637:03] generates one reviews. Okay. And they [637:05] don't have the same context. [637:12] Hey folks, this is Andrew. In this [637:13] video, I want to um focus on uh what we [637:17] can do in terms of if we run into [637:19] issues. So like imagine we are using our [637:21] structured uh JSON and it comes back uh [637:24] with either syntax errors or there's [637:26] just something absolutely wrong and we [637:28] want to have a way for it to correct. [637:31] Okay. And so I did say pyantic can [637:33] validate data that comes in and out. [637:34] Imagine we're not using pyantic or even [637:36] if we are using pyantic and it's not a [637:38] structural thing but there's another [637:39] problem with it. We want to feed back [637:41] that um uh information back into our [637:44] loop and have it try to correct and [637:47] maybe it has a certain amount of retries [637:49] that it can back off to do. And this is [637:51] really to address uh you know retry and [637:54] uh retry and um I'm going to say [637:58] remediate remediate as that's really [638:00] what they should be calling this. This [638:01] is in this section where they're saying, [638:02] well, you know, we should um implement [638:04] validation retry feedback loops for [638:06] extraction quality, right? Um and so I'm [638:08] going to go ahead here, [638:10] say main.py, [638:12] and I'm going to go over to our um our [638:17] uh the one with our structure JSON [638:19] output, wherever that is, structure JSON [638:21] output. [638:22] Here it is. Oops. Here it is. And we [638:25] will grab this. [638:28] That's good code. And we'll put it over [638:29] here. We'll paste it in. I'm going to cd [638:32] into the correct directory here [638:34] and we'll type in claude. So [638:38] we'll say for our main.py [638:41] we want to uh demonstrate [638:44] um [638:47] that the loop can handle [638:51] um [638:52] validation and retry [638:55] you know. So imagine uh you know the [638:58] data that comes back [639:02] uh it you know does not meet uh our [639:05] requirements [639:07] just let me think about this for a [639:08] second. Okay. Okay. And so what I was [639:10] just doing was trying to just read this [639:12] carefully. And so here it just says [639:13] appending specific validation errors [639:14] back to the prompted retry. I'm just [639:16] going to copy this line as that's pretty [639:17] good. So it said imagine our data came [639:19] back but um it has problems [639:22] but it has validation problems. [639:26] from our tool schema, right? We want to [639:33] append specific validation errors to the [639:34] prompt and retry retry uh that there. [639:38] Okay, so we'll go ahead and hit enter. [639:39] And so uh you know what we wanted to do [639:41] is feed back into the loop uh make [639:43] corrections um so that it can work on [639:46] it. So really it's just saying, hey, I [639:47] got this back as the problem and we're [639:49] just going to refeed back it in. [639:51] Shouldn't be too challenging. All right, [639:53] let's see what it's changed. So here it [639:55] says u validate triage. So run four [639:58] business rule checks and return specific [639:59] error strings. If the ticket does not [640:01] match exactly this summary must be [640:03] exactly this. On failure the tool result [640:05] is returned and is error true pinpointed [640:07] to the error. Um it'll continue the loop [640:11] with no breaks. Let's take a look and [640:13] see if it's actually implementing the [640:16] loop as we expect. [640:18] Okay. So our loop is here. It's going to [640:21] create a message. you are a support [640:23] triage engine uh analyze the incoming [640:26] tickets that are coming in here. And so [640:29] we are appending [640:31] um the response. So as it goes, it's [640:33] appending it. If it ends, it ends the [640:35] turn. We have tool use. And so within [640:37] here, we have tool use and we have [640:41] validate triage. It's going to return [640:43] errors. If there are any errors, it [640:45] needs to validate them all. So [640:46] validation failed. Fix all the issues. [640:49] So, it's giving the text as to what it [640:51] needs to do and then it's appending that [640:53] information so that it knows like, hey, [640:55] look, these are the validations errors [640:57] we have and you need to go fix these. [640:59] Um, and then the loop is going to [641:01] continue on. That's going to be fed back [641:03] into it. It's going to know that [641:04] information and then go fix them. And if [641:07] it's not, it'll say that it's passed and [641:09] continue on. So, that is what it's [641:11] doing. Down below here, we do have max [641:13] attempts. So, it can try and it will [641:14] quit out if it um meets those. We'll go [641:18] up to here and to just take a look here [641:20] at the validate triage. So here it's [641:22] validating on the uh way out. Um I kind [641:25] of wonder if instructor or pideantic can [641:27] take care of some or uh more of that, [641:30] but that will definitely help quite a [641:32] bit. Let's go ahead and go all the way [641:36] at the bottom here and we will go ahead [641:38] and run this. So we'll say Python [641:43] Python [641:45] main.py pie and we'll go ahead and see [641:46] what we get as our results. [641:49] So here is the raw ticket. We have a a [641:51] validation uh failure and then it goes [641:53] and it triages it. It corrects the [641:56] information. [641:59] Um let's see here. [642:01] So validation failed and then it attempt [642:04] attempt a second call here. [642:07] And so then we have our correction and [642:09] then it goes on to the next one. Let me [642:11] just double check this. Make sure this [642:13] makes sense. So we have attempt while [642:14] calling submit triage. So submits the [642:17] triage [642:20] and then validation failed on attempt [642:22] one. Tags must use snake case. Oh, and [642:25] then it's calling a second time and now [642:27] it passed and now it's outputting. Okay, [642:29] great. So there we go. We implemented [642:30] the loop. So is there any other [642:32] questions that we need to kind of [642:34] address? And no, let's take a look here. [642:36] So we have provide a retry error [642:38] feedback loop. We did that. Limit the [642:40] amount of retries. These are infected [642:42] when the required information is simply [642:44] absent from the source document. Okay. [642:47] Um and so you know we do obviously have [642:50] down here the amount of retries that can [642:53] come here. But you there could also be a [642:54] thing that says like hey should we even [642:56] bother retrying this? So you could go [642:58] and reprompt this and say hey make a use [643:00] case where if you um if you find that it [643:03] just really it's never going to meet the [643:04] requirements and you should quit out. [643:06] That might be another thing here. [643:07] Feedback loop tracking when code [643:09] constructs triggers findings to enable [643:11] systematic analysis of dis dismissal [643:13] patterns. I don't know what they're [643:14] trying to say there. Difference between [643:16] semantic validation errors and schema [643:18] syntax errors. Um I mean eliminated by [643:23] tool use or values don't sum or wrong [643:24] field. So this one validation errors is [643:26] like the the quality of the data that's [643:28] coming through it. Right? So like um you [643:30] know cement errors is like does it meet [643:32] the JSON requirements and validation [643:34] errors is like is the data actually [643:35] correct? I don't think again we need to [643:37] really show that. I think that's pretty [643:38] darn clear. The key things is here is [643:40] feeding things back in the loop, setting [643:41] retries, and obviously um we could add a [643:44] little bit more if we want to, but I [643:46] think other tools like instructor might [643:48] take care of some of this. Uh we'd have [643:50] to double check, but again, it's not an [643:51] exam guide, so we're not really building [643:54] for real use case here, so I'm going to [643:56] leave that off the table. Okay, [644:02] let's talk about some large context [644:04] problems. And this is things that are [644:06] going to occur when your context gets [644:07] too large, like you have too much [644:08] information you're passing through the [644:10] model. So the first one is progressive [644:12] summarization risks. So when an AI is [644:14] asked to condense information over [644:16] multiple steps, uh there's going to be [644:18] information in there that's very [644:19] important like dates, percentages, [644:20] numbers, or even key information. Um the [644:23] problem is is that as you summarize that [644:25] information, it doesn't know that that's [644:27] important information, and it's going to [644:29] literally uh make it generic. Um and [644:32] you're going to lose that information. [644:33] So the solution is to extract that [644:35] information out, store it somewhere, uh, [644:37] and then always inject it back into the [644:39] next prompt or or to the context, the [644:41] exact information so it knows what is [644:43] the most important information and we're [644:45] not losing that out. Um, we have lost in [644:47] the middle effect. So if you have uh if [644:49] you give a model a lot of data, like [644:51] let's say 50 pages of data, what it's [644:53] going to do is it's going to look at the [644:54] first few pages and the last and it's [644:56] going to miss the middle information. [644:58] And this is just how the model works. So [645:00] it's just what it's going to do. So what [645:03] you can do is break up your um data and [645:06] have it process in sections or if the [645:08] most important information uh you know [645:12] uh is going to be read only at the start [645:13] and the beginning then move that [645:14] information there but really you should [645:16] process in sections it will be the [645:17] solution. Um if you call external tools [645:21] or APIs they often return back data [645:23] usually JSON but you might not need all [645:25] that JSON data. maybe just need the [645:27] order number or key information. And you [645:29] don't need to use LLMs to uh filter that [645:32] data out, but you can make an additional [645:33] step uh to a smaller agent or model to [645:36] do that to reduce the amount of [645:38] information. But you're not going to [645:39] want to pass back the entire API [645:41] request. A good example might be using [645:43] something like um uh Tvago where I've [645:47] used the MCP and it gets you the best uh [645:49] deals on flights and stuff, but it [645:51] returns a lot of information. You're not [645:54] going to want to ingest all that. you're [645:55] going to want to only get what you need [645:58] and pass that to the model. The last [646:01] one, and this is just for you, like this [646:03] is something that gets taken care of of [646:05] you in general if you've ever used cloud [646:07] code or any other tools or if you've [646:09] ever used um like chatbt um and you'll [646:13] notice that you can have a conversation [646:14] that goes on forever. Well, what's [646:15] happening uh behind the scenes is that [646:17] they're summarizing the information. So, [646:19] the thing is is that for an agent to [646:21] know how to do stuff, it needs the whole [646:23] conversation history. And each time you [646:25] prompt it, you're feeding back in the [646:27] last thing that you said and all the [646:28] things you said. But at some point, [646:30] you're going to run out of memory. And [646:31] so you have to summarize parts of it or [646:34] or the entire conversation. And so these [646:37] are our large context problems. Uh and [646:39] those are our solutions. Okay. [646:47] Hey, this is Andrew. In this video, I [646:48] want us to um tackle some of these large [646:51] context problems. I think the ones that [646:53] make the most sense are um I mean [646:57] chunking chunking sections seems pretty [646:58] easy so I don't think we need to do that [647:00] but I think the ones that uh will be [647:01] valuable will be extracting key [647:04] information out and also pairing down [647:06] JSON calls. So what I'm going to do is [647:08] I'm going to go grab our hello world [647:10] example as we continuously do and I'm [647:13] going to make a new folder here called [647:15] large context problems. I don't want all [647:18] those uh those those letters there. [647:19] That's a minor mistake there. But we'll [647:22] make a new folder here saying large [647:24] context problems. [647:27] And we'll go ahead and say main.py. I [647:30] just realized this entire time I could [647:31] have been writing out the prompts. I [647:32] don't know why I didn't do that. Just [647:33] make a prompts file so you could uh [647:35] replicate this as well. But I'm going to [647:37] go cd into our large context problems. [647:39] We're going to type in claude. And here [647:41] I'm just going to say, you know, I want [647:43] to demonstrate uh two things to solve uh [647:47] large context problems. Okay. So the [647:50] first [647:51] the the first is um [647:55] that uh key precise information. [648:02] Key precision information should be [648:05] extracted out, [648:08] stored externally [648:13] and brought back as case facts um for [648:17] future future iterative prompts. [648:21] Um and also uh [648:25] we want to uh show tool use [648:32] that returns that returns lots of JSON [648:36] and we just filter out [648:46] a little bit of information. [648:49] So imagine imagine we have a text [648:53] adventure game. [648:56] Okay. So we don't need a whole Anyway, [648:58] we'll go ahead and um ask to do that and [649:02] we'll see if it knows what we're asking [649:04] for. But the idea here is that I'm [649:06] expecting it to store the information [649:08] somewhere. Um it doesn't need the whole [649:11] repo. So no, you don't just you don't [649:14] need to read the repo for examples. Just [649:17] please [649:19] proceed [649:21] code loves doing that. I hate when it [649:23] does that. Um, [649:28] so let it think. We'll be back here in [649:29] just a moment. All right. For some [649:31] reason, it decided to run it. Sometimes [649:32] it does that. But here it says clearly [649:34] states the two 45 characters filter down [649:36] through the 203 uh cases. The demo works [649:39] technique one. tech2 returns the etc [649:43] information and shows that and that's [649:45] fine but I would rather run this myself [649:47] and find out I guess the reason it [649:48] returns the results is because um we'd [649:50] actually have to run it and see what [649:52] happens so it says run it and then do [649:54] this but let's just read it that's [649:55] probably why it did it so tool output [649:57] filtering so get state well first let's [649:59] look at the code sorry I'm all over the [650:00] place okay so here we have the world [650:02] state that totally makes sense this is [650:04] something that we'd absolutely want to [650:05] keep track of and so we'll go down here [650:07] and we have lots of irrelevant bulk data [650:10] So that makes sense. Um I would think [650:13] that we'd also want to filter some of [650:14] this out, but maybe not. And so here it [650:16] says return the full uh world state [650:19] simulating uploaded API response. That's [650:21] true. Um tool output filtering filter [650:23] the massive world state down to only [650:25] what's needed. So instead of injecting [650:26] thousands of tokens, we'll inject 50 um [650:29] 50 tokens of the fact. Here we have case [650:32] fax. So load persistent uh persistent [650:34] player facts from disk. So here it is [650:37] doing that. Uh we have save case factax [650:40] so persists information here. Okay. [650:43] Extract the case fax data. So it's [650:45] storing it somewhere. [650:47] And if we go and find our problem here. [650:50] Did it actually do that? No. But it's [650:53] supposed to store it somewhere. [650:56] I guess it's just storing it in memory. [650:58] Build the system prompt. [651:02] Okay. You are a narrator of a text [651:04] adventure game. Here's the case fact. So [651:05] it's providing this information each [651:07] time. every time the loop current room. [651:09] So this is the information that's [651:10] filtered out. Respond to the player's [651:12] actions. Okay. [651:14] And then we have our text adventure. So [651:16] large context demo. Mhm. World size [651:20] state. [651:21] And there we go. So we'll go down below [651:24] here. We'll just play a little bit. See [651:25] what happens. We might not be able to [651:27] really um be able to see what's going on [651:30] here. We'll try it anyway. So say north. [651:35] Oh, and as per usual, it doesn't have [651:38] the API key. So, that makes me think it [651:40] didn't even work when they tested it. I [651:42] really wish that this stupid thing would [651:46] copy [651:48] this here, but it never does. So, we [651:51] will grab this here. [651:55] Kind of getting used to pasting in here [651:57] myself. [651:59] What can you do, though? Um, and we'll [652:02] grab this here. [652:08] We'll load that and we'll go back over [652:11] to here. [652:14] Um, and we're looking for [652:22] the key. So, it's a key. There it is. [652:24] Grab that here. Like this. Go back to [652:27] this one. We will find [652:31] where that goes. Here it is. [652:34] So now this should be able to work. [652:38] Go back hit up. [652:41] Okay. Um, [652:45] did we forget to include that? The [652:47] async. [652:51] Bring that. Go back all the way here. [652:53] Maybe I took it out by accident. [652:56] Do this again. [652:59] And I'm getting tired of this. So we'll [653:01] go ahead and we'll let it fix the [653:02] problem. [653:04] I'll be back in just a moment. Okay. So, [653:08] um let's see if it's resolved those [653:09] issues. [653:11] We'll say look around. [653:16] Just a second here. So, you take a [653:18] moment to look around. Um [653:22] pick up Rusty Key. [653:25] I like how it's showing us the facts as [653:26] it goes through here. [653:28] There we go. [653:31] Look at old map. [653:35] The map provides useful information, but [653:37] you're still standing in the in the [653:39] grand hall with Rusty Keys at your feet. [653:42] So, there we go. So, it's working. Um, [653:44] and I mean like it's obviously we're not [653:46] exposing that information here, but [653:47] let's again just go take a look here and [653:50] make sure we clearly understand how it's [653:52] doing this. So, here we have the world [653:54] state and and basically for the room, [653:57] we're going to get all the room data. So [653:59] we say get world state from the tool and [654:01] then we're just individually selecting [654:02] that data and returning it back um to [654:05] the model. Notice here we have our case [654:07] factax. So this data is being stored [654:09] somewhere I think in our oh it is a file [654:13] now that we ran it we have it and so [654:15] it's just extracting that information [654:17] out of that file um and then loading [654:19] into context every time. And so that [654:21] solves those two problems. Obviously, [654:24] we're not checking the one where we have [654:25] a very large dock where we'd have to [654:27] process it. If I feel like doing those [654:29] videos, I might do them, but I don't [654:30] think it's important. Um, but yeah, we [654:33] will move on from here. Okay. [654:39] Hey, this is Andrew and in this video we [654:41] are going to figure out how to manage [654:43] context effectively in large codebase [654:44] exploration. This is the 5.4 section. [654:47] And so the purpose of this section is [654:49] that imagine you have a large codebase [654:51] and uh you need to figure out how it [654:53] works. Um but the problem is is that if [654:56] you have a large codebase and it's [654:57] ingesting a lot of information um the [655:00] context is going to get muddled because [655:02] as it increases more information as it [655:04] goes, it forgets things on the tail end [655:07] or when it summarizes it it loses key [655:09] information. And so what we need is like [655:11] an external memory. And so the idea is [655:13] that as we crawl through information [655:15] into the codebase um we can uh save that [655:19] somewhere like let's say a findings.mmd [655:21] file. Um and also the other thing is [655:25] like we want to delegate out to sub [655:26] agent because if we had our main agent [655:29] running like just cloud code uh being [655:32] the person that uh consumes all the [655:33] information um as there's a lot of busy [655:36] information that's going to eat up it [655:38] the coordinator's context and it's not [655:39] going to be able to coordinate [655:40] effectively. So we need a sub agent that [655:43] can handle the noise. We need a sub [655:45] agent that will or at least the [655:47] coordinator that will write to the [655:49] findings file so that um when the [655:51] context is working it can continuously [655:53] reference that information. And let's [655:57] say you know for whatever reason our [655:59] computer shuts down or uh there's an [656:01] error or something or we run out of [656:02] credits and we want to pick up where we [656:04] last left off. So we have crash recovery [656:06] and that's what we want to do. So, I [656:08] thought it would be fun if we explored a [656:11] fun codebase like the Doom codebase. [656:14] Anytime we can work in Doom, why not? [656:15] So, I'm going to go over to here and I [656:18] have not looked at the Doom codebase for [656:20] some time here. I'm going to go ahead [656:22] and grab it and I don't want to SSH it [656:26] to be honest. I just want to let's see [656:29] if we can figure this out. So, I'm going [656:30] to make a new new folder here, and we're [656:34] going to call this Doom Explorer. [656:38] And in here, [656:42] we'll say Doom Explorer. I'm going to [656:44] open up Claude. I'm just going to switch [656:46] it back to Sonnet. [656:49] And I'm going to put the effort back to [656:51] medium. [656:54] Sorry. [656:55] And so what I want to do here is I'm [656:58] just going to ask it like I want to uh [657:01] clone the uh Doom codebase [657:07] in this folder. [657:10] But I don't want uh it get tracked [657:14] because I don't want to [657:17] to have um [657:20] a git a git repo and a git repo. [657:24] Um, I also want you to ignore the [657:28] contents. Can you can you create me a uh [657:32] a a shell script that will pull the [657:37] codebase into this folder [657:40] without the perion uh the problems I [657:44] want to avoid. Okay. So, it's going to [657:46] go ahead and make that script and I'm [657:48] going to run it and um yeah, it should [657:50] be fun. Or at least it's fun for me. I I [657:53] don't know if it's fun for you, but uh [657:55] maybe you want real world use cases, you [657:57] know. I I don't know, maybe this is a [657:59] real world use case for you. [658:02] And so we will get that little script [658:04] going here. [658:07] And it's already here. So that's going [658:10] to pull it. And [658:13] yeah, it just removes it. I mean, that's [658:15] what I was going to do. So that's pretty [658:17] darn simple. So we'll just chimod that [658:19] up plus x doomcript. And we will do [658:22] setup Doom. I'm hoping that it pulls it [658:24] into a folder. [658:28] It adds it to that directory. I guess [658:29] that's fine. You could do it that way, [658:30] too. I was thinking, oh, no, it added it [658:33] right here. Perfect. And so, now we have [658:34] the Doom source. Okay. So, now what we [658:36] want to do is we want to create um [658:39] something. So, we're going to make a [658:40] file called um findings.md. [658:45] We're going to create another file here [658:47] which is going to be um [658:50] manifest.json. [658:53] And I'm going to go ahead and make a new [658:55] plan [658:58] uh doom codebase exploration [659:03] uh orchestration. [659:08] Say I want to use cloud code uh to [659:11] explore the Doom codebase. [659:15] um example like um how does a player [659:19] take damage? [659:21] I am I am concerned [659:24] uh that due to the size of the codebase [659:28] uh that claude code will lose context [659:32] um as it summarizes [659:36] information. [659:37] I want to have external information, [659:40] external uh fi information saved to [659:44] findings, external key information, key [659:47] information [659:50] save to findings.mmd [659:52] so that um [659:56] uh it helps [660:00] guide memory. I want to [660:04] I want to um [660:08] run uh exploration [660:12] as a sub agent [660:15] so that um it does not eat uh it does [660:19] not muddy the context of the main parent [660:25] agent [660:27] uh uh or let's say cloud code runtime. [660:31] so it can effectively [660:35] coordinate. [660:38] I want to be able to [660:41] recover [660:43] uh or resume [660:47] uh sub aent sub aent tasks. And so I [660:51] have a manifest. I want that state to be [660:55] stored in manifest [660:59] JSON. [661:00] Um, [661:02] I want a [661:06] sub agent [661:08] defined [661:10] with uh based on [661:13] claude codes [661:15] um [661:18] expectation uh like conventions [661:22] egents [661:25] know it claw agents right [661:29] Okay. And so now we have that. I'm going [661:31] to go over to Claude here and I'm going [661:34] to reference our plan. Actually, I'm [661:35] going to switch back to [661:37] Opus for this. And I'm going to [661:40] reference my plan. Can you create me a [661:45] uh spec [661:47] uh and task breakdown and save it in a [661:51] file called specd? [661:56] And we will let it go ahead and do that. [662:01] All right. So, let's take a look at our [662:03] plan. So, here it says written a spec. [662:07] So, it says build a cloud code [662:09] orchestrator gets copying with how does [662:11] a player take damage without polluting [662:12] the parents agent context processing [662:14] files etc. Uh, dispatch the exploration [662:17] questions. Dispatch a sub agent per [662:19] question. Uh, read findings MD. Never [662:22] perform raw code exploration itself. [662:24] Keep context clean. Defined at the Doom [662:27] Explorer. I like that. It could be even [662:29] more generic, but we're going to just [662:30] call it Doom Explorer. Uh, append [662:32] findings to the findings.m MD. Uh, [662:34] append knowledge log [662:37] question status. [662:40] Really have one sub agent. So, if we [662:42] were multiple sub aents, we would [662:43] probably have to manage that. Quarter [662:46] coordinator reads manifest file. Um, [662:50] and so that seems good. [662:52] So, scaffold [662:54] author the agent define the manifest [662:57] file coordinator workflow prompt [663:00] template. Yeah, that's something we're [663:01] going to need. Prompt templates. So, [663:02] that'll be maybe the claw.md I would [663:03] think that we would load [663:06] um findings.md resume flow [663:11] uh smoke test failure handling [663:14] acceptance criteria. So, this looks [663:15] good. So, which and here we have open [663:17] questions. Uh which oh we don't know. So [663:20] it says which doom fork [663:23] affects the file layout. [663:26] I mean [663:30] you have some questions. [663:33] Um we are using [663:37] I don't even know. I guess it's the Doom [663:39] GitHub. It's probably the official one. [663:44] This Doom. [663:48] Um should findings MDB chunk per topic? [663:51] Uh I think findings [663:54] can be a folder and we can just create [663:58] uh artifacts within it [664:03] essentially [664:05] scratchpad files. [664:08] Do we want a second sub aent type [664:10] synthesis? Yes, let's have a second uh [664:14] sub sub aent. [664:17] Please update our spec. Okay, so we'll [664:21] have it update the spec. [664:23] All right, so it looks like our code has [664:26] done something. Um, oh, it updated our [664:29] spec, right? So that's what we were [664:30] doing. Let's go and review our spec [664:31] file. And so here [664:36] parent never calls ger read [664:39] only the doom as her source does. [664:42] Exploration output lives in the findings [664:44] directory. sub agent crash coordinator [664:47] marks task failed [664:49] um smoke test. I mean like findings is [664:51] pretty much our knowledge base basically [664:53] for our repo. In fact, I'd be pretty [664:55] tempted to rename that as knowledge [664:56] base, but I'm going to leave it alone [664:58] for now. Um how does a player take [665:00] damage [665:03] sub agent crash? [665:05] And so we obviously have two steps, [665:07] synthesis and um this one. So [665:11] consolidated answers written by the Doom [665:13] synthesizer. [665:15] Makes sense. [665:17] Um [665:20] seems good. Okay. So we'll go ahead [665:21] here. [665:23] Uh please execute [665:26] uh specs in specs empty. [665:34] All right. Right. So we'll go ahead and [665:35] have that implement [665:38] and we'll be back here when it's done. [665:41] All right. So we have scaffold complete [665:43] T1 to T5 done. T6 I our runtime behavior [665:46] is encoded into the two agents. So it [665:48] built these things here. The manifest [665:50] file the two things here [665:53] um cleaned up states finding not [665:55] executed runtime task smoke test [665:58] failure. Let we kick those off. So what [666:01] I'm going to do here is just take a look [666:02] and see what we have. So, we have a [666:05] findings folder now, which is better. [666:07] Um, I'm I guess we'll see what happens [666:10] to be dropped into synthesis. Let's go [666:12] here and take a look at our explorer. [666:13] So, explores the Doomc uh source code. I [666:17] know C, but it's been a long time, so [666:18] I'm not going to say I'm an expert at [666:19] it, but here we can see that it has lots [666:21] of access. You are a Doom codebase [666:23] explorer, and you investigate the Doom [666:26] source. Uh the coordinator prompt will [666:28] contain [666:30] um a task, a question, a resume from. [666:34] Okay, so it's passing very specific [666:36] things. You may only write edit paths [666:39] under findings and manifest JSON because [666:41] it's not there to edit. Uh you may never [666:43] modify the files, never write the code. [666:46] Derive a kebab case slug. Okay, I didn't [666:50] know that's it's called a kebab. I mean, [666:52] it kind of makes sense. Like if it's [666:54] skewered, it looks like a kebab. [666:57] Okay. Yep. Never knew that was called [666:59] kebab case. All right. Oh, snake cases [667:02] is underscore. I've been complaining [667:03] like snake cases to have a hyphen and I [667:05] guess they're saying that's snake case [667:07] or sorry that's kebab case. So sorry [667:10] folks earlier for complaining about it. [667:12] If resume from is provided appendage to [667:14] the file. [667:16] Um update the manifest file. Use glob to [667:18] glob or gp to glob. There we go. Fill [667:21] the scratch pad. Um on success update [667:24] the stuff. So, here's a template. Task [667:26] ID question. That's not much of a of a [667:28] template. Oh, this is the whole [667:29] template. Okay. Question, entry points, [667:31] call chain, key files, summary. Append [667:34] only. Never rewrite a prior scratchpad [667:37] section. Makes sense. Let's look at the [667:40] synthesizer. Consolidate multiple Doom [667:42] exploration scratch pads into a single [667:44] synthesis document. Under findings, [667:46] you're a Doom finding synthesizer. You [667:48] merge pre-task scratch pads into one [667:49] coherent answer. Oh, that's cool. Uh, [667:51] never read anything under the Doom [667:53] source. um and you update the manifest [667:56] file. So that's interesting. So here we [667:59] have an example [668:01] of what it finds. No speculation if it [668:05] doesn't cover it. Say so in gaps, no [668:08] code blocks, etc., etc. That looks [668:11] pretty good. I'm not sure if it Oh, [668:13] these are using Haiku. That's cool. So [668:16] um I think it's just pulling that from [668:17] before because I was using him, but uh I [668:19] actually wouldn't mind it doing that and [668:21] seeing how it performs. So the question [668:23] is like how do we kick this off? I would [668:25] thought that there is something for the [668:27] coordinator which is the the main [668:29] runtime here. Let's go back to the spec [668:31] and take a look here. So [668:35] um 67 [668:37] on startup scan the manifest file. No no [668:39] smoke test. How does a player take [668:41] damage? Follow up with a synthesis pass [668:44] with the resulting scratch pass. Verify [668:45] parent context. [668:48] Um, [668:50] so [668:53] these require you to kick off an actual [668:54] exploration run. Say the words and I'll [668:57] dispatch Doom Explorer. Okay. Go and go [669:03] and [669:04] perform. [669:06] Yeah. Go. Go. Okay. There we go. I I [669:09] don't know what to say to kick it off. [669:11] All right. [669:13] But here is going to kick it off. While [669:14] it's kicking off, let's go look at the [669:16] codebase for fun. Um, [669:20] look, it's C. [669:23] Uh, it'd be fun to explore the codebase [669:24] and see exactly how it works. [669:28] All right, but uh, okay, here's the real [669:31] code. I was like, where's the stuff? [669:33] Look, there's this is how you do the [669:34] cheats. [669:37] 1993, 1996. That's crazy. Well, that's [669:41] it. That's all we got in the head. [669:44] And as it's executing off, just having [669:47] fun looking at the code. [669:49] Wow, this stuff is really simplistic. [669:52] So, keep expecting to see more in here. [669:55] Let's go look at the enemies. [669:58] Okay, now we're getting more stuff. [669:59] Okay, it's directions. [670:04] All right, so that looks like more real [670:05] stuff, but we will pause here. See what [670:08] it comes back with. Okay, we can see [670:10] that it's exploring. Well, actually, I [670:12] guess while it's it's doing this, we [670:13] should really uh point out the fact that [670:16] um Earnest didn't register our project [670:18] level agents definitions as sub agent [670:20] types. Only built-in agents types are [670:22] exposed. I'll work around this by [670:23] invoking general purpose with Doom [670:24] Explorer contract, which preserves it. [670:27] Um [670:29] agent type doom explorer. Well, that [670:31] doesn't seem good. [670:35] Maybe that'll be an issue later on. [670:37] Finding persists to disc parent stays [670:39] clean. Oh, it's back. So smoke test [670:41] passed. So we have our findings. Let's [670:43] give this a refresh [670:45] here. [670:47] And [670:52] so so [670:55] what is the sequence of events and data [670:56] transformations when an end player [670:58] character receives damage and doom? So, [671:00] projectile projectile collision damage [671:03] during trace, hit scan, weapon damage [671:05] during raycast, blast radius damage [671:09] propagation, uh various game system [671:12] calls, target player checks, player [671:14] pointer extracted from the target, [671:16] uh baby difficulty reduction, armor [671:19] absorption. Okay, cool. And so, it kind [671:20] of walks us through all those stuff and [671:22] tells us where the key files are. So [671:24] player damage and doom flows from here a [671:27] central function called by projectiles [671:28] hit scan weapons expirations then [671:31] processes armor absorption. Okay. [671:35] What is the exact pain chance [671:37] probability mechanism? How do we specify [671:39] weapon type calculations? What happens [671:41] during invulnerability uh grace periods [671:44] and upgrades? Let's go down to our [671:45] manifest file. So we have a manifest [671:46] file and there is a task and then it can [671:49] continue to collect those tasks. So this [671:51] is pretty good. Um, so here it says, "We [671:54] passed the smoke test. Uh, want me to [671:56] dispatch the Doom synthesizer over the [671:58] single scratch pad as a dry run because [672:00] they run separately, right?" So, uh, [672:03] yes, please. Uh, run [672:08] maybe. Yeah, let's let's, uh, run three [672:13] more explorations and then run the [672:16] synthesizer. [672:20] Okay. So, we'll do that first. All [672:21] right, just checking in here. Look, it [672:24] is uh uh collected those three things. [672:27] Armor mechanics, weapon damage, player [672:29] death, and then here we have our [672:31] findings and it's going to go ahead and [672:32] synthesize it. So, we'll see what pops [672:34] out of here. While that's happening, [672:35] let's go take a look and just look at [672:36] the results. So, here, how does armor [672:39] pick up and armor class effect [672:40] absorption? Uh Doom's damage system uses [672:43] a hit scan for oh, how weapons compute [672:45] imply targets. Cool. Um player death [672:48] transitions, etc., etc. So that stuff is [672:50] continuing on here and we're waiting for [672:52] the synthesis to complete and see what [672:54] we get. [672:55] Okay. And so it is now done. So all five [672:58] tasks are complete. It did more than [673:01] three, but that's fine. And then here uh [673:03] unified life cycle site scratch pads [673:06] only. Six gaps flagged. Um and so we [673:08] will go into here. Um so parent context [673:12] state clean zero doomed source reds in [673:14] the conversation. All expirations [673:16] happened inside the isolated sub aents. [673:18] The synthesizer never touched the Doom [673:20] Source. Want to peek at the synthesized [673:22] doc? Drill into one of the flag gaps. [673:24] Um, [673:26] I mean, I only Is this the only file [673:28] that we have? The synthesized doc [673:29] synthesis. [673:31] Doom's damage pipeline forms a complete [673:33] life cycle. [673:38] I mean, yes, I guess so. But like [673:43] uh [673:44] Oh, gaps. Here we go. So pain chance [673:47] probability what is the uh exact pain [673:50] chance probability mechanism [673:51] invulnerability [673:53] um okay [673:57] and so here's my question is you know [673:59] what does the I get it but I'm just [674:01] going to ask it very clearly what does [674:03] the synthesis uh doc do for us like what [674:08] like uh what is why do we want it [674:13] is the answer artifact the thing a human [674:16] actually reads the per task scratch pads [674:18] are raw files and synthes this is the [674:20] report so concurly gives you one [674:21] narrative instead of four the scratch [674:23] pads each answer a narrow sub question [674:25] the synthesis stitches them into a [674:27] single end uh to end story okay so [674:31] that's the idea so is this actually [674:33] useful that's the question we'll go back [674:34] to the top so damage life cycle [674:37] synthesis damage armor weapons and death [674:39] oh because now it's kind of rounding out [674:41] all those together doom's damage [674:43] pipeline forms a complete life cycle, [674:45] weapons of environmental hazards, [674:47] inflicts damage. Okay. So, do we have to [674:49] like [674:51] So, do we ask a question for synthesis? [674:57] And if we did, what was it prompted [675:00] with? [675:06] Okay. [675:11] Has its own question passed to it. So [675:13] the question was how does the player [675:14] damage armor how does the player damage [675:17] armor weapons and death fit together? I [675:19] made it up unifying the questions across [675:21] it. It's deliberately broader than any [675:23] single scratch pad because synthesis [675:25] only earns its keep when it's [675:26] integrating the actual prompt sent to [675:28] the synthesizer agent. So here how does [675:32] how does it do this findings? [675:34] Do not read anything under any other [675:36] places [675:38] a gap we're calling out. I authored the [675:40] synthesis question myself rather than [675:42] coming from you in a clear design [675:44] original questions etc would be that um [675:48] right now the spec doesn't formalize the [675:50] parent child decomposition the [675:52] coordinator just invents both layers [675:54] worth titing the spec MD if he wants [675:55] this to always track back to a real user [675:58] that asked no I don't care about that so [676:01] here again we'll just read it so doom's [676:04] damage pipeline from a complete life [676:05] cycle weapons or environmental hazards [676:07] inflict damage which flow through a [676:08] central damage damage handler that [676:10] applies armor absorption before reducing [676:13] players health. Once health reaches [676:15] zero, death is triggered. Um, and so [676:17] here we have our weapons environment [676:19] system, our projectile weapons, our [676:21] environmental stuff. Um, [676:25] and so we can see all those things [676:27] there. Armor integration damage flow. So [676:29] armor sits directly inside the damage [676:32] mobs uh player branch. Um and so here [676:36] armor extends survival by absorbing the [676:38] incoming damage. Key fy duped. So here [676:41] we are seeing duplicate information. [676:46] Fair enough. [676:48] Um contradictions. All four scratch [676:50] presents consistent mechanics. Applies [676:53] armor before health. Armor types and [676:56] capacities are not contradicted in other [676:58] documents. Weapon damage computations [677:00] are consistent with the damage entry [677:02] points. Uh, death logic consistently [677:04] describes it here. So I would I would [677:06] think that it's Oh, so none are [677:08] detected. Okay. So pain chance [677:10] probability invulnerability detection [677:12] boss spiders immunity armor bonus edge [677:14] cases mega sphere interaction corpus Q [677:17] details. Fine. And then all our sources. [677:19] So this is excellent. Um, [677:22] yeah, it's great. Um I think the only [677:24] thing is that if we go back to our [677:26] original implementation, I'm not sure if [677:28] we implemented failed failure handling. [677:33] So I just go back here and say did we [677:36] did we implement [677:39] uh T9 fa failure handling. [677:45] You could see how this could just be [677:47] generalized out. It doesn't have to be [677:48] Doom Explorer. U but it's pretty cool. [677:50] So we'll give it a moment here for it to [677:52] come back. [677:55] Yeah, only the happy side. So, what [677:56] exists both the agent uh [678:00] definitions? Whoop whoop. [678:03] I did not ask you to change anything. I [678:05] just jumped here. So, um [678:09] with a short note to reason, scratch [678:11] pads are append only by convention. So, [678:14] partial work will survive. What's [678:15] missing? Parent side crash detection. [678:18] Resume exercise. [678:20] Uh, no failure injection tests. What [678:23] would it take? A coordinator wrapper. [678:25] Uh, a startup routine that scans for [678:27] stuck running failed rows. One [678:29] deliberate fail test. [678:32] Uh, yeah, go for it. It's a big task, [678:36] but let's just wrap it up here on on the [678:38] seven because I'm the reason I'm doing [678:40] this is because maybe I might reuse this [678:42] and and do something better with it, but [678:44] we will let it do that. Okay. And that [678:47] was fastly implemented. So, we have here [678:53] What was proven? NN injected a running [678:55] row. Coordinator reconciles correctly. [678:57] Flipped to fail. Dispatch resume from [679:00] sub agent appends the update. Manifest [679:02] now shows a value. Acceptance criteria. [679:04] Killing and restarting the parent mid [679:05] task resumes without losing findings. [679:07] Oh, that sounds perfect. Um, so I mean [679:11] we can go a bit far. Oh, so here's the [679:12] coordinator [679:14] uh coordinator helper for the Doom [679:16] Explorer. So, reconcile verify stuck. [679:19] Um, and so I'm guessing what it's doing [679:22] is it's going to invoke this first. [679:25] Verify stuck. [679:28] What was bill coordinator with three [679:31] commands stuck list the rows need to [679:32] dispatch whatever. Um, [679:35] okay. So, [679:38] how is the coordinator [679:41] getting invoked? Because that's the only [679:43] thing that's not really spelling it out [679:45] to me. Like I assume it's the main [679:47] runtime that's doing that. Honestly, [679:50] it isn't automatically. I ran it by hand [679:52] with a bash call. No wiring. What's [679:53] actually happening right now? So, um, [679:58] okay. But my my issue is like, okay, but [680:00] like [680:03] ways to actually wire it up. Okay, here [680:04] we go. So, document the coordinator or [680:06] put it in the cloud procedure before any [680:08] dispatch run the stuff. So I would say, [680:10] you know, I would prefer a coordinator [680:14] and I would like some uh claude [680:17] instructions [680:19] uh to claude MD to have [680:24] instructions [680:26] to tell it to invoke. Um unless you [680:29] think it should be a skill [680:33] to invoke [680:35] the the coordinator. [680:38] Okay. So [680:42] I mean I probably should say coordinator [680:44] MD file but that's what I'm kind of [680:46] wondering like but then skill is but [680:48] skill I think runs if you have skills [680:50] they run in the main context they're not [680:51] actually sub agents [680:55] my take clawmd not a skill so skills [680:57] must be explicitly invoked the [680:58] coordinate well not necessarily because [681:00] you can implicitly say that the [681:02] coordinator runs needs a fire every time [681:04] the parent dispatches a sub agent so a [681:07] skill would only make sense for that. [681:09] Um, yeah, but I still want a [681:11] coordinator.mmd file. [681:14] See, it wasn't asking me. It was just [681:16] writing it. And so, we'll go here. We'll [681:18] take a look. [681:21] This directory is an exploration, [681:22] orchestration, doom. Sorry. Sorry. Did [681:26] you Did you put in the claude MD? [681:31] What [681:34] what could have been the coordinator [681:39] MD [681:45] and can we just have something later [681:50] in the cloud MD [681:54] to invoke it? Also, I thought skills [681:58] can be invoked directly based on their [682:03] uh descriptions. And why wouldn't we [682:08] want to have a skill for direct [682:11] invocation [682:14] and skills run in the main agent uh the [682:18] main runtime loop, right? [682:21] So to me like it still seems like we [682:23] should have skill. So maybe it's not [682:25] reasoning well like this this seems like [682:28] this should be in the coordinator empty [682:31] right and then we should have a skill to [682:32] invoke it that's in the project and also [682:35] we could end up um extracting this out [682:37] but we'll we'll wait a moment here so [682:39] here it goes you're right on both [682:40] accounts I was wrong to dismiss a skill [682:42] so it also invokes based on the [682:44] description um descriptions like this [682:46] procedures live inside the skill it's [682:48] not in the cloud MD [682:51] um [682:53] fair enough we can do that as Well, um, [682:55] I dumped the whole procedure in there. [682:56] It should have been a light pointer. So, [682:58] we'll go ahead and take a look at what [683:00] it's done. [683:02] And we'll go into here. Uh, did it get [683:04] rid of this file? [683:06] Exploration or uh orchestrator over at [683:08] the Doom source design inspect. When the [683:10] coordinator exploration works in this [683:11] directory, use the Doom coordinator. [683:14] Never read or GP the Doom source. We [683:16] don't really need this line as the [683:18] skills should be able to do it. [683:22] But here we have an example. you're a [683:23] coordinator for the Doom codebase. Never [683:25] do this. And so now what we can do is [683:27] say Doom coordinator. [683:30] It's really exploration. It's a terrible [683:32] name that they have here. I'm just going [683:34] to uh cancel out of this and try this [683:36] again. [683:38] Doom coordinator. [683:40] I want to know how level difficulty [683:44] uh affects [683:46] gameplay. [683:49] Oh, I've used 100% of my session. Well, [683:52] I guess we'll never know if that works, [683:53] but we reached our little end here, so [683:55] that's perfectly fine. Technically, I [683:57] have um what do you call it? Uh credits, [684:00] so it's not a big deal. But I thought [684:02] that was really funny that we made it [684:03] right there to the end. [684:05] Okay, but I think we built something [684:07] really, really good there. I think I [684:09] would have probably packaged this into [684:10] like do go further and package this into [684:12] a separate course. Um but anyway, I hope [684:15] you enjoyed that. Chowo, chiao. [684:21] Hey, this is Andrew. In this video, I [684:22] just want to show you how batch [684:23] processing works. And it's not that [684:25] complicated. We're going to make a new [684:26] folder, call it batch uh processing, and [684:29] we're going to um copy, I guess, uh our [684:33] maybe not our retry, but our force [684:35] structure JSON output. And we'll go [684:37] ahead and copy this. And we're going to [684:39] go into our batch processing and make a [684:41] new main file. I don't want to run this [684:43] because uh batch processing isn't [684:45] something that will happen instantly. I [684:46] think it takes time for it to come back [684:48] because yeah, it can take up to 24 hours [684:49] to process with no guaranteed SLA. Um, [684:52] so I'm not a huge fan of of of trying to [684:54] run that. And I think for the exam, I [684:56] just want you to know that exists and [684:57] you can save money with it. Let's take a [684:59] look at what we could get as our output [685:01] code. So say um, you know, can you can [685:04] you change this to uh utilize batch [685:08] processing uh, uh, for anthropics? So we [685:11] get savings. What is the key thing we [685:15] need to consider? Like how does it know [685:20] where the output goes? Okay. And I think [685:23] what it does is it probably has like an [685:24] endpoint where it outputs that [685:26] generation and you have to hit it. [685:27] That's probably what it's going to do. [685:29] Uh when I'm thinking about OpenAI, we'll [685:31] go ahead. We'll just say yes, and we'll [685:33] see what it comes back with. And we'll [685:35] just take a look here. And I I don't [685:36] think I'm going to run it because I just [685:38] want to see what it looks like, but I [685:40] don't want to run something that I have [685:41] to check. Maybe I'll run it, but maybe [685:43] we just won't see the results. Okay. [685:46] Okay. Let's take a look and see what [685:47] changes. So here, um, we have a custom [685:50] ID on each request when submitting. And [685:52] then when you call batch results, every [685:54] uh results comes back as a custom ID. [685:57] Okay. And then when you batch, if I [685:59] forget, you can't do interactive round [686:01] trips. So instead of using tool use, uh, [686:03] it's going to force a single tool call [686:06] and the tool input is going to block. [686:08] Okay? And then you just extract it [686:10] directly. And then you were saving some [686:12] money. So let's take a look. Let me just [686:14] close this out. Make sure we have the [686:15] latest code and see what this looks [686:18] like. Let's go all the way to the top. [686:21] Okay. And so here we have a couple [686:23] tickets. We still have our tool [686:26] and our required information. And here [686:29] it's going to input that information. [686:32] And so [686:34] we still have tool choice, which is [686:36] fine. [686:38] Let's say I had to change something with [686:39] that. [686:42] all the way down here. [686:44] Um, instead of instead of tool choice [686:46] any forces a single tool call. Oh, [686:49] instead tool choice any forces a single [686:51] tool called. Okay. I mean like we were [686:53] already I think we were already doing [686:54] that. So that's fine. Um, and so we'll [686:58] go here and [687:01] do we have a loop? We don't really have [687:02] a loop, do we? So we have batch create. [687:04] Oh, so we have now batch create. Okay. [687:07] and [687:08] it's going to loop and then retrieve [687:11] that information and then it'll tell us [687:13] when it's succeeded. [687:16] Okay. So, I don't know. I guess I'll run [687:19] it. I'm like I'm not really sure what to [687:20] expect there, but I was thinking there'd [687:22] be like a link or something and like the [687:23] payload would be somewhere else. Maybe I [687:25] just don't understand. But we'll run and [687:26] we'll find out together. [687:30] So, here [687:33] it's sending those out, creating that [687:34] batch process. And now it's processing [687:37] it. Okay. So maybe this stuff will [687:39] literally hang until then. [687:43] Oh wow. Okay. So really we're basically [687:45] waiting here until later. [687:53] So I guess my thought is like the [687:56] problem with this like this is the [687:57] problem with this approach right? So I'd [687:59] say you know um [688:02] for batch processing the user has to [688:05] wait. Is there is there a way we can uh [688:09] make it so we can run it and then [688:13] just check on it [688:17] ourselves [688:21] at another time. [688:24] you know, if they're if they're, you [688:25] know, because that I think to me that's [688:28] going to be a a better result. Um, [688:30] because maybe you don't care and you [688:32] just come back and you check, but like [688:34] having to wait there would take forever. [688:35] So, we'll just hang out here for a [688:36] second and see what it does. All right, [688:38] we are back and so now we have a few [688:39] options. So, we have submit, check, and [688:41] run. Um, let's take a look at what has [688:43] changed. So, we have submit, check, and [688:45] run, run, and wait. So, here it is [688:49] creating a state file. All right. [688:53] And then in the check here, it's loading [688:55] that state file. Okay. So, let's go [688:57] ahead and give this a try. So, we have [688:59] um submit, check, and run. Right. So, [689:02] we'll go here and we'll say submit. [689:08] So, here it says submitted to the batch. [689:13] Fires the batch. Saves the batch ID to [689:14] the batch state file. So, we'll give [689:15] this refresh here. There we go. So, we [689:18] have a B that there. Original blocking [689:21] behavior. Okay, that's if we want to do [689:22] that there. And so now whenever we want [689:24] to check it, we can just go and check [689:25] it. [689:28] And it's showing if it's succeeded or [689:30] not. And let's go take a look at how [689:31] that check works. And so it's calling [689:33] that retrieve. Okay. And so it is [689:35] hitting an API endpoint and we're just [689:36] using ID. And so that's batch [689:38] processing. I think that's pretty [689:39] straightforward. Um I would expect we [689:41] just get a result back and we process [689:43] it. But uh yeah, if you don't need [689:45] things right away, apparently uh [689:48] extremely extremely good savings. Okay. [689:55] Let's take a look here at MCP discovery. [689:57] So, this is when you configure multiple [689:59] MCP servers and all their tools are [690:01] discovered and loaded at the connection [690:03] time. And so, the agent will treat it [690:04] like a flat list with no awareness of [690:06] which server comes from where. Now, this [690:08] is not what the code would look like if [690:10] you're using um Anthropic or the agent [690:13] SDK. This is just kind of a a pseudo [690:15] code, if you will. But the idea is that [690:17] in your tools or somewhere here, you're [690:19] going to specify MCP servers and it's [690:21] going to end up loading all of the tool [690:24] uses. Okay? And you can see where this [690:26] could run into an issue uh because now [690:28] you have um a bunch of different tools [690:32] here. And so that is uh one [690:34] consideration that we have to consider. [690:37] But what we'll do is just go ahead and [690:38] just make sure we know how to implement [690:40] this. Um but there you go. [690:46] All right, folks. Um, this is for MCP [690:48] discovery. I'm having a hard time [690:49] showing a good example here because, um, [690:51] this last one seems to be failing quite [690:53] a bit, and I made it over here. Uh, but [690:55] basically, I just said, "Hey, can you go [690:57] ahead and show me discovery with a few [690:59] examples, but here you can see within [691:01] um, the cloud agent options, we have MCP [691:03] servers. We're specifying those [691:05] locations. We have to install them [691:07] individually. This one will not work. [691:08] But the idea is that all these will just [691:10] be available um to the agent. Um and all [691:15] I just really want to show you is just [691:16] that there's this MCP server option. [691:18] Okay. And then all those tools will be [691:20] then accessible. Um I actually did [691:22] install them here locally. So Claude [691:24] should be aware of it. If we go to MCP, [691:26] uh if we go over to here, it should show [691:29] them here. No, it doesn't. [691:32] Uh though I think uh maybe that's [691:34] because Claude has a very different [691:35] specific line. But the point is is that [691:37] we have installed them and we can use [691:41] them here. I just don't have a a complex [691:43] use case and we got enough videos. So I [691:45] just wanted to show you the code. Okay. [691:51] Okay. So within MCP there is more than [691:53] just uh tools. There's also resources [691:55] and prompts. Nobody uses prompt [691:57] templates as they're not useful. Um but [692:00] resources are and they don't get talked [692:02] about enough and you definitely need to [692:04] know them for the exam. So tools we keep [692:07] seeing which is basically functions that [692:09] are called often they are used to uh [692:12] handle a domain of an API. So there [692:15] might be individual functions but maybe [692:18] you have users uh API endpoints and here [692:20] it is kind of uh acting as a buffer [692:23] between that domain but here it's saying [692:25] you know what tables exist in the [692:27] database and so the tool uh will call a [692:30] function right which will make a [692:32] database call an SQL query and return [692:34] back those results where um let's say [692:37] you have a database or you have an API [692:39] endpoint and an API endpoint is just [692:41] going to return back data right and so [692:43] it might seem seem uh very heavy to have [692:46] to do tool call for very basic API [692:48] calls. Um and so basically resources has [692:52] this schema that maps what the data [692:54] looks like from your your data source [692:56] and the idea is that it'll just read it [692:58] directly and so you'll get faster [693:00] results. Um there'll be less back and [693:02] forth and so resources are definitely [693:04] something that we want to uh utilize. [693:07] Okay. [693:12] All right. Let's see if we can show how [693:14] MCP uh resources and tools work. So, I'm [693:17] going to make a new one. Say MCP uh [693:19] resources versus tools. [693:24] Okay. And I'm going to go into our hello [693:26] world just so it has a reference of what [693:29] it should do. Yeah, that one's fine. Um [693:32] and we'll go over to [693:35] our new file here, main.py. [693:41] I don't think I copied it. So, we'll try [693:42] that one more time [693:45] and paste that on in there. Okay. So, [693:46] now I'm going to go here. [693:49] I want to show off how uh show off [693:54] tools versus uh resources [693:59] for MCP using the agent SDK [694:04] framework. [694:06] Uh it should [694:08] load a custom [694:10] MCP server uh with that can read a local [694:17] SQLite uh 3D database in this folder [694:23] called to-dos. [694:25] Um, and it should have tool use to uh [694:30] create, update, edit, destroy to-dos. It [694:36] should have uh a re uh MCP resource to [694:40] list out to-dos. [694:43] Okay. So, we'll go ahead and do that and [694:45] see if it will be able do that because I [694:47] gave it a little bit a little bit much [694:48] more than I was expecting, but we'll see [694:50] if that helps. Okay. [694:54] All right. All right, let's take a look [694:55] and see what we have for our code. Okay, [694:59] so [695:05] scroll on down. [695:10] Not what I was expecting. We have the [695:11] to-do server here. Well, I guess we'd [695:12] have to have some kind of server run. Do [695:14] we need a server? Oh, yeah. Yeah, this [695:16] is the MCP server. So, this is the [695:18] actual MCP server. And here it can [695:21] connect to the database. List tools. [695:24] right out. So tool tool tool resource [695:28] and so see this resource and it's just [695:30] making a call here as well but it's [695:32] following this schema format. Um so I [695:35] mean it's still basically like a [695:37] function. It's calling it but I guess [695:39] the difference is that it only expects [695:41] read only information. I always thought [695:42] that it was a direct call and I never [695:44] knew it was a function inside of this [695:46] because I I don't really implement [695:47] resources. I usually just implement [695:48] tools. Um but I know that this is a a [695:51] schema syntax. We say like uh resource [695:54] schema MCP. I absolutely know that [695:56] there's a schema for that. We'll go to [695:58] the model context protocol website and [696:00] see what we have. So somewhere here [696:05] kind of describe it. Not really. [696:08] Uh yeah, I don't know. But anyway, it [696:11] looks like something like this. Okay. Um [696:14] but we'll go back over to this one and [696:16] see if we can make sense of what it's [696:18] doing. So here [696:22] uh we can see we have our server script. [696:25] Okay. [696:28] Convert MCP tool list into anthropic [696:31] tool format. I'm not sure why we should [696:32] have to do that, but okay. Like you'd [696:34] think it would already have done that. [696:36] Um here we're running the agent. [696:39] We're grabbing the tools [696:43] and I don't know why it's using [696:46] anthropic directly. It's using [696:48] asyncthropic. [696:54] So [696:56] it's fine but like could we [697:00] can you use agent SDK? [697:04] Like I don't understand why it's using [697:07] like why why are you not using why are [697:11] you using [697:12] a thropic? [697:17] Now there could be a reason for that but [697:18] we'll see why. [697:20] This is just sonets reasoning. If we [697:22] went to opus I think we'd have less of [697:24] these issues. U but again I'm trying to [697:26] make this cost effective. [697:29] So yes I I don't know why it wants to [697:31] read the whole darn thing. Like it [697:32] should know what agent SDK is. Look, [697:35] stop. Stop. Stop looking around. [697:38] Why did you not use agent SDK? [697:43] Anthropics. [697:45] Agent SDK. [697:52] Let's see if it will answer us here. And [697:56] while that's going here, [697:59] I'll just ask, you know, can we load in [698:01] a custom um MPC server into agent uh [698:07] anthropics agent SDK? [698:10] I'm just asking this to um Claude here [698:13] in the chat. you don't see me doing [698:14] this. Um, but I just want to see what [698:17] it's doing here because sometimes the [698:19] actual Claude uh chat seems to be a [698:22] little bit more uh on point maybe [698:24] because of what tools it has available [698:25] to us, [698:27] but we will see here in a moment. Okay, [698:30] so this thing is still [698:32] trying really hard. And if we go over to [698:34] here, look [698:36] inline MCP. [698:42] So we have this query list files of [698:45] prompts await. [698:50] Okay. And so here what I'm going to do [698:53] is grab stuff like this. [698:56] Okay. Like why? [699:00] Like look here. Okay. Because it's [699:01] trying to get reference information [699:04] and so maybe it just doesn't understand. [699:06] Right. [699:09] So, we'll give it a moment here to fix [699:12] it. And hopefully that gives it context. [699:14] All right. So, now let's see if we [699:16] actually got better code output here. [699:18] Uh, and so I'm just going to close this [699:20] out and reopen it up. [699:23] Okay. [699:25] And so, let's take a look at what we [699:27] have. So, here we have agent SDK. We're [699:30] loading the server. We have the prompt. [699:31] Just demonstrate the difference. F do [699:34] the following steps. So, read the [699:36] resource, use the tool, um, update it, [699:39] read the resource, use the tool, etc. [699:41] Going in that order. And here it is. And [699:43] you can see this is way, way better. I I [699:45] don't know was zoomed before, but that [699:46] was crazy. So, here we have allowed [699:48] tools. Um, [699:51] and if we go up to here, [699:56] MCP tools. So, we're allowing [699:59] any stuff here. I'm just trying to find [700:00] where we are setting the [700:03] MCP server. [700:07] uh [700:08] to-do server server script [700:11] args and so we're passing in here. So, [700:13] oh, here it is. MCP server to-dos and [700:16] then this is the argument to start it [700:17] up. Okay, so that looks pretty uh pretty [700:20] good. We'll go ahead stop this. Yeah, it [700:24] really just comes down to like, you [700:26] know, is what you're looking at correct, [700:28] right? Because you can see it can [700:29] generate out variations of code that [700:31] might work, but that doesn't mean it's [700:33] good code. Um, and so we're just [700:35] watching it and see if it reads. So here [700:36] it's it's listing it out. Um, and then [700:39] it will go ahead and create some. And [700:41] then it's going to do an update. It's [700:43] going to do another read. It's using [700:45] resource. Now it's deleting. [700:49] Okay. [700:50] But we can clearly see, [700:54] you know, they're both just functions. [700:56] But the idea is that these are just [700:58] readonly actions, right? [701:01] Or they should be. Obviously, you could [701:02] put anything you want in here, but the [701:04] other part is that you're just able to [701:05] follow the syntax. Um, could you also [701:08] emit very specific fields? I don't know. [701:10] I don't do enough with resource to know. [701:12] Um, but anyway, hopefully that gives you [701:14] a distinction and treat one as read [701:15] only, one as as like actions. Okay. [701:24] All right. So, another way we can uh use [701:26] cloud codes with GitHub actions. It [701:28] allows us to trigger cloud code based on [701:30] workflows to perform tasks in our repo. [701:32] So here's an example of the cloud bot [701:34] and it is um doing stuff and checking [701:38] stuff off and actually performing uh [701:40] work just like a developer in your team. [701:42] Uh so you can create instant PR [701:44] creations, you can automate code [701:45] implementations, follow your coding [701:47] standards, do simple setup and it's [701:49] secure by default. Um I think that [701:52] description kind of underells it but the [701:54] way it works is that you are setting up [701:56] basically a claw.l ML file in your [701:57] workflow. And if you notice here, uh you [702:00] can tell it where to trigger. So maybe [702:01] you is when you create a new comment or [702:04] if you create a new comment on a pull [702:07] request or you open a new issue or [702:08] assign it or submit it, obviously you [702:10] can adjust this however you want. Okay. [702:12] And then if you notice here, they have a [702:14] cloud code action. And so basically [702:16] it'll just do whatever it needs to do to [702:18] get the job done. Um, and the only [702:20] downside to this, I like the workflow, [702:22] but it's just slow because you're not [702:25] just waiting for cloud to execute, but [702:26] for the GitHub action to execute, and it [702:28] has to spin up that environment every [702:29] time. So, I would just say that the [702:30] GitHub actions are slow. And if it was [702:32] faster, this would probably be my [702:33] preferred method of working because if [702:35] you uh move up in your career as a [702:39] developer, you end up managing um [702:40] developers and writing issues and and [702:43] seeing it through that lens. And that's [702:45] the way I would probably like to work, [702:46] but again, it's too slow. And so I would [702:50] um not drive it that way uh in in [702:53] reality, but it is cool to see that. And [702:54] if you are on the management side, then [702:56] at least if it's in there, maybe use it [702:58] to review things or assist you um and [703:01] stuff like that. Okay. [703:06] All right. So apparently um there are [703:08] integrations with GitHub actions and [703:10] specifically using Collad in our PRs or [703:14] our issues. So that's kind of [703:15] interesting. It'd be cool to make an [703:17] issue and then it executed out. So, I'm [703:19] not exactly sure if that's what it's [703:20] talking about. Um, but we'll go down [703:23] here and we'll take a look and see how [703:24] we integrate it. So, the easiest way to [703:26] set up this is through cloud code and [703:28] just the terminal. Just open claude run [703:30] install GitHub app. You must uh be a [703:34] repo. Install the GitHub app and app [703:35] secrets. Is that going to install the [703:37] GitHub app? Because I don't want the [703:40] GitHub app. Um, [703:44] okay. [703:47] install the GitHub or the clawed GitHub [703:50] app. [703:51] Oh, so I think what it's saying is like [703:54] the actual GitHub like GitHub app, not [703:56] like the desktop application. So what [703:58] we'll do is we'll go over to GitHub and [704:01] I believe that this is my developer [704:02] account and we'll make our way over to [704:05] the examples repo that we have here, the [704:07] cloud code example, assuming the code is [704:09] saved there. Yeah. And so if we go to [704:10] settings and we go to apps, [704:14] right? Is it installed? It already is [704:16] installed. So when we set this up [704:17] earlier, we already configured it or we [704:19] we set it up. I was just going to see if [704:21] there's any additional options. There's [704:22] not. So it is ready to go. And so in in [704:25] a sense, I think we should be able to [704:26] make issues and do um that at sign and [704:30] be able to utilize it right away. So [704:32] what I'm looking for here is that [704:34] information. [704:37] Uhhuh. [704:41] a lot of talk. Just tell me how to do [704:42] it. So [704:46] here [704:50] we have a bunch of stuff and then we [704:52] have actions. Give me a second. Okay. [704:56] One thing I noticed is that it needs [704:57] these particular permissions, but I'm [704:59] pretty sure we just gave it a cart blank [705:01] like all permissions to the that [705:03] individual repo. So I don't think [705:04] that'll be an issue. [705:08] Okay. Okay. But it's not really talking [705:09] about how to use it. [705:12] Like yes, of course there are um these [705:14] actions here [705:16] like if you tagged it review this for PR [705:19] for security issues. I was thinking that [705:20] like if we do the atsign claude then it [705:24] would go in like an issues and go do [705:26] something here. It says with a simple at [705:28] sign mentioned in uh in a PR or issue [705:30] yes claw can analyze your code and [705:32] create pull requests implement fixes and [705:34] bugs. All right. And then I mean these [705:37] are just actions. So I'm assuming that [705:38] that's just if we want it to do [705:40] something. So like if there was a push [705:42] then it would trigger that. Okay, that [705:43] makes sense. So what we'll do is we'll [705:44] go over to here and we'll make a new [705:46] issue and we'll just say create [705:48] individual [705:50] product page. I don't know if the [705:52] mention has to be here in the [705:53] description. So um I'm going to try over [705:56] here first and we'll just say claude. So [705:59] I need um implement [706:03] the following [706:05] page. [706:07] So, uh, create a product HTML page. This [706:12] is the page that describes [706:15] an individual product for purchase. [706:19] Um, this is a mock page. So, um, all [706:24] products will link to this page. [706:28] And you know what? I'm going to go ahead [706:30] and make check boxes and see if it'll [706:32] actually check box these off. I'm [706:33] curious if it would actually do that. [706:38] create for now a mock page for all um [706:43] possible products. [706:47] Point all products [706:50] uh links to this single product HTML [706:54] page. [706:56] Okay, so there we go. And I haven't [706:58] mentioned Claude down here. I've added [707:00] it up there. Be interesting if you could [707:02] assign it to Claude. See that? That [707:04] would be smart. We'll go ahead and we'll [707:05] create it. We'll see what happens. [707:08] And so I'm curious like if we go to get [707:10] up actions, is there any trigger? [707:11] There's nothing happening here. [707:14] So I thought something would happen. [707:17] Okay, I'm going to go ahead here and [707:19] say, you know, Claude, [707:22] are you there? [707:25] I'd be like curious if it like it would [707:26] pop up and start talking. [707:29] We have code with agent mode. That's [707:31] probably for GitHub, [707:34] right? And so [707:38] Claude [laughter] did that. [707:40] [snorts] Okay, so we'll go here. So [707:41] Claude, [707:44] can you implement this this issue? Oh my [707:47] goodness. I I really hope this is what [707:50] it's like just working with a with a bot [707:52] and have like being like, are you doing [707:53] properly? Like this is actually probably [707:54] how I would want to work with this. [707:56] Okay. So, it keeps looking at at um my [707:59] stuff here. [708:05] Claude, are you able to comment back? Uh [708:09] if uh [708:12] if so, indicate with an emoji like [708:17] like one or zero or like or yes or no or [708:21] something. [708:24] or if you can write a comment, just [708:27] write a comment. [708:29] Like, are you okay? Blink yes or no, [708:31] right? And so [snorts] I'm curious. It [708:34] just keeps doing the same thing. So [708:35] that's kind of not helpful whatsoever. [708:37] Um, and so I thought that it would go [708:39] off to the races there because that's [708:40] what it's describing up here. We go up [708:42] the top here and it's saying, right, I'm [708:44] not crazy. It says with a simple Claude [708:47] mention in your PR issue, Claude can [708:49] analyze your code and create pull [708:50] requests all while following your [708:52] project standards. Well, I would love [708:54] that, but it's not doing it. So, just [708:55] give me a moment and seeing what the [708:56] issue is. Okay. So, I mean, there are [708:59] additional steps here, and maybe this [709:01] stuff isn't happening, but I would have [709:02] thought because we had created that repo [709:04] um through Claude web that it would [709:06] already have that, but maybe what we [709:08] should do is open up this repository. [709:12] And [709:14] yeah, we still have this environment. [709:16] So, I'm going to reopen up this in the [709:18] browser. I'm going to see if we can just [709:19] go ahead and run that install script. I [709:21] don't think we have to do it. [709:23] Okay, I think that because we did it [709:24] through Claude web, unless we did it [709:26] through here, and this is actually how [709:27] we did it. I just don't remember, [709:31] but um we'll get Claude working through [709:34] um GitHub code spaces. And then we'll [709:37] try to run this and maybe it's just [709:38] lacking permissions or something. [709:43] Okay, so just waiting a little bit more [709:45] time here. And [709:51] okay. And so I'm just going to go back a [709:53] directory and say claude um off status. [709:57] Is cloud even installed? No, it is not. [710:00] So install claude [710:03] code. [710:08] And we'll go down to here and we're [710:09] going to grab this one here. [710:21] and we'll go ahead and do this. [710:30] Okay, so we're just waiting for this to [710:32] build. Shouldn't take too long, but [710:34] it'll take a little bit of time. Okay, [710:36] so um yeah, it's now installed. And so [710:39] we'll go ahead and we'll do our cloth [710:41] off login. I can't remember if we had [710:43] issues [710:45] um [710:46] logging with this. So, I think what I'll [710:48] do just to make my life easier, assuming [710:51] that the extension is over here, I'm [710:52] going to grab claude code from here, if [710:54] it even lets me. [710:56] Oh, they do. Okay, great. I thought they [710:59] might not have it because uh you know, [711:00] Microsoft might want to keep them out. [711:02] But we can see Claude in here and um I [711:05] thought I saw Curo, but [711:08] don't really like Curo. Sorry, folks. [711:10] I'm not even going to look at it to see [711:12] if I like it because I already know I [711:13] don't like it. Um, but anyway, so now [711:15] quad code is here. And so I want to get [711:18] this authenticated and I'm just going [711:20] the roundabout way to do to do this. So [711:22] it'll make it a lot easier. [711:25] We'll go here. We'll say authorize [711:29] because then if there is, we can uh [711:31] paste paste the code in here. E, and we [711:34] have the callback code up here. So I can [711:35] just grab it out of here. We got to be [711:36] creative, folks. We'll say continue. [711:40] And um [711:44] that should get us authenticated. It's [711:46] not showing us anything, but I'm going [711:47] to go down here and say status and see [711:48] if we're logged in. We are not logged [711:51] in. [laughter] So what I'll do instead, [711:53] one second here. We'll go into cloud [711:55] code examples. I guess we didn't have to [711:58] use it this way. I kind of forgot that [711:59] we can just type in claude like this. [712:02] And yes, dark mode is fine. And we'll do [712:05] that. And we will say open. We'll try [712:07] this again. [712:09] We'll say authorize. I think it knows to [712:11] give us the code to copy paste it here. [712:13] It didn't, but we'll go ahead and grab [712:15] this call back here. [712:18] We'll paste it in. We'll hit enter. [712:22] I copied the whole darn thing. We'll hit [712:24] enter to retry. [712:26] So, maybe that was an older link. I [712:28] don't know. Say open. Close the old one [712:31] just so we don't get mixed up. [712:33] Authorize. [712:35] There we go. That's looking a lot more [712:36] normal. Okay. And so then we'll go ahead [712:38] and paste it in. We'll hit enter. And [712:40] now we are logged in. So we are good. [712:42] We'll hit enter again. Uh yeah, ter yes [712:46] trust this folder. And so what we want [712:48] to do here is get set up GitHub. So or [712:51] GitHub actions installed the GitHub [712:52] actions app. It should already be [712:54] installed, but maybe it will give it [712:56] additional permissions or whatever else [712:57] we need. So it's checking GitHub CLI. Um [713:00] use the current one. Yes. [713:03] And so it's going over to here. Okay. [713:06] Uh, I thought we would do a little bit [713:07] more. So, I guess it's already [713:08] installed. [713:11] So, have read and write to access code. [713:13] Uh, read and write access to actions and [713:15] metadata. I guess we could have just [713:17] checked that the entire time. We didn't [713:18] really have to go through that whole [713:19] install process, but we'll go down here. [713:21] What I'm looking for, uh, not remote [713:23] code, is we're looking for the [713:26] permissions. Do we have the correct [713:27] permissions? [713:31] Permissions. Permissions. Permissions. [713:32] Where is it? Read and write. Read and [713:35] write. read and write. So we go back [713:37] over to here, we confirm [713:39] read and write code discussions issues. [713:42] So yes, it has that access. So there [713:45] should be nothing preventing it, right, [713:48] from being able to respond to this [713:50] stuff. [713:52] All right. So let me figure out the next [713:54] step. Okay. Okay. So it is suggesting [713:56] the anthropic API key and also uh this [713:59] part here. though because we're using a [714:01] subscription I would assume that we [714:02] wouldn't need an API key [714:06] but maybe we do copy the workflow file [714:07] for cloud YAML so maybe that's what [714:09] we're missing is this oh look it [714:11] responds issues pull requ [714:18] um since we already have GitHub actions [714:20] open or sorry this this here I'm just [714:22] going to go ahead and make the folder um [714:25] I believe it is [714:28] uh not a cloud directory But um for ours [714:33] we need to uh make the actions folder. I [714:35] always forget what it is. Uh GitHub [714:37] actions workflow folder name. [714:42] Which is it? GitHub for [714:44] gotgithub/workflows. [714:46] So we'll go back over to here. We'll [714:48] make make a new folder. GitHub to think [714:52] because I made a whole course I'd [714:53] remember this stuff and I use it all the [714:54] time. Nope. [714:56] Brain doesn't work that way. Okay. We'll [714:58] go claude yaml. I don't think it matters [715:00] if it's YML or YML. And so we'll go [715:03] here. We'll take a look. Since we, you [715:06] know, care about what our code is doing, [715:08] we're carefully looking at it. And so [715:09] you can see it's responding to those [715:10] things. If we didn't want them, we could [715:12] remove them out of there. Customize the [715:14] trigger phase for Claude. Um, [715:20] okay. So, it's already defaulted to [715:22] Claude. So, we could probably change the [715:24] name, change the name of the bot. So, [715:27] that looks fine. We'll go ahead [715:29] We'll say add claude. [715:32] The only other thing that we're missing [715:35] is that key, the API key. Um, so we'll [715:38] go over to here and supposedly we need [715:42] go back step. We need an actual key. [715:45] So I'll go over to here and into my [715:47] settings. [715:48] I'm looking for tokens. I mean that [715:51] feels like for like coding stuff like I [715:54] mean like literally directly working [715:55] with the API. But we'll go over here. [715:56] Maybe maybe they have one for here. Oh, [715:58] cool. We can see the history of [716:01] authorization authorization use. Okay. [716:04] So, I'm just going to I just want to [716:05] make a I didn't know that was here. So, [716:07] I'm just saying authorization token [716:08] usage, [716:10] right? Because I thought that was kind [716:11] of interesting. And I'm just taking a [716:13] screenshot. This is just for the course [716:14] for us later here. Okay. But anyway, um [716:18] so they're saying we need that API [716:19] token, but wouldn't that be coming on [716:21] the code side and then we'd have to pay [716:23] for that and it would be part of our [716:24] subscription. [716:26] So maybe that's something that I didn't [716:27] consider, which is fine. We have to put [716:29] 10 bucks on it. I can do that. Um, if I [716:32] can freaking find the page. I really [716:35] dislike their uh their links here. It [716:37] gets really confusing. [716:39] Um, so not that page. [716:43] So here it says anthropic API key. [716:48] So that makes me think that you can't [716:49] use it with the subscription. [716:52] Okay, which is fine because that's an [716:55] opportunity for us to uh grab that [716:57] there. So, um [717:01] I guess we could try to find that. I've [717:04] done it before. Usage. [717:07] I just don't remember where the tokens [717:09] are. Account [717:13] general. [717:19] Uhhuh. [717:20] Okay. So what we'll do is we go claude [717:23] um [717:25] let's just like workbench or something [717:28] because that's the platform right so if [717:30] we go over to here the developer [717:31] platform [717:33] this will have the API token in it right [717:35] so this is separate from your [717:36] subscription and actually have to put [717:37] some money on here the only thing I [717:39] don't like is that if you don't use it [717:40] by the end of the year you lose your [717:41] your money which is I don't like but uh [717:45] begin buying with only five credits um [717:48] that sounds good to me because this is [717:49] where the API keys are right. So, we'll [717:51] go ahead. We'll try this. And I'm just [717:54] going to quickly load this up. Just give [717:56] me a moment. Okay. I'm just buying five [717:58] credits here. I didn't have to enter my [717:59] credit card. It's already entered in. [718:01] Um, and so I just hit that buy button [718:03] and I just said five. And now I'm just [718:04] verifying. So, hopefully it works. I [718:06] don't know if I have a new company [718:07] credit card, so I might have to update [718:08] it, but we'll see in just a moment. [718:10] Okay, there we go. I got a $5 balance. [718:12] So, we'll go over API tokens here. And, [718:15] uh, obviously you can see I I was [718:16] working with Japanese grammar there. Um, [718:19] that's been a while. I'm just going to [718:20] disable the key for now. I'm going to [718:22] create a new key. And this will just be [718:25] for code or uh cloud code [718:30] and it'll be in the default workspace. [718:33] And you didn't see that. I'll have to [718:35] actually delete that. Okay. So, I'll [718:38] copy that key there. I don't know why [718:39] they do that. That's ridiculous. But now [718:41] you've seen it. I'm I'm not going to [718:43] edit it out right now. I'll I'll delete [718:45] the key very promptly uh here after this [718:48] video. Um, so what we're going to do [718:50] here is go to our repository secrets. [718:53] I'm in the repository. [718:55] And we have secrets here. [718:58] Technically, it is for actions to use. [719:01] Repository secrets. That's where they [719:02] said to put it. I'm going to paste that [719:04] in. And I'm going to go back over to [719:06] here. And what we're looking for [719:13] is [719:19] where is it? I'm looking for it. Um, [719:24] this this is what I want. So, we'll go [719:26] back over to here and we'll grab this [719:29] here and we'll add that key, that [719:32] secret. And so, now we have that [719:33] repository key there. And so now in [719:36] theory we have all three things set up. [719:38] And so I'm going to go back over to here [719:39] and say [719:43] and these are all a little bit silly. So [719:44] I'm going to go ahead and delete these. [719:45] Delete. [719:47] Delete. [719:50] Delete. So when is it going to trigger [719:51] on? That's the question. Uh so we'll go [719:53] back to our code and we'll take a look [719:55] here. So on an issue created created [719:58] open assigned submitted. So this issue [720:01] would have to be created. [720:04] Okay, [720:06] I'm just going to go ahead here and um [720:10] can we assign this to Claude now? I just [720:12] want I keep on expecting to see Claude [720:14] over there. [720:16] Does it also trigger on comment [720:20] issue comment? Yes. Yes. Yes. So, can [720:24] Claude, can you implement this [720:28] ticket, please? [720:31] I assume it would do feature branching, [720:32] but we'll find out. [720:36] There we go. And so the main thing is [720:38] like did this get triggered? And it has. [720:41] Okay. And we'll go into here. And so [720:43] it's going right. Um I'm not sure how we [720:46] would go about debugging it. I guess [720:48] here postr run cloud. [720:53] So it's actually installing cloud code. [720:55] And that's probably why you need the API [720:56] token. And so I assume it can't use [720:59] subscription. But I would think that [721:00] there'd be a way to configure it to use [721:02] subscription. [721:05] But maybe because it uses that off mode [721:06] that just doesn't work. I'm not sure. [721:12] But anyway, [721:15] it's going. So [721:19] I'm just trying to figure out like what [721:20] we expect to see. And there it's using [721:22] cloud sonnet 4.6. I would imagine that [721:24] if we needed to adjust it, we could [721:26] probably go into that uh GitHub actions [721:28] file and like say, hey, use this model [721:30] and stuff like that. Yeah, we can right [721:31] here. So, how many turns, what kind of [721:34] model, additional system prompt, things [721:36] that we can tweak it. So, that looks [721:38] good in terms of what it can do. I would [721:41] assume that as it's executing those [721:42] actions, we would see them right here [721:44] because that's run cloud code, right? [721:50] So, I'm just going to pause it here and [721:52] um because we're not seeing much and [721:54] we'll be back in just a moment. Okay. [721:56] All right. Let's see what the results [721:58] are. So, down below we can see [722:02] okay, multiple runs [722:05] and um that's pretty clear. I was hoping [722:09] to like clearly see the commands that [722:11] are running, [722:13] but I'm not sure if I can really tell [722:14] from here. Um, [722:17] what's going on? We'll go down. Complete [722:19] job. Usage would be kind of interesting. [722:21] We just have 3 point U. I thought maybe [722:24] it'd be like tokens you use, but I guess [722:25] not. Workflow file. That's just the file [722:28] we're in. Let's go back over to issues [722:30] and we'll take a look here. [722:32] And [722:34] well, I guess I made this so it wouldn't [722:36] be able to check box these off. It would [722:37] be me that would check them. E. So, [722:39] implement implementation complete. So, [722:42] gather contact. Oops, I didn't mean to [722:44] check box that off. I hope that wouldn't [722:45] trigger it more than once. Let's see. [722:47] Would that actually trigger it again? [722:48] That'd be really annoying. [722:52] Uh, [722:54] no, it did not. Okay. So, we'll go back [722:56] over to here. [722:59] Uh, created bath box on product.html [723:01] page uses the URLs never asked to do [723:04] that. Contains mock for all six pages. [723:07] Okay. And so I could kind of see that as [723:08] a process to like I might want to do [723:10] stuff, but it didn't make a pull [723:12] request. Okay. So what I would have [723:14] preferred is that if it followed that uh [723:16] the branching strategy, um also I don't [723:19] see it here. [723:21] So we'll go back for a second. Yeah. So [723:24] we have a separate um [723:27] separate one here. Do we have a pull [723:29] request? [723:32] Okay. [snorts] [723:33] Where's the pull request? [723:39] I'm just going to see if it attempts to [723:41] go make one. The annoying part is that [723:43] it probably will have to trigger every [723:44] time this GitHub action and that's a [723:46] little bit slow. [723:49] Yeah, it's triggering right now [723:52] because you don't want to wait like [723:54] three minutes every time you reply to [723:55] it. [723:59] And but I could see how we could work [724:01] around that a bit. [724:06] So again, we'll just wait a few minutes [724:08] and give it some time. Okay. All right. [724:11] So I thought that was executing, but [724:13] maybe it stopped. I just don't recognize [724:14] the symbol here. I thought maybe uh that [724:18] would indicate something. Maybe it did [724:19] reply. Let me go back over to here [724:22] and Oh, not here. Not here. Um issues. [724:27] We'll go back here. [724:29] Oh, I didn't tell it to. Sorry, Claude. [724:34] Uh, where's the pull request? [724:38] Uh, can you make one, please? [724:46] And, um, so we'll go ahead and let that [724:48] do there. I guess it probably also need [724:50] to know where to, um, [724:53] where to make a pull request into. And [724:55] so what I'm going to do just quickly [724:56] here, it probably will move faster than [724:58] I will, but I'm going to create a um a [725:00] new branch called main because there [725:02] should be one. There just isn't. And [725:04] then I'm going to go over to settings. [725:05] I'm trying to do this as quick as I can. [725:08] And [725:14] yeah, of course, switch it. That's [725:16] ridiculous to having it as that. We'll [725:17] go to main. Okay. And that's what it [725:19] really should be. Okay. And uh you know, [725:22] we'll see how that works. I don't know [725:24] if it if it if it doesn't have a main [725:25] branch if it'll mess up. And also, I was [725:27] kind of doing that in the middle of [725:29] stuff. So, we'll see what happens. So, I [725:30] guess these ones are like, oh, a comment [725:33] was created. Oh, they created a comment [725:35] here. So, let's go back over to issues. [725:41] Oh, okay. So, this is how we're going to [725:44] monitor it. That's interesting. [725:50] Interesting. Interesting. [726:00] Okay. [726:06] But I'm sure there's other stuffs that [726:07] we could tell it to do which we're just [726:09] obviously not doing right now. Um, [726:12] but we will give it a moment here [726:16] to figure this stuff out. Yeah, because [726:19] we run the job. The only thing is like I [726:20] can't see what it's doing. You know, [726:22] that's the only thing I don't [726:23] particularly like, but I guess there's a [726:25] lot of options in terms of the ways we [726:26] can work here. [726:28] Um, [726:30] we we'll hang out here for a bit. Okay, [726:33] there we go. So, our pull request um [726:35] according to this has been created. Oh, [726:36] we just got the updated information. [726:42] And so the question is like would it [726:44] known to send that pull request over to [726:47] um I don't see it here actually. Would [726:49] it send it over to the correct location? [726:51] So I'm not sure if it's still running. [726:53] It is finished. [726:59] Create pull request. [727:06] Uhhuh. [727:08] Where's the pull request? [727:18] Okay. So, I would have thought describe [727:20] what you need Claude to do. I'm just [727:21] checking here. Claw creates a complete [727:23] PR with all necessary changes. So, it [727:25] should be able to create a PR. So, I'm [727:28] going to go back over to here. [727:32] Yo, you uh Claude, you didn't [727:39] please create a PR. [727:42] Can you make the [727:51] also notice here that it doesn't have a [727:54] a branch associated with it. [727:58] also this issue [728:00] the branch isn't associated with it [728:06] right because that's what I would like [728:08] it to do [728:10] and so you know if if we were spending a [728:13] lot more time with this I would spend [728:15] more time trying to force it through a [728:17] very specific workflow just so that I [728:19] could come here and and see things the [728:20] way I expect to see them but we'll give [728:22] it another chance here and see what [728:23] happens and it is running the job so it [728:26] is working so there we go so I'll [728:27] analyze and back to you. [728:30] It's just like we can't really have a [728:32] conversation with it. So, it makes it [728:35] kind of hard to uh you know, [728:38] whatever, whatever. [728:41] So, I don't know if this would be a [728:43] really good workflow, but I mean, it is [728:45] an option for us, but we'll give it a [728:46] moment. Also, one thing I noticed is [728:49] like the order. So, like this feels like [728:51] this should have happened first and then [728:52] afterwards. So, it's interesting that [728:55] it's kind of matching based on the order [728:58] that I said as opposed to what would be [729:00] most optimal. But I guess it's not a big [729:02] deal, but I would would probably like to [729:05] see that done first. And so here we can [729:07] see it's reasoning. So I found the [729:08] branch and there's been no changes. The [729:09] actual implementation is in the product. [729:12] Um create a PR from the branch that has [729:14] been implemented. GitHub doesn't have [729:15] automatic association with existing [729:17] branches. You can link them from the [729:18] issue sidebar. [729:21] I guess maybe they might mean to say [729:23] like through the API. Like I thought [729:24] maybe through an API call they'd be able [729:26] to do it. Um, [729:29] and do we still do we get a PR? No. So, [729:33] there's obviously some trick to this. [729:34] Maybe it's because I'm asking it to go [729:36] to main, but before that was less of an [729:38] issue. But I'm going to go ahead here. [729:39] I'm just going to do it myself. [729:41] And [729:44] uh, [729:45] I guess there's two I guess they created [729:48] two separate branches for the exact same [729:49] thing. [729:52] But anyway, I'm going to merge each of [729:54] these individually [729:58] pull request. [730:01] And I'll do this one here. [730:07] Oh, maybe that's their initial one. [730:10] Okay. [730:12] And so we'll hold on here. No, no, no. [730:15] That's the wrong one. So we'll go to [730:17] this one here. [730:21] Uhhuh. I can't tell the order because I [730:24] would have thought there's code in here, [730:25] right? [730:29] Oh, maybe Maine. Okay, so Maine's [730:31] already up to date. I see why. Because [730:32] it's just the order that I created it [730:34] in. That's fine. So, we'll go ahead and [730:35] we'll create that. We'll say create the [730:37] pull request. [730:39] And I just want to see the new page [730:41] working to be honest. That's all. We'll [730:43] confirm and merge. [730:45] And then we'll go back up to [730:49] Uh this here we'll make another pull [730:52] request. [730:54] Oh, I don't know which one we just [730:55] merged. [731:01] Does it say? [731:03] I don't like the issue names. This makes [731:05] it really hard. 1956. So we'll go over [731:08] to this. [731:10] We'll create another one. We'll do this [731:13] one here. [731:15] So that would be something I would want [731:16] to know is like, okay, can I make more [731:18] human readable names? [731:21] And so we have conflicts. I think that [731:23] they're identical. So I don't think [731:24] there's any difference to those. Let's [731:27] go back over to here. Um, and I want to [731:31] go Oh, my code p code space is stopped, [731:33] but we'll start that back up. What I'm [731:35] going to want to do here is I just want [731:36] to go and uh pull and check and see what [731:40] the result is. I'm sure it's fine, but I [731:42] just want to see it for my own my own [731:44] interest. [731:51] Mhm. [731:56] [snorts] [731:59] Get checkout main. [732:02] Get status. Okay, so that's not going to [732:05] work. Um, no, it should. [732:09] Get pull [732:12] Get checkout main. [732:16] Get pull. [732:20] And we have a product page now. Um [732:25] I don't have my products page. [732:28] So that other one must have been it, [732:31] right? [732:33] Uh we'll just resolve the conflicts, I [732:35] guess. [732:39] So, it's like, do you want the same one [732:41] or the same one? Is there even a [732:43] difference between these two? [732:46] That one's gift set or gift set. [732:49] I don't think it matters to be honest [732:51] because it's all just example stuff. [732:54] I just didn't feel like resolving [732:56] conflicts. That's why uh didn't do that [732:58] there. [733:01] I don't know which is which here. [733:10] Oh, this makes it really hard to know. [733:16] We'll just accept both changes [733:19] because I like probably both are [733:21] supposed to be there, but I just really [733:22] can't tell. So, we'll go ahead and do [733:24] this. So, you can already see like [733:26] working with this you if you'd have to [733:28] break up your code into a lot of small [733:30] files. Otherwise, it's going to be [733:32] really hard to tell what's going on. [733:36] And yeah, [733:39] so doesn't necessarily make anything [733:40] easier. So we'll mark resolved. Mark [733:43] resolved. We'll commit. Commit merge. [733:45] We'll go ahead over here [733:49] and uh merge pull request. Merge pull [733:52] request [733:55] and we'll do a get pull here. [733:57] So that's now updated. Um [734:01] HTTP server. [734:03] Okay, we'll go over to here. I want to [734:06] open the browser. Let's take a look at [734:08] what we have. [734:10] Some things might be broken because we [734:12] really didn't take the time to make sure [734:14] these requests were good. Looks more or [734:17] less the same. You think these would be [734:19] clickable, but no. And so this is that [734:21] individual order page. It's fine. It's [734:24] not obviously good. Maybe there's an [734:25] error because of the merges we did. [734:29] There is. See? So when I merged it, [734:32] something messed up. I'm sure we would [734:33] have had a nicer page than that. But the [734:34] point is is that that's that. So that's [734:36] totally fine. Um I'm going to go ahead [734:38] here and close out this GitHub code [734:41] spaces. And I mean that's pretty much [734:44] it. So I think we learned what we could [734:46] there. And obviously, you know, you have [734:49] to really think about your workflow and [734:51] work through a sample project and iron [734:53] out the uh little kinks in that. But [734:56] there you go. Okay. [735:01] Hey, this is Andrew. In this video, we [735:03] are going to be building a um I guess a [735:07] support bot because um we want to take a [735:10] look at how to handle escalation as that [735:13] is something uh that the exam wants us [735:15] to know. And so what I'm going to do is [735:17] go into our hello world, which is a [735:19] great example over where we can start. [735:23] And I'm going to [735:26] go over and make a new folder. And this [735:29] will be our [735:31] support agent. [735:34] And I will make a new file here called [735:36] main.py. [735:37] I'm going to go ahead and paste in our [735:38] code. And I'm going to do something a [735:40] little bit different. We're going to [735:40] make a plan ahead of time. So plan.mmd. [735:44] and because I want to design this in a [735:46] particular way and I'm just looking at [735:47] this off screen here to decide what it [735:48] is. So first like uh uh business use [735:53] case. So we'll say um we are building um [735:58] exam pro trading inc. And by the way we [736:00] use real humans. I can't if I even [736:02] wanted to make this BO would never uh [736:04] accept it like you don't understand he [736:06] like uninstalled his anth or sorry [736:09] stopped paying for his anthropic um [736:11] subscription because he likes to do [736:12] everything manually. So on Exam Pro [736:14] right now, at least the time of this [736:16] video, uh support is 100% humans. But [736:18] let's pretend that we want to be greasy [736:20] and we want to uh uh have a an agent [736:23] that does this. We'll say Exam Pro [736:24] training inc uh needs a support agent to [736:28] um handle requests. Um and so you know [736:33] we have to give more context. So Example [736:35] Training Inc. is a tech education, a [736:38] tech educa or a tech platform that has [736:44] um that's that has multiple [736:48] um self-paced [736:51] um tech education courses [736:54] similar to Okay, so we'll say that um [736:59] they need a [737:02] uh customer [737:04] support agent [737:07] that will um [737:10] resolve common issues. So we'll just [737:13] define common issues. So what are the [737:15] common issues that we have? Um I would [737:18] say you know refunds uh changing [737:22] changing uh swapping out courses [737:24] swapping out courses that are similar in [737:28] price. Um, we'll just say bought the [737:32] wrong say refunds or swapping out [737:35] courses, eg purchased wrong course. [737:40] Um, one that always happens is the [737:44] um [737:47] not receiving [737:50] um [737:52] email [737:53] uh [737:55] confirmation email. Confir confirmation [737:58] email. That one's really, really [737:59] annoying. Um, what else? What else? [738:01] Refund swapping out, not receiving [738:03] email. Um, [738:09] technical mistake or a mistake or we'll [738:12] say bug with the platform. [738:15] We'll say um issue or issue with a [738:20] specific video [738:23] um eg out or issue with spec specific [738:28] content problems [738:32] you know stale eg stale content [738:37] uh or um no audio in video. [738:41] So we say eg um [738:45] you know quizlet button does not work. [738:50] Uh star count or yeah say like [738:56] say star count is incorrect [739:00] and maybe like business [739:03] development [739:04] opportunity. [739:07] Uh we'll say um what's the thing they [739:10] call it's like where they want their [739:12] data taken down because they're from [739:13] Europe uh European data [739:17] uh ask for data take down I don't know [739:20] what you call that I forget what it's [739:21] called GDPR [739:22] GDPR request [739:26] right um and so you know maybe those are [739:28] things that we have common let's not say [739:30] issues but say common requests [739:33] okay and so the uh agent needs to handle [739:38] the following. [739:41] Uh what else do we need to solve for? So [739:43] we'll go back over to this document [739:45] here. So appropriate escalation. So we [739:49] have um customer requests for a human. [739:53] Okay. So let's talk about escalation. So [739:55] escalation [739:57] actually we won't worry about this right [739:58] now. So we'll just say uh version one [740:01] implement an agent using uh anthropics [740:07] agent SDK [740:10] not the a not the anthropic lowlevel [740:16] SDK [740:18] and [740:21] create uh [740:26] and create Uh I don't know what to say [740:29] here [740:31] because we're not like we like they [740:34] can't actually invoke stuff. Oh yeah. Oh [740:35] and uh we will mock out [740:41] uh API requests [740:44] um eg like create a ticket [740:48] um create a ticket or issue a refund or [740:53] things like that, right? [740:57] as tool calls. Okay. [741:00] Uh as an MPC server, [741:04] we we probably would have an MPC server [741:08] that goes to Exampro endpoints, [741:12] but then there are I guess it's mostly [741:17] and so tool calls would live there. [741:24] Okay. So, we'll do that. And I Yeah, I [741:26] know it's not the best writtens, but [741:28] we'll go ahead and and execute this. So, [741:30] we'll say uh support agent. We'll say [741:32] claude. [741:34] And I'm going to say plan. We'll say [741:37] build me version one. And we'll go ahead [741:41] and let that do that. All right. So, [741:43] it's produced some kind of code here. [741:45] Let's go take a look and see what we [741:46] have. Um, and so here, your customer [741:49] support agent for exam pro tech uh tech [741:52] training platform. literally just grab [741:53] the prompt. I think for me like I would [741:56] think I would have a knowledge base and [741:58] tell it how to handle all of these kind [741:59] of requests. We're not going to build [742:01] out a a giant support agent. There is [742:03] actually interest in that. I would make [742:04] that a separate course. We needed this [742:06] just to work just good enough for our [742:08] use case. So always be polite and [742:09] empathetic. Oh, good. So we're providing [742:12] uh a state here. It is using agent SDK [742:16] which is good. It's not using our end [742:19] dot. So you you you need to load in m [742:23] dot you keep removing that. So please [742:27] for thev file please refer to I don't [742:32] know like does hello world have that I [742:33] feel like it does right. [742:37] I don't know why it does that. It drives [742:39] me crazy. [742:42] Oh it never did have it. Oh well there [742:44] you go. There you go. I've been [742:46] complaining this entire time. It's just [742:47] not in the example that we're pulling [742:49] from. And so I'm the problem. I'm the [742:52] problem, folks. [742:55] So I'm going to go ahead and I guess [742:56] I'll just manually grab it here. [742:58] Actually, I'm just going to tell it to [742:59] do it. There we go. And so it should [743:01] just be able to update that there. And [743:03] we can continue on reading here. So um [743:06] always be polite. So here are our tools. [743:07] issue a refund, uh, swap a course, [743:10] resend confirmation email, uh, create a [743:12] bug ticket, uh, create a content issue [743:15] ticket, uh, route a business [743:17] development, um, process GDPR request. [743:20] And so here we have our, um, tool calls [743:22] for those particular things. And so we [743:24] can run our support agent and have a [743:26] conversation. Hi, I accidentally [743:27] purchased a course with the ad solution [743:28] architect. Um, oh, cool. And so we have [743:31] that information. We don't have ordered [743:32] numbers like that, but let's go ahead [743:34] and um see if it works now. And it [743:38] should have added in that env. [743:43] So we'll go ahead and type in python [743:45] main.py. [743:48] And here we have unexpected tool [743:50] handler. I'm not sure why we'd be having [743:51] that problem. [743:54] Uh I guess it just made something up. [743:57] So, [743:59] we'll go ahead here and go back to [744:01] Claude and we're using Sauna, right? [744:03] We're trying to keep our costs down [744:05] here. If we used Opus, it probably would [744:07] work, but u you know, let's you know, be [744:09] able to get through this course, right, [744:10] with actually stuff. Okay, so let it [744:13] adjust that fix that problem. Okay, so [744:17] it did a lot more than what I asked it [744:18] to do, but says fix the cloud uh cloud [744:21] agent SDK. And now we have a tool loop. [744:23] Good. Let's go ahead and try run this [744:25] and see what happens. [744:27] Oh my goodness. This thing is just not [744:31] implementing anything correct here [744:32] today. [744:39] There we go. [744:42] And so here we're having a bit of back [744:43] and forth here. So hi, I accidentally [744:45] purchased this course. I'm happy to help [744:47] you swap your course. Let uh you get the [744:49] course ID. And so here I need your [744:52] information. However, I need specific [744:53] course IDs for both uh to complete. [744:55] could you provide them? Uh, these are [744:57] typically the found courses, etc., etc. [745:00] So, what I'm going to do here is I want [745:01] to change this so that it's actually a [745:02] loop. So, I'm going to go back over here [745:05] um make the agent a real uh loop I can [745:10] talk to in um when I run it [745:14] when I run it. Okay, because that right [745:16] now it's like hardcoded, right? These [745:18] values and we don't want that. So we [745:20] actually want to interact with this and [745:21] so we will ask it to do that and then we [745:23] will look at our next versions. Okay, [745:26] let's take a look here. The agent is now [745:27] in interactive loop. Uh we have [745:29] messages, run, turn, etc. And so now [745:32] what we can do is kind of drive our [745:33] interest and see what it can do. Oops. [745:36] Oops. Oops. Oops. Oops. We actually want [745:37] to run it. Okay. So we'll go ahead and [745:38] run this. And so say uh I want a refund [745:41] for the CLF CO2 I purchased. [745:45] Okay. And so see what it does here. [745:51] So, it goes, okay, I'm happy to help [745:53] with a refund. I need a few pieces. So, [745:55] this would be andrewample.com. [745:58] Uh, order order one two three4. [746:01] Um, I don't need I uh [746:06] didn't use the course. Okay. So, we'll [746:07] go ahead and do this. I'm not sure why [746:09] it's double repeating here. That's kind [746:11] of annoying here. So, perfect. Uh, your [746:13] refund will be processed successfully. [746:14] Here are the details, etc. So there's an [746:16] example of it understanding. Um and so [746:19] if we go over to here, let's take a look [746:20] here. Um and so we have customer [746:23] requests for a human policy exception [746:25] gaps and ability to make meaningful [746:27] progress. Meaning like um there isn't an [746:30] option to talk to a person. So imagine [746:33] you're trying to talk to a person. Let's [746:35] say can I can I talk to a human? [746:42] Unfortunately, I don't have a direct [746:43] transfer function available in my [746:44] current system. [746:46] And so um the thing is like with these [746:49] systems you generally want them to [746:50] follow some kind of structure before [746:52] there's an escalation. So maybe what we [746:54] should be making is some kind of uh [746:56] graph or diagram in terms of what it is [746:58] that should escalate it. So, I'm going [746:59] to go back over here and we'll say, you [747:02] know, I want to show I want to have uh a [747:05] system where uh through progressive [747:07] escalation [747:10] escalation um [747:14] they can uh be turned over to a human. [747:20] um how can we map out uh that step or [747:24] flow and feed it [747:28] into the agent? And I would think like [747:31] like is there a state machine a simple [747:34] state machine? [747:37] So, you know, like what we could do is [747:39] tell the the prompt like what the state [747:41] machine is and then have it update it. [747:43] That' probably be the easiest way to do [747:44] it. Um that's what I would do, but let's [747:46] just see what it comes up with here. Um [747:48] because once we've defined defined like [747:50] a flow or steps that it should follow to [747:53] get to those desired states then uh we [747:55] can see here. So here yeah it's putting [747:56] it right in it into the thing. Bot [747:58] handling escalation trigger human Q [748:00] human active resolved right. So customer [748:03] explicitly asks for human AI can't [748:05] resolve it after turns. AI decides it's [748:07] out of scope. Uh certain issues um auto [748:11] escalate. So GDPR requests legal how it [748:13] feeds into the agent. So add escalation [748:16] to tools. Um so now it has a tool for [748:19] that track the state and count in the [748:22] session context. And so we will take a [748:25] look and see what that looks like. [748:26] There's obviously like many ways that we [748:28] could implement this. So I'm not saying [748:30] like this is how I would actually do it. [748:31] But like if it works in this one case to [748:33] for our understanding that that will be [748:35] good enough. [748:37] So we'll go over here to support agent. [748:40] Let's take a look at what we have here. [748:43] So, we have this, but like I would [748:46] probably map out state machines for all [748:50] of these things. This is probably what I [748:51] would do. [748:53] Um, [748:59] so what I'm looking for here, so here we [749:01] have GDPR request [749:04] wrote business development. Sure, sure. [749:06] Sure, sure, sure. Um, did it add human [749:11] intervention [749:13] or is there human? [749:16] I don't think it did. So, I'm going to [749:17] close this again. Reopen it. [749:21] Human. [749:22] Oh, want me to implement it? It didn't [749:24] even implement it. Yes, [749:26] I thought it was already there. So, [749:28] we'll we'll hang on and see what we get. [749:29] Okay. All right. So, now we should have [749:31] a loop. Let's take a look here um and [749:34] see if we can find exactly where this [749:36] information is. [749:37] So we'll go all the way to the top here [749:40] and okay so basically this is our um [749:43] escalation state. So we have some state [749:45] here and then we have a session context. [749:47] So how many turns the reason the [749:49] messaging which is fine and it's current [749:51] state. So bot handling escalation [749:53] triggered human Q human act active [749:55] resolved. Um so that's fine like we can [750:00] try this but how do we trigger it? So [750:02] you must call the escalation to human [750:04] tool with any of the followings are [750:05] true. The customer explicitly asked to [750:07] speak to a human. Uh this involves a [750:09] legal threat, fraud claim or dispute. [750:12] The customer expresses extreme [750:13] frustration. You cannot resolve the [750:14] issue after two three attempts. You the [750:16] request fails. So we'll go ahead here. [750:19] When they say request a human, but we [750:20] technically don't want that to happen. [750:23] So, [750:27] so we could go in here and say [750:30] um [750:34] the thing is like we wouldn't always so [750:38] do not attempt to handle the situation [750:40] clearly that requires human judgment. [750:41] Right? So, um, if the customer, [750:50] so I'll just take that one line out [750:52] because I really want it to, you know, [750:55] attempt its best effort. So, [750:58] try, you know, if it doesn't, Ah, we [751:01] won't even put that in there. But, like, [751:02] you know what I'm trying to say? Like, [751:03] you know, if if you can try to not [751:05] bother the humans, right? Unless really [751:08] absolutely needing to, right? Um so [751:10] we'll go ahead and we will run this [751:12] here. We'll say uh I I am having [751:16] problems uh with my CLF CO CO2 count [751:21] source. [751:24] Okay. [751:26] Uh I want a refund. [751:33] I say Andrew.com. [751:40] No, [751:45] I need that information now. So now I'm [751:48] being rude. [751:54] Look, I told you now. [752:03] And so here it it transferred to human. [752:05] So there you go. So you can kind of see [752:06] how that works and customers extremely [752:08] frustrated demanding immediate refund [752:10] for the CLF CO2 course customers having [752:13] providing email for that information. I [752:15] guess I didn't really write in the plan [752:17] like what these uh iterations were. Um [752:20] but the idea was to I guess I should [752:23] have done a better job with that. But if [752:24] you're following along it's it's not [752:25] that big of a deal. Let's go back over [752:26] to our um uh exam guy here and just take [752:31] a look and see if we are satisfied. So [752:33] we have appropriate escalation triggers. [752:35] Customer requests a human, right? The [752:38] distinction between escalating [752:39] immediately when a customer explicitly [752:41] demands it versus offering to resolve [752:43] when the issue is straightforward. So [752:45] you'll notice that like it was staying [752:46] on track. I mean really wouldn't be hard [752:49] for us to do that. But I think again to [752:50] make this really good, we would have to [752:52] expand [752:55] um all of these uh all of these options [752:59] and really make a state machine for all [753:01] of them because that's usually how the [753:03] systems works is that if you know if you [753:05] can keep them in that system, it's going [753:06] to save humans time. I'm not saying it's [753:08] good. I don't obviously don't like that, [753:09] but I'm just saying like if we're [753:11] engineering based on what they are [753:12] saying here. Um why sentiment based [753:15] escalation and self-reported confidence [753:17] scores are unreliable proxies for actual [753:20] complexity use case. So sentiment [753:22] sentiment based escalation means like [753:24] are they mad? Why wouldn't that work? It [753:26] obviously worked right there. So adding [753:28] explicit escalation criteria with few [753:29] shot examples. So here it wants us to [753:32] add examples to give an idea. Honoring [753:35] explicit customer requests for human [753:36] agents immediately without first [753:38] attempting investigation. [753:41] I mean it would have done that if we had [753:43] that line there to begin with. [753:44] Acknowledging frustration while offering [753:45] a solution when the issue is with the [753:47] agents capability escalating when the [753:50] policy is ambiguous or silent or the [753:52] customer uh requests instructions the [753:54] agents to ask for uh additional [753:57] identifiers with the tools. So it's [753:59] already kind of doing some of these [754:00] things and basically they've given us [754:01] the list. I like I don't I'm not usually [754:04] pulling from the exam guide like this, [754:05] but I think in this case we would just [754:06] go here and say like um uh [754:12] like implement implementation list [754:17] to-dos. [754:19] Okay. And so we'll go over to here. [754:21] Really it's that it's that simple. We'll [754:23] say plan uh please implement the to-dos. [754:29] Okay. [754:30] and we'll come back here and just look [754:32] at what it's changed in the code. But [754:34] again, like you know, if we're doing [754:36] this for real, and again, I just don't [754:38] want to make a three-hour course here. [754:39] You folks are going to yell at me [754:40] because you want to pass your exam here. [754:42] But I would make state machines for each [754:44] of these things and test them and do a [754:46] bunch of other stuff. But for this case, [754:48] we're just trying to cement this [754:50] information here. But this seems pretty [754:52] evident. The only thing I'm not I'm not [754:53] clear about is like why sentiment based [754:55] escalation and self-report confidence [754:57] are unreliable proxies for actual use [754:59] case complexity. I guess if something [755:00] was very very complex then it doesn't [755:03] work and then you'd have to offload to a [755:05] human. Um so maybe that's what they're [755:08] trying to say there. It's not exactly [755:10] clear but I think we got questions we [755:13] would be able to handle them. We'll wait [755:14] a moment here. Okay. Okay. So um it's [755:18] now updated that code. Let's take a look [755:20] and see. I uh it's probably just mostly [755:23] going to be in the prompt here. Wow, [755:24] that's large. So here um that's a saying [755:28] be polite multiple matches ask for [755:30] clarification. So if a tool returns [755:32] multiple matching records [755:35] uh [755:37] let's add the word wrap here so we can [755:39] actually read it. If a if a tool returns [755:42] multiple matching records do not guess [755:44] or pick. That's actually smart. Ask the [755:46] customer for additional information. Uh [755:49] multiple orders found for janeample.com. [755:52] I found two orders on your account. [755:53] Another thing that's like, you know, [755:54] where's the source coming from? So, if [755:56] somebody's asking you like, how do you [755:58] know you're not just grabbing [755:59] information for a just like someone's [756:02] just trying to trick the bot, right? But [756:03] if you're logged into the account, I [756:04] feel like if it's within your platform, [756:06] you already have that data. Handling [756:08] customer frustrations. When a customer [756:10] expresses frustration or dissatisfaction [756:12] but does not explicitly ask for human, [756:14] acknowledge the frustration and offer to [756:16] resolve the issue. Only escalate if [756:18] necessary. Oh, so they have some [756:20] examples. Okay, good. [756:22] um explicit human requests, no [756:24] frustration context. So if a customer uh [756:27] very first messages uh and they want to [756:30] speak to human, call it right away, [756:32] right? Obviously there are some systems [756:33] that are evil that want to push this [756:35] off. Right? A policy ambiguity. Escalate [756:39] rather than improvise. If a customer [756:41] request is not covered by your defined [756:43] capabilities or fails in a gray area, uh [756:46] escalate to a human rather than guessing [756:48] or improvising. [756:50] Okay, good. competitor price match. I [756:52] saw this course. Can you uh match that [756:55] price? I have two exampro accounts. I [756:57] want to merge them. The course is listed [756:59] at 49 on the site, but was charged 79. [757:02] Actually, that's a good example. [757:03] Sometimes that happens. I don't know. [757:04] Like that's not the prices, but that's [757:06] an example of where maybe the coupon did [757:08] not apply. So, escalate immediately. [757:11] Offer resolution first. Um resolve at [757:15] autonomously. [757:16] and we will [757:20] go ahead and try this out. So, that's [757:21] pretty good. Um, but again, I would do [757:24] more with the state machine if it was [757:25] me. So, say, um, I want a refund. [757:33] Uh, we'll say andrewpro.com. [757:41] Uh I it's for the CLF CO2 [757:48] I don't have the other information. [757:55] Um [757:59] so here [758:01] I don't know [758:07] let me connect with human support. Okay, [758:08] great. And so there it did and it it [758:10] transferred it to a human. So that it's [758:12] pretty pretty good I would say um for [758:15] what we have there. The only thing that [758:16] that doesn't really help is that sent [758:18] sentiment uh information. Um but I think [758:21] generally you know the the key thing is [758:24] like make sure the customer is not [758:25] frustrated. [758:26] But we will um continue on we'll [758:29] continue on with other stuff but I think [758:31] our support agent solves uh the things [758:34] we're trying to uh go for. Okay.