How to Deploy, Secure, and Automate Full-Stack Web Apps – Course for Beginners

[00:00] Taking a web application from local [00:01] development to a live secure production [00:04] environment can be daunting if you've [00:06] never looked under the hood. In this [00:09] comprehensive step-by-step course, you [00:12] will learn to manually configure an [00:15] Ubuntu server, set up essential runtimes [00:18] and search services, and establish [00:21] automated CI/CD pipelines. By doing [00:24] everything hands-on before introducing [00:27] automation, you will gain a deep [00:29] foundational understanding of full stack [00:32] deployment and how to reliably host [00:35] complex systems. Immad Sadi developed [00:38] this course. Hi everyone, I hope you are [00:41] all doing great. In this course, we are [00:44] going to talk about full stack [00:46] deployment. This is a long course but [00:49] trust me if you stick around and follow [00:52] each chapter slowly you will learn a lot [00:55] of things because I try to do things [00:57] manually and to go step by step even if [01:00] you haven't tried anything and the [01:03] problem that here that I'm trying to [01:05] solve and that I will try to teach you [01:07] is how to take a web application that [01:09] you have developed locally and deploy [01:12] it. I have done this exercise three [01:14] times for websites that I developed for [01:17] myself and that I deployed for users to [01:20] use and I will try to teach you all the [01:22] things that I have learned across [01:24] different domains. So let's look at the [01:27] slides and let's see what we will learn. [01:30] Let's define the problem in detail. [01:32] First you have your web application. [01:35] That's the prerequisite for watching [01:37] this course. You can develop it using [01:40] Python, Vue.js JS or other web [01:42] frameworks and you want this application [01:45] to be used. Here we have users so they [01:49] they need to access the web application [01:51] but they can't because you are running [01:54] it locally on your laptop so they can't [01:57] access it. What do you need to do? You [01:59] need to take your source code and deploy [02:02] it somewhere. So this is what we call [02:04] the production environment. Here you [02:07] rent a server where you can launch your [02:10] application code. You can use platforms [02:13] like Digital Ocean, Cloudflare, [02:16] Lenode or others. There are plenty of [02:18] platforms that we can use. And now your [02:22] users can access the application that [02:24] you have deployed and they are going to [02:26] be happy. This is the problem. How do we [02:29] do this? We have the web application. So [02:31] we need a few steps to follow in order [02:34] to deploy the web application and this [02:37] is exactly the problem that we are going [02:38] to solve in this course. What is the [02:41] solution? In our case, we are going to [02:44] rent as I said a server from let's say [02:48] digital ocean, but you can do the same [02:51] thing with other platforms. And this [02:54] server can be either virtual or [02:56] physical. We are going to explain this [02:57] later. And some of these managed [02:59] platforms simplify the process for us. [03:02] So for example, you can have your code [03:04] on GitHub and you just connect it to [03:07] this to one of these platforms and they [03:09] will deploy the code for you. You have [03:11] platforms like Netlifi, Cloudflare, [03:15] Lenode as I said and others. But the [03:18] problem if you are a beginner and go [03:20] directly to one of these managed [03:22] platforms is that you won't understand [03:25] exactly what is happening under the hood [03:28] because they have a layer of abstraction [03:30] that removes all the burden and all of [03:34] the things that you need to configure [03:35] manually. And this is great for people [03:38] that have the experience with Linux and [03:41] have deployed things manually across the [03:44] years. But if you are be a beginner, I [03:47] will highly suggest to not start with [03:49] these platforms and try to do things [03:51] manually to understand what is happening [03:54] because transitioning from a manual [03:56] process to using one of these platforms [03:59] is very easy. So we are going to look [04:02] behind the scenes here. As as you can [04:04] see from the slide, I am banning these [04:06] platforms because we are going to rent a [04:09] server, a brand new server. We are going [04:12] to install the operating system on it. [04:15] In this case, we are going to use Abuntu [04:17] and I will show you how to install the [04:20] required software. So, we can install [04:22] Python, NodeJS and other software that [04:26] we need to run the application. And we [04:28] will focus also on making sure that the [04:31] web server is secure and that no one can [04:34] access it. And we will see other things [04:36] as well in this course. And here I want [04:38] to mention or I want to emphasize this [04:41] core philosophy that we will follow [04:43] during the course is that before we [04:46] automate anything we have to do the [04:49] process manually because that will teach [04:51] us why we need automation because if you [04:53] don't do that you won't see the benefit [04:55] of automating processes and also I have [04:58] developed a book that goes uh with this [05:01] course is uh not something the [05:04] theoretical as I said I have applied [05:06] this three times s on three websites [05:09] that I have deployed and I am trying to [05:12] give you all that knowledge in a [05:14] distilled format. So I used a lot of [05:16] resources like Google, I read books, I [05:20] also found some code snippets and I [05:22] filtered all of that into one simp one [05:26] course that you can follow with all the [05:28] things that I have learned and also the [05:30] book if you want to read it. And as I [05:33] have said, I have worked on this course [05:36] for more than 8 months now to try to [05:39] give you something that you might find [05:41] helpful and to go in to see everything [05:44] manually and to learn about everything [05:46] that happens behind the scenes. And in [05:49] this course, I will focus on deploying a [05:53] specific web application that I have [05:55] used and that you can find the source [05:56] code online on GitHub that uses fast API [06:00] in the back end and Vue.js JS in the [06:02] front end and as an extra layer we added [06:05] a search engine using M search but this [06:08] is just an example to show you that you [06:10] can deploy a complex system but if you [06:13] are using Ruby or other programming [06:15] languages the process will be the same [06:17] you just need to change the software [06:19] that you need to install and the [06:21] commands to run it. This course is [06:24] divided into six modules. In the first [06:27] one which I call the foundation we are [06:30] going to as I said in this case we are [06:32] going to use digital ocean but you can [06:34] do the same thing with the other [06:35] platforms. We are going to rent a [06:37] server. I will teach you what is an SSH [06:40] key. This is something that we are going [06:42] to use in order to authenticate to uh [06:44] the machine that we are going to rent. [06:47] And here something very important. This [06:49] is for security reasons. We are going to [06:51] remove the ability to connect using a [06:54] password. Instead, we will only allow [06:56] using an SSH key. And we will also block [07:00] users from connecting as the root user [07:04] because if someone is able to use that, [07:08] they might do something nefarious in [07:10] your server and we don't want to allow [07:12] that. Instead, we are going to create a [07:14] new user that you can use in order to [07:16] connect to the virtual machine. And on [07:18] top of that, we are going to configure [07:20] firewall to block anyone who is trying [07:24] to access your your web server. Why is [07:26] that? Because once you once you create a [07:30] web server, it is going to be available [07:31] online and anyone can see it. So we need [07:35] to set up the firewall and block people [07:37] that are trying to access your web [07:40] server. In this second module which is [07:42] called the application runtime, here we [07:45] are going to work with our code. So we [07:47] are going to take it from let's say [07:49] GitHub or from our laptop and we are [07:52] going to deploy it to the server. So we [07:55] are going to copy it and paste it there. [07:58] Because we are using Python, we need to [08:00] install we need to install it and [08:02] configure the virtual environment. And [08:04] something very important here. We are [08:06] going to start with a small server. It's [08:09] not it is not going to have a lot of [08:11] RAM. So we will end up having some [08:14] problems with memory. So we will [08:16] eventually when we try to install [08:18] software that needs compilation that [08:20] will that will make the uh Linux run out [08:23] of memory. So in that case we are going [08:26] to learn something new which is swap [08:27] memory which will allow us to take some [08:31] some space from the SSD and help to in [08:35] order to help us to install some some [08:37] software that requires a lot of RAM. And [08:39] in order to manage the back end we are [08:41] going to use a mix of these software. We [08:44] are going to use supervisor, J unicorn [08:46] and UV core. Don't worry about that. We [08:49] are going to learn why do we need them [08:52] in module 2. So I'm just showing you [08:54] what the things that we are going to [08:56] learn. And after that we are going to [08:58] use engineext as a reverse proxy and we [09:01] are going to connect it to the back end [09:04] using Unix sockets. In the third module, [09:07] we are going to focus on media search [09:09] because this is the data that we that [09:12] the web application needs. I will show [09:14] you how to export and dump the data that [09:17] we have locally so that you can take it [09:19] from your local machine and put it on [09:21] the server. We are going to self-host [09:23] mail search. We are not going to use a [09:26] cloud offering that will give us the [09:28] ability to get me search. No, we are [09:31] going to install it and we are going to [09:32] manage it on the same server. And here [09:35] we are going to create another user that [09:37] will manage me search for us. And we are [09:39] going to use systemd to create a [09:42] background service that will launch me [09:45] search for us and that we can use in [09:47] order to manage me search. After that we [09:49] are going to use or we are going to [09:51] create a batch script that will [09:53] automatically create backups for us so [09:55] that if something bad happens we can [09:58] just read the backup and load it in [10:00] order to continue. And finally, we are [10:03] going to I will show you a nice [10:05] dashboard that you can use in order to [10:07] manage the data you have on me search. [10:10] And we are going to use something very [10:12] awesome which is called SSH tunneling [10:15] that will allow us to connect this [10:17] dashboard to your server without having [10:21] to open a port or something like that. [10:25] In module four, we are going to talk [10:27] about global delivery and app security. [10:31] And here we are going to start by [10:33] registering a domain and configuring [10:36] DNS. After that we are going we need a [10:39] certificate so that we can get HTTPS [10:42] that will make uh the connection between [10:46] your users and your web server secure [10:48] because if you don't do that you they [10:50] will only see HTTP and their browser [10:53] will flag that connection as insecure. [10:56] And with let's encrypt the certificates [10:59] are only valid for 3 months. So here I [11:02] will show you how to create a script [11:03] that will automatically renew the [11:05] certificate for you without having to [11:07] run any commands. After that we are [11:10] going to use cloud player to serve our [11:13] front- end assets because it is static [11:16] and that will help us reduce the latency [11:19] across uh the world because when when [11:23] you pick a server you pick a location in [11:25] in my example I picked Germany. So that [11:28] means people from China or other [11:31] countries that are very far away they [11:33] are going to see high latency. In order [11:36] to fix this problem, we use CDNs and in [11:39] this case we are going to use cloud [11:41] player. In module 5, we will focus a lot [11:43] on automation. Here we are going to [11:46] connect to GitHub because up to now we [11:50] have cloned the project using a command. [11:55] So we copied the code and we pasted it [11:57] on the virtual machine and we ran the [12:00] commands to start the back end and front [12:01] end servers etc. We did everything [12:03] manually. But in this module, we are [12:06] going to automate that. So we will push [12:09] the code to GitHub and we will just hit [12:11] a button to deploy the web application [12:13] without having to go to the server. So [12:15] this is what we are going to learn here. [12:18] But first we need to learn how to [12:19] protect a branch. So if you are working [12:22] on a codebase, you have let's say your [12:25] main or master branch, you should never [12:27] push the code directly to it. You need [12:29] to create a new branch and then merge [12:31] the code. So I will show you how to [12:33] protect that main branch so that you [12:35] cannot so that you don't push code that [12:38] is broken accidentally. After that we [12:40] are going to use a project called [12:41] pre-commit that will allow us to create [12:45] some rules that we can run when we [12:47] commit code. For example, one of the [12:50] rules that we can add is checking [12:51] linting and formatting so that if your [12:53] code is not formatted well, you will not [12:55] be able to commit. After that, we are [12:58] going to create a pipeline using GitHub [13:00] actions. And that will allow us to run [13:03] the same um tests like linting, [13:06] formatting, running unit tests, etc., [13:10] but on GitHub. And that will allow us to [13:13] block a merge request or a pull request [13:16] if it doesn't satisfy all these tests. [13:20] And that's how we can detect if our code [13:22] is broken or not. If it isn't, if [13:24] everything is green, then we can merge [13:26] to master and from that we can deploy [13:28] and we will do other things. Uh we will [13:30] add integration tests, end to end tests, [13:33] etc. So don't worry about that. We will [13:35] focus a lot. Uh this this model is going [13:37] to be very long, but it is going to be [13:40] very very useful. And finally in the [13:43] last module, we will focus on [13:45] optimization and maintenance. Here I [13:47] will show you how to read the enginex [13:49] logs and display that data nicely using [13:51] a project called go access. After that I [13:54] will show you how to monitor the memory [13:56] usage using btop++. This is an amazing [14:00] project. I really like it and you will [14:02] see once we once you reach module 6 you [14:04] will see how btop can help us understand [14:08] what is happening on our web server and [14:11] I will show you how to ban IP addresses [14:14] that are hammering your website. So [14:16] sometimes when you analyze the logs [14:19] using go access you will see some IP [14:21] addresses that are trying to reach your [14:24] server thousand um more than a thousand [14:27] times a day or a minute and that's that [14:29] is not human. So a human will not will [14:32] not go to your website and try to [14:34] refresh the page a thousand times. [14:35] That's impossible. So these are [14:37] automated bots that are trying to steal [14:39] credentials or to see if there are [14:41] weaknesses in your web server. So we are [14:43] going to block them in order to first [14:46] not waste time serving them things that [14:48] they they are not going to read and to [14:51] yeah reduce the the burden on our [14:54] server. And we will also use an amazing [14:56] tool called locust. This is a Python [14:59] library that allows us to simulate users [15:02] trying to access our website because [15:04] this will allow us to estimate how many [15:07] requests per second or per minute we the [15:10] server can sustain before crashing or [15:13] not being able to serve as many users as [15:16] possible. So this is going to allow us [15:18] to understand the weaknesses of our [15:21] server and we will do other things. is [15:23] we will analyze the disk usage um [15:26] perform some log retention policy [15:28] changes etc. So as you can see this [15:30] course is packed with information. So it [15:34] is going to be more than 10 hours long. [15:36] So go through it step by step and don't [15:38] try to rush. I I have tried to make it [15:41] accessible to anyone. So even if you are [15:44] a beginner don't worry about that [15:46] because this is why I have made the [15:48] course in order to teach you how to [15:50] deploy things. And at the end you will [15:52] see that you can also go automate uh [15:56] more stuff but I just try to pick your [15:59] interest and show you what you can do. [16:01] And after that as I said if you don't [16:03] want to do this process manually at [16:05] least you did it once. So now you [16:07] understand after that you can use [16:08] platforms like Netifi get GitHub pages [16:11] to deploy your applications without [16:13] having to worry about renting a server [16:16] or something like that. And in terms of [16:18] resources don't worry about that. [16:20] Everything is open source. I will make [16:23] sure to put the links in the video [16:24] description. You have two GitHub [16:26] repositories. The first one, which is [16:28] what I'm showing you right now, contains [16:31] the code that we will deploy and the [16:34] second GitHub repository contains the [16:36] content of the book. So if you want to [16:40] just read and you don't prefer watching [16:42] videos, you can go to that repository, [16:46] click on the book folder and you'll find [16:48] all the chapters right there with the [16:50] code. Everything everything is there in [16:52] a nice format. And if you need some help [16:55] or if you just want to say something [16:58] after watching the course, feel free to [17:00] connect with me. You can find my [17:02] LinkedIn on the video description or you [17:05] can send me a an email if you want to. [17:08] You will find all my information in the [17:11] video description. I hope that I have [17:13] motivated you enough to watch the [17:15] course. I hope that you enjoy it and [17:17] that you learn a lot from it. Now, let's [17:19] start with the first module. [17:22] Welcome to module one where we will set [17:25] up the foundation of the cloud [17:27] environment. This is where your code [17:30] will live. In this sub chapter, we will [17:33] focus on cloud setup and access. Let's [17:36] get started. Before we deploy any code, [17:40] we need a secure environment. In this [17:44] course, I will be using digital ocean. [17:47] But these concepts apply everywhere. [17:50] Whether you are using AWS, Lenode, or [17:54] even a Raspberry Pi at home, the Linux [17:57] commands remain the same. By the end of [18:00] this video, you will have a server [18:03] running Ubuntu 2404 LTS. We will [18:07] generate secure SSH keys, connect to the [18:12] server, and most importantly, we will [18:15] completely disable password [18:18] authentication. [18:19] This ensures that your server is locked [18:22] down from the very first moment. Now [18:25] let's go to digital ocean in order to [18:28] create our first droplet. [18:31] Before creating the droplets, I want you [18:34] to know that we have a GitHub repository [18:37] where you will find everything I'm [18:40] sharing with you in this course. So for [18:42] example here inside the book folder we [18:45] have the first subchapter and you have [18:49] all the steps here. So if you prefer [18:52] reading articles you will find those [18:55] inside the book folder and the slides [18:58] and everything else are available in the [19:01] repository as well. You will find the [19:03] link in the video description. If you [19:06] have never used Digital Ocean before, if [19:09] you click on the link in the video [19:10] description, you will be able to get [19:13] $200 for free just to test their [19:16] platform. And that amount is available [19:18] for 60 days. Click on get started. That [19:22] will take you to a login page where you [19:26] can create your account. In my case, it [19:29] it logs me directly because I already [19:31] have an account. And here we need to [19:34] click on projects. As you can see, I [19:36] already have two projects. I have my [19:39] personal website is hosted here. And in [19:43] your case, this list will be empty. [19:46] First, click on new project. [19:50] In this form, give the project a name. [19:53] Select the environment type and the [19:55] purpose of this project. I will call the [19:58] project test. The environment type, [20:02] let's say development. Here in the [20:05] purpose drop-down menu, I'll select it [20:08] web application and the description is [20:10] optional. So click on create a project. [20:14] Let me zoom in a little bit. Here they [20:16] say move resources into test. So here we [20:20] don't have any resources. So we will [20:22] skip this for now. And as you can see [20:25] the project has been created. And now we [20:28] can create resources. And the resource [20:32] that we are going to create is a [20:34] droplet. Droplets are machines that that [20:37] can run our code in the cloud. This [20:40] means that we will take our [20:43] application's source code put it on this [20:46] droplet that is hosted by digital ocean [20:49] and the code will be running inside [20:52] inside of that machine. So now you can [20:55] click on create droplets or you can do [20:58] it from here. Create droplets. It's the [21:02] same thing. [21:05] Now we are in the create droplets page. [21:08] The first thing that we need is to [21:11] choose a region. For example, in my [21:14] case, I live in Morocco. So I will [21:16] choose Frankfurt because Germany is [21:20] close to Morocco. That's why I chose it. [21:23] you can do the same thing. So for [21:24] example, if you live in Europe, you can [21:26] select either London or Frankfurt or [21:29] also Amsterdam. Those three will work [21:31] perfectly. But if you are trying to [21:35] deploy this application next to users in [21:38] another country, then try to choose the [21:40] the region based on where your users are [21:44] going to be uh located. And here for [21:47] example in Frankfurt, we have just one [21:49] data center. So there there are there [21:51] aren't other options. Next, choose the [21:55] operating system. Here we have a lot of [21:58] options, but I will select Ubuntu. And [22:02] here we have plenty of versions. I will [22:04] choose the LTS version, and I will [22:07] select the latest one. Now you need to [22:10] choose the size of your virtual machine. [22:14] You can pick between shared CPU and [22:18] dedicated CPU. Inside each option, you [22:22] must decide the type of of storage, how [22:26] much memory you want, uh how many CPU [22:30] cores you want to have in this machine, [22:32] etc. And here if you select dedicated [22:35] GPU CPU, uh you will see that the price [22:39] increases drastically because inside [22:43] this option, the resources are reserved [22:46] only for you. But shared CPU, you will [22:49] be sharing resources with other people. [22:52] That's why it's not as expensive as [22:55] dedicated CPU. Just for testing [22:58] purposes, select choose shared CPU, [23:02] regular, and then select the lowest [23:06] tier, $4 a month. And if you have just a [23:09] simple web application that you want to [23:11] host, then this option is fine. uh for [23:16] example I have de deployed many websites [23:18] using just this plan and it works [23:21] wonderfully if you see that the [23:24] application can't handle this the high [23:26] traffic that you are seeing after that [23:28] it's possible to to to get more [23:31] resources so this doesn't if you select [23:33] this now it doesn't mean that you are [23:35] stuck in this offer and you'll have to [23:37] recreate everything from the beginning [23:40] no you you can still uh upsize the the [23:43] droplet and add more resources If you [23:46] need next, so here we have additional [23:48] storage. You can see that here we get 10 [23:50] GB of storage. If you think that this is [23:53] not enough, you can click on add volume [23:56] and select the amount that you want to [23:58] add. But this is not free. So you will [24:00] be paying $1 per month per 10 GB. In my [24:04] case, I don't need this, so I'll just [24:06] remove it. 10 GB is enough. Uh if you [24:10] want to have backups, you can click on [24:12] this. And again, this is not free. So uh [24:16] if you need this enable it and choose [24:19] the right options but in any I don't [24:22] think this is needed for testing [24:24] purposes. So uncheck it. And now we [24:28] arrive at the authentication section. So [24:31] here we have two options. We can use SSH [24:35] keys which is as you can see the default [24:37] option selected by Digital Ocean or if [24:40] you want you can use passwords. But as [24:43] you can see here, we have a warning that [24:45] says please store your password securely [24:48] because if because passwords can be cast [24:51] if they are not strong, but SSH keys are [24:54] much stronger. So let's use SSH keys and [24:58] I'll show you how to create one uh next. [25:01] So now we need to create an SSH key. So [25:04] I already have one here, but I will [25:06] create one by clicking on this new SSH [25:09] key button. You can see that they give [25:12] us here the steps that we need to [25:15] follow. I will show you how to do that. [25:17] Now one once we generate the SSH key, we [25:21] will put it here. Give it a name and add [25:24] it. But before I create this, I want to [25:26] go back to the slides in order to [25:28] explain how uh authentication with SSH [25:32] keys work. Let's look at what actually [25:36] happens when you try to connect to your [25:39] droplet. As I said, we will be using SSH [25:42] keys instead of passwords because they [25:44] are much harder to steal. The process [25:48] starts with two files that we call [25:50] private key and public key. And as you [25:54] can see, we have two containers. This [25:56] one represents your machine, your laptop [26:00] that you have next to you. And here we [26:02] have the server that we are creating on [26:05] Digital Ocean or you can create on AWS [26:09] or other cloud providers. The private [26:11] key lives on your laptop but the public [26:14] key will be inside the server. So the [26:18] private key here think of it as your [26:20] physical house key. You never share this [26:23] with anyone. So this needs to be kept [26:26] secret. Then you give the server a [26:31] public key. Think of this as the lock on [26:34] the door. Anyone can see the lock, but [26:37] because they don't have the key, they [26:39] cannot open it. When you run the SSH [26:42] command, your laptop contacts the server [26:46] and says, "Hello, I would like to [26:49] connect using this specific ID." When [26:52] the server receives this request, it [26:55] replies with a challenge because the [26:58] server is suspicious. It it doesn't know [27:00] if you are the real person or if this is [27:02] a hacker that wants to hack the the [27:04] server and extract valuable information [27:07] from it. So the server sends a challenge [27:11] basically a random string of characters [27:13] that we call a nons and basically says [27:17] prove you own that private key. sign [27:20] this random string for me. So if the the [27:24] requesttor is legitimate, that means [27:28] that it has the private key, it will be [27:31] able to basically sign this string and [27:35] send it back to the server. So here the [27:40] the laptop will say or the real person [27:42] will say I signed this using my private [27:44] key. Here is the signature. Now the [27:47] server will receive this and will try to [27:50] do the math to verify if this is uh if [27:53] the private key is correct. If that's [27:56] the case, the server will grant access [27:59] because it verified that the signature [28:01] is correct. Otherwise, it's going to [28:04] block this request and the anyone except [28:08] you will not be able to log to your [28:11] machine. So this is how SSH SSH [28:14] authentication works. And now I'll show [28:17] you how to create the these two pairs of [28:20] keys uh the private and public keys. [28:23] Paste this command. As I told you if you [28:26] go to the to GitHub you will find this [28:31] markdown file and here you will have [28:33] everything. Um, so if I scroll down a [28:35] little bit, you can see that I'm here we [28:37] we have the same steps that we followed [28:40] and here you can see that we had the [28:42] commands. So don't worry about them. You [28:44] will find the everything in here so that [28:46] we can just copy copy them easily [28:48] without typing them manually. So the SSH [28:52] key gen command is used to generate SSH [28:56] keys and we have plenty of algorithms [28:59] that we can use to generate these pairs [29:01] of keys and one of those is called [29:04] ED25519. [29:06] So this is the one that we are going to [29:07] use. It's faster and more secure than [29:11] RSA but you can also use RSA if you [29:14] want. And here - F is basically the file [29:18] name. All SSH keys are stored in the SSH [29:22] directory. And if you don't give it a [29:24] name, you might risk overriding your [29:27] keys. So here I will call it test SSH [29:32] key. And here we can add a comment. This [29:36] is helpful if you want to identify the [29:38] keys. And now I can hit enter. And as [29:42] you can see it says generating public [29:44] key with this algorithm. And here I need [29:47] to enter a passphrase for this specific [29:51] key. And here they tell me if I don't [29:53] want to add a passphrase, I can just hit [29:55] enter to move to the next uh step. When [29:59] should you add a passphrase and when [30:02] should you ignore this? I strongly [30:04] recommend adding a passphrase for your [30:07] personal key. So if you want to log to [30:10] the machine, type some commands etc. do [30:14] do some maintenance in that case you [30:16] will be logged in manually. So in that [30:19] case it's great to have a passphrase [30:21] because even if someone was able to get [30:25] your key they won't be able to login [30:27] because the server will ask for a [30:29] passphrase. Basically it's another layer [30:31] of security that will prevent anyone [30:34] from using your ser or accessing your [30:37] server. But if you want to have a a [30:42] machine or a script that will [30:44] automatically deploy your application, [30:47] in that case you don't want to have a [30:48] passphrase because the script will be [30:51] blocked. I will show you how to create [30:54] two keys. So this one that we are [30:57] creating is going to be used manually [31:00] but we'll have another one that will [31:02] call it let's say GitHub or something [31:05] that we will use in GitHub to deploy our [31:08] application automatically and in that [31:11] case we don't need a passphrase. So for [31:13] now I will add the passphrase and I will [31:15] show you that we will be asked to enter [31:18] it when we want to log the droplets. I [31:23] have entered the passphrase and here [31:26] they tell me to enter it again. Great. [31:29] So as you can see the key random image [31:32] is so that means that the the keys were [31:35] generated. As you can see I have listed [31:38] all the keys inside the SSH directory [31:41] and I search just for this and you can [31:44] see that we have two files. This one pub [31:48] means public key and this one is the [31:50] private key. This one should stay on [31:52] your laptop but this one we will copy it [31:55] and put it inside the droplet. Great. So [31:59] now copy this and we will use the catch [32:03] command to see the content of this [32:06] public key. So here make run this and [32:09] copy everything and after that I will [32:13] show you where to put that uh on digital [32:15] ocean. I have copied the key and now I [32:18] will put it here and I will give it a [32:21] name. So I will start by entering the [32:24] name. So test SSH key manual. So here [32:30] you paste your SSH key and after that [32:32] click on add SSH key. I have done that [32:35] and as you can see I have another SSH [32:37] key here. And now if I want to add [32:40] another key, I can click on new SSH key [32:42] and repeat the same steps in the [32:45] terminal to generate another one. But [32:47] make sure to have another name and to [32:51] not enter a passphrase because that one [32:54] we will use later to deploy the [32:57] application automatically. Or if you [32:59] don't want to do this here, I will show [33:01] you how to do it within the droplet. [33:04] Here we can select this first option add [33:07] improved metric monitoring. It's free so [33:10] why not? And here finally we have this [33:14] section. So here we can create multiple [33:16] droplets if you want to but in our case [33:20] uh one is enough. You can change the [33:23] host name. So this is the name that will [33:26] appear inside the droplet. We can we can [33:30] change that. Let's call it uh let's keep [33:33] it as Ubuntu test project. If you have [33:37] multiple droplets, using tags will help [33:40] you filter them quickly. And after that, [33:43] we should assign this to the test [33:45] project that we created. So everything [33:48] is okay. Now click on create droplet. [33:53] Open your list of projects and click the [33:56] project that you have created. As you [33:58] can see here is the droplet and here [34:01] this is the running indicator. So this [34:04] means that this droplet is in the [34:08] creation phase but once this turns green [34:12] it means that it's running. You can [34:14] click on the name and as you can see now [34:17] it's running. We have no data because we [34:20] have just created this. You have your IP [34:23] address. You have everything here. And [34:26] as you can see, if you want to upsize [34:29] this later, you can click on the upsize [34:31] droplets button. And here you can in add [34:34] more memory, more storage or in uh [34:38] increase the the number of CPU cores if [34:41] you want to. So I have refreshed the [34:43] page and as you can see now it says [34:45] active instead of uh new. Great. Now I [34:49] want you to copy this IP address. Just [34:52] click on the copy button. And you might [34:55] be wondering why am I blurring the IP [34:58] address. Even though this is just a test [35:01] droplets that I will delete later, it's [35:04] a good habit to keep your server IPs [35:06] private to prevent unwanted traffic or [35:10] attacks. So, copy the IP address and [35:13] open your terminal. We are going to use [35:16] the SSH command to SSH to the droplet. [35:21] So first we type SSH [35:24] I and here we need to specify the [35:28] private key and the keys are stored in [35:32] the SSH directory and here I have called [35:35] mine test SSH key and after that we [35:38] specify the user that we want to log in. [35:43] So in the machine we have uh by default [35:45] we'll have just the root the root user [35:47] but after that we'll I will show you how [35:49] to create other users and here we type [35:53] at and we paste the IP address. Now hit [35:58] enter and you should be able to login. [36:02] So because this is the first time we are [36:05] trying to login, we get this warning. [36:08] And here you can just type yes. And as [36:11] you can see, I'm not able to log in uh [36:14] because I have added a passphrase. I [36:17] need to specify it before logging in. So [36:21] now I will add my passphrase. And here [36:24] on Ubuntu there, they we have this [36:27] option that will automatically unlock [36:28] the key. Um so if you want to login [36:31] quickly, that's fine. and just click on [36:33] unlock and as you can see I am in uh [36:36] let's verify so this is the memory usage [36:39] it's has 40% we don't have any swap [36:42] memory this we are using 22% of the disk [36:46] space and yeah it seems to be working [36:50] you can see that we have some updates [36:52] that we can perform and that's great so [36:56] as you can see we are logged in as the [36:58] root user and this is the name that we [37:01] have specif ified when we created the [37:03] droplet. So everything seems to be [37:06] working. Okay. So since we are logged in [37:09] as user, we can type in sudo app update. [37:14] So this is a an abuntu. [37:17] So this is a machine running Ubuntu and [37:19] this is how we update it. Okay, good. So [37:22] now we will type sudo upgrade. [37:26] Okay, the type yes and hit enter to [37:29] start installing the packages. If you [37:32] see a configuration screen during the [37:35] update, select keep the local version [37:39] currently installed. Don't don't select [37:42] any other option because we want to [37:44] preserve the working SSH configuration [37:47] that Digital Ocean set up for you. In [37:50] the Digital Ocean page, we added a SSH [37:55] key manually. So that updates the SSH [37:58] configuration and we want to keep that [38:00] but also add the new modifications that [38:02] this update process is trying to [38:05] perform. Keep the differences and yeah [38:08] let the the updates continue. Great. So [38:12] the update has finished. Now we need to [38:16] reboot the machine. So just type the [38:18] reboot command. This will complete the [38:22] upgrade process and start using the new [38:24] kernel and packages. So you can see that [38:28] the connection is closed. So I was uh [38:31] ejected from the server because it needs [38:33] to reboot. But don't worry, I can run [38:36] the SSH command again to login back to [38:40] the machine. So let's wait a little bit [38:43] or let's go back to Digital Ocean just [38:45] to check uh if it is ready. So I'll [38:49] refresh the page. Okay, it says it says [38:51] active. Let's hit enter. And voila, I am [38:56] back. And as you can see, it says zero [38:59] updates can be applied immediately. This [39:02] confirms that the update process works [39:06] perfectly. Now we are going to create a [39:09] nonroot user and configure SSH access. [39:14] Why do we create new users instead of [39:17] using the root user? Running everything [39:20] as root is dangerous because sometimes [39:23] you can make mistakes or if you don't [39:27] know Linux a lot and you might be using [39:30] LMS without knowing the commands that [39:32] they give you, you might do something [39:34] that will maybe corrupt your file system [39:37] or do something irreversible and that's [39:40] not good. And if a hacker was able to [39:44] log into your machine, he can do every [39:46] anything. We want to limit the things [39:49] that we can do inside the VM by creating [39:53] a nonroot user. And if you want to [39:56] perform something that requires pseudo [40:00] privileges, we can give the user that [40:03] ability momentarily. So the command to [40:06] create a user is add user. I will call [40:11] that user Emad. So that's my name. Uh [40:15] make sure to change that. And as you can [40:17] see now we need to provide a password. [40:20] This means that if I try to to run the [40:24] same command sudo opt update the Linux [40:28] machine will ask me to enter my [40:29] password. So this is really great [40:32] because let's say that someone wants to [40:35] login with this specific user that I [40:37] created, they will not be able to do [40:40] commands that require sudo privileges [40:42] because they don't know the password. So [40:45] create the password uh retype it again. [40:49] And here you can add additional [40:50] information like the full name. So let [40:53] me just add that uh room number. I don't [40:56] need that. For the things that you don't [40:59] want to add anything, uh, just hit [41:03] enter. And finally, let's type yes to [41:06] create the user. So, as I said, we want [41:09] this user to be able to run pseudo [41:11] commands. So, we will use the user mod [41:14] command. And here we will type the the [41:18] user. And now we can see that we are [41:21] still logged in as the root user, but we [41:23] want to log in as this user. As you can [41:26] see that I'm inside the the droplet. In [41:30] order to exit, I can type exit and hit [41:32] enter or I can just hit control and D. [41:37] Good. So the connection to this server [41:40] is closed. Now I can go back and instead [41:44] of uh login logging in as root I will [41:48] use my new user. Hit enter. And as you [41:52] can see we were we weren't able to [41:54] connect. Why is that? And here it says [41:56] too many authentication failures. In [41:59] your case, you might see just permission [42:01] denied public key. In my case, I see [42:04] this because I have a lot of public [42:05] keys. So the the server was or my laptop [42:09] was trying to log in with multiple keys [42:13] and it wasn't able to do that. So why [42:16] aren't we able to connect as this new [42:20] user? This happens because the SSH key [42:24] that we added in digital ocean lives in [42:27] the root user. This means that we need [42:30] to take that SSH configuration and also [42:34] give it to my new user. So let's log [42:38] back as the root user and we can verify [42:41] this. We can go inside the root user and [42:44] we have again the SSH folder and here we [42:48] have authorized keys and authorized keys [42:51] is a file. It's not a directory and we [42:55] can verify this by typing ls-la [42:59] and you can see that here we have a dash [43:01] that means it's a file. If it says d [43:04] that's a directory. So authorized keys [43:06] contains all the public keys that we [43:08] want to use on this droplet. In our [43:11] case, it contains just one because uh [43:14] when we created the droplets, we added [43:16] one key. But we can add more if we [43:18] wanted to. But my new user doesn't have [43:22] this directory. So if I go to home and [43:25] this user new user, you can see that [43:28] there isn't an SSH directory. This means [43:32] that we we won't be able to login as [43:34] this new user. But luckily, we can solve [43:39] this issue by just copying this SSH [43:43] directory from the root user to this new [43:46] user. And the command to do that is cp [43:50] preserve mode. And here we want to copy [43:54] this directory to this new user. And [43:58] preserve mode is an option that we can [44:00] add in order to preserve the file [44:03] permissions. So you can see that here. [44:06] Uh so let me just run this command. And [44:08] instead of running ch mode 600 700 etc [44:13] on this new directory. Preserve mode [44:16] will do that for us. So in just one [44:18] command we preserve the the file [44:20] permissions. Who can access this file? [44:23] Who can read it? Who can execute it etc. [44:25] And if now I can look at this. So let's [44:30] use ls and let's see what we have inside [44:34] the sh folder. You can see that we have [44:38] the authorized keys. That's good. But [44:40] the problem is that if I look at who [44:43] owns that file, you will see that the [44:46] root user is owning this file. So we [44:49] want to change the ownership and give [44:51] that to this new user that you have [44:53] created. And in order to do that, we use [44:56] the ch own. And here I will also add [45:01] capital R. And I want this new user to [45:06] own that file. And here I'll go to home [45:09] and SSH folder. And now if I go back to [45:13] ls-la, you can see that we have that [45:17] this user is owning this file. Now we [45:21] can exit. Uh again hit Ctrl D go back to [45:26] the previous command and here type in [45:29] the new user not root and now hit enter [45:33] and as you can see we are logged in and [45:36] the prompt shows my user and not root [45:41] and if I want to let's look at root dosh [45:47] I'm not I'm not able to do that because [45:49] that's not my directory but Because I [45:53] have the pseudo privilege, I can type in [45:55] pseudo and enter my password. And as you [45:57] can see, I'm still able to look at [46:00] things that I'm that I'm not supposed to [46:02] look at or per do maintenance uh stuff [46:06] like that. But if someone hacks your [46:08] machine and is logged in as this user, [46:12] new user, they won't be able to do this [46:14] because they need to know the password. [46:17] Awesome. Uh just one note, if this [46:20] didn't work for you, if uh for example, [46:23] you weren't able to uh look inside the [46:26] SSH directory, that means that you did [46:29] something wrong. Just rewind and make [46:32] sure that this step works because in the [46:34] next step, we are going to disable root [46:37] login. We will not allow even us, we [46:40] won't be able to login as root. We will [46:42] only login as this new user. This will [46:46] uh help increase security a lot because [46:49] no one will be able to login as roots [46:51] because as you saw root doesn't require [46:53] providing a password and that's a big [46:56] security problem. If someone is able to [47:01] login as that super user let me show you [47:05] how to stop this from happening. We will [47:08] use pseudo again. I'm going to use nano [47:11] as matx editor and I'm going to go [47:14] inside the shd config. [47:18] We want to change uh the following I [47:21] will say parameter and in nano in order [47:24] to search you can click on or press ctrl [47:28] w paste the thing that you are looking [47:31] for and hit enter. You can see that by [47:33] default this value is set to yes which [47:36] means that we can login as root but we [47:39] want to change that to no. We also want [47:42] to disable password authentication even [47:45] though we have specified in the creation [47:49] of the droplets that we want to use an [47:50] SSH key but it's it's also a great idea [47:54] to set it here and not rely on the cloud [47:58] provider to do that for you. So these [48:00] are the two things that you need to [48:02] change. You need to change permit root [48:04] login to no and password authentication [48:07] to no. Now in order to save this you can [48:11] click on Ctrl S or if you are on an [48:14] older machine you can hit Ctrl O just [48:18] hit enter and after that Ctrl X. You can [48:22] verify that again if this works. Just [48:24] enter. And you can see that the the [48:27] values are now set to no. [48:30] So now we need to restart the SSH [48:34] service. And now let's test this. So I [48:37] will type exit. And now I will try to [48:40] login as root. Okay. Let's see what will [48:43] happen. Uh so as you can see um my [48:45] request was rejected because we have [48:48] blocked root access to or we blocked [48:52] logging in as root. But if I go back and [48:56] try to login as my new user I am able to [49:00] do that. So that's awesome. So now we [49:03] have lots of layers of security. We have [49:06] disabled password login. We are using [49:09] SSH keys. And even if we use an SSH key, [49:13] we need to provide a passphrase. And [49:15] within the machine, we created a new [49:18] user in order to prevent us from doing [49:21] things that are unreoverable. [49:24] And if we want to do things that require [49:26] sudo privileges, we need to provide a [49:29] password. So this means that if someone [49:32] was able to breach a layer, they won't [49:36] be able to uh do something harmful [49:39] because if even if they have the SSH [49:42] key, they need to provide a passphrase [49:44] or even if they logged in, they need to [49:46] provide uh the password. I want to [49:49] explain to you how the SSH configuration [49:53] works. We have this main configuration [49:57] file that we have changed. Remember we [50:01] have disabled root login by setting [50:05] permit root login to no and we also [50:08] changed password authentication to no. [50:12] So this is the main configuration file. [50:15] But if I go up you will see that we have [50:18] this line include and here we include [50:22] all these configuration files that are [50:25] inside this directory. We call this drop [50:29] in configuration files. Now let's try to [50:32] look at this folder to understand what [50:35] we have there. As you can see I am using [50:38] tmox. This is why I have two window [50:40] panes so that you can see everything uh [50:43] in one place. Here I have copied that [50:46] path and let me enter my password. And [50:49] here I need to list all the [50:52] configuration files. You can see that I [50:54] have two of them. And if I use this [50:58] command, as you can see, let me zoom in [51:01] a little bit. So here I am searching for [51:03] this property that we have changed to [51:07] no. And you can see that it we already [51:09] have it in these two files. What does [51:12] this mean? When we created our droplet, [51:16] we selected SSH authentication [51:20] and digital ocean created this file for [51:23] us because on Linux we treat these [51:28] drop-in configuration files before the [51:31] main configuration. Let me explain. You [51:34] can see that here we include all these [51:37] files before changing all these fields [51:40] manually. This means that even if I come [51:43] here and set let me see so if I set this [51:47] to yes this will not take effect because [51:51] if I go for example inside this file so [51:55] let's look at it here we have this [51:58] command and here I will just try to [52:01] print what we have inside that file and [52:04] here I need to type 50 yes so as you can [52:07] see this file contains just this [52:10] configur this property and it is set to [52:13] no. So this means that this will [52:16] override any value that we have in the [52:19] main configuration file and you can see [52:21] that we have also another file which is [52:24] called 60 cloud image and here we also [52:28] have this property set to no. So as you [52:31] can see we have two files that set this [52:34] property to no. Which one should we [52:37] take? Because this is the SSH service, [52:40] we are going to order the files [52:43] alphabetically. [52:44] So here in this example, 50 cloud in it [52:47] will come before 60 cloud image. We are [52:51] going to read the properties inside this [52:54] file and we are not going to look at the [52:57] other ones. So here if we find that [52:59] password authentication is set to no, we [53:02] are going to take it and we will drop [53:04] the other files and the main [53:06] configuration file. This is why I said [53:08] that the values that we have in these [53:11] drop in file configuration are going to [53:14] override what we have in the main SSH [53:18] configuration file. So you might be [53:21] asking why did we edit the main file if [53:25] Digital Ocean has already created this [53:28] for us. That's a good question. We edit [53:32] the main configuration file as a safety [53:34] net. If you ever want to do this in the [53:37] future without using a cloud provider, [53:40] in that case you will not have these [53:42] files configured for you. So you will [53:45] need to either create one uh manually [53:48] and put it inside this directory or you [53:52] come inside the configuration [53:55] file the the main one and you change the [53:57] values manually. This way you guarantee [54:00] that your server is secure regardless of [54:02] the hosting provider. So now I want to [54:06] simplify the way we access the machine. [54:09] So if I hit again Ctrl V, I need to [54:12] always specify the SSH key and then the [54:16] user and uh the IP address. That's a lot [54:20] and there is a way to simplify this and [54:23] I'll show you how to do that. On your [54:25] local machine, you can see that I'm no [54:27] longer connected to the VM. I have [54:30] closed the connection. I want you to [54:32] type this command nano uh SSH config. [54:36] You can see that here I already have [54:38] some entries but I want you to go to the [54:41] end of the file and I want you to copy [54:45] and paste the following. So here we have [54:48] placeholders that we need to change [54:50] here. Uh the host. So this is just a [54:53] name. Uh this is a shortcut that we will [54:56] be using in the SSH command. So here [54:58] I'll call this test website. Here I need [55:02] to provide the droplet ID. Uh sorry the [55:06] droplet IP address. I'll copy it from [55:10] digital ocean. Uh okay. I will replace [55:13] this placeholder with that IP address. [55:15] And here I will replace this with my new [55:18] user. Here the identity file is going to [55:22] be the private key. I think I've named [55:24] it test ssh key. Uh but I'm not certain [55:29] but I I can check this. So save the file [55:32] again. You can do it by hitting Ctrl S [55:36] or Ctrl O and then enter. And after that [55:39] hit Ctrl X. So let me check. Yeah, it's [55:43] test SSH key. Okay. And let's go back to [55:46] the configuration. And yeah, it's uh [55:49] it's correct. Awesome. So you can see [55:51] that now after adding this to the config [55:55] file we will be able to log to the [55:57] website just by typing SSH and the [56:01] shortcut name that we have created. So [56:04] now I can just type this and as you can [56:07] see I am in and this is easier to [56:09] remember instead of uh remembering which [56:13] key you use and what is the IP address [56:16] of that droplet. And now I can just type [56:19] this and I'm in. You have successfully [56:22] configured a clean Ubuntu server and [56:26] secured it with SSH keys. However, your [56:30] server is still exposed to the open [56:33] internet. In the next subchapter, I will [56:37] show you how to work with the firewall [56:41] to lock down the network. You will learn [56:44] how to configure UFW to block unwanted [56:49] traffic, set up fail toban to stop brute [56:53] force attacks, and use the recovery [56:57] console if you ever get logged out. [57:01] Welcome back. In the previous video, we [57:04] launched our server and set up secure [57:07] SSH access. However, right now our [57:12] server is like a house with the front [57:14] door locked, but every window is wide [57:19] open. The server is sitting on the [57:22] public internet and by default it will [57:26] accept connections on any port. In this [57:29] sub chapter, the firewall strategy, you [57:34] are going to close those windows and [57:37] lock the parameter. Here's our plan for [57:40] this session. First, you will address [57:43] the problem. A fresh Linux server is too [57:47] open. It accepts traffic from anywhere. [57:52] To fix this, you will configure UFW, [57:56] which means the uncomplicated firewall. [57:59] You will set it to block all incoming [58:02] traffic by default and only open these [58:06] specific ports you trust. Next, you will [58:10] install fail to ban. This tool adds a [58:14] layer of intelligence to your firewall. [58:17] It automatically bans bots that try to [58:21] brute force your SSH keys. Finally, I [58:25] will show you the digital ocean recovery [58:28] console. This is your safety net. If you [58:31] ever accidentally configure the firewall [58:34] wrong and block yourself out, this tool [58:38] will save you. Now, setting up a [58:41] firewall is great, but it doesn't stop [58:44] people from trying to get in. If you [58:47] look at your server logs, you will see [58:50] bots trying to guess your password. [58:53] thousands of times a day. We need a way [58:56] to stop them automatically. This is [58:58] where fail to ban comes in. I will try [59:02] to explain this using this diagram that [59:05] I created. We have our server and we [59:09] have an attacker. So this is a user who [59:14] wants to get access to our server. You [59:17] can see inside the server we have the [59:19] uncomplicated firewall. We have fail to [59:22] ban. We have the SSH service that runs [59:26] on this port. And we have this log file. [59:30] So the first thing that that this [59:32] attacker will try to do is to connect to [59:35] this server using the SSH service. This [59:39] is going to fail because they don't know [59:42] the our SSH key and maybe they don't [59:46] know the user. In the previous video, we [59:49] have blocked access to this server using [59:53] the root user. So if they try to SSH as [59:56] the root user, they will be blocked. So [59:59] if that happens, the service will write [60:02] to the log something like this invalid [60:06] user Ubuntu from this IP address using [60:10] this port. So yubuntu is the user that [60:13] the attacker tried to use in order to [60:16] connect to this machine and this is uh [60:20] his IP. So if we see this we know that [60:23] there are some bad actors who are trying [60:26] to log to our server. So fail to ban is [60:30] a service that we need to install. I [60:34] will show you how to do that. And it's [60:37] it's going to monitor this log file. And [60:41] if this attacker is making multiple [60:44] requests, we can detect that. And if [60:47] fail to ban sees that uh we have too [60:50] many errors coming from this attacker, [60:53] it's going to tell the firewall to block [60:55] this IP. And now if this attacker tries [60:59] to login again, we are going to block [61:02] that connection. It's not going to enter [61:04] the server. And this way we limit [61:08] attacks like these just by reading [61:11] what's happening inside the log file. [61:14] That's it for the theory. Now let's open [61:17] the terminal connect to the machine and [61:19] let me show you how to set up UFW fail [61:23] to ban and other things. Before I start [61:26] the practice session, I would like to [61:28] mention that I updated the first chapter [61:32] and now it contains the second [61:35] subchapter and it's called the firewall [61:37] strategy and you have all the steps [61:41] written here and we are going to follow [61:43] them in this session and this GitHub [61:47] repository is going to be shared and the [61:50] link is going to be in the video [61:52] description. Now let's open the [61:54] terminal. Last time we configured the [61:58] SSH configuration to allow us to login [62:01] directly without specifying the user and [62:04] without specifying the IP address. So [62:07] now I can run this command and I am in. [62:11] Let's start by looking at the authentic [62:14] the.log file. To do that I will use [62:18] sudo. I will use the cat command to [62:21] print the content of that file. I need [62:25] to go to var log and art.log. [62:30] It's going to ask me for my password. I [62:32] will provide it. And as you can see, we [62:35] have a lots of stuff. And we see that [62:39] invalid message. Let me try to filter [62:44] the output to get only uh the rows that [62:48] mention this word invalid user. And as [62:51] you can see, we have a lot. Remember [62:54] that's my user is IMAD. It's not Oracle, [62:58] it's not Solana or other stuff. And as [63:02] you can see, we have plenty of IP [63:05] addresses that are repeating. Let's take [63:07] this one as an example. You can see that [63:10] this attacker is trying to connect as a [63:14] test user after that. Yeah, it's it's [63:17] trying to connect uh again. And if you [63:21] look at the time, so yeah, they are [63:23] making hundreds or maybe even thousands [63:27] of requests a minute or a day. And [63:30] that's not good because they are here [63:33] the the log file is going to be to get [63:35] bigger and it's going to consume uh [63:38] storage and we don't want that. It's [63:40] going to be great if we can block these [63:43] to not fill this log file. So you can [63:47] see that I have just created this uh [63:50] specific droplet for this course and as [63:53] you can see I'm getting bombarded by uh [63:57] attackers who are trying to log the [64:00] machine but luckily we have services [64:04] that will automatically ban them. So [64:07] let's get started. First, I would like [64:10] you to look at the status of the UFW or [64:15] the uncomplicated firewall. To do that, [64:18] we can use pseudo UFW [64:22] status. And as you can see, it's [64:24] inactive. This is the problem that we [64:27] have. The firewall is is not active. We [64:30] need to activate it in order to block [64:34] the open ports and allow only the ones [64:37] that we care about because the web [64:40] application that we are going to deploy [64:42] is going to be running on one one or two [64:46] ports. We don't need to open every [64:49] single port to the public internet. But [64:52] before activating the firewall, we need [64:56] to allow SSH connections. If you don't [65:00] do this step, you will lock yourself out [65:02] of the server immediately. So run this [65:05] command sudo ufw allow open ssh. And [65:10] this is going to allow port 22 to be [65:15] accessible. Otherwise, if we enable the [65:19] firewall without allowing SSH, we will [65:22] not be able to run SSH uh the SSH [65:26] command to connect to the server. and [65:28] that's not going to be uh good. In that [65:31] case, we need to go to digital ocean, [65:33] open the recovery console and fix this [65:36] problem. But if you do this, then [65:40] everything is going to be fine. Now I [65:43] can go back and run enable to start the [65:49] uh firewall. So as you can see, we get [65:52] this message. It says command may [65:54] disrupt existing SSH connections. You [65:56] can uh let's say yes. Okay. So it says [65:59] firewall is active and enabled on system [66:03] startup. Let's validate this. We have a [66:07] command that we can run. It's called [66:08] verbose. And this is the output that we [66:12] are getting. We have default deny [66:15] incoming, allow outgoing and disable [66:18] disabled routed. And the status is [66:21] active. That's what we want to see. And [66:23] we have uh we have this table. So by [66:27] default you can see that we deny [66:30] everything. This is the most important [66:33] rule. It means that every single port on [66:37] your server is blocked unless you [66:40] explicitly allow it. The the exception [66:42] list is allow in. So here we allow open [66:47] we allow SSH on port 22. This is the [66:50] exception. But the other ports are all [66:53] closed. We cannot use them. This means [66:56] that port 80 for HTTP and port 443 for [67:01] HTTPS are blocked. But in a future [67:04] episode, we are going to open them. But [67:06] for now, we only care about SSH. Great. [67:10] So now we have the firewall. We can test [67:13] it. I will exit the machine like this. [67:17] And now I will try to SSH again by using [67:21] the SSH command. And as you can see I am [67:24] in. I hope that you followed the steps [67:26] in order. And if somehow you are unable [67:30] to reach the server, don't worry. I will [67:33] show you how to use the digital ocean [67:35] recovery console to fix the issue. But [67:38] for now uh we are going to install fail [67:41] to ban to deal with this problem of [67:44] attackers trying to log to the machine [67:48] and b yeah yeah filling in the disk [67:51] space because the SSH service is writing [67:55] a lot of stuff uh into the log file [67:59] because here we are using [68:02] luckily fail toban is available in its [68:05] default repository and can be installed [68:08] told by using the following command sudo [68:12] up install fail to ban dashy means yes [68:15] confirm everything. Let me enter my [68:18] password. Awesome. So we can see that [68:22] the installation [68:24] was successful and now we can verify if [68:28] fail to ban is running or not. As you [68:31] can see, I run this command pseudo [68:34] service and the service that we in that [68:36] we want to inspect is fail toban. And [68:38] they want to look at the status. And as [68:41] you can see here in the active row, it [68:44] says running. Awesome. That's what we [68:46] want to see. Now, hit Q. By default, [68:51] fail to ban only bans attackers for 10 [68:55] minutes. This is too short. and bots [68:58] that are hitting the web automatically [69:01] are going to come back later. So let's [69:04] increase this default value value to one [69:07] day. Let's create a local configuration [69:11] and edit this default value. So if we [69:15] try to run sudo nano etc fail to ban, we [69:21] can look at the jail.coniguration [69:25] file. Awesome. And as you can see uh we [69:28] have some default values and here they [69:30] are telling telling us how to override [69:33] this. So we can see that they say [69:34] provide customizations in a jail. file. [69:39] So this means that we should not updates [69:40] the jail.config because if the system [69:43] updates the values are going to be uh [69:46] overridden and we don't want that to [69:48] happen but the local file is uh not [69:51] going to be changed. And if we scroll a [69:54] little bit, yeah, we can see that we [69:56] have plenty of values that we can change [69:58] and the band time is set to 10 minutes. [70:01] Uh I will explain the other values, but [70:04] let's start by creating the local file. [70:07] Here is the command and I want you to [70:09] copy and paste the following text. Let [70:13] me explain to you these parameters. So [70:15] enabled means that we want to turn on [70:18] protection for SSH band time. As I said, [70:22] it's going to ban attackers for a [70:25] specific duration. Here it is set to one [70:28] day. Max retry allows a user to fail [70:32] five times before being banned. This is [70:35] important. Don't set this to one or a [70:38] low value because you are you can also [70:42] get banned. Let's say that I forgot to [70:45] pass the SSH key and I did that let's [70:49] say three times. If max retry was set to [70:52] three, I will be banned from the server. [70:54] I will not be able to log in. Keep it at [70:57] around three or five. Don't set it to [71:00] one. If you do once one mistake will be [71:03] logged out. And if that happens, you [71:05] need to go to recovery console, open the [71:08] terminal, and unban yourself. But we [71:11] don't want to do that. That's a lot of [71:13] work. Find time is set to 10 minutes. [71:16] And this is the window of time in which [71:18] those five failures must occur to [71:21] trigger a pan. So let's say that t is [71:24] equal to zero. Time is equal to zero. Uh [71:27] so the interval is 10 minutes. If we get [71:30] an attacker who failed to authenticate [71:33] five times during that duration, we are [71:35] going to block it. But if that attacker [71:38] failed only four times, we are not we [71:40] are not going to block it because uh the [71:42] minimum value is set to five. uh if we [71:46] get five failures during that 10 [71:48] minutes, we can ban them otherwise we [71:50] will wait for the next interval and [71:53] start the count from zero again. Now [71:56] save the file and exit. I also want you [72:00] to enable the failtobend service to [72:04] start automatically when the server [72:06] reboots. So do this and start it using [72:11] the systemctl start command. Now we can [72:15] verify that the ban time is equal to 1 [72:18] day by running this command failtoban [72:22] client get shd ban time and it gives us [72:27] 600. [72:28] 600 is basically 60 seconds time stamp [72:32] which is 10 minutes. Let me retry [72:34] running the enable and start commands [72:37] again. Let's go back to get SHD band [72:40] time and it still gives us 600 seconds. [72:44] The problem here is that I forgot to [72:48] restart the service and in order to do [72:51] that we need to run the restart command. [72:54] So this will allow fail to ban to read [72:57] the new configuration that we created. [73:00] Now if I go back and as you can see we [73:02] get 86,400 [73:05] seconds which is 1 day in seconds. Now [73:09] let's see if fail to ban is working. We [73:12] have this command status SHD. As you can [73:16] see we get this output. We have the [73:18] status for the jail SHD. Great. [73:21] Currently failed total failed. We have [73:24] the and currently banned. So we can see [73:26] that fail to ban has banned these two [73:29] IPs and here we have total banned [73:32] currently banned. Now these two [73:35] attackers are not going to be able to [73:38] reach the SSS SSH service because the [73:41] firewall is going to block their [73:44] requests. Awesome. So this means fade [73:47] toban is working as we expected. Now [73:50] I'll keep this running. I'll now [73:53] transition to showing you how to access [73:56] the digital ocean recovery console and [73:59] I'll come back and see if we catched [74:02] some bad actors. You might be wondering [74:06] why do we need to use the recovery [74:09] console. I'll give you an example. If [74:11] you accidentally run UFW enable before [74:17] allowing open SSH or if you configure [74:20] SSH keys incorrectly and lose access to [74:24] your private key. In that case, you not [74:27] be able to log in to the machine and the [74:30] recovery console is provided to us by [74:33] digital ocean to give us direct access [74:36] to the machine through the browser and [74:40] this bypasses SSH entirely. To access [74:44] the recovery console, click on the [74:47] droplets, click on the access tab, [74:50] scroll down, and here you have the [74:53] recovery console. Let's launch it. Okay. [74:56] So, here it tells us that we need to log [74:59] in. First, I want you to start typing [75:02] randomly just to verify that the keys [75:07] that are pressed matches what you are [75:10] seeing because sometimes if you have a [75:13] different keyboard layout, yeah, this [75:15] might not work well. So in that case you [75:18] need to basically start typing your [75:22] login username and password and after [75:26] that delete everything and start typing [75:29] your user. So here I want to login as a [75:33] and the password is going to be this [75:36] one. As you can see I am logged in and [75:40] now I can type in the same command that [75:44] I've showed you earlier. sudo ufw allow [75:47] open SSH it's already allowed so nothing [75:51] will happen um because I am using pseudo [75:53] I need to provide this and as you can [75:56] see it says skipping adding existing [75:58] rule but in your case it's not going to [76:00] say this it's going to add it it's going [76:03] to add this rule for SSH now we are fine [76:07] we can exit and let's close this tab and [76:12] now you can run the SSH command and you [76:15] can login without any problem. Let's go [76:18] back to Digital Ocean again. If for some [76:21] reason you want to reset the p the root [76:24] password, you can do that. But we don't [76:27] have that, so there is no need to. And [76:30] you can open the droplet console by [76:34] specifying the user and you can launch [76:37] it. So if you don't want to use your [76:39] terminal, you can use everything through [76:42] the web page. And as you can see I am [76:44] connected as this user. Let's try to [76:48] connect as root. Awesome. So it failed [76:52] because in the previous session we [76:54] blocked accessing this machine as root. [76:58] Awesome. So everything is working as [77:01] expected. What is next? You might be [77:04] wondering. That's a good question. So [77:06] far you have a solid foundation. You [77:10] have a secure server that blocks [77:12] unauthorized traffic and automatically [77:16] bans attackers. As you can see, I have I [77:20] rerun the status shd command. As as you [77:23] can see, now we blocked four people. [77:26] This is the new IP address that we [77:29] blocked. So, this means that fail to ban [77:31] is working. And as you can see, it's [77:33] automatic. We don't need to do anything [77:35] manually. And that's awesome. And you [77:37] might be wondering why this IP was not [77:41] added to the previous two. So these two [77:44] got unbanned because the band duration [77:47] is set to 10 minutes before I changed it [77:51] to 1 day. And we can verify this by [77:54] running this. As you can see it says [77:57] unban. So [clears throat] if I look at [77:59] the fail toban logs I can filter by this [78:02] word unban. And as you can see these all [78:06] these IPs were uh were unbanned. And you [78:09] can also look at this specific new ID. [78:12] And as you can see it was banned. So [78:15] that's that's confirms it. And here I [78:18] just want to explain this. If you will [78:21] see that these two got unbanned because [78:23] before changing the configuration to set [78:27] the band time to one day, these two were [78:29] already banned. That's why they they [78:31] were unbanned in 10 minutes. But this [78:33] one will stay here and will be unbanned [78:36] the next day. Let's just verify that it [78:39] is still there. Yeah, it is still here. [78:42] And four, this basically keeps a total [78:45] count of how many IPs were banned. It [78:48] doesn't mean that we have four IPs that [78:50] are currently banned. The currently [78:53] banned is what you need to look at and [78:55] it shows us that we have just one IP [78:57] that that is banned from accessing or [79:00] from trying to communicate with the [79:01] server. In the next module, we will [79:04] shift focus from security to software. [79:07] You will learn how to prepare the [79:09] environment by installing Python and [79:11] NodeJS. Setting up swap memory to handle [79:15] builds on a low RAM server like ours and [79:20] configuring J unicorn and supervisor to [79:23] keep your application running reliably [79:26] in the background. Hi everyone. In this [79:30] video, you are going to put your server [79:32] to work by turning it into a web server. [79:37] You will learn a lot from this video. [79:39] I'll show you how to move files from [79:42] your local machine to the virtual [79:44] machine, use SSH tunneling, find and [79:48] kill zombie processes, and more. In this [79:52] video, you will be deploying a full [79:54] stack application that uses Vue.js JS in [79:58] the front end and pass API for the back [80:01] end. The source code is available on [80:04] GitHub and I will make sure to put the [80:06] link in the video description. I am [80:09] using my personal website in order to [80:12] show you the exact same steps I took to [80:16] deploy it. The goal here is not to [80:18] deploy my specific website on your [80:21] server, but use this project as a [80:24] reference. Especially if you don't have [80:26] your own project. If you have your own [80:29] app, but you don't use Vue.js and pass [80:32] API, it's fine. You can adjust a few [80:36] commands in this tutorial to make it [80:38] work with your specific setup. Now, [80:40] let's go back to the slides because I [80:41] want to show you the road map for this [80:44] session. We will transfer the project to [80:48] the server using the SCP command and I [80:52] will show you how to organize the file [80:55] permissions securely. Next, you will [80:58] install the necessary runtimes. We will [81:01] need Python for the back end and NodeJS [81:04] for the front end. I will show you how [81:07] to set up swap memory. This is an [81:10] important step for smaller servers in [81:13] order to prevent out of memory crashes [81:16] during the build process. We will use [81:19] SSH tunneling to securely preview the [81:22] application on your local browser before [81:25] opening it to the public. And finally, [81:29] we will configure J Unicorn and [81:31] Supervisor to make the backend robust [81:34] and production ready. Now let's go to [81:37] GitHub because we need to copy the [81:39] source code. Click on the code button [81:42] and copy this link. Now open your [81:45] terminal. Since this is just a test, uh [81:48] for me, I will put this in the download [81:51] folder, but feel free to clone this [81:53] repository in any directory. Run the get [81:57] cloned command and here paste that link [82:00] that we copied and hit enter. As you can [82:04] see, we are cloning the repository and [82:06] the command finished. Great. Now we need [82:09] to move inside this folder. So I will [82:12] use the cd command and I will type in my [82:16] this name. And now I am inside that [82:20] repository. And as you can see we are on [82:22] the master branch. Now let me type ls. I [82:26] want to see what we have inside. As you [82:28] can see we have the backend folder and [82:30] front end folder. These are the two [82:32] folders that are that contain the s the [82:36] code for the the website. The other [82:39] stuff are not that important for now. [82:41] These folders are inside my laptop. I [82:45] need to move them to the server that we [82:48] created on digital ocean. We could move [82:51] them one by one, but that is slow and [82:55] repetitive. Instead, we are going to [82:57] compress these files and folders into [83:01] one compressed file. In order to do [83:04] that, we will use the tar command. So [83:07] here I am going to call this compressed [83:10] file source code and we need to exclude [83:14] some folders that are not important and [83:17] that will take a lot of space. For [83:18] example, the node modules contains the [83:21] dependencies that the front end needs. [83:23] So we don't need this. we can generate [83:25] it directly on the server. Same for [83:28] Python, we have the virtual environment. [83:31] We need to exclude that. We also have pi [83:34] cache and the getit directory. So we [83:37] don't need this these folders. That's [83:39] why we exclude them. And here we have a [83:43] dot and this means that we want to [83:46] compress everything inside this [83:49] directory. So if you wanted to compress [83:52] just the back end folder, we you need to [83:55] change this dot and put back end or if [83:58] or something else. But here we want to [84:00] take everything and put put it on the [84:02] server. Now hit enter. And as you can [84:06] see, we were able to compress everything [84:08] inside this compressed file. Now we need [84:11] to copy this from our laptop to the [84:15] server. In order to do that, we are [84:17] going to use the SCP command, which [84:20] stands for secure copy. And here, [84:23] because we configured SSH alias in the [84:27] previous chapter, we don't need to type [84:29] the IP address of the server, we can [84:32] just type in the name. So here I will [84:35] take the source code. So I'm going to [84:38] copy this and I'll put it on the [84:42] website. And here we don't need to type [84:45] root uh the user at and then the IP [84:48] address. We can just uh use the alias [84:51] the alias that we created before. And [84:54] here I will put it inside the temp [84:57] directory. Now hit enter. And as you can [85:00] see we have a progress bar that shows us [85:02] that we are uploading this to the [85:04] server. And we it's finished. So now I [85:08] can go SSH to that website or to that [85:13] server using this command. As you can [85:15] see I am in let's verify that we have [85:18] that zipped file and as you can see we [85:21] have it inside the temp directory. Now [85:24] let's organize the project files. We [85:27] should move this compressed file [85:29] somewhere. But where exactly? The best [85:32] practice for web applications is to use [85:35] a neutral location like /var/www [85:40] or a dedicated folder like web app. This [85:44] approach keeps your application logic [85:47] separate from your personal user files [85:50] like SSH or bash history and prevents [85:55] permission issues. So let's create this [85:58] folder with the make directory command. [86:02] So I will use a make directory command [86:05] and I'll put everything inside web app. [86:08] Now we need to move the compressed file [86:10] into this new folder. For that we are [86:13] going to use the move command. I will [86:16] take the path to that compressed file [86:19] and I will move it to web app. Great. [86:22] Now we can um yeah yeah yeah let's [86:25] change directory let's move to web app [86:28] uh let's type ls and as you can see we [86:30] have that compressed file now we need to [86:33] decompress it using the tar command [86:35] again and here is how we do that hit [86:38] enter and let's see yes we have [86:42] everything now we can delete this [86:44] compressed file because we no longer [86:46] need it let's type its okay great so we [86:48] have everything here that's great now [86:51] Before uh we continue, let's type ls-la [86:55] and hit enter. Here you need to see that [86:58] your user is owning these files. As you [87:00] and as you can see, my user is owning [87:02] them. If you see root is owning these [87:05] files and folders, make sure to run this [87:08] command sudo chown. So change ownership [87:12] dash r. We want to do this recursively. [87:14] And here you put in you put the user and [87:17] the group. in in in my case it's going [87:20] to be a MAD and here you specify the [87:24] folder but since we are inside web app [87:27] um we need to yeah change directory but [87:30] or we can just use dos okay and [87:34] hopefully when you when you run ls-la [87:37] you you'll be able to uh see that the [87:40] ownership has changed and your user is [87:42] owning these files. This is very [87:44] important so make sure to do that. [87:47] Awesome. We are now ready to install [87:49] dependencies. Let's start with the back [87:52] end. First we need to install Python [87:54] tools because those are required for the [87:57] back end. Here we are working with the [87:59] with an Ubuntu machine. So this is how [88:01] we can install them. So we are going to [88:03] install pip the virtual environment and [88:06] dev. And here dashy will just accept [88:09] everything and we don't need to yeah [88:11] confirm this operation. Awesome. We [88:14] installed the those tools. Now I'll [88:17] clear the screen. Let's go inside the [88:20] backend folder. And now let's create an [88:24] virtual environment. So we will use [88:26] Python tree dash M. We will use the VNV [88:30] module and we will create a virtual [88:32] environment called VNV. Hit enter. Okay, [88:36] great. Let's activate the environment. [88:39] So we will use source bend/bin/activate. [88:45] If you see the name of the virtual [88:47] environment here before your name is the [88:50] user's name that means that you [88:52] activated the virtual environment [88:54] successfully. Now if I let's type ls [88:58] because I want to see yeah we have the [89:00] requirements.txt file. Those are all the [89:03] the dependencies that we need to launch [89:06] the backend server. In order to install [89:08] them, we will use pip install-r [89:13] and we will provide the requirements.txt [89:16] file. Let's wait for this to finish. And [89:19] here make sure that before you install [89:21] the dependencies that your virtual [89:24] environment is activated cuz we want [89:26] everything to be contained inside this [89:28] project. Awesome. We were able to [89:31] install everything. That's great. Let me [89:34] uh clear the screen. And now we can [89:37] start the back end to yeah make sure [89:40] that everything runs without any issue. [89:42] And here we are working with an with a [89:44] fast API application which means that we [89:47] need to use uh u core. Uh this is this [89:51] command or this library is installed [89:53] will be installed after running pip [89:55] install-r [89:56] requirements.txt. [89:58] And here we have uh so the entry is the [90:03] entry point is inside this file and the [90:06] yeah the app is called app. We are going [90:09] to provide 0000 to the host because we [90:12] want we want to make this the server [90:16] accessible outside the virtual machine. [90:19] And finally we want port and let's use [90:24] 8,000 because this is what the front end [90:27] expects. So now hit enter [90:30] and yeah it seems to be working. If you [90:33] see application startup complete then [90:35] that means that you are in a in a good [90:37] shape and you can see that u is running [90:39] on this IP 0000 [90:43] and yeah if you try to click on this [90:46] link nothing will happen because uh [90:48] let's see let me show you as you can see [90:51] nothing happens because this server is [90:53] running inside the virtual machine it's [90:55] not running locally on our computer uh [90:59] close the page go back to the VM but if [91:02] We can't access this the server outside [91:05] of the VM. How can we test it? Well, in [91:09] order to test it, we need to to open [91:11] another terminal. I will use again the [91:14] sh command and I will go inside the VM. [91:19] But before I test it, I want you to run [91:22] the uh UFW [91:25] status command. uh provide your password [91:29] and I want you to make sure that the [91:33] firewall is active and that you have [91:36] just SSH. So these are the ports that [91:39] are allowed. But we don't have port [91:41] 8,000 or other ports. That's why we were [91:44] not able to access this server outside [91:48] of the the VM. But we can at least test [91:51] using the curl command. So we are going [91:53] to send a request to this to the server [91:57] and here 127.001 [92:00] means local host basically inside the VM [92:03] inside the server and 8,000 is the port [92:07] and this is going to make a get request [92:10] to the server running on this port. So [92:13] yeah, let's see what we'll get. So it [92:15] says detail not found. Uh maybe because [92:20] we need to do health. Uh let me go [92:22] inside the web application back end and [92:27] let's look at the main file. Let's use [92:31] uh or let's use maybe nano main.py. [92:36] Let me make the screen smaller. I want [92:39] to see the end points that we have here. [92:42] So yeah, we need to use / API/hel [92:46] in order to get status is okay. Okay, [92:49] let's zoom in again and here we need to [92:54] before help we need to add API and as [92:58] you can see we get we are we get status [93:00] is okay which means that the backend [93:02] server is responding and if you look at [93:04] the logs you see that we were making get [93:07] requests this we we were getting 404 [93:10] because the endpoints didn't didn't [93:13] exist so the root endpoints or the /hel [93:17] endpoints don't don't exist That's what [93:19] we got 404. But when we try to talk to [93:24] this endpoint API/halth, we got 200. [93:27] Okay, because it exists in the source [93:30] code. Great. So this confirms that the [93:33] backend server is running and that's the [93:35] we installed all the dependencies that [93:38] we need. Hit Ctrl C because we want to [93:40] stop the server. We no longer need it. [93:42] Now we will move to the front end. We we [93:45] are going to install the dependencies [93:47] there and do the same thing. We need to [93:49] verify that it works and after that we [93:51] will combine them together. Uh you can [93:54] see that we are inside the backend [93:55] folder. Let's move uh one layer up. [94:00] We have the virtual environment is still [94:03] active. We can leave it like that or if [94:06] you want you can um write type [94:08] deactivate like so. And now we need to [94:12] go inside the front end folder. I am [94:15] using Vue.js in the front end. This [94:17] means that we need to install NodeJS. [94:21] And the best way to install NodeJS is [94:24] with NVM. Uh if you don't know, NVM is a [94:28] Node package manager. So just type nvm [94:31] GitHub. There it is. And the [94:34] installation the installation is easy. [94:36] You just have you just run this curl [94:39] command. So copy it, go back to the [94:42] server and paste it here. So this is the [94:45] latest version of NVM. Now in order to [94:48] install it, as I said, we just hit enter [94:51] and voila, it's done. Now the the [94:54] problem is that when you install NVM, [94:56] you need to create a new terminal to [94:59] basically get the command. Because now [95:01] if I run NVM, you can see that it's not [95:04] recognized. But we have a solution. we [95:06] can uh use source uh and here I am using [95:11] bash so I will target bash rc [95:16] and this so if when you run this it's [95:18] going to basically load in nvm so now if [95:22] I type nvm you can see that uh it works [95:24] and let's verify that the version is [95:27] correct yes it's 040.4 4. Awesome. Now [95:30] we can we need to install the late the [95:34] latest long-term support version of [95:37] node. And in order to do that, we run [95:40] nvm install and we we uh yeah we provide [95:44] LCS because we that's what we want to [95:46] install. And this is the latest LTS [95:50] version from node. And if you want to [95:53] yeah verify if that's correct, you can [95:56] go to nodejs.org. or uh click on get [96:00] NodeJS and as you can see so the latest [96:02] LTS version is 2413.1 [96:07] let's verify that and yeah that's [96:09] correct so 2413.1 [96:11] we also need to install PNPM this is the [96:15] command that I use to install the [96:16] front-end dependencies I don't use npm [96:20] and this is how you install that make [96:22] sure to install the latest version of [96:25] PNPM version 10 and not prior prior [96:28] ones. Hit enter. And here we want to [96:31] install it globally. Okay. So, it's it's [96:35] working. Uh it says that there is a new [96:38] minor version of npm available. That's [96:40] not important. So, let's click that. And [96:43] now let's verify we are inside the front [96:46] end directory. Type pnpm install and hit [96:50] enter. And as you can see, yeah, PNPM is [96:53] very fast and it's uh I really like to [96:56] work with it. And voila, we have [96:58] everything installed. And now in the [97:01] front um so when working with Vue.js or [97:04] any front end framework, uh you need to [97:07] run the the build command to ensure that [97:10] everything builds. So let's do that. And [97:12] here we are going to hit a out of memory [97:15] problem. So this is why we need to [97:17] create a swap memory. But let's verify [97:19] if that's what will happen or not. Okay. [97:22] So as you can see the terminal is stuck. [97:25] Uh but let's wait a minute. If you want [97:28] you can also go back to the other [97:30] terminal and maybe run hub. But as you [97:34] can see the yeah the the VM is [97:37] struggling. So the I can't run things [97:40] smoothly and that means that I'm running [97:42] out of memory. And yeah, this this this [97:46] command will fail at any second. So [97:49] let's just wait. Yeah, so this one is [97:52] stuck. Let's go back to here. [97:55] And let's wait. Uh I am back to the [97:58] other terminal. And as you can see, the [98:00] memory is full. Uh and this is not [98:03] updating because it's uh struggling. And [98:06] as I told you, the command failed. And [98:09] that's because we don't have enough [98:11] memory to build uh the front end. Is [98:14] that the end? Uh we have a solution. So [98:17] when uh on Linux we have something [98:19] called swap memory. Basically we take [98:23] some portion from the SSD and we use [98:26] that as additional RAM. It's going to be [98:30] slower but it's going to help us uh deal [98:33] with these situations where we run out [98:35] of memory. Let's run the df command. [98:39] Going to show you uh so all available [98:43] and yeah let's type it h to make it [98:46] human readable. As you can see here we [98:49] have 8 GB of disk storage and we have 5 [98:54] GB available. I'm going to take two GB [98:57] from this partition and I'm going to [99:00] create a swap file because we don't need [99:03] all this storage uh to run the front end [99:06] and back end. And this is how we create [99:08] a swap file. So we use the f allocate [99:12] command. We specify the size and this is [99:17] the the file that we are going to [99:18] create. And here so let's just run the [99:21] command first and let me provide my [99:23] password. It worked and we didn't get [99:25] any error. That's great. And I want to [99:28] explain something this. So when you see [99:31] / swap file, this doesn't mean that we [99:34] this is a folder. No, it's a single file [99:38] acting as a virtual RAM. So this means [99:42] that you can just you can't do this. You [99:45] can do this. Um just type in in / LA. [99:49] And as you can see, it's a file. If it's [99:51] that D that means that it's a directory [99:54] but yeah it is a file and it's owned by [99:57] root. Why did we create a file instead [100:00] of a dedicated disk partition? Because [100:04] we can easily resize or delete this file [100:08] if we wanted to if we don't need it [100:10] anymore. This is the flexibility of [100:12] working with files. But we need to set [100:15] up uh some permissions. Uh again let me [100:17] type in ls-la. You can see that these [100:20] are the permissions. We need to change [100:22] them and uh to do that we are going to [100:25] use sudo ch mode 600 and swap file. I [100:30] will explain what the 600 means. Now if [100:33] I type ls-la you can see that some of [100:36] these characters have changed. So w [100:39] means write, r means read and x means [100:43] execute. But here we don't have x. And [100:46] the first three let's say letters are [100:50] related relative to your to the root [100:53] user. The second three are relative to [100:57] uh where is the yeah the second tree are [100:59] here are uh for your user. So in my case [101:02] it's emad and the last ones are for [101:05] other users. So here uh so we did 600 we [101:10] which means that root will be able to [101:13] read and uh to read from swap file and [101:16] write to it but the other users are not [101:20] able to touch this at all. So we created [101:23] the swap file. Now we need to make this [101:26] file as a swap space and to do that we [101:29] are going to use the make swap command [101:32] and we are going to provide that file. [101:35] Hit enter. As you can see, it says [101:37] setting up swap space and here is the [101:39] size and it gives us an ID uh which is [101:43] not important. Now let's enable it. So [101:46] we created the swap space. Now we will [101:48] use swap on and um swap file. Hit enter. [101:53] How do we how do we verify this? We are [101:56] going to use the swap on command. And [101:58] let's type show. And yeah, we have it [102:02] here. So this is the file name and this [102:05] is the size. It says that the file the [102:07] type is file. That's good. And here we [102:10] have priority. The priority level is set [102:13] to minus2. In our case it doesn't matter [102:16] because this this is um used only when [102:20] we have multiple swap files. So if um we [102:24] had another swap file that had a lower [102:27] priority level, then Linux will [102:29] prioritize this one. And when it's [102:31] filled, it's going to move to the other [102:32] one. But in our case, it's not relevant [102:35] because we have just one swap file. By [102:38] default, this swap configuration will be [102:41] lost when the server reboots. To ensure [102:44] that the swap file loads automatically [102:47] every time the system starts, you must [102:49] add it to the FS tab file. Uh let me [102:53] show you the FS tab file that I'm [102:55] talking about. It's this one. Now let's [102:58] try to uh let's see the content. Okay, [103:00] so this is what we have. Uh and we are [103:03] going to yeah put uh this string inside [103:08] there and we are going to use one [103:09] command to do that to make things [103:11] quicker. Here we have echo. It's going [103:13] to spit this string. We use the pipe [103:16] command to take the output from this [103:18] command and give it as input to the t [103:21] command. Let me let's hit enter. Okay, [103:24] great. So it worked. Let's go back and [103:27] as you can see we appended this string [103:29] to the end of the fsst tab command. Now [103:32] that we have added extra memory let's [103:35] try to build the project again. So let's [103:37] run pmppm run build. Uh I will also yeah [103:42] let's uh open another terminal and let's [103:44] go ssh to the server and here I want to [103:49] run htop. Okay. And as you can see htop [103:52] also detects the swap memory. So [103:54] currently it is empty but look what will [103:57] happen when we start the build process. [104:00] Let's see what will happen. As you can [104:02] see we are already filling the swap [104:04] space and look how the the physical RAM [104:08] is quickly filling up. Nothing is [104:11] freezing. That means that it we it [104:13] works. So if you look at the total uh [104:17] space consumed it's yeah greater than [104:19] the total memory that we have inside [104:23] this uh machine and it seems that it [104:26] worked. Yeah it built. So yeah that's [104:28] that's awesome. Before making this [104:31] tutorial sometimes node was not using a [104:35] lot of memory so the build failed due to [104:39] an error called heap limit error. If [104:42] that somehow happens to you, then before [104:46] uh running pmpp run build, we need to [104:49] pass this variable. So here we are going [104:52] to set max all space size to 2 GB. Uh we [104:56] do that before uh starting pmppm. Um in [105:00] my case it worked but if for some reason [105:03] again uh you get that heap limit error, [105:06] you can solve that by using this [105:08] command. Awesome. So we built the [105:11] project and as you can see we have the [105:13] dis folder. It contains everything we [105:15] have in the front end. Uh let's move [105:18] inside that directory cd dist. Uh let's [105:22] see we should have yeah we have our [105:24] assets there. We have the index html [105:28] some images sitemap uh and stuff like [105:31] that. Now we can create a temporary [105:33] Python web server on port 8000. Uh let [105:37] me use Python and we want to use this [105:40] module https [105:43] server and the port is going to be at80. [105:46] Uh I need to use python tree. Okay [105:50] great. So we are serving the built files [105:54] uh on this port. Again we cannot access [105:57] this. So if you try to open this on your [106:00] local machine, it's not going to work [106:02] because the firewall is active and we [106:05] are not uh letting traffic through this [106:09] port. But is are we done? Um maybe yeah, [106:13] we can use the cur command if we want. [106:15] So we can do something like this. But [106:19] yeah, we are we have port 8080 this [106:22] time. And as you can see, we got uh some [106:24] HTML back. Uh so that's that means that [106:27] it's it's working right but we want to [106:30] see this we want to see the UI because [106:33] the back end uh it's it's fine if we run [106:35] cur commands on the back end but when it [106:38] comes to the front end we need to see it [106:39] with our eyes to validates that it is [106:42] working but you might say well we cannot [106:45] do that because the firewall is active [106:48] but I have a solution for you and it's [106:50] called SSH tunneling. Let's go back to [106:53] the slides because I want you to learn [106:56] about this amazing feature. Okay, here [106:59] we are and let's look at SSH tunneling. [107:02] I have this diagram that I made in order [107:05] to explain this concept for you. We have [107:08] our laptop is sitting here. It can be [107:12] running maybe Linux, Mac or Windows. It [107:15] doesn't matter. We have the SSH key [107:18] pairs that we created. Uh here we have [107:21] the private key and we will be uh [107:24] accessing the the website through a [107:28] browser and here on the other side we [107:30] have the server. It's can be running on [107:32] digital ocean, Azure, GCP, AWS doesn't [107:36] matter. We also we will have our front [107:39] end running here. Uh in our case it's [107:42] Vue.js JS but it can also be react or [107:45] any other framework or just pure HTML [107:48] CSS and JavaScript and we have our [107:50] public key in the middle we have a [107:53] tunnel that we call SSH tunnel how do we [107:56] see the app without opening uh let's say [108:00] port 8080 the we use SSH tunneling for [108:03] that think of this as a secure private [108:06] subway line that runs inside your [108:09] existing SSH connection we don't need to [108:12] change the firewall because we are [108:14] already allowed to access the server via [108:16] SSH to create a tunnel. We use the -l [108:21] flag. And this syntax tells SSH to bind [108:26] port 8080 on your laptop. It's this one [108:30] colored in pink to port 8080 on the [108:34] server. After running this command, a [108:37] tunnel will be created. Now if I open my [108:40] browser and type localhost and specify [108:44] port 8080 in my browser the request is [108:48] not going to stay on my computer. It's [108:51] going to travel through the tunnel. It's [108:54] going to reach the server the this uh [108:57] and here it's going to reach the server [108:59] on this port 8080 because this is what [109:01] we specified here. It's going to see [109:03] that we have a server running. This is [109:06] why we have uh yeah we run that Python [109:09] command to create a temporary server. [109:11] Okay. So it's going to forward that to [109:14] the front end. The front end will [109:16] respond with that HTML. It's going to go [109:18] back through the tunnel. It's going to [109:20] hit the browser and the browser uh sees [109:24] HTML. It's going to load it. So this is [109:26] the concept uh of SSH tunneling and it's [109:29] really great. And as you can see, we [109:31] didn't update the firewall configuration [109:34] to allow port at it enabled directly [109:38] because we are already using SSH. But if [109:41] we stop this command, the tunnel will go [109:44] yeah will go away and these ports are [109:47] going to be are not going to be [109:48] connected. Awesome. So yeah, let's do [109:51] this. Uh uh let's create. So here I'm [109:54] already inside the machine. I need to [109:57] exit by using Ctrl D or exit. And now [110:00] yeah, let's run let's write that [110:03] command. So it's SSH-L. [110:06] Uh let's type the local port on my [110:09] machine. Here we need to specify [110:11] localhost AT80 and let's use my alias my [110:16] website. As you can see, we are inside [110:19] and now we have a tunnel. Let's go back. [110:23] You can see that we have the server [110:24] running. Now let's go. Yeah, let's open [110:28] Firefox. And now we can go to localhost [110:32] at80. And voila. As you can see, it [110:35] works. You saw that we we got some [110:37] errors because we are making uh requests [110:40] to the back end, but we don't have a [110:42] back end running. Uh yeah, so you can [110:45] see that if I go to the blogs page, [110:47] yeah, nothing will show up because we [110:49] don't have the back end running yet. But [110:51] that's not a problem. What we want to [110:53] see is uh basically the UI. Does it look [110:57] right? Um did we load the images? Um is [111:01] the font loaded? Do we have animations [111:04] running? Stuff like that. This is what [111:06] we wanted to verify. But connecting the [111:08] front end to the back end or to other [111:11] services uh should be uh easily done [111:15] when we connect it to the back end. [111:18] Awesome. And yeah, let's go back. Uh now [111:20] what will happen if I do this? So let's [111:23] close this. Now we have no SSH tunnel. [111:27] Let's go back and let's refresh. Can see [111:29] that it doesn't work because now the the [111:32] two machines are not connected. Great. [111:35] So everything is working perfectly. [111:38] Nice. We installed the Python the back [111:41] end and front end dependencies. We made [111:43] sure that both servers run correctly. Uh [111:47] everything worked really well. [111:50] Now we need to move to installing J [111:54] unicorn and supervisor. But why do we [111:57] need them? Let's go back to the slides [111:59] again to explain this. Here is the [112:02] diagram that I created for this slide in [112:05] order to explain this. Uh as you can [112:07] see, we have three components. We have [112:09] supervisor, junior and uvicorn. We have [112:14] used the uvicorn command to run the [112:17] python server. After that, we used curl [112:20] to ensure that the the back end responds [112:24] correctly. So that's done. But we did [112:26] this manually. What will happen if the [112:28] server crashes? We need to again SSH to [112:32] the server and run the UV command again. [112:35] Is this scalable? No. What will happen [112:38] if your server crashes at 4:00 a.m. or [112:41] something? Are you going to uh wait [112:44] until 8:00 a.m. when you wake up and [112:47] launch it? No. We want the system, we [112:49] want the server to do this for us [112:51] automatically. And this is why we need [112:53] these three components or this [112:55] architecture. At the bottom, we have our [112:58] workers. The worker he's is is Uicorn. [113:02] The the worker is going to run your fast [113:05] API code. So if someone visits the [113:08] website and wants to get blogs, the [113:11] worker will get that request. It's going [113:13] to talk to me search and it's going to [113:16] get the response and send it back to the [113:19] user. Or let's say that someone wanted [113:22] to maybe do a search query or maybe uh [113:26] like a blog. U yeah that that worker [113:30] will also do that. And let's say that we [113:32] had a huge traffic um let's say that we [113:36] we have maybe 100 people trying to [113:39] access the server at the same time. Then [113:41] the load is going to be balanced through [113:44] these workers here. Let's say that we [113:46] have three workers. So this one is going [113:48] to get maybe 33 requests. The other one [113:50] is going to get 40 and the other one [113:52] gets 30. So yeah and J Unicorn will do [113:56] that. It's going to balance the load [113:57] through all these workers. Now we know [114:00] the role of Uicorn but why do we need J [114:03] Unicorn? J Unicorn acts as the manager. [114:07] It spawns multiple Uicorn workers and [114:11] this is can be configured. So if you [114:12] want to you have just one URON instance [114:15] or worker you can do that. If you want [114:17] to have three or 10 or whatever yeah you [114:21] can do that. Um you can just change the [114:23] configuration. We will see this after [114:26] explaining this slide. Let's say that [114:28] this worker crashed for some reason. [114:31] Jorn will detect that. It's going to [114:34] delete it and replace it with a new with [114:37] a new uh worker. And this is going to be [114:41] done automatically for you. You don't [114:43] need to enter to SSH to the server and [114:46] start the worker manually. That's not [114:49] scalable. But this this is above that we [114:53] have the supervisor or we can also call [114:56] it watchdog. Uh in engineering this term [114:59] means a hardware component that [115:02] basically will hit the reset button if [115:05] the system freezes for some reason. [115:07] Let's say that something bad is [115:09] happening. The unicorn is not doing [115:12] good. The unicorn curs are also [115:16] crashing. Supervisor will detect that [115:18] and uh hit the resize button which means [115:21] that uh we are going to recreate [115:22] everything from uh scratch just to [115:25] ensure that yeah the the our application [115:28] is healthy again and is ready to receive [115:31] requests from users. So yeah, this is [115:34] the this is why we need J unicorn and [115:37] unicorn and also supervisor and you saw [115:41] we tested manually. Always do that. Try [115:44] to test uh your application manually by [115:47] running the commands like what we did. [115:50] So we run uh uicorn to start the backend [115:54] server. After that we moved to the front [115:57] end server. After validating that we can [116:01] automate this process. Let's go back to [116:03] the server and let's do this. Okay. Uh [116:07] now let's stop the front the temporary [116:10] front end server because we don't need [116:12] it. Uh let's also go back to back end. [116:18] Activate the environment and let's [116:21] install J Unicorn. I'm not sure if I [116:24] already have it in the requirements [116:26] file. Let's just double check. And yes, [116:28] we have it. So yeah, that's great. So we [116:30] don't need to install it separately. Uh [116:33] let's make sure that it is there. Yeah, [116:35] it is there. So we it just didn't [116:38] receive any application. So that's fine. [116:40] So we have J Unicorn and we also have UV [116:43] core. Now let's try Yeah, we need to [116:45] create a J Unicorn start script that [116:49] will tell J Unicorn how to run uh your [116:52] application. Let's call it J Unicorn [116:55] start. And let's use nano. I will paste [116:59] all this directly here. But don't worry, [117:03] you you should be able to find this on [117:06] my blog. Yeah, I will make sure to [117:09] deploy it uh in the future. So if you go [117:11] to blogs, you will be able to see the [117:14] individual articles here. So if you want [117:17] to copy paste them uh quickly, you can [117:20] do that. Don't worry about it. Or you [117:22] can go to GitHub. I have this repository [117:25] where I yeah basically documents these [117:28] uh videos in a in text format. So we [117:32] have foundation. So now we are in [117:35] chapter 2. So you can click here and [117:37] you'll should yeah will find everything [117:39] here. So can just copy the commands and [117:42] uh paste them uh in your VM. So okay [117:46] let's continue. We forgot to add the [117:49] shebang. So let's add it. Uh we are [117:53] going to use bash. We have some [117:55] placeholders that we need to change. [117:57] Let's start with the user. So my user is [118:01] IMAD and the group is also IMAD. So [118:05] let's provide that. We have the number [118:08] of workers. Let's set it to one because [118:10] we don't have a lot of traffic and we [118:13] are just testing. So as you can see I [118:16] have this comment. increase this if you [118:17] have more traffic or or a more complex [118:20] application. But when you increase the [118:22] number of workers, you will consume more [118:24] RAM. So keep that in mind. The [118:26] application directory is web app back [118:28] end. That's correct. And we have this [118:31] stocket file. We'll come back to this [118:33] later. And here we need to give the [118:35] project a name. Let's call this my [118:38] website or something or a mad website. [118:41] And here the worker is going to be [118:43] Uicorn. That's fine. here forward allow [118:47] ips. Yeah. So here here we just want to [118:50] pass the IP of the user directly uh [118:53] because yeah we will talk about [118:54] engineext and unique sockets later so [118:57] don't worry about this. Here we are just [118:59] showing that we are starting this [119:01] website. We move to the directory. We [119:05] activate the environment. Here we create [119:07] the socket file. Uh let's go back to [119:10] sock file. We need to create this. Uh so [119:12] this is important because it's going to [119:14] communicate with engineext but we will [119:17] talk about this later don't worry about [119:18] it and here start junicorn so here we [119:22] have the execute command j unicorn is [119:25] inside the virtual environment we could [119:27] have just uh yeah since we activated the [119:29] environment we could just type j unicorn [119:32] that's fine the application is inside [119:34] the main file and we pass these [119:37] arguments so the name of the project the [119:40] number of workers [119:42] the worker class which is uicorn. Here [119:45] it is. Uh time out the user and the [119:48] group. We bind this to the socket file. [119:52] Uh we forward the IPs and we have some [119:55] login. Great. So now save this. Uh I'll [119:59] hit Ctrl S and then Ctrl X. And now we [120:02] need to make this executable. So let's [120:05] do sud ch mode plus x or before that I [120:09] want I want you to understand the ch [120:12] mode command. So before that let's run [120:14] ls-la j unicorn start. You can see it's [120:18] not executable. But if I run ch mode [120:22] plus x j uh unicorn start and run the [120:28] ls-la you can see now we have x and x is [120:32] is available in the three users. So ruse [120:35] can execute this file my user ID can [120:39] execute this file and other users can do [120:42] the same. Great and you can see that I [120:45] am the owner of this file. So we [120:47] configured the unicorn. Now we need to [120:50] configure supervisor. We could again we [120:54] could run this uh script manually but if [120:58] it if for some reason J unicorn crashes [121:01] we need to come back and uh rerun it [121:04] again. We want we don't want to do that. [121:06] We want supervisor to do this for us. [121:09] Supervisor is not installed by default. [121:13] Yeah, it's not recognized but we can [121:16] install it using the op command. So [121:18] let's do that. Supervisor. [121:21] Let me provide my password. And it is [121:24] installing. Great. Let's wait a little [121:26] bit. Uh let's also create the logs [121:29] folder because we we want supervisor to [121:32] save logs um there so that we can uh if [121:37] something hap bad happens we can uh see [121:40] what happened there. So that's important [121:42] and again verify that you are owning [121:45] that uh directory. So run this and yeah [121:49] we are running that since we don't use [121:52] the sudo command we will be uh owning [121:54] everything but just in case uh it's a [121:56] good practice to keep checking the [121:58] ownership. In order to configure [122:00] supervisor we need to create this [122:03] configuration file. So supervisor lives [122:06] inside the etc folder and here we want [122:09] to create our own configuration and let [122:13] me paste this. Uh okay so we have so as [122:17] you can see we want supervisor to run [122:19] the J unicorn start start command. Uh so [122:23] the only thing we need to change here is [122:25] the user. It's going to be emad and here [122:29] we need to change this placeholder and [122:32] let's call it a mad website. Okay. And [122:36] the log file we are going to store to [122:38] yeah to store logs inside this. This is [122:40] why we created this directory. And yeah [122:43] um save this. Hit ctrl x. Great. Now we [122:47] can start this service. Let's use pseudo [122:51] supervisor [122:53] ctl. And here we are going to reread the [122:56] new configuration. It says available. It [122:59] detected this uh new config that we [123:02] created. And now we need to let's run [123:05] this. Great. So we added this to the [123:08] process group. And now let's check the [123:11] status. It should be running. So if I [123:13] type status uh and here you need to type [123:16] that project name. Great. It says run [123:19] in. It didn't crash. That's great. Let's [123:23] see. So, we have logs. So, let's look at [123:26] logs. Supervisor log. The file exists. [123:29] Awesome. Let's do cat or maybe tail. [123:33] Awesome. So, it's working. You can see [123:34] that. Uh, let me let me look at the full [123:38] output. So, these are just this is just [123:40] configuration. Um, yeah, great. And here [123:45] the application started. So we are [123:47] starting J unicorn. Uh it's listening at [123:50] this socket and here we have just one [123:52] worker and the application startup [123:55] complete. So this means that it's [123:56] working. We can try and uh yeah let's do [124:00] some curl commands. Oops, we don't have [124:02] it here. Uh that's fine. Okay. Uh this [124:06] is the curl command that we used [124:08] earlier. Uh let's try to run it. And it [124:11] fail. It says fail to connect to local [124:14] host. Why this command failed? We have [124:17] tested this before and it worked fine. [124:20] Let me show you why this failed. [124:23] Remember we have uh yeah let's look at [124:26] the G unicorns tart uh file. You can see [124:29] that we are using this socket file. So [124:34] in order to communicate with the server, [124:36] we need to use the socket file that was [124:39] created here in back end. And let's [124:41] verify that it is there. Yeah, as you [124:44] can see, we have JU Unicorn stock and if [124:47] I look at it, yeah, it's it's it's a [124:50] different type of files in Linux. You [124:53] can see that it says S, which means [124:55] socket and this is used um it's like a [124:58] pipe. It's used to uh by different [125:00] applications to communicate with each [125:02] other. In our case, we want uh the [125:04] backend server to communicate with [125:06] EngineX. Uh but we don't have EngineX [125:10] yet. We are just preparing for that [125:12] step. uh but uh to fix that we can do [125:15] this. So with curl we can specify the [125:19] Unix socket file and here here is the [125:22] the socket file that we want to use and [125:25] same here we are going to use this [125:27] endpoint and curl will use this socket [125:30] as a way to talk to the backend server [125:34] and the backend server will send the [125:36] response through the socket to curl and [125:39] curl would display that. As you can see, [125:41] I I run that and I got status. Okay, [125:45] awesome. Now we can uh run tail again [125:49] just to verify and yeah, we got this. So [125:51] we uh so this is the get requests. We [125:55] got 200. Okay. And yeah, it took less [125:58] than 2 milliseconds. Awesome. So yeah, [126:01] everything is working. Now if we run [126:04] htop or I want you to install this. So [126:08] we have a replacement um we have many uh [126:12] applications like htop with good UI. One [126:16] of them is called Btop and I really like [126:19] Btop. Uh it looks like yeah this is a [126:22] Btop and it's available on a lot of [126:25] Linux distros. It's also available in [126:27] Auntu. So that's great. And to install [126:30] it we can as you can see it looks really [126:33] really nice. So we are going to use this [126:34] in order to monitor the processes, [126:36] memory usage, etc. And to install it, we [126:40] can run the APC command uh sudo up to [126:43] install btop. Awesome. Now run btop and [126:46] as you can see uh it looks really nice. [126:48] So here you have the CPU usage, you have [126:51] memory usage. Uh let me decrease the [126:54] yeah the the font size. Okay. So we have [126:57] uh so we have 8 GB of storage. We are [127:00] using 65%. [127:03] This is the swap um space that we [127:05] created. It's not used. So we are just [127:08] using 6%. This is the available memory. [127:11] It's uh yeah almost 500 megabytes. We [127:14] are using 35% of the memory. Uh and yeah [127:18] we have also the network. Uh so we have [127:20] the download and upload speeds. Um and [127:24] here we have the processes. This is what [127:26] we care about. And here you have filter. [127:29] With the arrow keys, you can uh let's go [127:32] to memory because that's what we want. [127:34] And this is going to sort everything by [127:36] memory. If you want, you can hit R to [127:39] reverse this. But we want to yeah, we [127:41] want this sorted in a descending order. [127:44] And here type filter and run uh and type [127:48] J unicorn. Why? We want to see the [127:51] processes that are running. Here we have [127:53] just one worker but you you can see that [127:56] we have two processes. Why is that? So [128:01] one J unicorn creates two services. So [128:05] one will manage the workers. So one of [128:08] them is the worker and the other one is [128:12] uh the one controlling those workers and [128:15] as you can see they are using uh roughly [128:18] 24 megabytes. That's not a lot because [128:20] the application is not doing anything [128:22] right now. But this is why I mentioned [128:26] that if you increase the number of [128:27] workers, [128:29] you are going to consume more memory. [128:32] Why I why I installed bet? Because I [128:34] want you to I want you to learn about [128:38] zombie processes. Um yeah, this is [128:41] something very important because [128:42] sometimes uh this happened to me. I [128:46] didn't pay attention and I my memory [128:48] usage was rising up quickly because I [128:51] was I was not restarting J Unicorn [128:54] correctly uh because sometimes when I do [128:57] when I work on a new feature uh I push [129:00] that to the server and uh restart the [129:03] back end that created zombie processes. [129:06] Basically some uh G unicorn processes [129:09] were left here. they weren't doing work [129:11] but they were consuming memory because [129:13] they were from the old deployments. So [129:15] let's see uh we don't have this problem [129:17] right now. Uh we we have just created [129:20] this uh but I will try to do uh to [129:23] basically create zombie processes just [129:26] to show you how this looks like and how [129:28] to deal with this problem. Uh but first [129:30] we need to go and SSH through the [129:32] website. Great. So if you don't want to [129:36] look at this, we can run um this command [129:39] ps and we are going to grab and we are [129:43] going to use I think aad websites. We [129:47] have two of them. So yeah, this one and [129:50] the other one. Uh this looks ugly [129:53] because we have a lot of text but no [129:55] problem. It confirms that we have two [129:57] processes running. Now let me manually [130:00] let's first go to web app back end and I [130:04] want to run this manually just to create [130:07] more processes. Uh let's do this. So J [130:10] unicorn now hit enter. This is going to [130:12] create uh yeah this is going to run [130:15] again the the back end and it's going to [130:17] create more processes. Uh let's wait a [130:20] little bit. You can see that the memory [130:21] usage is getting higher and higher every [130:24] single time. Okay, good. And now if we [130:27] want to we can also go inside uh that [130:30] file [130:32] G unicorn start. And um yeah let's [130:36] change this from 1 2 3 save exit. And [130:40] now we need to restart. We will run this [130:42] command because we changed the [130:44] configuration. We need to read that and [130:47] restart again. Let's do this. So restart [130:50] and let's uh use my project name. Uh I [130:53] need to pass my password. Okay, it says [130:56] stopped. Let's look at this. Okay, so uh [130:59] the old processes were gone and now we [131:01] have three new ones. Uh we should have [131:04] just the manager is going to be [131:06] controlling the the workers and we [131:09] should have three workers because this [131:10] is the new configuration that we [131:12] created. But we also have how many? So [131:15] one we have 1, two, three, four. But we [131:17] have three orphaned processes or we can [131:21] also call them zombie processes that are [131:23] look consuming memory but not doing any [131:26] useful work. And if you get this somehow [131:31] uh this is how you solve this issue. You [131:33] can't just come here and hit okay to [131:36] kill the process because it's not clear [131:40] which one is uh doing work. I mean you [131:44] can sort of uh get that information by [131:47] looking at the memory because the the [131:50] workers that are busy are going to be [131:52] using more memory but yeah the you need [131:55] to confirm this and I'll show you how to [131:57] do that. Uh let's run uh ps. [132:01] As you can see we have uh yeah we have [132:04] plenty of processes. Great. Now I have [132:08] this uh command that we can run. uh it [132:11] it's basically giving us the same info [132:13] that we have here in Btop. I'm just [132:15] showing you that running commands like [132:17] these, it's not that fun, but looking at [132:20] this uh in a terminal app, it's really [132:23] uh useful and it's fast. But you can do [132:26] the same thing with just Linux commands. [132:29] And here are the processes and how much [132:31] memory they are using. We can also try [132:33] to uh sum the memory. So we are using [132:37] almost 300 mgabytes. But yeah, we have [132:40] some processes that we need to remove to [132:42] free some memory. So the quick fix is to [132:45] kill all J unicorn uh processes. And to [132:49] do that you can either do you can do it [132:52] here uh by hitting K and and you say yes [132:56] I want to kill this but uh if you want [132:59] to kill everything you can use uh [133:01] process kill and this is the name of the [133:04] process. uh but this will cause [133:07] downtime. Your your application your [133:09] users won't be able to access your [133:12] application. Uh the front end will try [133:14] to make requests to the back end but [133:15] it's it's not going to respond. So they [133:17] will get errors. So this is the easiest [133:21] uh you do this and then supervisor will [133:24] start everything up because it sees that [133:26] something bad happened. Where are the J [133:28] unicorn services? It's going to panic [133:30] but it's going to uh yeah run them. [133:34] Let's actually do it. So let's look at [133:36] this. So we have uh all these J unicorn [133:39] services. Let's kill them. Now let's go [133:41] back here. You can see they are gone but [133:44] supervisor create them created them [133:46] quickly. You can see that we have four. [133:48] That's the correct amount. We have the [133:50] manager. Uh one of them is the manager [133:53] and the other ones are workers. And you [133:56] can see this is why we used supervisor [133:58] in order to uh solve this issue for us. [134:01] Great. Yeah. This is the the easiest uh [134:06] solution and the quickest but if you it [134:08] will cause some downtime but yeah it [134:11] depends on your application. If you [134:12] don't have a lot of users just do this [134:14] one it's fine. Uh but if you have more [134:17] users then yeah we shouldn't do it. Let [134:20] me show you the other option. So I will [134:23] again run the J unicorn start command. [134:28] Uh okay, we created more or more [134:32] processes. [134:33] Let's restart. Okay, so you can see that [134:36] the restart command is the one that's [134:38] causing the issue. It's yeah, it re it [134:41] reads the new configuration, [134:43] but it doesn't delete the old processes. [134:47] It just creates new ones. So this is the [134:50] the big issue that we need to solve. And [134:52] this is what I was doing uh in my [134:55] websites running currently on [134:56] production. I was just pushing new [134:59] changes and I run restart before [135:03] removing the old ones and this created a [135:06] lot of orphaned uh processes. So we can [135:09] see that now we need to remove uh three [135:11] more processes. Okay, I show you uh I [135:14] have showed you the easy method. Now [135:17] let's try to solve this uh surgically [135:21] without causing any uh downtime. First [135:25] we need to ask supervisor for the [135:29] process ID of the J unicorn process [135:32] which which we can also call master [135:34] because it is controlling the other [135:36] workers and to do that we use sudo [135:40] supervisor ctl P ID and here uh you type [135:43] your project name so this is the P ID of [135:47] the master process this is very [135:50] important in in your case it's not going [135:52] to be the same it's going to be a number [135:54] But just uh remember it. Now let's look [135:57] at the process hierarchy. We can use [136:00] this command. Okay. So yeah, we have a [136:03] lot of text. We can also do the same [136:05] thing here uh in Btop. You can see that [136:07] you have this tree column. Just uh hit E [136:12] and as you can see it groups everything. [136:14] So this is the master process. You can [136:17] see that it's uh it is controlling these [136:19] three processes and these ones they [136:22] don't have a parent. So they are just [136:24] here they are orphaned processes that [136:27] are consuming a lot of memory uh but uh [136:31] doing no work. So we need to kill them. [136:33] So now this makes it easy. So I can just [136:36] hit K and kill them one by one. But we [136:39] can also see this in the ps-f [136:42] command. Uh yeah I know this is this is [136:45] a lot of text. So let me just copy this [136:48] and let me open a text editor. So I'll [136:52] open them in VS code. Uh so yeah this [136:56] this stuff is not needed. So you can see [136:58] that these are all the commands that we [137:01] uh are used in order to run the s the [137:04] yeah the junior core service. We don't [137:07] want them. So let's just delete this [137:08] stuff. Uh we don't need this two [137:12] uh and this. Okay great. Let's zoom in a [137:15] little bit on this part. Okay great. So [137:19] let me also go back to the terminal. Let [137:22] me also copy the master ID. So this is [137:26] the master process ID. It's this one. [137:30] You can see that how many we have three [137:33] of uh we have 1 2 3 4 5 6 7 and eight. [137:38] Where is the the process the manager? [137:40] Here is the manager. So uh and here I [137:43] forgot to add the column names. Uh I [137:47] think uh let let me go back. I just want [137:49] to confirm something. Let's go here. [137:52] This is the process ID. This is the [137:55] parent process ID. RSS. Yeah, we don't [137:58] need this one. So let's remove it. We [138:00] just want these two. Let's see. We have [138:04] uh yeah, these are the three UVON [138:07] workers and they are attached to this [138:11] master process ID. And if we look at [138:14] this, it's going to point out to this [138:16] one. And this is that fourth J unicorn [138:20] service. It's this this is the same as [138:23] this. We have these. So let me double [138:26] click on this one. As you can see, here [138:28] is the process ID. It's 531 402 531 402. [138:33] And it is attached to the parent. Uh [138:36] let's look at the parent. Yes, it's this [138:38] one. 531399. [138:41] This this is your group. It's exactly [138:44] it's exactly this. You have one process [138:47] ID and one master process and their [138:51] children. But we have these three [138:54] strugglers, these three orphaned [138:56] processes. They are these ones. So 274, [139:00] 272 and 273. Here they are. And the PP [139:04] ID is equal to one. Hm. What does this [139:07] mean in Linux? P1 [139:11] is the system in it process. This means [139:15] these processes lost their parent but at [139:19] least they were adopted by the init [139:22] process because they they need to be [139:24] attached to something. This is how you [139:27] detect orphaned processes. If they are [139:30] not attached to a process ID uh if they [139:34] lost their parents they will be attached [139:36] to in it. And now we can safely delete [139:38] these. And we have the process ids. So [139:41] we can do it uh here with the help of [139:44] the top. Let's do it once for this one. [139:47] So we hit okay. And yes. [139:51] And it should Yeah, it's gone. Bye. We [139:54] no longer have that zombie process, but [139:56] we have the other two. Uh if you want to [139:58] do this with the terminal, you can just [140:00] uh use pseudo kill. And here you provide [140:03] the IP addresses or the process ID [140:06] sorry. Uh you pass them like this one [140:09] after the other. Now hit enter. Go to [140:12] Btop and they are gone and we freed [140:15] memory. You can see that uh we we had to [140:17] do a lot of stuff in order to detect the [140:19] processes that uh are not used by uh our [140:23] back end. But this this is useful if you [140:27] don't want to cause any downtime at all. [140:29] Your application code is on the server. [140:32] We built it. We run it and we everything [140:37] went smooth. The back end is managed by [140:39] supervisor and listening on a private [140:43] socket file. However, no one from the [140:46] outside world can access your [140:48] application yet. This is a problem that [140:50] we will fix in the next sub chapter. We [140:53] are going to talk about engine X [140:56] security headers and stuff like that. [140:59] See you in the next episode. Hi [141:01] everyone. In the previous section, you [141:04] successfully started your backend and [141:07] set up a process manager. Your API is [141:10] alive and listening on a Unix socket [141:14] file. However, your users cannot access [141:17] your website. Your front end is also [141:20] just a folder of static files sitting on [141:23] the server waiting to be served. In this [141:27] sub chapter, we are going to talk about [141:30] reverse proxy and security headers. [141:33] First, we need to sync the front end and [141:36] backend URLs. After that, we will [141:39] install EngineX to act as a reverse [141:42] proxy. I will explain this once we [141:45] install Engine X. I'm going to explain [141:48] to you what does 502 bad gateway means. [141:53] So this is an error that we might get if [141:56] we don't set up enginex correctly. And [141:59] finally we will move to the security [142:01] part. In this sub chapter we are going [142:04] to focus only on the HTTP security [142:07] headers mainly CSP and there are other [142:10] ones but CSP is the one that that [142:13] requires a lot of work and we will focus [142:15] on it at the end of this video. Now [142:18] let's SSH to the machine because we need [142:21] to start by syncing the front end and [142:23] back end and I will come back to the [142:25] slides in if I want to explain some [142:28] diagrams. As you can see I am logged in [142:31] to the machine and before we configure [142:34] engine X you need to prepare your code. [142:36] You want your front end and back end to [142:39] act as if they live on the exact same [142:42] server both in local development and in [142:45] production. This avoids complex cross [142:48] origin issues. Let me show you what I am [142:51] talking about. As you can see, I need to [142:53] go to the web app directory because last [142:57] time we moved the source code here. I [142:59] will change ls to cd. Now I am inside [143:03] that directory. Let's start with the [143:05] front end. Let's go to front end. And [143:09] here we need to go to source. And we [143:12] have main.js. So we need to open that [143:17] file. Let's use nano and let's see. As [143:21] you can see here we are using an [143:23] environment variable v a base url. Let [143:27] me show you the content of that [143:29] variable. Let's do ls. And here I need [143:33] to go back to front end because here we [143:37] have the environment file as you can. So [143:40] let's do ls-la. [143:42] Yeah, you can see that we have env.ample [143:46] and we also from that we create the env [143:49] file. But let's look at env.ample [143:52] by using cat. So let's provide the file [143:57] and as you can see this is what we have [144:00] here. The back end runs on this port. [144:04] When you are developing this locally the [144:07] back end will be running on a different [144:08] server. And in the front end we use [144:10] Axios to make HTTP requests and it needs [144:15] a it needs to know where the the API [144:17] lives and here it lives in port 8000. [144:22] This is good but it introduces cross [144:26] origin issues. If we try to send a [144:29] request to this port, the browser will [144:31] block it because it sees that you are [144:33] trying to communicate with a different [144:35] origin and even though we are running on [144:37] local host, it it's not going to be [144:40] allowed because the front end runs on [144:42] port 8080 [144:44] but your back end runs on port 8,000. So [144:48] that's how I started developing this but [144:50] I will show you a simple way to avoid [144:52] this problem. We are going to use vit [144:55] and with vit we have something called [144:57] proxy and it will make this very simple. [145:00] So this example we no longer need it. [145:04] That's why I wanted I wanted you to open [145:07] the main file and change it. So let's do [145:10] that quickly. Now you can see that here [145:12] I am using axios and the base URL is we [145:16] set it here. I I will remove all of [145:18] this. I don't need it. And here instead [145:21] I will just put slash. By doing this the [145:25] browser will automatically send requests [145:27] to the current host. So if I am running [145:29] on local host this will will tell the [145:32] browser to send it to local host. If I [145:34] am running on my domain it's going to be [145:38] that domain. That's the first change [145:40] that we need to make. Now save the file. [145:43] Exit uh with Ctrl X Y and enter. Great. [145:48] Now we need to configure vit. So let's [145:50] look at vit. It's not here. Uh it's it [145:53] must be here. Yes. So we have vit [145:55] config. Let's open it with nano vit [145:58] config. Now I want you to scroll down [146:01] here. You can see that we have server. [146:03] And this is uh where we set the port. By [146:06] default vit starts on on a different [146:10] port. I think 4,000 something. But we uh [146:13] I usually change it to 8,080. [146:17] And here we can add another field to the [146:20] server block and we we call it proxy. [146:23] Here is the the field that we need to [146:25] add to server. You can see that it's [146:27] called proxy. And here the backend API [146:31] is yeah it has this prefix / API and we [146:36] tell V to send that request to this URL [146:40] or or to this target. This means that if [146:43] the front end tries to send a request to [146:48] the API, vit will intercept that and [146:52] target this URL. This way we can change [146:56] origin without having to worry about [146:59] cross origin issues. Let me explain this [147:02] with another diagram that I created. You [147:05] can see that here we are on a local [147:07] machine. we are developing the the web [147:09] application and here I I am using let's [147:13] say Chrome and I want to make a an HTTP [147:17] request. So I I am in the browser. If I [147:20] want to load static files basically the [147:23] web page I will call I will be here. So [147:27] the path will not be API. I will just [147:29] talk to the front end. The front end [147:32] will send back the HTML, CSS and [147:35] JavaScript files. Vit will take that and [147:38] send it back to the browser and the [147:40] browser will display the web the website [147:43] for me. Good. But let's say that I am [147:45] that I want to search for an article or [147:47] something that that request will need to [147:50] talk to the back end. So vit will see [147:53] the the request that I'm trying to make [147:56] or the browser is trying to make. If it [147:58] sees that it has this prefix / ai, it's [148:02] it's going to send it to the back end. [148:04] So, VIT sits between the browser and [148:08] your front end and backend servers and [148:11] this implementation [148:13] solves the cross origin issues. Awesome. [148:17] Now, we need to go to the to the back [148:18] end. Let's do that quickly. So, we are [148:22] in the backend folder. Let's see what we [148:24] have. So, we need to look at main.py. [148:26] Let's open it with nano. And here we [148:29] need to remove some things. You can see [148:31] that here we have a coarse middleware. [148:34] This is what we needed before using the [148:37] proxy feature that vit uh gives us. So [148:41] we need to remove this class. And now [148:44] let's go down a little bit because here [148:47] as you can see we used that middleware [148:49] here because we we allowed the following [148:53] origins. So this block need we need to [148:56] remove it. And in order to do that, hit [149:00] alt a go down and hit ctrl k. That will [149:04] remove the those multiple those lines. [149:08] Now we need to yeah save this and go [149:12] back. Okay. So now the we don't have [149:16] course issues. everything will the front [149:18] end will be able to communicate with the [149:20] back end because as as you as you saw we [149:23] used uh the proxy feature that v comes [149:27] with. So after making this change the [149:29] back end we don't need to to do anything [149:31] in the back end but the front end must [149:33] be uh built again. So we need to build [149:36] the front end. So go back to the front [149:38] end folder and run pnpm build. Why we [149:42] need to do that? because the build [149:44] process would regenerate the the HTML, [149:47] CSS and JavaScript files and it will [149:49] bake in the changes in those generated [149:52] files. But the back end doesn't need [149:53] that because it doesn't have a uh build [149:57] step. So just run pmppm build and let's [150:00] wait for the build process to finish. [150:03] Awesome. The build finished. Now the [150:06] first step in this video uh is done. We [150:10] synced the back end and front end URLs. [150:13] And now we can move to using EngineX. [150:16] But why do we need enginex? EngineX, you [150:21] can think of it as a front door for your [150:24] server. When a user types your domain [150:27] name into their browser, the request [150:29] will hit X first before loading HTML [150:33] files or before talking to the back end. [150:36] So engine X is going to intercept that [150:38] request and it's going to act as a [150:41] reverse proxy meaning it will decide [150:45] what to do with that request. Let me [150:47] show you another diagram that I created [150:49] to explain this in detail. I am back and [150:53] let's let's go to the next slide. Here [150:55] is the diagram that I created in order [150:58] to explain why we need engine X. As you [151:02] can see, it's similar to the diagram [151:04] that I showed you that explains vit [151:07] proxy. Here we start with the user with [151:10] a browser. We send a request before [151:14] talking to the front end or back end. We [151:18] in EngineX will intercept that request. [151:21] If it sees that the path is slappi [151:26] something, engineext will talk to the [151:29] Unix socket and that Unix socket will [151:32] send the request to J Unicorn and J [151:35] Unicorn will send it to the UVORN worker [151:39] which runs our back end. You can see [151:41] that this is the part that we talked [151:43] about in the previous sub chapter. Let's [151:46] go back here. When we talked about SSH [151:50] tunneling, we mentioned that the front [151:52] end is running here and the back end is [151:54] running on the server and we explained [151:57] how that works. We have the supervisor, [152:01] we have J unicorn and our workers and [152:04] here we mentioned that we created a Unix [152:07] socket file and I told you that I will [152:10] explain to you where that will be used [152:13] and in this video we are using that to [152:15] communicate with EngineX. Let's go back. [152:18] You can see that we have the unicorn. [152:19] Here I simplified the diagram. I am [152:22] showing just one worker, but you get the [152:24] idea. You have your Unix socket. It is [152:27] used in order to allow EngineX and J [152:31] Unicorn to communicate between each [152:32] other. You can see that here in the [152:34] description I I said it's a two-way pipe [152:38] connecting EngineX and J Unicorn. Uh so [152:41] let's say that the user is searching for [152:44] an article. Let's say that the endpoint [152:46] is / API/ articles. That request will be [152:50] intercepted by EngineX. It sees that it [152:53] has the / API prefix. It's going to send [152:56] that to Unicorn, but it's going to do it [152:59] through the Unix socket file because [153:02] here we are inside the VM. And in order [153:04] for two programs to communicate with [153:06] each other, we used Unix sockets files. [153:09] So now the J J unicorn receives a [153:13] request to the / API/ articles endpoint. [153:17] It's going to give that to the fast API [153:19] endpoint. If we have an endpoint, great. [153:23] The code will run. We'll we'll try to [153:26] fetch articles. Now we will send that [153:28] response to engine X. So we will move uh [153:32] the response will go to G unicorn. We'll [153:35] will go through the socket and back to [153:37] EngineX. engineext sees the response. [153:40] It's going to send it to the browser and [153:42] yeah, you will see articles. So that's [153:44] why here we use unique Unix sockets and [153:48] you can see on the other side we have [153:50] just the static files and if we if if we [153:53] have a request to an end points that [153:55] doesn't start with / API we will go here [153:58] we will fetch the static files which are [154:00] HTML, CSS and JavaScript and the browser [154:03] can display that. So that's why we need [154:05] enginex and as you can see it works like [154:08] uh vit but this runs on production. You [154:11] use vit locally when you develop your [154:14] application but on the virtual machine [154:17] or on the digital ocean droplet we use [154:20] engine x. It's the same concept but one [154:23] runs on your laptop and the other one [154:25] runs on the virtual machine. I hope that [154:28] my explanation was clear. Now let's [154:31] install enginex. We will do it this way. [154:34] We are we are on Ubuntu. So we can [154:36] install this directly from the [154:38] repository. I will provide my password. [154:41] And let's wait for a few seconds. [154:44] Awesome. We installed EngineX. You need [154:47] to understand that EngineX uses [154:49] configuration files to know how to route [154:52] traffic. Let's create a new [154:54] configuration file for your website in [154:58] the sites available directory. Now let's [155:01] do this. pseudo nano etc engine x and we [155:06] have a directory called sites available [155:09] and here you put your project name let's [155:12] call it my website maybe.com uh com we [155:18] are going to build this configuration [155:21] block by block so that you can [155:22] understand everything let's start with [155:24] the upstream block as you can see I just [155:27] copy copied this from GitHub and you [155:31] just I we just need to change this [155:34] placeholder. I will use the same name [155:37] that I used for the configuration file. [155:40] And if you are wondering where to get [155:43] the code or yeah the the things that I [155:46] copy paste, you will find them on this [155:48] GitHub repository. Here it is. So if you [155:51] go to full stack deployment handbook, [155:54] you will see that here I am documenting [155:56] everything. Click on the book folder you [156:00] have and here you have the chapters. So [156:02] far we completed chapter one. Now we are [156:04] in chapter number two. So if you click [156:08] on the uh the second markdown file let's [156:11] search for upstream. [156:13] You can see that here is the upstream [156:15] block. I you can copy it from here and [156:18] paste it in nano and uh go from there. [156:22] So so that you can uh go quickly. So [156:24] don't worry about the code. Everything [156:25] will be shared in the video description. [156:28] First, as I said, start by changing that [156:30] placeholder here. I use my website com, [156:33] but you can use any other name. So, this [156:37] acts as a variable that we can reuse in [156:40] other blocks. Instead of typing every [156:42] single time Unix, web app etc., we will [156:46] just use this variable to make the code [156:49] clean. Here you can see that we are so [156:52] here this is a variable. Here we have [156:54] fail timeout is equal to zero. This is [156:58] an important setting because it tells [157:00] engineext to never mark the back end as [157:03] down even if it fails to respond. So uh [157:06] if you have a lot of traffic and your [157:09] back end is starting to respond slowly. [157:11] Sometimes the engineext will drop the [157:14] request if it if it doesn't respond [157:16] quickly. But here we tell engineext [157:19] don't worry uh even if the back end is [157:21] down don't fail. Okay, now let's add a [157:25] different block. It's called the server [157:28] block. This tells EngineX to listen for [157:32] incoming web traffic on port 80, which [157:35] is standard HTTP, and to respond when [157:38] someone asks for your specific domain. [157:41] Right now, we haven't created a domain [157:44] name, but that's not a problem. We will [157:48] use the droplet IP here in the server [157:51] name. But later this will be changed [157:54] from the droplet IP to the domain name. [157:58] So let's do this. If you don't know how [158:00] to get your droplet IP, we have [158:02] different ways of getting this but you [158:04] can go to digital ocean. Make sure to [158:07] log in here in the projects. Make sure [158:10] to select the project where you created [158:11] the droplet. And here as you can see we [158:14] have the IP address. So copy it and [158:17] let's go back to nano and let's paste it [158:20] here. I will remove this and replace it [158:24] with the droplets IP. You can see that [158:26] we also have some log files. And here we [158:29] need to change this. Uh these are also [158:32] placeholders. [158:34] And here I will use again let me copy [158:36] this my website com and let's place it [158:40] here. Let's also do it here. This is [158:43] helpful because if something breaks, we [158:45] can check the error logs to understand [158:48] why that happened. Here I have a comment [158:52] that says we will add the location block [158:54] some other blocks here next. But as as [158:57] you can see we are doing this step by [158:59] step to understand every every line that [159:02] we are writing. Okay. Now I will remove [159:04] this comment again. I will use Ctrl K to [159:07] do that. And let me place the location [159:10] block. Remember here we have slash which [159:14] means that we want to serve the front [159:16] end. If I open the browser and go to [159:18] this IP address and I add a slash at the [159:21] end, engineext will catch that and will [159:25] serve my front end to the user. Now [159:30] let's try to understand what we have [159:31] inside here. We have root. This tells [159:35] engineext where to look for files. In [159:37] this case, it points to the dist [159:40] directory because when you run pmppm [159:43] build, the files that are generated are [159:46] going to be inside the disc folder. Try [159:49] files is important because here we have [159:52] a single page application. It tells [159:55] engineext to find the exact file the [159:59] user asked for and that file is going to [160:02] be here dollar uri. If it is not there, [160:05] it's going to look for a directory. So [160:08] if it this is a directory, it's going to [160:11] look inside it to find the file. If it [160:13] can't find the file, it's going to use [160:16] index.html [160:17] as a fallback. Let me explain this. So [160:20] here I have my personal website. I can [160:23] open it. You can see that here we have [160:25] slash at the end which means that we are [160:27] serving the front end files. If I want [160:30] to open something else like maybe my [160:34] resume, I can do it. So I can do /resume [160:38] and you can see we get it. But how does [160:41] this work? If I go back to the machine [160:44] or to the VM, let's save this file. [160:47] Let's hit Ctrl X. Let's look at the this [160:51] folder. What do we have here? We can see [160:53] that I have a lot of files. So we have [160:57] some images. We have robot txt. Let's [161:00] let's also try to get robots txt. robots [161:02] txt. You can see that I am able to see [161:05] this file. Why? Because engine x knows [161:08] where to find those because of this. Uh [161:12] let's go back to nano because of this [161:15] location block. So try files will uh URI [161:19] will be robots txt [161:22] and uh we know that we should look [161:24] inside the disc folder. If we find it [161:26] good, we will show it. But if we can't [161:28] find it, we will show index.html. [161:31] Okay. And uh yeah, index html is also [161:35] should be here. Uh you can see here here [161:38] it is. So that's that's how that's uh [161:41] the reason why we added this location [161:43] block. And this will serve the your [161:45] front end. We will do another one for [161:48] the back end. And it's going to look for [161:50] / API instead. I will copy it and I'll [161:53] paste it here. You can see that it is [161:56] similar. we will use the location block [161:58] and instead we will look for / [162:02] okay now we have um a lot of directives [162:05] the these are called directives so proxy [162:08] pass is a directive and here we have we [162:11] need to change this placeholder from [162:13] your project name to this string so [162:17] proxy pass will the request over to your [162:22] J unicorn socket that's what I explained [162:25] in the slides we are here. So we use the [162:28] unique socket to allow EngineX to [162:32] forward the request to J unicorn and [162:35] this is what we are doing here in this [162:37] directive. The proxy set header we have [162:41] two of them are crucial here because [162:43] they take information about the real [162:46] user who is trying to access your [162:48] website because if we don't do this we [162:51] will all we will send the IP address of [162:54] your virtual machine and we don't want [162:56] to do that and finally we have proxy [162:59] redirect of tells engineext to trust the [163:02] redirect URLs exactly as uh your backend [163:07] sends them. So if your back end tells [163:09] EngineX to redirect, EngineX will trust [163:11] it. It's not going to do uh something [163:13] else. Okay, great. So now we have the we [163:16] we have two two locations, one for the [163:18] front end and one for the back end. So [163:21] we can save the file and let's exit. We [163:25] have two options to test this [163:26] configuration. We can do it live here on [163:29] the server. But if you aren't unsure or [163:32] if you already have a working website [163:34] and you want to introduce changes to [163:36] your configuration, if you do if you [163:38] have a mistake or if the configuration [163:41] is false, that will cause downtime and [163:44] we don't want that. There is another [163:45] option which is to use an a website [163:50] called EngineX Playground. EngineX [163:52] playground is a website that allows you [163:55] to simulate your routing rules safely in [163:58] your browser without uh having to do [164:00] this on your production server. You can [164:03] see that here we have we put the [164:04] engineext configuration. Uh here we have [164:07] events HTTP. We as you saw we didn't [164:10] write these. I'll explain to you why. [164:12] But we need to have we need to add them [164:15] for the configuration to work. the the [164:17] the blocks that we created are these. We [164:20] created server location. We created two [164:23] of them. Here we are listening on 80 [164:25] because we we used uh HTTP. Here we are [164:29] serving the front end and here we are [164:30] serving the back end. This playground [164:33] has a built-in testing back end running [164:36] on port 7777 [164:38] and we can use this as our back end. [164:40] Here you write your curl commands to [164:44] yeah send HTTP requests. We'll start [164:47] with local host. Make sure to use local [164:49] host. You can't use your own domain [164:51] because this is a playground and we are [164:54] testing things internally. To test the [164:57] cur command you hit run. And as you can [164:59] see the output says serving view.js JS [165:03] files which is correct because here we [165:05] have slash and the this location block [165:08] will catch that and it will in this case [165:11] return this string. Uh but let's add uh [165:14] maybe something else. So let's add API [165:18] v2. Will this work? Let's see. Let's hit [165:22] run. And that works because here we have [165:25] / API and that catches this. And here [165:28] you can see that we have u yeah this is [165:31] a response from the back end. Uh let's [165:33] do something else maybe let's do API [165:39] v1 or slash articles. Uh let's remove [165:42] this. Let's send the request and it [165:45] works. So this block will catch this. [165:47] But maybe let's do something else like [165:50] maybe resume or something. Yeah that [165:53] will be catched by this block. So [165:55] everything is working. And if you add [165:58] complex um I don't know if you work on [166:00] complex configurations this playground [166:03] will be very useful. So we know that our [166:06] configuration works. Now we can close [166:08] this go back to the VM and I want to [166:13] tell you why in our configuration we [166:16] don't have the events block and HTTP. [166:20] Okay, let's close the file. And here on [166:23] Ubuntu, we use a modeler system for [166:26] EngineX. There is a master configuration [166:29] file located here. Uh let's look at it. [166:32] Let's do cat etc. EngineX enginex [166:38] doconfig. Hit enter. So this is uh this [166:42] is the master configuration file. Let's [166:44] scroll to the top. You can see that we [166:46] have events. [166:48] We it's already here and we have the [166:50] HTTP block and if we scroll down uh we [166:54] include some things and what do we [166:55] include? If you look at this path it's [166:57] etc engineext sites enabled here we have [167:00] an asterisk to load every configuration [167:03] inside that directory and if you [167:06] remember the configuration that we [167:08] created is inside this folder. So that [167:11] means that this master file will load [167:14] our configuration. That's why we didn't [167:17] we don't need to add events or HTTP. It [167:20] is it's already there. And one thing [167:22] that you need to understand is that the [167:24] engineext will run as this user. It's [167:27] not going to run as my user or the user [167:30] you created. No, it's going it has its [167:32] own user. It's going to use it. So, it's [167:34] ww [167:36] data. Now, we need to enable this site. [167:40] Um, let's see. So, we have pseudo nano. [167:43] Okay, good. But I want to use uh ls [167:47] instead of nano. So let's see what we [167:50] have inside the engine x. You can see [167:52] that we have two folders. We have sites [167:55] available and sites enabled. So sites [167:58] available uh we define the [168:00] configuration. So just so that engineext [168:03] knows that there is a maybe a [168:05] configuration that's that that it needs [168:07] to take into consideration. But for it [168:11] to work we need to put it in enabled. [168:13] And here we are not going to copy paste [168:15] it because that's that's not [168:18] [clears throat] a good bracket because [168:19] you would need to maintain both. Instead [168:21] we are going to use a symbolic link. Uh [168:24] this is uh just a shortcut that will put [168:27] that file here and if we want we can [168:30] just change the configuration inside [168:33] sites available and that change will uh [168:35] take effect in the other directory. So [168:38] we we'll have it in two places but we [168:40] can change it once and it will take [168:42] effect in uh other places. Here is how [168:45] we create a symbolic link. We use ln [168:49] dash s here we need to change this. Uh [168:54] let's see I need to put my configuration [168:56] my name the name here. Uh let's go here [169:00] and let's do it in size available [169:02] because I can copy it easily. Let's copy [169:06] this and let's paste it inside sites [169:09] enabled. So we are creating a symbolic [169:12] link from here to this directory. Now [169:16] hit enter. Great. Let's look at let me [169:20] copy this uh or let's copy the other [169:22] one. Let's do etc. And here I will add [169:26] ls-la. [169:28] And as you can see here it says l which [169:30] confirms that is a symbolic link. As you [169:32] can see we have this it is pointing at [169:35] this file which means that if this file [169:37] change changes this uh file will also [169:40] change because it is a basically a [169:43] shortcut. We are almost ready. So now we [169:45] need to make sure that the configuration [169:47] is correct. So run this command and what [169:50] you should see is this. It should say it [169:53] should say that the syntax is okay and [169:56] the the test was successful. If you see [169:59] an error or something, please fix it [170:01] because if you use a configuration that [170:04] is not correct, it's the your websites [170:09] will not work or people will not be able [170:11] to access your website. Awesome. So now [170:14] let's reload X so that it it can read [170:17] the configuration. And now we cannot [170:20] access the website. So I can take my IP [170:24] address. Let's copy it. Uh let's go here [170:27] and let's paste it. You can see that [170:29] it's not going to work because we have [170:32] the firewall. We need to allow traffic [170:34] through port 80. And in order to do [170:37] that, we have we can do this. We can [170:40] allow EngineX. You we use sudo here. We [170:43] use the uncomplicated firewall and we [170:46] tell it to allow EngineX. Let's do that. [170:49] Okay. And now let's verify the status. [170:51] Sudo UFW [170:54] status. Uh before as you saw as you can [170:57] see we we had just open SSH so that we [171:00] can SSH to the machine but now we also [171:03] have enginex which means that we can [171:05] talk to port 80. Now if I go back if I [171:08] go to the website as you can see it's [171:10] already worked. Before it it it wasn't [171:13] uh working. It couldn't find the the [171:16] machine or the the firewall was blocking [171:18] it. But now as you can see I can refresh [171:21] the page and I uh I see the front end. [171:24] You can see that the website is not [171:25] secure because we are on HTTP. We will [171:28] solve this in the upcoming chapters and [171:32] you as you can see we have some errors [171:34] because we don't have articles yet in [171:37] the brows in the application but the [171:39] front end is running and the back end is [171:41] also running. So if I click on blogs or [171:45] courses you can see we cannot we cannot [171:47] get them because they are not here. And [171:50] in my web application I am using mill [171:53] search and we will talk about that in [171:55] the next chapter. But we also have a [171:58] problem here. So uh let's see. So if I [172:02] open the console, let's zoom in a little [172:05] bit. You can see that I am not able to [172:09] make a request to the API endpoint. [172:13] Interesting. So let's go to the VM and [172:17] let's understand why is this happening. [172:20] Remember we added log folders. So here [172:25] let's do this. You can see that here we [172:27] have error logs for our website. Let's [172:30] take this and let's put it here and [172:33] let's look at what we have here. We [172:35] don't have anything. Okay, that's [172:37] interesting. And let's look at access. [172:39] So here access log that's me trying to [172:42] access the website. So you can see that [172:43] I made a post request to these [172:45] endpoints. So here we have tracking and [172:48] here we have search and yeah and I was [172:51] using uh Mozilla I'm using Firefox. So [172:55] yeah that's uh what you see in the [172:58] access logs but we don't see any errors. [173:02] Let's look at the supervisor logs. For [173:04] that we need to go to back end. Let's [173:08] see. Yes, we have logs. Let's go inside. [173:11] What do we have here? We have [173:12] supervisor.log. [173:14] So let's let's do tail supervisor.log. [173:19] And as you can see we see the problem. [173:21] So here it says that uh failed to [173:24] establish a new connection to me search. [173:28] So here it is me search communication [173:30] error. That's what that's why uh what I [173:32] said but we will fix this in the [173:34] upcoming chapter. Let's make sure that [173:38] we can also reach the back end. So here [173:41] the back end starts at API slash and let [173:45] me do health because we have an endpoint [173:47] that tells us if the back end is alive [173:49] or not. So as you can see we are [173:51] receiving okay which means that this is [173:53] working. Uh we can confirm this in uh [173:57] the main file we have this endpoint [174:00] API/halth [174:01] and yeah it means it is working. If you [174:05] followed the previous sub chapters, you [174:09] shouldn't get a bad gateway error. But [174:11] if you do, that might happen because of [174:13] the engineext user doesn't have the full [174:16] permissions to to access the Unix socket [174:19] file. Uh remember the Unix socket is [174:22] here. You can see that it is called [174:24] junicorn.sock. [174:27] And if we try to let's look at the [174:29] permission. So if we do ls-la [174:32] juniors [174:34] sock, here is what we have. Uh I want to [174:37] correct a mistake that I made in the [174:39] previous video. So here the first three [174:43] characters are for the current user. So [174:46] the current user is IMAD. If you are [174:49] connected as root then it's going to [174:51] apply to root. The second tree are for [174:54] the group and the final three are for [174:57] other users. So the mod here is 777. [175:02] Why? Because here uh we have three of [175:05] them. So every person or every user can [175:08] read, write or execute this file. This [175:12] is not good. We need to make sure to [175:16] restrict this. This is why the we didn't [175:19] get bad gateway because everything is [175:22] open. Everyone can uh access read this [175:24] file. But we need to change this or [175:28] maybe if you have a different mode then [175:31] you might get a bad a bad gateway. Let's [175:33] try to do this. So uh I'll change this [175:37] intentionally just to show you how to [175:39] debug this problem. Let's use ch own and [175:43] let's change the ownership to 700 [175:48] and let's point it to let's do it for j [175:50] unicorn.stock. What does 700 mean? So [175:54] this is in binary and let's take these [175:58] three. The first character is one, the [176:00] second character is two and the third [176:03] character is four. If you sum those [176:05] three numbers, it's going to give you [176:06] seven. So if all of them are on, if you [176:10] want to allow read, write and execute, [176:12] you put seven. If you don't want to [176:14] allow anything, you put zero. So here it [176:17] means that for the current user, it can [176:20] do everything anything with this file. [176:23] But for the the group they can't and for [176:26] other users they cannot uh do anything [176:29] with it. If you want to allow people in [176:32] in the group to just read the file maybe [176:36] read and execute in that case we will [176:38] put five because four + 1 is five. Let's [176:42] do this. So ch okay so it said uh [176:46] operation is not permitted. Let's use [176:48] pseudo. Okay that works this time. Let's [176:51] go back to ls-la. [176:53] Sorry, I did ch own. I I should have um [176:57] changed this to ch mod because these are [177:00] called mods. Uh so yeah, this time it [177:02] worked. Uh let's do ls- array. And as [177:05] you can see it changed. Let's verify [177:07] that. So the three characters here for [177:10] other users are not available which [177:12] means they are set to zero. The second [177:15] three are read and execute. That's [177:17] that's uh five. And uh the last three [177:21] for my user is seven. So I can do [177:23] whatever I want with this file, but [177:25] other users cannot. So this is very [177:27] important. And remember you engine x is [177:32] not inside this group. It's not my user. [177:35] It's ww- data. Okay. Now let's try to do [177:40] this. Uh, as you can see, we get the 502 [177:44] bad gateway error because EngineX is [177:47] trying its best to send this request [177:50] that I am trying to make to API/halth [177:54] to J Unicorn. Let's go back to the [177:56] slides. It received / API/halth. It [178:00] tries to send it to J Unicorn, but the [178:02] Unix sockets, we changed the mode now to [178:06] not allow EngineX to do anything with [178:09] it. So that's why it was blocked and uh [178:12] if it is blocked then it's going to [178:14] display this bad gateway error to the [178:16] user. Now how do we solve this? So we [178:19] can add engine X to my group that will [178:22] solve the issue. If you want you can set [178:24] it to 777 but that's uh yeah that's not [178:28] secure. Any user can do whatever they [178:31] want with the file. Let's add uh ww data [178:34] to my uh group. And let's let's also uh [178:38] let's look at the error logs. Uh I want [178:41] to see what we have there. Uh as you can [178:44] see we also get uh something in the [178:46] error logs. Great. So this confirms as [178:49] as you can see it says critical [178:51] permission denied. So it it tried to [178:53] connect to the socket but it failed [178:55] because it doesn't have the full [178:57] permissions. Awesome. So this make uh [178:59] this confirms that changing the mod will [179:02] either allow engine X to be able to [179:06] connect to the socket or not. So let's [179:09] fix this. We have different ways but I [179:11] will do it in this way. So I will just [179:14] use this command sudo user mod- a g and [179:20] here I have this you put here put your [179:23] user my user is emad I will put it here [179:25] and I will add ww- data to my user now [179:30] we have another command called get end [179:33] group and here put your user and as you [179:36] can see you should be able to see ww- [179:39] data awesome so this confirms that it [179:41] works Now let's run uh let's do that ls- [179:46] array and ww data is part of my group so [179:50] it is a uh let's see so we have this [179:53] tree so it's able to read and execute [179:57] okay let's go back to the browser let's [179:59] refresh uh it is still not working maybe [180:03] let's try to reload engine x so here hit [180:06] ctrr and start typing reload it's going [180:08] to find it in the history [180:12] Let's reload. Okay, we still we are not [180:15] able to connect to the socket. I want to [180:17] go back to uh ls-la [180:21] and let's do it on J unicorn. And here [180:23] we have a problem because before I ran [180:27] ch mode and that changed the yeah the [180:30] user to 750 and that's not correct. So [180:33] we need to fix this. So let's do uh ch [180:38] own. I want to own this file, not 750. [180:41] 750 is not correct. So that was a [180:43] mistake that I did because instead of [180:45] typing chod, I typed ch own. And here, [180:49] let's do it on genicorn. [180:52] Let's run ls-la. Okay, now that's fixed. [180:55] Okay, let's go back again to the [180:57] website. Okay, it's still doesn't work. [181:00] And maybe because we also need write [181:03] permissions. So that's how you debug [181:04] this. Um, let's do it. So let's go back [181:08] to here and let's change this to 770. [181:13] Let's look at ls-la. Let's go back to [181:15] the web application. Okay, now it [181:18] worked. We need also write permission [181:21] because we need to write to the Unix [181:22] socket file. That makes sense. We write [181:25] data to this file so that G unicorn can [181:27] read it. And again, G Unicorn will write [181:30] data to Unix file so that engineext can [181:32] read it. Okay, great. So that works. And [181:35] as you can see, the other users will not [181:37] be able to reach this file. If you [181:40] followed uh the same steps as I did, you [181:43] should be uh able to get this to work. [181:45] Awesome. So, we have finished with the [181:49] first part of this video. Now, we will [181:52] talk about security headers. We saw that [181:56] we were all already secure because we [181:58] have the firewall. it was blocking port [182:02] 80, but now we allowed port 80 so that [182:05] we can access the website. But security [182:07] doesn't stop at the firewall. Browsers [182:10] have built-in security features, but [182:13] they only activate them if your server [182:15] tells them to. And if you want to read [182:17] more about this, you can go to HTTP [182:20] security headers, and you will find this [182:23] documented here on the developer Mozilla [182:27] website. And as you can see, they have a [182:29] lot of headers. They will explain them [182:31] to you in detail, but I will show you [182:34] because we have a lot of uh headers. [182:36] I'll just show you the main ones that we [182:38] need to have. And there are also [182:40] websites like security headers.com and [182:45] the http observatory from Mozilla that [182:49] will test the security headers to see if [182:52] they are uh on your website or not. If [182:55] they aren't, you will get a low score. [182:58] But if they they find them, then they [183:00] will tell you that your website is [183:01] secure. Same thing with security [183:04] headers. You just need to paste here [183:06] your domain name or uh in our case, we [183:09] don't have a domain name. So we could [183:10] try to put uh just the IP address. Uh [183:15] let's try this. Let's try it here. I'll [183:18] try it on HTTP Observatory. Ah, so it [183:20] says that it cannot scan IP addresses. [183:23] Okay. So that means that we need to do [183:25] it with a domain name. Okay. But here [183:29] let me actually take the full the full [183:31] link. Yeah, it doesn't work. So with [183:33] HTTP Observer 3, we need the full [183:36] domain. But I can uh use my own website. [183:39] Let's try to scan it. And as you can [183:42] see, I get A+, which is great. It's even [183:45] higher than the 100. So uh the security [183:48] here is great. But let's go back to [183:51] security headers. This is the websites [183:53] that you are that we are working with. [183:55] We have created the new configuration, [183:57] but you can see that it lacks a lot of [183:58] headers here. The great thing is that [184:00] they show you the main ones. They have [184:02] the list of missing headers. They tell [184:04] you why you need them. And here F. So F [184:08] is the lowest score that you can get, [184:11] but you can climb uh from F to A+. And [184:16] here what I like about HTTP observatory [184:18] is that if you click on scan history [184:21] they show you the progression. So in the [184:23] beginning I got I got the because I [184:26] wasn't aware of these headers or I [184:28] didn't add them in the beginning but [184:30] after that I started to add them [184:32] gradually to make uh my website secure [184:36] and now I have the highest score which [184:38] uh yeah which which gives me the A+ [184:40] grade. We will do the same thing for [184:43] this website that we created now. Yeah, [184:45] let's start working with these simple [184:46] ones because the the complex one is CSP [184:50] content security policy. That one we [184:52] will leave it at the end because it [184:53] requires a lot of iterations to get it [184:55] right. Now let's go back again. Hit [184:58] CtrlR. Let's do pseudo nano. Okay, we [185:01] find this is the configuration. We will [185:04] start with strict transport security. [185:07] This one forces the browser to always [185:10] use HTTPS when talking to your website. [185:14] And here it will uh force that for the [185:17] next two years. So this value uh is 2 [185:21] years. You might be asking we don't have [185:23] HTTPS yet. We are just on HTTP. That's [185:26] not that's not a problem because this [185:28] header is ignored by browsers if they [185:31] see that you are uh accessing the [185:34] website over regular HTTP. That's fine. [185:37] But this will be active or it's going to [185:39] be used by browsers when we switch to [185:42] HTTPS. So keep it like that. And here we [185:45] want to also include subdomains. Uh we [185:48] don't have that but if you plan to add [185:50] subdomains then uh we also want to [185:53] include them. Okay. So this is the first [185:55] one. Now we can yeah let's let's save [185:58] the file. Uh hit Ctrl X. Remember we [186:01] need to do sudo engineext-t. [186:04] Okay. It says successful. Now we can [186:07] reload. Reload EngineX. Now let's go [186:10] back to the browser. Let's scan the [186:15] website again. Okay. So yeah, it didn't [186:18] change because that header wasn't in [186:20] this list. That's fine because now we [186:23] will add X frame options, X content type [186:27] options, and another one called cross [186:30] origin cross origin resource policy. [186:33] I'll copy all of them. I'll go back to [186:36] nano. Let's go back to nano and I will [186:39] add them here. So here I have some [186:41] comments if you want them. Uh I will [186:43] just remove them from here but they are [186:44] in the markdown file. Let's start with [186:47] corp. This one allows sharing. So if you [186:50] want to allow other browsers to share [186:52] your resources then you can leave it [186:55] this way. Otherwise you uh you you might [186:59] need to block it. And to do that you [187:01] just need to copy this header. Go back [187:04] to the browser, search for it. You will [187:07] find the Mozilla uh the documentation [187:09] there. And here as you can see you have [187:12] uh you have an explanation and you also [187:14] have the directives that you can that [187:16] you can set. So you have same site, same [187:18] origin or cross origin. In our case, we [187:21] are using cross origin because we want [187:23] to allow other websites to maybe link [187:25] our articles, images or something like [187:27] that. But if you want to block them, you [187:29] put same sites which means that you you [187:32] allow only yourself to yeah reuse uh [187:35] some of your resources. Okay, let's see [187:38] what we have in nano. And as you can [187:40] see, we allow other origins and origin [187:42] means other uh domains. Uh I will remove [187:45] the comment because I don't need it. We [187:47] have here anti- clickjing. Uh we use [187:51] Xframe options for that. If you want to [187:53] read more about clickjing, you can find [187:56] this Wikipedia page which explains it. [187:59] Basically, this is uh how the attack [188:02] works. So, the user is presented with a [188:05] false interface where their input is [188:07] applied to something they cannot see. [188:10] You might visit a website, they will [188:12] show you this page where you see they [188:15] are telling you claim your prize. But [188:17] this page is layered on top of another [188:20] page. So when you click on claim your [188:23] prize, you are not clicking on this [188:25] button. You are clicking on another one [188:27] which is invisible to you. And here they [188:30] say confirm bank payment. So they can do [188:33] uh shady things. They can take your data [188:36] or something like that. So that's a [188:38] problem and we don't want to do that. We [188:39] we want to to prevent this. To do that [188:42] we add xframe options and we tell same [188:46] origin. As you can see we are not using [188:48] cross origin uh or other directive. Uh [188:51] let's look at the other directories [188:53] quickly. Let's search for it here. Um [188:55] Mozilla. [188:57] Okay. So here this is what we have. So [188:59] we have either deny sim origin or allow [189:02] from origin. Awesome. So we have that. [189:04] We also have MIM sniffing. This one [189:08] forces the browser to strictly trust the [189:10] file type declared by the server. [189:12] Sometimes attackers might upload [189:16] malicious files maybe like a script but [189:19] they might tell the browser this is not [189:21] a script this is an image and this is [189:23] called MIM sniffing. MIM means uh the [189:27] the format types. So let's search for [189:29] MIM. If you are not familiar with MIM [189:32] let's search for MIM type and yeah it's [189:35] uh media types. So this uh uh yeah it [189:39] can mean pawns, images, text, scripts [189:43] etc. So attackers will trick your [189:46] browser to think that that script is an [189:48] image and in that script they might load [189:51] some uh malicious code or something like [189:53] that. So we want to deny this. We don't [189:55] want them to sniff the file types. We [189:58] added three extra headers. Now save the [190:01] configuration. Let's do engineext-t. [190:04] Everything is good. Let's reload. Nice. [190:07] So, let's go to the browser. Let's close [190:10] these tabs. We don't need them. Let's [190:11] scan. Great. Now, we have D. So, you can [190:14] see that we have two green check marks, [190:15] but we need to add another ones. Uh, [190:18] these two are simple. So, let's add [190:19] them. And let's leave CSP at the end. [190:23] Okay. And let's add the headers just [190:26] below them. So, here we have this one is [190:29] for privacy. When a user clicks a link [190:32] on your website, uh for example, if I [190:35] link a maybe a Wikipedia article on my [190:39] website, if a user clicks it, the [190:42] browser will send a referral header to [190:45] the new site. So if you want to protect [190:47] user privacy by only sending your your [190:50] root domain name, not the full URL path, [190:54] you can do this. So this uh then wiki [190:57] here if they click on that Wikipedia [190:59] link they will Wikipedia will only see [191:02] my domain name they will not be able to [191:03] see the full link. So yeah we can add [191:06] this and here this one is simple. So if [191:09] your website doesn't require using [191:11] camera microphone dual location payment [191:15] you can disable them because these are [191:16] things that I don't want and that's [191:18] great. So if someone tries to maybe hack [191:21] your website and enable the user's [191:24] camera that's that will not happen [191:26] because we are blocking blocking that [191:29] with the help of the permissions policy [191:31] header. Now save exit the file verify [191:35] that the configuration is correct. Now [191:37] reload let's go to the website and let's [191:40] refresh. Awesome. So it says C and in [191:44] order to jump from C to A+ we need to [191:47] work on CSP which is the final boss. [191:50] This one is complex but I'll show you [191:52] how to do it uh step by step. Okay, [191:55] let's go back to our website. So this is [191:58] the front end. Everything looks good. Uh [192:02] let me open another tab. I want you to [192:04] search for this. So we find it here and [192:07] we also uh you can also see it here. So [192:10] yeah, uh if you want to read more about [192:12] cross sites, that's not what I wanted to [192:14] search for. I wanted to search for [192:16] content security policy CSP. Uh yes, [192:21] here it is. You can see that this [192:23] article is very long because in order in [192:26] order to build it, there isn't a one [192:30] there isn't one CSP that everyone can [192:32] uses. This depends on what you are using [192:35] in your website. That's why you need to [192:37] build it progressively. [192:39] open your uh website. So I will delete [192:43] uh the tab that we don't need. Okay, we [192:46] just want these two. Now we are in [192:48] Chrome because here we have the ability [192:52] to change this CSP very quickly. I'll [192:56] show you how to do this. Now I want you [192:59] to look at the very top of the network [193:02] request list. Here we don't have a [193:04] domain. That's why we have just uh the [193:07] IP address. Right click, show all [193:10] overrides. Uh here I will start fresh. [193:13] Now select let's click on select folder [193:15] for override. So I will create one here [193:20] and select it and let's yeah allow this. [193:24] Now go back to network. Here I want to [193:27] click on override headers. Yes, that [193:30] worked. And here let's add content [193:34] security [193:36] policy because this is the one that we [193:38] want to work with. And here in the [193:40] header value let's start with default [193:44] source and let's do none. And let's not [193:46] forget the semicolon. Now hit enter. Now [193:50] I want you to click on the dot headers [193:53] pile here. It should say star because we [193:57] want to apply it to everything. And [193:59] let's refresh the page. Okay, you can [194:02] see that we cannot view the website. H [194:05] and we have a lot of errors. So we can [194:07] see that here it says refused to load [194:09] the image, refused to load the [194:12] stylesheets, refused to load the script. [194:14] So now that's what we need to do the and [194:17] also make sure if you want you can open [194:19] this uh headers file in a text editor so [194:22] that you can change it here easily. I'm [194:25] opening it in sublime and yeah I see the [194:28] same thing. So I'm applying it to [194:29] everything and I am just playing with [194:32] this header and default source none will [194:35] block everything. So this is the [194:37] strictest CSP and as you can see it [194:39] doesn't work but we start from here and [194:41] we start relaxing it in steps. So you [194:44] can see that there are many things that [194:45] are blocked. There are images that are [194:47] blocked some styles. You can see that [194:49] this image comes from our origin. So [194:52] this one is fine. Uh let's see we here [194:55] we are blocking this side sheets because [194:57] we are it comes from cloud flare so this [195:00] is not our website it's coming from [195:02] another place but we know that we need [195:04] this font it's awesome so we can keep it [195:06] that's not a problem but this will show [195:08] you the things you will decide do you [195:11] want to keep it or block it uh same [195:14] thing for scripts these are all coming [195:16] from our assets folder so yeah these are [195:18] these things can can be kept but uh if [195:21] you if we see that there are other [195:23] strange change things we need to decide [195:25] if we should keep them or not. So let's [195:27] start with the uh with the images. So I [195:30] will come here. You can see that here [195:32] default source I will change it from [195:34] none to self. I saved the file. Now I'll [195:38] go to the browser. Let's refresh. So [195:40] that's solved a lot of problems. Why? [195:43] Because you see that we don't have [195:46] assets. The image was loaded because [195:48] those things were in our disc folder. So [195:52] when we built the website we already had [195:54] those images uh those stylesheets [195:56] everything was there. So when you use [195:59] self it means our origin everything in [196:02] our origin uh should be allowed to load. [196:07] Now we aren't we have uh different [196:10] errors. So for forget about these ones [196:13] uh because as we saw before we don't [196:15] have media search that's why they are [196:16] failing. But focus on these. Now we [196:19] cannot load this font. So this is the [196:21] fonts that we are using in our website. [196:23] Yeah, let's fix this one. Go back here [196:26] and we have another directive. We need [196:29] to come here. Let's add a space and [196:32] let's put style source. This one will [196:34] allow us to load styles. Again, we can [196:37] start with self. And if we want, we can [196:40] add more things here. But let's save [196:42] this. Let's go back to the website. Uh [196:44] or instead, let's let me put it here. [196:46] And let me load Chrome like this. Yeah, [196:49] that's better. And I will move this docs [196:53] this panel uh to the bottom. Now I've [196:56] saved this. I will refresh the page. Uh [196:59] we still have some problems with uh [197:02] Google. So we will copy this link and [197:05] paste it here. We we we need just the [197:08] the origin. We don't need the full URL. [197:10] So we can see that the origin is [197:12] fonts.googleapis.com. [197:14] That's what we need. Now save. Refresh [197:17] the page. And as you can see now we are [197:20] able to load the font. We don't have any [197:22] any error. Now we have another [197:24] stylesheet issue but this time we will [197:27] just take this domain. Take it copy and [197:30] let's add it after Google. Sorry let's [197:34] add it here before the semicolon. Save. [197:37] Refresh the page. Awesome. So now we are [197:40] we are able to load font awesome icons. [197:43] We are able to load the font. And as you [197:46] can see, we don't have those errors, but [197:47] we have different errors. So here we are [197:50] not able to load the font URL because it [197:53] violates the following default source [197:56] self. Okay. So you can see that here we [197:58] don't have uh let me decrease the font [198:00] size. We have default source. We have [198:02] style source but we haven't yet added [198:05] font source. So let's do that. I will [198:08] come after the semicolon. I will add [198:10] this and I will add another semicolon. [198:13] Save this. refresh the page. Okay. So, [198:16] let's open this and let's see what's [198:18] happening. So, again, we have the same [198:21] do origin. Let's copy it. Let's paste it [198:24] after self. Save. Reload. Okay. So, we [198:27] have a different domain that's coming [198:29] from Jstatic. That one is fine. Again, [198:33] if you see that there is a domain or an [198:35] origin that is suspicious, then you feel [198:38] free to block it. But if you think that [198:40] this is something that you are using and [198:44] it comes from a reputable website, it's [198:46] not something suspicious, then add it to [198:48] the list. So now we solved the problem [198:50] with fonts. Now let's look at images. [198:54] Okay. So here we have some X SVG files [198:58] and in order to keep those we can just [199:00] add data. I will come where is it? Do I [199:04] have image source? No, we don't. And [199:06] here they are telling us. So note that [199:08] image source was not explicitly set. So [199:11] again let's do the same thing. Uh I will [199:13] copy font source. I'll come here and [199:16] here I'll need to change it to image [199:18] source. Save. Reload. Okay. That still [199:22] doesn't work because we need to add this [199:25] here. We don't have a website but uh we [199:28] can add where is image. Uh let me make [199:31] this smaller. Yeah, it's here. I can [199:33] come after this and add data. Let's [199:35] refresh the page. And voila. As you can [199:38] see, we were able to fix uh all every [199:42] error. And now these are just coming [199:44] from the back end. But we are not done [199:45] yet. What we what you need to do is to [199:48] test your whole website. Make sure to [199:50] click on every single page. See if the [199:53] errors are uh if you don't see the goal [199:57] is that we we should not see errors [199:59] coming from here. Okay. So as you can [200:02] see, we have other problems. Okay. So we [200:04] have inline style issues. Let's go back [200:08] to sublime text. I want you to find [200:12] style source. Do we have Yes, we have [200:15] style source. So we have self font. [200:18] Google and here. So I will add a space [200:20] at the end because here they say that we [200:23] are using so yeah we are using inline [200:26] styles but that this is blocked by [200:28] default. So we need to add unsafe [200:30] inline. [200:33] Let's save it first and let's reload the [200:35] page. Okay, we have zero errors. Great. [200:38] But this is unsafe. Here's the thing. [200:40] Even though this is unsafe, it's fine [200:43] here in our case because in our Vue.js [200:46] application or maybe you uh you might be [200:49] using React or other frameworks. You you [200:52] are using inline styles or inline [200:54] scripts and those are fine because if [200:56] you don't do those, your application [200:58] won't work. But yeah, by default they [201:00] are blocked but we can add them. Let's [201:02] continue testing the application. Let's [201:04] go here. Okay, that's fine. So, the [201:07] other pages are not working because they [201:09] need to communicate with me search. But [201:11] at least uh every other page is working [201:14] uh correctly. Good. Now, after doing [201:17] this, we can let's let's copy the value. [201:21] I'll hit Ctrl X. I no longer need this. [201:24] So, I'll clear this. Uh I'll just keep [201:26] this. Now, I'll go back to engine X. I [201:30] need to update the configuration. So [201:33] let's do that. I will come down and here [201:38] I'll type add header. It's going to be [201:41] content security policy. I'll add quotes [201:47] and I'll add the other ones. As you can [201:49] see, I have added the CSP. I'll save the [201:53] file. I'll exit. Let's verify that the [201:56] syntax is okay. And it isn't. So this is [201:59] why we need to do this. Let's go back. I [202:02] forgot to add something at the end. [202:04] Yeah, I forgot to add always. And why do [202:08] we do this? Because we want to send this [202:09] header in all the response codes. So if [202:13] we get 200, 2010, [202:16] all of them. So uh this header will be [202:19] there. And we can also add something [202:21] here at the end. I will add another [202:24] semicolon. I will add upgrade insecure [202:27] requests. This one again forces the [202:29] browser to change from HTTP to HTTPS. [202:33] Now we have HTTP. So this doesn't matter [202:36] but later uh after adding SSL [202:39] certificates stuff like that we will be [202:42] able to have HTTPS. Now we are just [202:45] preparing save the file exit. Verify [202:49] okay the syntax is okay this time and [202:52] let's reload X. Okay so this is the [202:54] moment of truth. I will take this. Let's [202:58] put it here. We need to make sure that [202:59] the the website loads correctly. And it [203:02] is it is loading. Let me move this to [203:05] the left. Uh I will clear the logs. I [203:08] also go to the other pages. Yeah, [203:09] everything seems to be working [203:11] perfectly. Okay. So now let's scan [203:14] again. And yes, we got uh a um maybe uh [203:19] on Firefox might have gotten a plus, but [203:22] this is the highest score that we can [203:23] get. And as you can see, everything is [203:24] green. And the yeah these are the this [203:28] is the response that uh we received. So [203:30] as you can see this is what we have [203:31] added here is upgrading secure requests [203:34] and yeah they're also telling you if you [203:36] want to add other requests but this is [203:37] the bare minimum you need to have these [203:40] and as you can see most of them are easy [203:42] but content security policy is the one [203:44] that takes time because it is unique to [203:47] each website. Your application is now [203:50] live. Engine X is serving your view [203:52] front end and routing API calls to your [203:55] fast API back end. However, we haven't [203:59] yet configured search. This is a search [204:02] engine that I am using to s to allow [204:05] users to search for articles. It's [204:08] similar to elastic search uh yeah to [204:10] elastic search but it doesn't consume a [204:12] lot of memory. So we are going to [204:14] configure that in the next chapter. But [204:16] if you don't care about mail search, you [204:19] can skip to the to the other chapter, [204:22] chapter number four where we are going [204:24] to create our domain name, add SSL [204:27] certificates and make our websites [204:29] secure by accessing by making um making [204:33] users access accesses through HTTPS [204:36] instead of HTTP. See you in the next [204:38] chapter. Welcome to module 3. At this [204:42] point, we have a secure server and we [204:45] have deployed the front end and back end [204:49] successfully. There is one missing piece [204:52] if you are using my project which is to [204:55] include a search engine. In my case, I [204:58] am self-hosting M search because it is [205:02] lightweight, doesn't consume a lot of [205:04] memory and it is fast. A quick note on [205:07] this module. If your specific project [205:10] doesn't need a search feature, you can [205:13] technically skip this chapter. However, [205:16] I highly recommend following along [205:18] anyway because the skills you are about [205:21] to learn like creating a system user and [205:24] deploying background services that are [205:27] completely hidden from the internet are [205:30] essential server skills you will use [205:32] anytime you need to host your own [205:35] database or yeah something else. Here is [205:38] our road map. First the search engine [205:42] that we are going to download is me [205:44] search. There are other options like [205:46] elastic search but those consume a lot [205:49] of memory and here we are playing with a [205:51] small virtual machine and we can't just [205:54] host elastic search. So we are using me [205:57] search because it's lightweight it's [206:00] fast and most importantly it doesn't [206:02] consume a lot of memory. Next we are [206:05] going to log this database by creating a [206:09] dedicated system user. I'll explain the [206:13] difference between a regular user and a [206:16] system user in the upcoming slides. Then [206:19] you will configure systemd to run me [206:23] search as a background service. This [206:26] will allow us to perform daily backups. [206:30] Make sure that me search is running even [206:32] if it if the service fails and other [206:35] things. As I said, we are going to [206:37] schedule automatic snapshots. I'll show [206:39] you how to do it. You can specify the [206:41] duration. Do you want to do it daily, [206:43] weekly? It depends on how much your data [206:46] changes. Finally, we are going to use [206:49] SSH tunneling again to access the daily [206:53] search database uh using a dashboard. [206:57] Now let's go back to GitHub. I want to [207:00] show you what we are trying to do. We [207:03] are now in module 3 data and search. You [207:07] will you'll find the commands here. So [207:10] even if you get lost or if you find long [207:12] commands, don't worry about them. Uh you [207:14] will find everything inside this [207:16] markdown document. As I said, if you [207:19] don't have a project and you rely on my [207:22] own inside the backend folder, I have [207:25] everything for you. So even if you don't [207:27] have data that you want to put in me [207:30] search, you can just come here in the [207:32] seed folder and I have created this [207:36] documents list. As you can see, we have [207:39] some documents that we can index uh in [207:42] mill to me search. Okay, so don't worry [207:44] about this. Everything is inside these [207:47] two repositories. Great. Now, open your [207:50] terminal and let's SSH to the machine. I [207:54] will use [207:56] my alias. Uh the alias that I'm using is [207:59] my website. I have successfully logged [208:02] into the machine. Uh let's run ls. Yeah. [208:06] Uh let's go to web app. Okay, so we have [208:08] everything. Great. Let's start by [208:11] installing me search. If you click on [208:15] the curl option, I mean you have a lot [208:17] of options, but we'll go with this one. [208:19] Copy this command. Paste it here inside [208:23] the machine. It shouldn't take a lot of [208:26] time. Yeah. So it worked. Let's see. So [208:28] we have me search. Uh let's see if it is [208:32] executable. [208:33] Yes. If we are the current yeah for my [208:36] current user it is executable. Now we [208:38] need to move this binary into user local [208:43] /bin. Let's do this. So we have mainly [208:46] search here. Let's move it to user local [208:51] bin. I need to provide my password. [208:54] Awesome. Why we do this? Because it [208:56] allows us to run search from any place. [208:59] So I can now just type med search d- [209:02] version and it works. I don't need to go [209:05] to uh say say I downloaded the binary in [209:09] the downloads folder. If I don't move it [209:12] to user local bin, I have to always go [209:14] to downloads in order to interact with [209:16] the binary. But now when you put it in [209:20] uh user local bin, it allows you to run [209:23] it from any place like we did. Now if I [209:26] type ls as you can see we don't have the [209:28] mely search binary but I was able to run [209:32] the me search uh version command without [209:35] any error. So now we have me search. [209:38] Let's create a system user. Before we do [209:42] that I would like to come to go back to [209:43] the slides to explain the difference [209:45] between a regular and a system user. I [209:49] have created this table to explain to [209:52] you the difference between these two [209:54] types of users. Here are the features [209:56] and here are the types of users. So what [209:58] is the purpose of a standard user when [210:01] we started this course? We created a [210:04] user. We are not using the root user but [210:07] we created another user that we can use [210:09] to do maintenance, install software. [210:12] Yeah, stuff like that. We are using a [210:14] standard user to administer to do [210:17] maintenance. So this is the purpose of [210:19] this user. But a system user is used to [210:24] run background services only. So if I [210:27] want to run M search uh in a background [210:30] service, I need to run to use a system [210:32] user for that. Can we use the shell in a [210:36] standard user? Yes. As you saw, I was [210:38] able to SSH to the machine and uh type [210:41] commands. But if I try to do the same [210:43] with a system user, I won't be able to [210:45] because we deactivate that. This means I [210:48] can't login as a system user and start [210:51] typing commands. So this is the same [210:53] thing that I that I mentioned here. So [210:55] login is possible with a standard user [210:57] but it isn't allowed with a system user [211:00] and generally the standard users have [211:03] their uh home folder inside/home [211:07] but system users are yeah put somewhere [211:10] else. So this is the difference between [211:12] them and because M search is a software [211:15] that we want to run as a background [211:18] service we want this specific system [211:21] user to be responsible of handling that. [211:25] Okay. So let's create uh a system user [211:28] and we can call it med search because it [211:30] uh its job is to uh uh run this software [211:35] for us. And in order to do that, we will [211:37] use the user add command. Here I will uh [211:42] specify the directory where we want to [211:45] put this user. As I said, we are not [211:48] going to put it in home. We will put it [211:50] here. var lib mail search. We will [211:53] disable login. So we won't give it shell [211:56] access. And finally, we will call it [212:00] search. Mele search. I hope I didn't [212:02] make a mistake. Me search. Me search. [212:05] Yes. So now let's hit enter and it [212:09] worked. Now let's try to look at this [212:13] directory. As you can see the I don't [212:15] have the permissions. Why? Because I am [212:17] using my standard user aad and this [212:21] folder or yeah this directory is owned [212:25] by me search the system user we created [212:28] and we can verify this. And if we do [212:30] sudo as you can see this user mini [212:33] search owns this this folder. So we [212:37] cannot go inside it. But yeah with with [212:40] root we can do that and we can also [212:42] verify it here. As you can see the [212:45] permissions are they are they are [212:47] constrained. They are they are allowing [212:49] just this user to write to this [212:52] directory. Great. So this confirms that [212:54] everything is working. Now we need to [212:56] create a folder inside [213:00] uh that will hold data. We need to [213:03] create the data folder because me search [213:06] needs somewhere to save data. So let's [213:08] do it. Uh let's verify the ownership. [213:12] This is very important. ls-la [213:15] as you can see. Uh let me go back. Okay. [213:18] So data is owned by root and we want it [213:22] to be owned by med search. So let's do [213:24] it. We we have used this each own [213:28] command before. Here we are uh changing [213:31] the user and group from root to M search [213:35] and we want this user to own everything [213:37] inside this directory. That's why we are [213:40] using dash r. So now if I rerun this [213:43] command as you can see now the data [213:46] folder is owned by mini search. Awesome. [213:49] So now we installed the mini search [213:51] binary. We created a system user. Now we [213:55] need to put some data inside. So if you [213:59] already have your own instance, you you [214:02] don't need to use let let me go back to [214:05] GitHub. You don't need to use this JSON [214:08] file. You can just uh export the data [214:12] from your local main search instance and [214:14] use that. If you if you don't have data, [214:18] then you can use this. I'll show both [214:20] options. so that uh everyone can uh [214:24] follow along. Let's start by exporting [214:27] data from a local me search instance. [214:30] For that I will create another terminal [214:32] window. Now I am uh on my local machine. [214:36] I will start the mearch service and I [214:39] will show you how to export data from [214:42] it. I will use T-Max to create two panes [214:46] here. Okay, I moved where I installed me [214:50] search and this is how I am using it. [214:52] Um, okay. So, as you can see it started. [214:55] Let me show everything and here. Yeah, [214:57] it is listening on this port. Maybe [215:00] search by default uses port 7,700. [215:03] Uh, in this other terminal we can talk [215:07] to this server. We can use the terminal [215:10] or if you want you can also use their [215:13] SDKs. They have one for Python, [215:15] JavaScript and other programming [215:17] languages. But for now, to keep things [215:19] simple, we will just use the yeah the [215:23] terminal and we'll just run a curl [215:25] command. And as you can see here, we [215:27] have something that we call dumps. And [215:30] here I need to provide my master key. [215:32] And since we are on a local machine, [215:35] yeah, I am just using their default [215:38] master key. Plus in a production [215:41] environment we will change this into a [215:43] secure key. But for now it's fine. Uh [215:46] when what you are working with this [215:48] locally no one will steal data from you. [215:50] So it's fine. Keep it like this as the [215:52] default. Yeah. So now I can't talk to [215:55] this. But before running the command I [215:57] just want to explain the difference [215:58] between dumps and snapshots. If you [216:02] search for me search dumps snapshots, [216:05] you will find this page where they [216:07] explain the difference between these two [216:10] concepts. A snapshot is an exact copy of [216:14] your database. You use a snapshot to [216:17] create backups. So this is what we are [216:19] going to schedule later. We are going to [216:22] uh allow me search to save a backup of [216:24] the database daily. So that's let's say [216:27] some if uh for some reason something bad [216:30] happened to me search and you lost data [216:34] you can recover by using these [216:36] snapshots. A dump on the other hand is [216:40] just a set of instructions that are used [216:43] to recreate the database from scratch. [216:45] So this is the difference between the [216:47] two and dumps are the safest way to [216:50] migrate data across different machines [216:53] or versions. This is why in uh my local [216:57] terminal I am using the dumps. I'm going [217:00] to create a dump with this curl command. [217:03] So now let's run it. As you can see we [217:05] get a task uid. Uh in m search tasks are [217:09] asynchronous which means that they take [217:12] time to finish but search at least gives [217:15] us a task u so that we can track the [217:18] task and the task that we are performing [217:21] is creating a dump. And if you want to [217:24] follow a task, what you need to do is to [217:27] change here dumps to tasks and you [217:32] provide the U ID and in in our case it's [217:35] 4,94 [217:38] and you yeah keep keep the authorization [217:41] key and hit enter. And here I forgot to [217:44] change post to get. So let's try to do [217:47] that. And as you can see the status is [217:50] succeeded which means that we were able [217:52] to create a dump. That's great. And we [217:55] can also if you want you can also verify [217:57] this in the mini search server. Uh so [218:01] here we started with 20 workers. And [218:04] here as you can see we the server [218:06] received the the task and here it says a [218:09] batch of task was successfully completed [218:12] with one successful task which means [218:14] that everything worked great. Now we [218:16] just need to find this dump. As you can [218:19] see, I have my mini search binary inside [218:22] this folder. If I hit if I try to run [218:25] ls, you can see that we have a dumps [218:28] folder. Let's go there and let's see [218:31] what we have. Let me also remove the [218:33] first pane. We don't need it. As you can [218:34] see, I have created a lot of dumps. [218:37] Let's look at this one. So, we uh as you [218:40] can see, they label them by date. So we [218:42] are in 2026 which means this is the one [218:44] that we created. Today is March 20 which [218:47] is correct. So this is the one that we [218:49] created. Great. Now we need to take this [218:52] dump and move it to the virtual machine. [218:56] As I mentioned you can if you don't want [218:58] to use curl you can use one of their [219:00] SDKs. If if you go to this page I just [219:04] searched for create dump and I got I got [219:07] here. If you click on curl, you can get [219:10] as you can see they have a lot of [219:11] programming languages. Let's select [219:14] Python. And as you can see when you [219:16] create the client, you can just call the [219:18] create them. Uh the response will be a [219:20] task u uh same same as what we got in [219:25] the terminal just by using the curl [219:26] command and after that you can just wait [219:28] for it to finish. In the beginning it's [219:31] going to be incued but after uh a [219:33] successful execution it's going to [219:35] return succeeded. Uh so they talk about [219:37] authorization. So yeah, everything you [219:39] need to know about uh how this works. [219:42] Now let's go back to the terminal. We [219:44] need to move this to the machine. Uh so [219:48] let's use the SCP command. Let's paste [219:51] the name of that dump. I will use my [219:54] alias, my website, and I will put this [219:59] uh temporarily in the temp directory. [220:02] Okay. Okay. Great. So that worked. Uh [220:05] let me copy the name again. I'll go back [220:08] to the virtual machine and let's find [220:11] that if if the dump is uh inside the [220:13] temp deck directory or not. So let's use [220:16] ls temp and let's paste the name of that [220:20] file. And as you can see we have it [220:23] great. Now we need to move this dump. So [220:26] let's uh go up. I will remove ls and [220:30] replace it with mv to move this file [220:34] into var lib mail search. As you can [220:38] see, I cannot do this because I don't [220:42] own that directory. So, let's use sudo. [220:45] Okay, it's it's there. And again, don't [220:48] forget to use ch own. Uh let's search [220:51] for it. Hit ctrlr. [220:54] Start typing ch. We found it. Now hit [220:58] enter. Let's verify that the permissions [221:01] are correct. If we copy this, let's use [221:06] ludo ls-la [221:08] and paste that directory. Let's see. [221:11] Yes. So this dump is owned by mail [221:14] search. We could have created a dumps [221:16] folder, but for now this is temporary [221:19] because we just want to transfer the [221:20] data once and remove this file. We don't [221:23] we don't need it later. That's why we we [221:25] didn't create the dumps folder inside [221:28] the virtual machine. Now let me show you [221:31] how to run me search as a service [221:33] because right now if uh it's not it's [221:37] not running we need to manually do it [221:40] and if a problem happened that cause [221:43] many search to stop we need to SSH back [221:46] to the machine and rerun the command [221:48] again and this is not good. We want the [221:52] system to do this for us like when we [221:55] used supervisor J unicorn to manage the [221:59] back end uh the fast API back end for us [222:02] without doing any work. So this is what [222:05] we are going to do now. We will use the [222:08] mini search system user to run search as [222:12] a service. Here we are going to use [222:15] systemd to create the service. We are [222:17] not we are not going to use supervisor [222:20] like we did with J unicorn because uh [222:25] systemd sits at a lower level. Let me [222:29] explain this. Uh I have a slide that [222:31] explains this. So here we are talking [222:32] about systemd versus supervisor. As you [222:36] can see we have many layers and here I'm [222:38] just showing the operating system level [222:41] and the application level. So operating [222:44] the operating system level is lower and [222:47] here we have the services that are [222:49] running and system D manages these [222:52] services. We are going to create a [222:54] service that we will call mini search [222:56] and the goal here is that we want system [222:58] D to start this service whenever we [223:02] reboot whenever a mini search crashes. [223:05] Uh so yeah systemd will be the manager [223:08] and we use systemd in uh when we work [223:12] with things that are low level at the [223:15] application level for example with our [223:18] backend API we use supervisor for that. [223:21] So yeah this is the difference between [223:23] the systemd and supervisor. Okay now [223:26] let's go back to the terminal and let's [223:29] create this service. If you want to [223:32] learn more about systemd, you can just [223:34] type systemd and they have here their [223:37] website where you can learn more about [223:39] it. It's really awesome and uh it's also [223:42] easy to use. Let's go back to the [223:44] terminal. Now we need to create a secure [223:47] key. Uh if you remember here when [223:51] locally I am just using this dummy [223:54] master key that uh yeah the med search [223:57] team uh shows you uh in their [224:00] documentation but only do this if you [224:03] are working locally because no one will [224:05] steal your data because everything is [224:07] local in production. We need to create a [224:11] secure master key and that this is what [224:14] we are going to do in uh at this stage. [224:17] Okay, let's go back to the BM. Uh, in [224:20] etc., we need to create millarch because [224:23] we don't want to put the master key in [224:26] the service file. We want to secure this [224:29] so that we can just call a variable that [224:32] will contain the master key. And we are [224:35] going to do that here in this search [224:37] directory. And inside we will put uh an [224:41] env file where we are going to create uh [224:45] two environment variables. I'll put them [224:48] here but we will change the values. And [224:50] by default me search will search for any [224:54] environment variable that that uses this [224:56] prefix melee. [224:59] So here the environment is production. [225:01] This is important. And now we need to [225:03] change this placeholder to a secure key. [225:07] You have many options to generate a a [225:10] secure uh key. Uh one of the options is [225:14] to use OpenSSL. So when you run it, it [225:17] gives you this random number that you [225:19] can use. So yeah, just do this. Copy it [225:22] and put it there. I will do this and I [225:24] will come back. Okay, I have created the [225:27] key using the open SSL command. I have [225:33] opened the env file and I have pasted [225:36] that key there. I'm not I am not going [225:38] to show you the key and you should also [225:41] do the same. Keep your keys secret. No [225:44] one should see them. Now we need to make [225:46] sure that the me search user is owning [225:49] again these files because we are going [225:51] to use that user in the service. We know [225:54] how to do this. We are going to use the [225:56] chall command and yeah we want to own [225:59] this file. We also we are also going to [226:02] change the permissions. Uh I think by [226:05] now you know what this what 600 means. [226:07] It means that for the search user we [226:10] wanted to be able to read and write from [226:12] this file but we want to block access to [226:16] any other user. Let's just do ls-la to [226:20] confirm this ls-la. [226:23] As you can see everything is prohibited. [226:25] So we just uh the mail first user will [226:27] be able to read and write. won't be able [226:29] to execute. But uh and also for the the [226:33] group and the other users, they are [226:35] blocked. They they don't have the [226:37] permission to even see read this file. [226:40] This is very important because we want [226:42] to keep things isolated. We don't want [226:45] anyone to touch the NV file or to read [226:48] the secrets from there. Now we are ready [226:50] to create the service file. And as I [226:53] said with systemd it's it's really easy [226:56] and this is how we do this. We are going [226:58] to create a service file. You can name [227:01] it whatever you want but at the end you [227:03] need to add dots service and you put it [227:05] inside the systemd folder. I will paste [227:08] this text and I will explain what it [227:10] does. And again uh we have two options. [227:14] The first one deals with users who have [227:18] data and they extracted it from their [227:20] local uh med search instance like what [227:23] we are doing now. And the other option [227:25] is for users who don't have uh data and [227:29] they just want to use the the JSON file [227:32] that I provided. I'll come back to that [227:35] to that second option. But for now, this [227:37] is what we have in a systemd service. We [227:40] need to provide a description. So here [227:43] um provide something that is that [227:46] explains what this file is doing. So [227:49] here we are going to run the mail search [227:52] engine and here what does after mean? [227:56] This line tells Linux do not attempt to [228:00] start this service until the server's [228:03] networking stack is fully active. This [228:05] sometimes is important for things that [228:07] need internet access. But if you if you [228:11] are running something that doesn't [228:12] require it, you can remove this. This is [228:14] what it means. So if you want your [228:16] service to start maybe after some other [228:18] services you you do that uh here. Now we [228:22] have the service. So the type here is [228:24] set to simple and here is where we put [228:28] the user. As you can see we are allowing [228:31] only me search to run this service for [228:34] us. This is why we were creating all [228:36] these env files. Uh this is why we [228:39] created this working directory and we [228:41] made sure that me search is the own is [228:45] the is the owner of uh these files and [228:48] folders that we created. Uh let's say [228:50] that someone hacks your server. They [228:52] won't be able to look inside for for [228:55] instance the env file. This allows us to [228:58] isolate media search as much as as as [229:01] possible. Okay. Uh so I have this [229:03] comment here. uh we will load those [229:07] environment variables. The working [229:09] directory is here and here we have the [229:11] execute start. This is the command that [229:13] will be executed. As you can see we are [229:15] pointing to the M search binary. We can [229:18] uh so here I'm putting the the full path [229:21] but you saw that we can just use M [229:23] search and it's going to be recognized [229:25] the database path. So here M search will [229:28] store all its data in the data uh [229:31] folder. Here we have dump directory is [229:34] dumps like uh what I have here locally [229:38] and these two lines are what we want or [229:42] this is yeah this is uh what we what you [229:45] will use if you want to import the data [229:48] that you exported locally. So here the [229:51] argument is import dump and here you [229:53] need to specify the the path to that [229:56] dump file and this ignore dump if [229:59] database exists is very important. So if [230:02] you have data in me search that will [230:05] raise an exception because this might [230:07] cause me search to override your data. [230:10] So just for safety it's going to block [230:12] it here. What you say is ignore this. So [230:14] if you have data in me search please [230:16] don't import this. We don't need it. And [230:20] finally uh we have this is very [230:22] important restart on failure. As I said [230:25] if me search failed for some reason no [230:29] problem. We don't need to go to the [230:31] machine and restart it. System D will do [230:34] it for us thanks to this line. As you [230:37] can see the the service is easy. I just [230:40] need to change this placeholder. I need [230:42] to find the name of the dump and put it [230:45] here. So here is the name. I'll copy it [230:48] from here. Let's go back and let's paste [230:51] it. Let's remove this placeholder. Let's [230:54] paste this. Save. Exit. Let's run damon [230:59] reload and enable. So after creating a [231:03] service, we need to enable it. Uh so it [231:06] created a system a sim link. That's [231:09] good. And now we can start it. Okay. Now [231:13] let's go back and let's do status cuz [231:15] status will show us if something bad [231:17] happened. And as you can see, yes, [231:19] something bad happened. So here it says [231:22] failed. This is not a problem. This is [231:24] part of the process. Uh don't worry [231:26] about it. Okay, let me show you how to [231:28] debug this. We can use journal CLL the [231:32] name we put the name of the service and [231:34] here 10 minutes ago. So yeah, it should [231:36] be fine. Uh let me remove this. Okay, so [231:40] it tells me that I am not able to see uh [231:44] any message. Let's add sudo. Okay, now [231:47] that works. So this is what we have [231:49] here. As you can see it tell let's see [231:52] it says failed to execute [231:54] user/loc/bin/aily [231:57] search. Okay so that means it wasn't [232:00] able to run the binary. Okay let's see [232:04] why. Let's type- la user local bin and [232:10] me search. Ah okay. So here the problem [232:13] is that uh we don't have the uh execute [232:17] permission. So execute is given just to [232:19] my user but maybe search is is going to [232:23] be in the others section. It's not going [232:26] to be in my group but it's going to be [232:28] here. And as you can see we don't have [232:29] X. So we need to provide that. Okay. So [232:32] this is the issue. Now let's change this [232:35] and let's use sudo ch mode plus X. Now [232:40] let's go back. Okay. Now now now it [232:43] should be working. Now it should work. [232:44] Uh in order to restart the service we [232:47] use systemctl restart. Let's look at the [232:50] status and voila it is running. So it [232:54] says active running. Awesome. So now [232:57] let's see what we have here. So here it [232:59] gives us some info. It says the instance [233:01] UID. Here it says that a master key has [233:04] been set but it shouldn't show it. Yeah [233:07] it's not showing it. So request search [233:10] won't be authorized unless you provide [233:12] an authentication key. This is the ex [233:14] this is exactly what we have done here [233:16] locally when we when you try to make a [233:18] post request or something you need to [233:21] provide uh a key to be able to [233:24] communicate with the server. So this [233:26] confirms it uh and yeah they have some [233:29] other info uh like documentation some [233:32] stuff like that but what's important is [233:34] what we have here. So here we have [233:35] starting one workers uh locally uh it [233:39] said starting 20 workers. Uh let's [233:41] verify that. Yeah, it says starting 20 [233:44] workers because I have a lot of cores. [233:47] If you we look at the memory, let me [233:50] show you and resources. As you can see, [233:51] I have 20 cores. So that's why locally [233:54] it it's going to use all of them. But we [233:57] are using a poor VM. It doesn't have a [234:00] lot of CPU cores. That's why it's going [234:03] to work with what we have, which is one [234:06] core, and that equals one worker. Uh [234:10] yeah, it's it's working and it is [234:11] listening. If we go here, it's listening [234:14] on port 7,700. [234:16] Awesome. So that means it's working. So [234:19] now just hit Q to exit and let's [234:23] continue. We have talked about option [234:25] one. But now let's deal with the with [234:28] other users who don't have their local M [234:31] search instance. This section is is made [234:34] just for those people. So for the other [234:36] ones who have their own local instance, [234:38] they can skip this section if they want [234:40] or they can stick they you will learn um [234:43] you will learn a lot. First let's verify [234:46] if I have the JSON here. Uh let's look [234:49] at web app. We need to go to back end [234:52] and yeah I have the seed folder. Uh [234:55] let's look at it. And we have [234:57] documents.json. [234:59] Awesome. You can read the readme if you [235:02] want. Let's let me show you the readme. [235:04] And here let's look at and it's it's [235:07] going to tell you what you need to run. [235:09] And here you you are not going to index [235:12] them manually but I have a uh I have [235:15] created a script for that. I called it [235:17] see the mainly search and it is here. It [235:20] is located inside the scripts folder and [235:23] this will uh yeah will index the [235:26] documents for you. Let's go inside back [235:29] end or yeah what's what was command? [235:32] Yeah, let's just take it like this. I [235:34] just need to go inside [235:37] web app. And now I can run this. Okay, [235:41] it didn't work because I need to [235:43] activate the virtual environment. This [235:45] is the command source van bin activate. [235:48] No such file or directory because I need [235:51] to go inside the back end. Uh let's do [235:54] it now. Okay, as you can see, we have [235:56] van which means that we were able to [235:59] activate the environment. Okay, now [236:02] let's go back to this command and let's [236:04] remove back end because we are inside [236:06] that directory and let's run this. Okay, [236:09] let's see. So here this is just a text [236:11] see the mainly search with real data. [236:14] Okay, here is the art the indexes we are [236:16] going to create. Okay, so here we are [236:18] trying to connect to mainly search. Uh [236:20] so here it says index articles already [236:23] exists because I have imported the dump [236:27] successfully connected. Uh but here we [236:30] have an error but it says that we have [236:33] an invalid API key. That's [236:35] understandable because I have created a [236:38] new master key. So let's see what we [236:41] have inside this file. Let's use nano. [236:45] Oh sorry uh I need to keep the full [236:48] path. Let's use nano. Okay. So here we [236:52] are using I have a settings. I have a [236:56] configuration and from that [236:58] configuration I load the API key. So as [237:02] you can see here we have API key is set [237:05] to settings dot mainly search master [237:08] key. This is why. So here I assume that [237:11] we are loading a default master key. Uh [237:15] okay that's why it didn't work. Let's [237:17] see. So let's go inside settings just to [237:19] show you what what we have there. Let's [237:21] go inside core. Let's look at config. [237:25] Yeah, as I told you, we are using the [237:27] default master key. That's why it didn't [237:30] work because because we are using a [237:32] strong master key. And here, let's see. [237:36] So, we have an environment variable. We [237:38] have an environment variable that we can [237:41] open. And there we can put the new [237:44] master key. Or if you want, you can do [237:46] it temporarily. You can just change this [237:49] and the script will work. Let me change [237:51] it inside the env file. And the env file [237:55] should be here. Let's go back to back [237:57] end. Let's use ls-l [238:00] and let's see do we have env. Okay. So [238:03] we just have env.example and from that [238:06] we need to create env. Let's use cp [238:11] env.ample and we will create av file. [238:15] Let's open that file. And as you can see [238:18] this is what we have. The environment is [238:20] set to development. We can change that [238:22] to production. Mainly search URL is [238:25] correct. The index is correct. But I'll [238:28] create um uh another one. Let's call it [238:31] articles too because as you saw from the [238:33] logs, we already have articles. And here [238:36] I need to change this. And remember use [238:38] the same key that we have put inside the [238:43] env uh inside the env file. Uh let me [238:46] show you. So we have this file where [238:50] hopefully you have created a strong key [238:52] and you put it here. I'm not going to [238:54] show you my key but I will just open the [238:56] file again. Copy it and paste it inside [238:59] this env file that we created in the [239:03] backend folder. I am back and I have [239:07] changed the master key inside thisv [239:11] file. Now we can try to run the Python [239:15] script again. Let's do it. So, Python [239:18] scripts seed med search. And as you can [239:21] see this time, we didn't get any error. [239:24] It says this is the index that we are [239:26] going to create. It created it. It [239:29] successfully connected to my search. And [239:31] here, as you can see, documents were [239:33] added successfully. Awesome. And I will [239:36] show you later how to verify this with a [239:39] dashboard. We can do it with the with [239:41] the curl commands or the SDKs, but it's [239:45] going to be very easy with a dashboard. [239:48] Now, I want to do something. Uh let's go [239:51] back to the service file here. We have [239:56] we imported the dump successfully. Now, [239:59] we don't need it. We can just get rid of [240:01] these two lines because I as I said, we [240:03] import the dump only once. We don't [240:05] import it every single time. We made we [240:08] made sure that we have the index the [240:11] documents were there and yeah after uh [240:14] verifying that you can safely clean this [240:16] service by removing those two lines. Now [240:19] save the file exit. We need to tell the [240:23] dam that we need to reload the damon [240:25] because we have we changed the service [240:28] configuration and we also we need to [240:30] restart the service itself. Awesome. [240:33] Remember that we can access the web [240:37] application using our IP. If you don't [240:40] remember the IP address, you can always [240:43] come back to Digital Ocean and find it [240:47] here. I'll copy it. I'll paste it here. [240:50] Let's go to the front end. So it's going [240:53] to be [240:55] sorry I'm not sure why I did that but [240:57] let's just put the uh the IP address [241:00] because we are using engineext and [241:02] engineext will take care of uh routing [241:05] the traffic but what why I want you to [241:08] come back here because last time if you [241:10] remember we weren't able to yeah we [241:14] weren't able to talk to the to get [241:16] articles from the back end because me [241:19] search was not set up Now we have M [241:23] search but we need to connect it to the [241:25] back end in order to get articles cuz [241:28] right now as you can see we we don't [241:30] have anything. Now we just need to [241:32] restart supervisor because it doesn't [241:35] have the new environment variable. This [241:37] is why it's not it's not able to connect [241:39] to the me search server. And to do that [241:43] we are going to use supervisor DTL. If [241:47] you don't remember the project name that [241:49] you used, you can always type status and [241:52] that will give you the pro the yeah the [241:54] name of that project and currently it is [241:57] running and it was running for 10 days [241:59] without any problem. Now we need to [242:02] restart that. So let's use restart and [242:05] let's put the name. You can use restart [242:07] all but if you have many websites on the [242:10] same server [242:11] uh it's not a good idea. So it's always [242:13] great to yeah just restart what you [242:16] want. Okay. So it said stopped now it [242:19] says it is saying started. Let's also [242:22] follow the mini search service logs. [242:25] This is what we have. It is saying [242:27] invade the API key because that's when [242:30] we run supervisor we use an old keys in [242:33] the NV file. But hopefully now it should [242:36] it should work. Uh let's verify this. So [242:39] I am here. I'll refresh the page and [242:42] voila, as you can see, I am able to get [242:44] my articles and if we look at the logs, [242:48] yeah, this is the latest request that we [242:51] got. It's a post request. It's coming [242:54] from Python because I am using the [242:55] Python SDK and the status status code is [242:58] 200 and as you can see media search is [243:00] fast. Uh so it was able to do this in [243:04] less than 100 milliseconds and also the [243:07] benefits of self-hosting misarch is that [243:09] the communication is local because my [243:11] back end is in local host and misarch is [243:14] also uh in local host. If I hosted [243:16] misarch on another server maybe uh that [243:19] will add some latency but here [243:21] everything is inside one box. Awesome. [243:24] Let's also verify other stuff. Let's go [243:26] to the homepage because we should see [243:28] some blogs here. Yeah, that's correct. [243:30] Let's all let's click on maybe this one. [243:34] Yeah, let's click on this one. We have [243:36] the content. We can also add some claps [243:41] just to verify that it the data is [243:43] saved. We also get recommendations which [243:46] is great to see. The number of claps [243:48] should have changed from 0 to five. [243:51] Awesome. It is working. This is really [243:53] great to see. We also have some filters. [243:55] We can sort by popularity. So this sorts [243:59] by the number of views. We can change [244:01] that to ascending. So we have seven 7 11 [244:05] 29. Uh we can change that to um number [244:09] of claps. So these ones these these [244:12] articles don't have any claps in them. [244:15] But this one has the most number of [244:17] claps. It has 15. This one has six five [244:20] etc. We also can filter by year. And we [244:24] have yeah we can filter by tags. And [244:26] here we are using faceted distribution. [244:30] Now we can let's go back to the machine [244:32] and let's stop this because we don't [244:34] need it. And as you can see the the [244:36] request that we were performing were all [244:38] successful. They all returned 200. [244:41] That's all I wanted to share with you in [244:42] this first subchapter in module 3. You [244:47] now have a working uh mini search server [244:51] hosted on your virtual machine. We [244:54] connected it to the back end. We created [244:58] a server, a service that will handle [245:01] managing the mele search service for us. [245:05] And we tested it in the front end and it [245:07] worked wonderfully. In the next sub [245:10] chapter, you will learn how to maintain [245:13] this database. I mentioned that we can [245:16] use the Python SDK or just plain curl [245:20] commands, but it's not enjoyable. I'll [245:23] show you a better way to maintain your [245:26] server by uh connecting it to a uh [245:30] goodlooking dashboard. We are also going [245:34] to set we are also going to set up M [245:37] search to automatically create daily [245:40] backups for us in order to protect the [245:43] data and we are going to use SSH [245:47] tunneling again so that we prevent [245:50] opening firewall ports and this is [245:53] important because we want we don't want [245:55] to host the dashboard. You can host the [245:57] dashboard on your virtual machine but we [246:00] in our case we don't want to do that. We [246:02] just want to connect to it. See you in [246:04] the next sub chapter. Welcome back and [246:08] today we are going to manage your med [246:12] search database securely. We are going [246:15] to use this amazing open-source web [246:18] dashboard. You can click on this link. [246:20] It will take you directly to GitHub. [246:23] Here it is. Uh you can give them a star. [246:26] uh clone the code, run it locally if you [246:29] want to, but the project is really [246:31] awesome and I use it to maintain my [246:35] database that I am using on my [246:36] production server. There are other [246:38] dashboards, but for me, this is the one [246:40] that I liked. And this is how it looks. [246:42] In your homepage, you can give your [246:45] instance a name. It shows you the [246:48] database size, status, and the version [246:51] of the mill search binary. You can also [246:55] create a dump, see the tasks, create [246:58] keys and view your indices. It's really [247:01] awesome. These are some some of the [247:03] features that are present there. You can [247:04] search documents, upload ones, edit the [247:08] fields. Uh and all of this is done [247:11] visually. You don't need to run any [247:12] command. And again, in order to connect [247:15] to it, we are going to use the same [247:17] trick. We are going to use this amazing [247:19] SSH channel feature so that we don't [247:23] open open any ports. And this time as [247:26] you can see we will have we have mill [247:28] search running on the server. It runs on [247:31] this port and we want to be able to [247:33] access or to manage this using this [247:36] dashboard. The dashboard is going to be [247:38] running on a browser on our local [247:41] laptop. And with SSH Simon, we will [247:44] connect the ports 7,700 from our machine [247:48] to uh 7,700 [247:50] on the server and they can communicate [247:52] freely. But if when we stop the SSH [247:56] connection, not we can't manage the med [248:00] search instance. We already worked with [248:02] SSH tunneling. So it should be uh very [248:05] easy for you to reuse again uh in this [248:08] sub chapter. Okay. Okay, now let's go [248:10] back to the virtual machine. I want to [248:13] start with automatic backups. So, let's [248:16] open the server or the service. I want [248:20] you to add two lines here. SK schedule [248:24] snapshot and snapshot directory. Here [248:28] the value in schedule snapshot is in [248:30] seconds. I set it to 86,400 [248:34] and this is exactly 24 hours. uh if you [248:37] want to save every hour or yeah if you [248:40] want to change the duration feel free to [248:42] do that just change this value and we [248:45] want to save data to the snapshots [248:48] directory but currently we don't have it [248:51] so we need to create it and also I [248:53] believe we should also create the dumped [248:55] directory uh if you want for to to [248:58] create some dumps yeah let's do this [249:00] let's save the file we uh we can create [249:04] the snapshots directory like this And [249:07] let's also create dumps if we don't have [249:10] it. Okay, great. And now let's change [249:12] the ownership. I will change everything [249:16] inside my search, not just snapshots [249:19] because I have also created dumps and I [249:21] made a typos. Uh let me show you how we [249:23] can do this. We can use move. Let's use [249:26] sudo move. Let's take this path and [249:30] let's rename it. Instead of dumps, it's [249:32] going to be dumps. Uh let's just verify [249:34] that it it's worked and let's look at me [249:38] search and yes we have dumps we have [249:41] snapshots and data and we can delete [249:44] this dumps file because we no longer [249:46] need it. We used this we imported the [249:49] data now it's it's not useful anymore. [249:52] So let's use sudo remove and let's put [249:55] the file there. Sorry. Okay. So let's go [249:59] back. Ah I wasn't able to delete it. [250:01] Why? Ah, I did sudo ls. I need to do [250:05] sudo remove. Now let's go back. Awesome. [250:08] So everything is there. Let's just [250:10] verify the ownership. Who owns those [250:14] folders? M search. Awesome. Now we can [250:17] continue. Whenever you update the [250:20] service file, make sure to run damon [250:23] reload and reload the service itself. [250:27] Now, Med Search will quietly save a safe [250:31] copy of your database every single day. [250:35] And I want to mention that we will keep [250:37] just the most recent snapshots. We won't [250:40] have a lot of snapshots because that [250:43] will uh fill up your hard drive over [250:46] time. You we just need one and that's uh [250:48] that's sufficient. [250:50] Let me exit and let's use this command. [250:53] SSH-L7,700 [250:58] to 7,700. This will will create a [251:01] tunnel. [251:02] Ah, so it says address already in use [251:04] because I am running Milly Search [251:06] locally. So I need to stop that server [251:09] and re retry again. So let's retry. [251:13] Okay, now it worked. It didn't uh it [251:16] didn't tell me that the port is [251:17] occupied. Now let's go back to GitHub [251:20] because as I mentioned you don't need to [251:23] host the dashboard locally because this [251:27] the creators of this project are already [251:30] hosting it uh on Versel for free and as [251:33] I said I am using it uh both locally and [251:36] on my production server and in order to [251:40] add your own instance you can just click [251:43] on the plus button give it a name I will [251:47] call [251:48] test instance. The host is correct. It [251:52] should be 7,700. [251:54] This one is pointing at your local port. [251:56] And here I need to provide the API key. [251:59] For that I need to go back here and um [252:03] copy the code the API key. Again I have [252:06] copied the key. I have pasted it. Now I [252:09] click on confirm. Okay. I can save that [252:12] but doesn't it's not needed. Now just [252:15] click on your instance. One thing the me [252:18] search server should be running. So you [252:20] can always run status if it is if it [252:24] says running you are good to go. And as [252:27] you can see I uh here are my indices. So [252:29] I have articles articles number two and [252:33] articles test. This was also on my dump. [252:36] And I have everything. Now I can click [252:38] on let's say articles. It shows me the [252:41] distribution. Uh we can click on [252:43] documents. And here it shows you the [252:45] fields. You can click on updates. Let's [252:48] say that I want to update the view count [252:50] from 14 to something else. I can do it [252:53] without any problem. So maybe let's just [252:56] do 999 or something. Submit. It's going [252:59] to create a task. We can look at that [253:02] here. So it's it says a few seconds ago [253:05] and it succeeded. Uh let's go back to [253:08] here. Let's go up. Let's go to blogs. [253:12] Let me see. Do we have what was that [253:14] blog? Let's go back. So, it says elastic [253:17] search change heap size. Elastic search [253:21] change heap size. Do I need to refresh? [253:24] Oh, no. Maybe I used the wrong index. [253:27] So, let's go back. Okay. Let's click on [253:29] articles. Yeah, I think I am using [253:31] articles too instead. Uh, let's look at [253:34] documents. Okay. Yeah, let's change this [253:37] one. So, update from 7 to 999. [253:41] submit. Go back to the website. Refresh. [253:45] And as you can see, we have 9999. With [253:47] this dashboard, you can easily manage uh [253:50] your documents. And here, if you want, [253:52] you can upload once. You can upload your [253:54] documents here. Just paste the JSON and [253:57] hit submit. And that will uh add a new [254:01] article or product, whatever. Uh for in [254:04] my case, I am using it to add articles. [254:07] Uh so that's my content. But if you use [254:10] it for other purposes, it doesn't [254:13] matter. You can just just put your JSON [254:14] here and hit submit or you can drag and [254:17] drop the file directly. Uh let's let's [254:19] look at settings here. Uh these are [254:21] these are the the settings. And this is [254:24] very important. When you create a dump, [254:26] the settings needs to be the same. And [254:29] just like that, we finished module [254:31] number three. As you can see, using me [254:34] search is very easy. It's just one [254:36] binary. It is really fast. [254:39] uh it's uh we have really awesome [254:42] dashboards that we can use to maintain [254:45] this database. We tried to create a [254:49] system user to isolate the database as [254:52] much as possible so that we don't leak [254:54] data, we don't leak keys, stuff like [254:57] that. And even if someone hacks to your [255:00] server, they won't be able to extract [255:04] anything from it. And we used SSH [255:06] tunneling to access this dashboard. [255:09] Which means that if I stop the server as [255:13] you can see I stopped it which means [255:15] that yeah port forwarding is no uh sorry [255:19] SSH tunneling is no longer a thing. That [255:22] means that yes I won't be able to [255:24] connect to this instance. So if I click [255:26] on it it says connection failed because [255:29] we don't we create a temporary tunnel [255:32] that allows us to manage this. Once we [255:34] are done, we close it and no one will be [255:36] able to talk to me research. So even [255:39] though it is inside the machine, it is [255:41] isolated. It cannot talk to the outer [255:45] world. It just talks to the back end [255:47] locally. In the next module, module [255:50] number four, we are going to talk about [255:53] global delivery and app security. This [255:57] is important because as you can see [255:59] right now if uh let's say that you [256:01] deployed this to production your users [256:04] will have to type your IP address to [256:08] access your website. This is not good. [256:10] We need to change this to be something [256:13] like this. So so that they can just type [256:16] a name and [256:18] in behind the scenes this name will be [256:20] converted to the uh to your IP address. [256:24] So this is what we are going to fix in [256:26] the next sub chapter and we are going to [256:28] talk about DNS records creating custom [256:31] domains uh adding certificates to uh [256:35] finally uh have a secure connection [256:37] because right now it is not secure it is [256:40] we are just working with HTTP uh in the [256:43] next module we will finally upgrade to [256:46] HTTPS and that will add another layer of [256:48] security to your website. [256:52] Welcome to module four. [256:54] Up until now, our application has been [256:58] running perfectly. But to access it, we [257:02] need to type the IP address in the [257:05] browser. This is not user friendly. In [257:08] this video, we are going to buy our own [257:13] domain name. I'm going to show you how [257:16] to transition from HTTP to HTTPS. We are [257:20] going to use Let's Encrypt for that. We [257:24] are going to modify the engineext [257:26] configuration and at the end I'm going [257:29] to show you how to autorenew your SSL [257:32] certificate. [257:34] This is the agenda that we are going to [257:36] follow. As you can see, we are going to [257:38] start with purchasing a custom domain. [257:42] We are going to configure DNS records. [257:44] We are going to add an A address and a [257:47] CNAME record. I'm going to explain [257:49] these. We are going to install three SSL [257:52] certificates via list encrypt. And [257:56] finally, we are going to automate the [257:58] renewal process because by default, the [258:03] certificates are going to expire after [258:05] 90 days and we don't want to do the [258:08] renewing manually. We want the machine [258:11] to do it for us. We are going to use a [258:14] chrome job for that. Before we start [258:17] practicing, I would like you to go to [258:19] GitHub. You'll find the links for this [258:22] repository in the video description. [258:25] Click on the book folder and come here [258:27] to the fourth file. As you can see, I [258:30] have committed the changes. Everything [258:33] is here. You will find the steps and the [258:36] commands that you need to run. [258:38] Everything is described in this [258:40] document. So, don't worry. Just follow [258:43] with me and uh the resources are on this [258:46] repository. Let's register a domain. In [258:50] my case, I'm going to use pork bun but [258:53] if you want you can use namecheep or god [258:56] daddy or other registars. There are many [259:00] but in my case I will use pork bun [259:03] because this is the one I use and their [259:06] pricing is transparent. They they show [259:10] you the real price. Sometimes the other [259:12] ones they show you a very low price but [259:15] the problem is that when you try to [259:18] renew in the next upcoming years they [259:22] the price becomes higher. So that's not [259:24] good. And here in pork bun as you can [259:27] see you can type the domain that the [259:29] name that you want to register and if it [259:31] is available you can uh you can take it. [259:35] I will I will sign in because I already [259:37] have an account and I will create a I [259:39] will try to register a new domain name. [259:42] I am back. As you can see, I have logged [259:45] in with my account. I already have two [259:48] domain names. This is the one that I [259:51] showed you in the beginning of this [259:53] course. If I go there, it takes me to my [259:57] personal website. And as you can see [260:00] here, uh, if you look at at the bottom [260:03] left corner, the link is HTTP. It's [260:07] insecure. But when I click on it, it's [260:10] going to use HTTPS because we [260:13] reconfigured EngineX so that it takes [260:17] HTTP requests and redirects them to [260:19] HTTPS. And as you can see, if you click [260:22] here, it shows you that the connection [260:24] is secure. And this is what we are going [260:26] to do with the new domains that I will [260:29] create now. Okay, I will close this. [260:32] This is the domain name that I would [260:35] like to register and I want a com one. [260:41] Let's search and let's see if it is [260:42] available. Okay, let me zoom in so that [260:46] we can see clearly. As you can see, I [260:48] have typed the name here and look what [260:51] we found. As you can see, I can I can [260:54] get the domain for $11 per year. And [260:59] this is what I was talking about the [261:01] renew process. As you can see, the renew [261:03] is the same as the price that I will pay [261:06] in the first year. But if you if you [261:08] look at this one, I am going to pay just [261:11] under $2. But after this year, I will be [261:15] paying almost $30. And this is why I [261:19] mentioned in the beginning that the [261:20] pricing here in pork one is transparent. [261:23] You can see everything. In my case, I [261:25] will take the com one because yeah, this [261:28] is what I want. And I want to also get [261:32] ma but sadly yeah this one is uh country [261:36] specific because I live in Morocco. I [261:37] would like to also take that one. But [261:40] yeah, you can try your luck. Uh maybe if [261:43] you live in another country, you can try [261:45] it. Maybe you will find uh yours here or [261:47] maybe use a different uh register. Uh I [261:51] know that I will not find that one. So [261:53] I'll just skip [261:55] and in my case yeah I think this one is [261:58] is fine. So now what we will do we will [262:01] click on the plus button that will add [262:03] it to the cart. Now click there and here [262:08] again pay attention to what they are [262:09] showing you here. So the registration [262:12] here is for one year. I want to pay just [262:14] one year. Uh after that they will notify [262:17] me to pay for the the second and the [262:20] upcoming years if I want to stick with [262:22] them. So the renewal is the same price. [262:25] That's correct. And here they are giving [262:28] you other things. They always do this so [262:30] that you that you pay more. But don't [262:33] don't add anything. These are the things [262:36] that they show you here are not are not [262:37] important. Okay. So this is uh this [262:41] looks good. Now all we need to do is to [262:44] scroll a little bit and here click on [262:47] continue to billing and finish the [262:49] payment and you can also ver verify the [262:52] price before doing this. I will click on [262:56] continue to billing. Here you have many [262:58] options. You can use PayPal if you have [263:00] it, crypto or your Visa card. I will [263:04] select PayPal and I'll I'll finish the [263:06] process and I'll come back. I am back. [263:09] As you can see, this is the message that [263:11] you should see after completing the [263:13] purchase. They will tell you great [263:15] success. Congratulations on your brand [263:17] new domain. That's uh that's good. Now, [263:20] we can go back. We can click on account [263:22] domain management. And as you can see, I [263:26] got my new domain. Uh they say that it [263:29] will take some time. That's fine. We'll [263:30] come back to it. Now, we need to [263:32] configure DNS domain name system. But [263:37] before I do that, let's go back to the [263:39] slides because I want to explain DNS how [263:42] DNS works. I created this diagram to [263:45] explain this. Let's say that I am this [263:48] user. I want to check this website. I [263:53] will type it on this browser that I have [263:56] here. I will hit enter. What happens [263:59] after I do that? The browser will talk [264:02] to the DNS server. In our case, it might [264:06] be pork bun. It might be cloud player. [264:09] It depends on what you want to do. For [264:12] right now, we have created we have just [264:15] bought a domain and pork bun. They have [264:18] their own DNS server. Behind the scenes, [264:21] what will happen is that we need to to [264:23] give Porkbun a an IP address. And this [264:27] IP address is the one that we that we [264:31] can take from Digital Ocean. It's the [264:33] machine that we created in the first [264:35] module in this course. And this will [264:39] will allow pork bun or other DNS servers [264:43] to create this mapping. They will map [264:46] the name the domain that you have [264:49] purchased to that IP address. After [264:52] typing this domain name, the browser [264:56] will tell the DNS server where is this [265:00] address. The DNS server will look in in [265:03] its databases. It will if it finds it, [265:06] if it finds that address, it will reply [265:09] with the IP address and the browser will [265:12] type that IP address automatically. It's [265:15] the same. Before we were typing the IP [265:18] address directly, but now the DNS server [265:21] will give that information to the [265:22] browser. Now the browser has the IP [265:25] address. It will communicate with your [265:28] Linux server. So it's going to tell it [265:30] please give me the content and the the [265:33] server will reply with the content and [265:36] you will be able to see the the front [265:38] end that you built. This is how DNS [265:41] works and this is what we are going to [265:43] do to create this mapping. We need to [265:46] configure the domain that we bought in [265:48] porkbun. We call this mapping the a [265:51] record. Let's go back to porkbun and [265:55] let's see if we can change this. I'll [265:57] refresh the page. Okay, everything is is [266:01] all right. As you can see, we have DNS [266:03] here. If you click on it, it allows you [266:07] to modify it. As you can see, we have uh [266:11] a few default ones. Yeah, and I think [266:14] this is a newer one. I will I'll go back [266:16] to the classic view because this is [266:18] Yeah, this is the view that I used here [266:21] uh on GitHub. Let's uh let's see. Yeah, [266:24] as you can see, before it was like this. [266:26] And yeah, we get a model. So I I'll [266:29] stick to the old one. You can see that [266:32] we have a few records. These are default [266:34] records that Porkman created for us. We [266:37] need to remove some of them. I will [266:39] start by removing the alias and CNAME by [266:43] clicking on this trash icon. We will [266:46] also remove the last two, these text [266:48] ones. That's what we want to do. Now [266:51] scroll up and here let's add the first [266:54] one is the address record. Here host [266:57] leave it blank but here in the answer [267:01] value we need to put the IP address. [267:04] Let's go to digital ocean because I [267:06] forgot the IP address. Okay, here is the [267:09] the droplet. Here is the IP address. I [267:12] copied it. I will paste it here. Here we [267:15] have time to live. Leave it at 600 [267:18] seconds. Um, you can add notes if you [267:21] want. And now click on add. Okay, let's [267:25] scroll down. As you can see, we have the [267:27] new record here. That's great. Now, [267:29] let's add another one, which is going to [267:32] be CNAME. This record maps one domain [267:35] name to another domain name. You use [267:38] this one to ensure that if someone types [267:42] www [267:44] in front of your domain, they can still [267:46] reach your website. So this is going to [267:48] map www to uh your domain name. I can [267:52] show you this with my own website. If I [267:54] type my domain name, I can go there. [267:57] That's fine. But I can also do let's [268:00] remove this part and type ww dot. [268:05] And as you can see that redirects me to [268:08] my yeah to my website. And it's it's [268:10] working. To accomplish that we add we [268:13] add the CNAME record. Here in host we [268:16] add www. [268:19] And here in answer you need to type the [268:21] domain name and we can just copy it from [268:23] here and paste it and hit time to live. [268:27] We shouldn't change that. Now just add [268:31] this record. Let's scroll down. As you [268:34] can see we have it here. This is the [268:36] mapping that I was talking about. If I [268:38] type this domain name porkbang will [268:41] return this IP address to the browser. [268:44] So that so that it can communicate with [268:46] my droplets and that's it. We configured [268:49] DNS as I said I used pork bun but what [268:52] we have learned here applies to the [268:55] other domain registars the only thing [268:57] that will change is their user interface [268:59] but other than that everything I think [269:02] should be should be the same. We need to [269:06] SSH into the server. Let's do this. I [269:10] will use this command to SSH directly. [269:14] Great. I am now inside the machine. We [269:16] need to configure SSL certificates. How [269:20] do we do that? First, we need to install [269:22] the searchbot client. To do that, we can [269:26] type this command. As you can see, it [269:28] installs searchbot and this Python 3 [269:32] searchb engineext. Uh this is a plug-in [269:36] that allows searchbots to modify the [269:39] enginex configuration for us. We will [269:41] see that in action but for now let's [269:43] just install this. Provide the password [269:46] and let's wait for a few seconds. Okay. [269:49] Now let's request certificates for this [269:53] domain and subdomain. As you can see I [269:55] have placeholders. I need to replace [269:58] them with this domain name. I will take [270:01] it and paste it here. And uh here I will [270:04] keep the www do and I'll paste the [270:08] domain name. Here I made a mistake. I [270:10] need to remove the.com from here and [270:13] from here. Let's hit enter. The first [270:16] thing they are asking for your email [270:19] address. This information is used [270:22] strictly for urgent security notices and [270:26] renewal warnings if uh maybe the [270:29] automation fails. After that, they will [270:31] ask you if you agree with their terms of [270:34] service. Type yes. And for this [270:37] question, they are telling you, do you [270:39] want to share your email address? Just [270:41] type no. The account is registered. And [270:44] now they are requesting a certificate [270:46] for my domain name. Let's wait for a few [270:49] seconds. And okay, this is the message [270:52] that we want to see. Successfully [270:54] received certificate. And as I mentioned [270:58] the certificate will expire after 90 [271:00] days. Today is April 4th and 90 days [271:05] will be yeah it's 3 months. So yeah 4 + [271:09] 3 is 7. That's correct. But don't worry [271:12] we are going to automate the process of [271:14] renewing the certificates. Let's open [271:17] the engineext configuration and let's [271:20] see if searchbot changed the changed it [271:24] or not. Hit enter and let's scroll down. [271:28] It seems to me that it's it's the same. [271:32] Source B did not change it. That's not a [271:35] problem. I wanted to show this issue [271:38] because as you can see in the previous [271:41] chapters, we didn't have a domain. [271:43] That's why we created just a dummy name [271:47] here. But the new domain that we created [271:50] is called this. and searchbot didn't [271:54] find something uh there. That's why it's [271:57] it uh yeah it couldn't change it. We can [272:00] see that search bot because it couldn't [272:03] find the domain name inside the the [272:07] configuration. It created the default [272:10] configuration. Let's remove that because [272:12] we don't need it. Now open the [272:15] configuration that we created before. [272:17] And here we need to update server name. [272:20] As you can see before because before we [272:22] didn't have a domain name, we used the [272:24] IP address. But now we need to change [272:27] that. Let me show you how we do it. I [272:30] will come here. Let's hit Ctrl K to [272:33] remove that line. I will replace it with [272:35] this one. Server name stays the same, [272:38] but here we replaced the IP address with [272:42] the domain name and the subdomain name. [272:45] Save the file. Exit. Let's ver let's [272:49] verify that the syntax is correct. [272:51] Always do that before reloading engine [272:53] X. And now I will search for reload [272:57] engine X. And to search you just type [273:00] something, hit CtrlR and that will show [273:03] you the previous commands in the [273:05] history. Hit enter. This is going to [273:07] reload EngineX because whenever you [273:10] change the configuration, always check [273:12] the syntax and after that reload. Now [273:14] let's run searchbot again with the same [273:17] command and this time it will find the [273:19] configuration because we ran this [273:22] before. Searchbot is asking us if we [273:25] want to reinstall. If we want if we [273:27] choose one that will attempt to [273:29] reinstall and two will just renew. We [273:32] want to reinstall. So let's type one and [273:34] hit enter. As you can see this message [273:37] shows us that third boss was able to [273:39] find to find the our custom [273:42] configuration. And now if we open that [273:44] file, we should be able to see the new [273:49] stuff that SB added. Let's scroll down. [273:52] Great. As you can see, it worked. We [273:54] have some new lines. Let's remove them. [273:57] Let's go up. Yeah, everything here [273:59] stayed the same. We have the server [274:01] name. We We have log files. These are [274:04] the HTTP headers that we added before. [274:08] Here we are serving the front end. We [274:10] serve the API. That stuff didn't change. [274:13] But here, here is the new stuff that [274:16] search bots added. And as you can see, [274:18] this comment proves that we have listen [274:21] 443 SSL. We have some SSL certificate [274:25] stuff. Let me explain what we have here. [274:28] Before in the server block, we were [274:30] listening for port 80. Searchbot [274:33] migrated that to port 443. [274:36] And for this port is the standard port [274:40] for HTTPS. You can search that on the [274:42] web to verify that information. But port [274:45] 80 is for regular HTTP and it is [274:49] unencrypted. That's why it is it is not [274:52] secure. Great. So that's nice. And here [274:56] um these these new lines just point to [274:58] the to the keys that search B created [275:02] for us. We are including this [275:05] configuration. Yeah, this stuff it's not [275:07] that important because it is managed by [275:09] search. The most important thing here is [275:12] 443. It will make our website secure. [275:15] Now if we scroll down, we have another [275:16] server block and this one is used for [275:19] redirecting. Here we have two if blocks. [275:22] If the host is this subdomain, we are [275:25] going to redirect to HTTPS [275:28] and 30. Let's search for it. 301 means [275:33] moved permanently and it is used for [275:36] redirection. And this is what we want to [275:38] do here. If someone visits www do we [275:42] want to redirect them to https and here [275:45] we put the domain name and again this is [275:48] managed by searchbot and here if we type [275:51] just the domain name we will catch that [275:54] and we will redirect to this link and as [275:57] you can see here for this server block [275:59] we are listening on port uh port 80 [276:02] which is the regular HTTP which is not [276:05] secure. Yeah, when we catch that [276:07] traffic, we will redirect it to HTTPS. [276:10] Awesome. Now save the file and exit. Now [276:14] let's automate the certificate renewal. [276:17] Let me clear the screen. We have this [276:19] command that we can run to test the [276:23] renewal system. We want to do this to [276:25] ensure that it works. And we are adding [276:28] dry run because this is just a test. [276:30] It's not going to change anything. Let's [276:32] verify that this is going to work. They [276:34] found that the account is registered. [276:37] That's good. And as you can see here, [276:39] they are saying simulating renewal. So [276:41] this is just a simulation. And they are [276:43] saying congratulations all simulation. [276:46] All simulated renewals succeeded. This [276:48] is what we want to see before [276:50] proceeding. Now let's create a new [276:53] chrome job. We are going to type sudo [276:56] chrome tab-e. [276:58] We want to do this in for the root user. [277:02] This is the first time we are doing [277:03] this. So the this command is showing us [277:06] what text editor to use. I like to use [277:09] nano. I will type one and hit enter. Now [277:12] let's go down and let's add this line. [277:15] If you don't understand the chrome tab [277:18] syntax, you can use this website called [277:22] chronab.guru. [277:24] And here you can paste the chrome tab [277:27] notation. And this is the one that I [277:30] added in the chrome tab. Why I like this [277:33] website is because they show you they [277:35] try to convert this into a sentence that [277:37] you can understand. And this expression [277:41] means at exactly midnight every Sunday. [277:46] And this is the command that we want to [277:48] run every sun. Every Sunday we want to [277:52] run the renew command. And we are adding [277:55] quiet because we don't want to generate [277:58] a lot of system logs. We want to do this [278:00] quietly. Now save the file. Exit. Good. [278:04] Now Linux will run this job every [278:08] Sunday. And again, as I said, you don't [278:10] need to do this manually. The system [278:12] will take care of the renew process. But [278:16] we have a small problem. Let's say that [278:19] before the expiration of the [278:22] certificates, we were able to [278:24] automatically renew it. Engine X will [278:26] not be able to see it because we need to [278:29] reload it in order to load the new [278:32] certificates. In order to solve this, we [278:34] will create a post renewal hook. I will [278:38] explain what that means. Let's create [278:40] this file. Look where we are creating [278:43] it. We have etc. Let's encrypt. This is [278:46] what we are using for the certificates. [278:49] And here they have a renewal hooks [278:51] folder. And we want to do this in the [278:54] post. So after renewing this is what [278:56] post means after an operation. After [278:59] that we are going to reload engineext. [279:01] This is a simple script that will run [279:04] this command for us whenever the [279:06] certificates are updated. Save the file [279:09] and exit. But don't forget to make it [279:12] executable so that searchbot can execute [279:15] this. And you can always verify if that [279:18] works by using ls-la. [279:23] And what you want to see is X in yeah [279:26] three positions and okay you are now set [279:29] up. You can just kick back and enjoy [279:31] your sleep. I think we have finished the [279:35] full setup. Now we can test if this [279:38] works or not. Let's go back. You [279:40] remember that before we used to go to [279:43] Digital Ocean, copy the IP address, [279:46] paste it here in order to uh see the the [279:50] website. Now this does not work because [279:53] we have uh updated the engineext [279:56] configuration to look for the domain [279:59] name. Let's do that. I will or I can do [280:02] it from here. I can click on this and [280:04] let's hope that it works. Yes, it did. [280:06] As you can see, I have typed the domain [280:09] name. This is my website. Uh as you can [280:13] see, it is working. Uh the front end and [280:15] back end are both connected. Let's load [280:18] an article. Let's scroll down a little [280:21] bit. This is a long article here. I can [280:24] add some claps. Let's go up. Let's [280:27] ensure that this works. Yes, it says 11. [280:29] That's the new claps. Yeah, everything [280:32] is working. And we can also try to force [280:36] this to go to HTTP. Uh let's use that. [280:41] And here I will type the domain name. [280:43] What will happen? This will go this will [280:46] redirect to HTTPS automatically. Boom. [280:48] That worked. As you can see, the [280:50] connection is secure. Awesome. This is [280:53] great. Uh, that works. And we can also [280:56] verify www do. Let me remove this part. [281:01] And let's hit enter. And that also [281:04] redirects without any problem. Amazing. [281:08] Now we have uh successfully created [281:12] purchased a domain. We changed the DNS [281:17] configuration. We added the A and C name [281:20] records. We used let's encrypt to make [281:24] the connection encrypted. But we are not [281:28] done yet. In the next subchapter, we are [281:31] going to talk about CDN layer. Why do we [281:35] need to talk about that? Because the [281:37] problem is that for me, this droplet [281:41] that I created is hosted in Germany. I [281:44] live in Morocco. So if I want to access [281:47] this website, it's going to load fast. [281:49] But if let's say that someone is from [281:52] China or Australia, if they want to to [281:55] go to my website, the load time is going [281:57] to be very slow. They need to wait for [282:00] seconds instead of milliseconds to yeah [282:03] to load the the content of the website. [282:06] That's a problem. We are going to fix it [282:08] in the next sub chapter with the help of [282:11] Cloudflare. We are going to use it to [282:14] cache the static files on their servers [282:17] because they have servers all over the [282:20] world and that will mitigate this [282:22] problem. So that person in Australia [282:24] that before was used to wait in seven or [282:28] 8 seconds for the website to load, we [282:30] now only wait for maybe a second or [282:33] less. And that's what we are going to [282:35] focus on in the next uh video. Welcome [282:39] back. In this video, we are going to [282:42] talk about the CDN layer and caching. [282:47] Here is what we are going to work on. In [282:50] this video, I'm going to show you how to [282:53] route traffic through a content delivery [282:56] network. There are many, but we are [282:59] going to use Cloudflare for this [283:01] tutorial. We are going to configure [283:04] strict end toend SSL encryption. I will [283:08] show you how to unmask the visitor IP [283:11] address in engine X because now [283:13] Cloudflare will sit between the user and [283:17] engine X. This is an important step that [283:20] we should not forget. And finally, I [283:24] will show you how to implement caching [283:26] strategies for single page applications. [283:30] Let me show you why we need to rely on [283:34] CDNs, especially for applications that [283:38] don't require a backend. Last time I [283:41] bought this domain to demonstrate how to [283:45] link your domain to your VM. And as you [283:50] can see, the application loads and uh [283:52] here is my website. You can use this [283:55] free utility that allows you to check [283:59] the load times. Here you can test. Let's [284:03] test this website. Keep from all [284:05] checkpoints and click on start test. [284:10] Here is the results that we got. As you [284:13] can see, this shows us the result. Shows [284:17] us a lot of information. But as you can [284:19] see, it tests the load time of our [284:21] website in different locations. You can [284:23] see that here Frankfurt is in Germany [284:26] because I am hosting this server on a [284:30] virtual machine in Germany. That's why [284:32] the load time is basically zero. But if [284:36] we go um let's see. So here the the [284:40] worst load response time is almost 5 [284:44] seconds which is in China. It makes [284:47] sense because the distance between [284:48] Germany and China is very big. We we [284:52] will also see here u yeah as you can see [284:55] we have Japan we have yeah other [284:58] countries that are very far but the the [285:01] countries that are countries that are [285:03] close to Germany uh especially in Europe [285:07] Africa those will uh see very uh very [285:10] low uh load times. This is a good tool [285:14] that you can use to test how fast your [285:17] your website loads for users. I wanted [285:21] to start with this because it shows the [285:23] problem that we currently have. And if [285:27] you want to solve this, one option is to [285:29] host your website on many servers. For [285:32] example, you can host one maybe [285:34] somewhere in Europe. That will give [285:37] people in Europe [285:39] very high load times. Maybe you put [285:42] another one in um USA, you put another [285:45] one in Africa, another one in Asia. But [285:48] that will drive the cost up very [285:51] quickly. CDNs solve this issue for us. [285:54] They allow us to serve static files to [285:58] users all over the world. And here cloud [286:01] player, this is what we are going to [286:04] use. They have plenty of servers uh [286:07] scattered around the world. And let me [286:09] show you this diagram that I made that [286:12] explains how this works. We have two [286:15] situations. The first one is before [286:18] applying CDN and the other one is after. [286:21] Let's now look at the first diagram. [286:24] Let's consider that the server here is [286:26] hosted maybe here in somewhere in [286:28] Morocco. And let's say someone here in [286:31] Asia wants to check my website. The load [286:34] time will be let's say almost 4 seconds. [286:38] And this user will see a smaller load [286:41] time but still it's not it's not very [286:43] responsive. 1.2 seconds is bad. But if [286:46] someone is here in Morocco or Algeria, [286:49] the the website will load instantly for [286:52] them. This is the problem that we have [286:55] distance that we need to travel and that [286:58] takes time. With CDN, we'll solve this [287:01] by including edge servers. These servers [287:05] are located close to the end user. This [287:10] person who lives in Asia will connect to [287:14] a server [287:16] the closest server to them. We call them [287:19] edge. And that edge server will [287:21] communicate with the main server. And [287:24] these edge servers are provided or are [287:28] given to us by Cloudflare. And here they [287:31] will cache your static files just the [287:34] front end so that users can load them [287:36] instantly. And as you can see, with the [287:39] help of these edge servers, we were able [287:42] to bring down this 3.9 seconds to almost [287:46] zero. This is the benefit. And I will [287:49] show you how to do this in this video. [287:52] And the great thing about this is that [287:54] we have the before situation. After [287:56] applying the test, we will come back [287:58] here. We will uh start the test again. [288:01] And hopefully we will see that for [288:03] example for Beijing it will go from 5 [288:05] seconds to maybe uh 1 second or less [288:09] hopefully and uh we will see that at the [288:11] end of this video. Now go to [288:13] cloudplayer.com. [288:15] Let me zoom out a little bit so that you [288:17] can see. And here make sure to create an [288:22] account. If you don't have it you can [288:23] loging in for free. They have a free [288:25] tier which we we are going to use in [288:27] this video. I will log in and I'll come [288:29] back. After you log in, you will see a [288:33] add button. Click on it and choose [288:36] connect a domain. Here I will type my [288:40] domain name. I will choose this one. [288:44] Keep import DNS records automatically [288:48] checked. This will allow Cloudflare to [288:51] automatically fetch the A and C name [288:55] records that we created uh in pork bun [288:59] or in your preferred domain registar. [289:03] Here we have a section uh about AI [289:06] crawlers. If you don't want AI companies [289:10] to scrape your websites, select block on [289:14] all pages or you can select the other [289:17] options. I don't want them to scrape my [289:20] content. So, I will select this. And [289:23] here we have this option that will [289:25] instruct AI bot traffic with robots txt. [289:28] You can read more about this. Uh it's an [289:31] it's a setting that Cloudfare will add [289:34] to robots txt to block AI companies from [289:39] scraping your website. As you can see, [289:41] they have a few few options. So, they [289:43] have AI input, AI train. If you want, [289:45] you can read read it here in the [289:47] documentation. I'll keep that enabled [289:50] and I'll hit continue. As I mentioned, [289:52] they have a free plan that you can use. [289:56] They also have other plans. If you want [289:58] to scale your website, you need to move [290:00] to maybe the pro plan. But for this [290:04] tutorial, the free plan will be enough. [290:07] And here they are showing you the how [290:09] much you will pay monthly or yearly. We [290:11] don't care about that because yeah we [290:13] will be on the free pan anyway. This [290:16] step is very important. Cloudflare here [290:19] is listing the DNS records. It's found [290:22] on pork bun. Here we need to verify the [290:26] status. Here you can see that we they [290:28] show you two icons. Uh so if the icon is [290:32] orange like we see here, it means it's [290:34] proxied which means traffic will go [290:37] through the cloudflare network network. [290:39] And this is what we want. Here we have [290:41] the A and C name records. They are [290:44] pointing to the correct values. And here [290:47] the proxy status is proxied. This is [290:49] correct. This is what we want to see. [290:51] Email traffic uh for the MX and TX [290:56] records. Here you want them to be DNS [290:59] only. We don't want we don't want [291:01] Cloudflare to proxy them because [291:04] Cloudflare does not proxy email traffic. [291:08] If you do this, your email will stop [291:10] working. So, please keep it as DNS only. [291:14] And if you have any NS records, in my [291:18] case, I don't have them. You need to [291:19] delete them. You will have this delete [291:21] button. Just click on it and delete [291:23] those records. Uh I can show you how [291:25] this looks in the documentation. Click [291:28] on the book folder. Come here to uh the [291:32] fourth module. Search for NS. And yeah, [291:37] here is the page. You can see that here [291:39] if you see these NS records, just click [291:42] on the delete button to remove them. [291:45] Okay, now we can go back. I'll click on [291:47] continue to activation. And if you are [291:50] wondering, there is here a warning that [291:53] tells you this host name is not covered [291:55] by a certificate. This is not a problem. [291:57] We created a certificate in the previous [292:00] tutorial. We used Let's Encrypt for [292:03] that. So yeah the and as you can see the [292:06] website is secure we have um as you can [292:08] see it's verified by let's encrypt just [292:11] click on continue to activation and at [292:14] this step we need to connect to porkban [292:18] because we need to uh update some stuff [292:20] I'll I'll log into pork and I will come [292:23] back I am in the domain management page [292:27] here as you can see we have uh the my [292:30] new domain I'll click on ns by default [292:33] We have a few name servers. We need to [292:36] replace these by the the ones that [292:39] Cloudflare has given us. Just click on [292:42] this link. It's going to copy the name [292:46] servers. Remove these depot ones and [292:50] start pasting the name servers that uh [292:54] Cloudflare has given you. And you can [292:56] follow the steps. You can read here. So [292:58] they tell you to log into the DNS [293:00] provider. In my case, it's Porkban. And [293:02] after that they tell you to find the [293:04] name servers section. Here it is. And [293:08] after that add these and delete the [293:09] other uh name servers. I have done that. [293:12] Now I just need to click on save name [293:14] servers and hit okay. Now return to [293:18] Cloudflare and click on I updated my [293:22] name servers. In this page click on [293:25] check name servers now to see if the [293:28] name servers have been propagated. Uh if [293:31] you read the text, they tell you that [293:32] this uh takes around maybe an hour or [293:35] two or sometimes even a day. But if you [293:38] want, you can click on check name [293:40] servers now. And uh as you can see, [293:44] Cloudflare is saying that they are [293:45] checking the name servers for my domain. [293:48] So they are telling me to wait a few [293:50] hours. But that's not a problem. We can [293:52] continue the configuration. We will come [293:54] back and uh verify that everything is [293:57] proxied through Cloudflare. Now on this [294:01] left panel we can click on SSL uh /TLS [294:06] click on overview and here make sure [294:08] that your domain is selected here I have [294:12] two uh I will select the the new one [294:14] that we added and here we need to set [294:16] this to full strict uh hit confirm. Why [294:21] do we need to select this instead of [294:24] full? Because as I said in the previous [294:26] tutorial, we used let's encrypt to [294:30] secure uh our website and uh we saw that [294:34] when we were add when we were verifying [294:36] the DNS uh Cloudflare was giving us that [294:39] warning because they don't know if we [294:42] have a a certificate and uh by choosing [294:45] full strict cloud player will know that [294:47] we have our own certificates and will [294:50] accept it. Now I want to show you a [294:53] utility called dig. Uh you can search [294:57] for it. It's it's called as I said dig [295:00] and you can read about it uh here on [295:03] Wikipedia. [295:04] It's a utility that we can use to verify [295:08] if the internet is seeing the new name [295:12] servers that we got from Cloudflare. [295:14] First make sure that it is installed by [295:17] typing dig-v. It's going to give you the [295:20] version. If you don't have the utility, [295:23] you can install it like this on Ubuntu [295:26] by installing DNS utils and that will [295:29] give you access to this utility. To use [295:32] it, you type big plus short. We want to [295:36] verify the name servers. And here uh I [295:39] will put my domain name. As you can see, [295:41] it is showing me the these four that are [295:44] coming from Porkbun, which means that [295:47] the new name servers haven't propagated [295:50] yet. So, we will have to wait for a few [295:53] hopefully for a few hours and not a full [295:55] day. You can also go back to Cloudflare, [295:59] go to overview. Here you should see that [296:03] this message that tells you that your [296:05] domain is now protected by Cloudflare. [296:08] That this is what we want to see. [296:10] And if you made any mistake, you can [296:13] always go back here. For example, if you [296:15] want to update the DNS records, just [296:19] click there or you can find it here on [296:21] the left panel. And here are the [296:23] records. You can see we have all the [296:25] records and also the name servers. You [296:28] can also use another website called [296:30] what's mydns.net. [296:33] We can do the same thing here. I can put [296:36] my name uh I can put my domain name [296:39] here. Here I will select NS because this [296:41] is what I want to verify. And if you hit [296:44] enter, as you can see, it is starting to [296:47] propagate. There are some locations [296:50] where the Cloudflare name servers [296:53] haven't propagated yet. But yeah, the it [296:56] it will take a few hours for uh every [297:00] location to see the new name servers. [297:02] Let's go back to dig to see what we get. [297:05] We are still getting uh yeah we are [297:08] still getting the old name servers but [297:10] yeah it's going to uh it's going to show [297:12] up uh in a few hours with this website. [297:16] We were we are able to confirm that uh [297:19] the new cloud player name servers are [297:21] starting to propagate. Let's go back to [297:25] this website. I will take a screenshot [297:29] for this previous test and uh I will [297:33] rerun it again and let's see if uh [297:36] things have changed or not. Okay. Uh we [297:40] see some improvements. Uh obviously uh [297:42] yeah uh the DN the name servers haven't [297:45] propagated fully but we should be able [297:48] to see some uh differences. I'll take [297:51] another screenshot and I'll put the two [297:54] images next to each other. On the right [297:56] side we have the before [297:59] and here this is after. As you can see [298:02] let's start with Beijing. Before it was [298:04] 5 seconds now it is 3.4 seconds and this [298:07] will keep improving. Um maybe the first [298:11] time you try to load the website that [298:13] will take maybe uh let's say 3 seconds [298:16] but after that the files are going to be [298:18] cached which means that they are going [298:19] to load very fast and uh since uh the [298:22] name servers are still propagating [298:25] I will maybe I'll try to rerun this uh [298:28] afterwards uh and just before ending [298:31] this tutorial and we'll see if things [298:33] will improve or not. But you can see [298:34] that there are some locations that [298:36] improved uh drastically. Uh especially [298:39] Beijing. Uh let's look at maybe Hong [298:43] Kong. So Hong Kong is now it takes just [298:46] 1 second. Soul takes 0.9 seconds. Okay, [298:50] good. We see some improvements which is [298:52] a great thing to see. Well, I will come [298:54] back to this website later. I will leave [298:56] it here. And now let's go back to the [298:59] slides. This is uh what I observed on my [299:03] personal website. You can see that uh I [299:06] used the same tool and here I I see big [299:10] changes because I let the name servers [299:13] propagate for maybe a day before running [299:16] the test again. For Tokyo, as you can [299:19] see, we were able to reduce the time [299:22] from 1.3 seconds to 0.3. [299:25] So again, same situation. Jakarta, same. [299:29] and Beijing it went from 4 seconds to [299:31] 1.5 seconds. This is the benefit of [299:34] using a CDN especially if your website [299:37] is [299:38] doesn't even if it has a back back end [299:41] uh attached to it there is no problem [299:44] your front end will load quickly but [299:47] because we introduced cloud player we [299:50] now have a problem I have created this [299:54] illustration to explain you the issue [299:57] let's say that we have a user this is [300:00] their IP address now if let's say that [300:03] they want to check my homepage, the [300:06] request will go through Cloudflare with [300:09] the IP of that user. That's not a [300:12] problem. But when Cloudflare will try to [300:16] send that request to my application, [300:19] it's going to send it to EngineX. But [300:21] the problem is that we are not sending [300:23] the user's IP. Instead, we send [300:26] Cloudflare's IP address and this will be [300:30] sent to the back end. And uh that's [300:32] problematic if you want to do I don't [300:35] know if you want to track uh IP [300:38] addresses. This is bad because let's say [300:40] that you you received 100 requests from [300:44] let's say 50 users your backend will see [300:47] the same IP address and this is the [300:50] problem that we will try to fix. And [300:52] luckily, if you read the Cloudflare [300:54] documentation, they tell you that you [300:57] need to add CF connecting IP to your [301:00] EngineX configuration. And that will [301:03] allow you to unwrap the IP to get uh [301:07] yeah to get the user's original IP [301:10] address. And now your backend receive [301:12] the correct IP and you can store that in [301:15] your in a database or uh yeah uh use it [301:18] in your analytics or something like [301:20] that. Let me show you how we do that. I [301:23] have connected to the machine. Let's [301:26] open this file. Let me type my password. [301:30] And here I want you to copy this from [301:33] the GitHub repository and put it here. [301:36] Let me show you where you can get that. [301:39] Here is where to get this code block. It [301:42] is available in uh my GitHub repository. [301:45] Let me go back here. If I scroll down, [301:48] you can see that we have this line which [301:51] uses this header to get the real user [301:54] IP. And don't worry about this. If you [301:56] want, you can find the link to the [301:59] documentation. Just click on this that [302:02] will take you to this page and you can [302:04] read about this header in detail. Here [302:08] we have the Cloudflare IP addresses. And [302:12] here I added a note that shows you these [302:16] IP addresses and they you can also see [302:19] the dates that maybe if they change them [302:21] or if they added more you can just come [302:23] to this page, copy them and put them in [302:27] this configuration file. Good. Now we [302:30] can save the file. Exit. This file will [302:33] be loaded automatically. So there is no [302:36] need to go to our website configuration [302:39] and uh include this. Now we need to [302:41] verify the syntax to ensure that [302:43] everything is correct. Good. And now we [302:47] need to reload uh engine X. But before I [302:50] do that, I just want to go to the [302:52] website because now we are using uh [302:55] Cloudflare. I want to show you the [302:57] situation before and after. We already [303:00] we have the logs. So we can check them [303:02] to see the IP address that we are [303:04] getting. Let's check the access logs. I [303:08] want I will do tail instead of cat and [303:12] let's see what we are getting. This is [303:15] the the IP that I was talking about. We [303:17] can see them here. Uh let me go to the [303:21] website again just to make sure that we [303:23] are receiving Cloudflare's IP addresses. [303:27] Okay, I have visited the web page. I am [303:29] using tail-fs [303:33] and let's start refreshing the page. You [303:34] can see that here we get we are using uh [303:37] some I uh we are receiving some IP [303:40] addresses. We confirmed that we are [303:42] receiving one IP address which is coming [303:45] from Cloudflare. We might receive [303:47] different ones but all of them are [303:48] coming from Cloudflare because here we [303:50] have uh plenty of uh IP ranges. Now [303:55] let's stop this and let's reload X. [303:58] Okay, I've reloaded. Now uh let's let's [304:01] watch and let's see what we get after uh [304:05] refreshing the page. Aha. As you can [304:07] see, I am getting my own IP address now [304:10] instead of the server's IP. And I can [304:14] refresh. And uh yeah, as you can see, I [304:17] am getting my IP address. And I'm I'm [304:19] also receiving other IP addresses. These [304:22] are coming from either bots, probably [304:25] bots because no one knows my website and [304:28] not real humans. Uh these are just [304:30] automatic bots that are uh targeting [304:33] your website because they are trying to [304:35] see if they can get some credentials [304:39] from your website. I can show you this [304:40] if we scroll uh let me use maybe cat or [304:44] something to see the full uh log. I can [304:48] also take this to grab and search for [304:51] env. As you can see, this person or [304:55] yeah, this boss is trying to see if they [304:58] can find some env files on my uh web [305:02] server. Um I mean even if they they try [305:05] they won't be able to get anything [305:06] because we are not serving those files [305:08] when we created our engineext [305:10] configuration. But you need to be aware [305:12] that if your server is public, it's [305:15] going to be hammered by bots. Awesome. [305:18] Now the thing that we need to do is um [305:21] to start working on caching. When you [305:25] develop a front- end application, there [305:28] are some pages that are not going to [305:31] change. Those can be cached. But the [305:34] pages uh that are dynamic, we should not [305:38] change them. For example, if I change my [305:41] index.html, [305:43] this is a single page application. If I [305:45] add another feature, let's say to one of [305:47] my blogs, I want to I want users to be [305:51] able to see it. If I cache index.html, [305:55] they won't be showing they won't be [305:57] receiving the new updates because [306:00] browsers they are the they are designed [306:02] to cache resources so that they load [306:05] fast for users. But sometimes if this is [306:08] not implemented correctly uh they might [306:11] not receive uh new updates from you and [306:15] this is what we are going to focus on. [306:17] Uh now let's open my engineext [306:20] configuration and here we need to start [306:23] adding some caching. We have a single [306:28] page application. I'm going to divide [306:30] the caching strategy into two parts. The [306:34] first one we will uh we will never cache [306:37] index.html [306:39] because the whole application is [306:41] injected there and if uh something new [306:45] was added to the front end it's not [306:48] going to show up if we cache index.html [306:51] but assets like images logos those can [306:54] be cached forever. So let's start uh [306:57] applying this. Let me show you how we do [307:00] this. I will come here and I will add [307:04] another location block. This block will [307:08] tell the browser to always communicate [307:11] with the server to see if there is a new [307:15] version of index.html. [307:18] If if we have one then the browser will [307:22] try to load it. And here in the caching [307:25] strategy we are using no cache. uh so [307:29] that we tell the browser to not store it [307:32] and instead try to communicate with the [307:36] server every time uh this file is [307:39] requested. And as you can see here we [307:41] are using equal because we want to focus [307:44] just on this specific file. We can also [307:48] add this location block to cache assets [307:52] forever. Here we are the the assets are [307:55] located in the b folder. Here the [307:59] caching strategy is now we are setting [308:02] it to public. Max age is set to this [308:06] very big value. Uh it is in seconds. If [308:10] you convert it to years it's going to [308:12] give you exactly one year. And here we [308:15] are using immutable. This is an [308:18] instruction that tells the browser that [308:20] this file will never change. Let's save [308:24] the file. Let's exit. Uh now we need to [308:29] verify that the syntax is okay. Good. [308:32] And we can reload. Let's go to the [308:36] browser. Let's refresh the page. Just [308:38] ensure that everything is uh working [308:41] without any issue. Okay, great. Now what [308:44] I want you to do is to open the network [308:48] tab. I will zoom in a little bit here. [308:50] Select all or you can select the static [308:53] files. Refresh the page and let's see [308:56] what we have here. I'll select HTML, CSS [308:59] and JavaScript. If you look at this [309:02] column transferred, you see that the [309:04] browser is telling you that most of [309:07] these static files were cached. That [309:10] makes sense because we have added the [309:12] the caching strategy but the root file [309:16] which is uh the index.html HTML we [309:20] always uh we are always communicating [309:23] with the server to get the latest [309:26] version. And as you can see here in the [309:28] cache control we uh it is set to no [309:31] cache which means that the configuration [309:34] that we added to engineext uh has been [309:36] loaded uh successfully and in the [309:39] response we get uh index.thml the HTML [309:42] and as I mentioned in Vue or other [309:45] frameworks we inject uh the application [309:48] inside this div with the id app that's [309:52] why we need uh it's very important to [309:54] not cache this file amazing this worked [309:57] now I want uh I want to mention the pros [310:01] and cons of using a CDN like Cloudflare [310:06] or yeah other CDN's uh the benefit here [310:10] is speed. We saw that before we weren't [310:15] able I mean for users in China they [310:18] needed to wait 5 seconds to load the [310:22] website and my website is not heavy. [310:25] Imagine if the website was uh very huge [310:28] or not optimized that 5 seconds uh would [310:33] have turned to maybe 7 seconds or or [310:36] more. Um but now thankfully with the [310:39] help of Cloudflare we were able to bring [310:42] that down from 5 seconds to 1.5 seconds [310:46] which is amazing. And as you can see the [310:48] the numbers have changed because [310:50] probably the name servers uh have [310:54] propagated that's why we see this [310:56] dramatic shift. Uh yeah, as you can see [311:00] even here uh when trying to run the dick [311:03] command I no longer see the pork bun [311:05] name servers instead I am seeing the [311:08] cloud player name servers amazing this [311:11] didn't take even an hour and we can see [311:14] a huge boost in uh performance that this [311:17] is very amazing another benefit is [311:20] security even uh even though we are [311:23] using the free plan we are able to uh [311:28] get some security. Um, CloudFlare will [311:31] protect your website from DDoS attacks [311:34] and other types of attacks. It's going [311:36] to block AI crawlers from scraping your [311:40] website uh which is amazing. But the [311:42] problem is uh this adds complexity. We [311:46] have introduced another step before [311:49] reaching uh our application. And if uh [311:53] let's say that someday a uh a problem [311:56] happened and your application is not [311:58] accessible you have to verify everything [312:01] and um yeah because you don't know if [312:03] the problem is in cloudflare or is uh in [312:07] your application and sometimes if cloud [312:10] if there is an outage in cloudflare your [312:12] application will not be accessible and [312:14] this happened many times uh in the [312:17] recent months uh many people who relied [312:21] on Cloudflare suddenly couldn't uh or [312:25] users couldn't re reach their [312:27] applications. So these are the pros and [312:30] cons of using this. But I mean I would [312:34] say the main benefit is that you won't [312:36] pay a lot of money to host your [312:38] application in uh many web uh on many [312:42] web servers scattered around the world. [312:45] Your website is now globally [312:47] distributed, [312:49] passed, and protected by Cloudflare's [312:52] network. You have successfully finished [312:55] the performance setup. However, before [312:58] we move on to automating your [313:01] deployments, we need to ensure that your [313:05] server is completely locked down against [313:07] hackers and automated scanners. As you [313:10] saw in the engineext logs, we saw that [313:12] there are many bots who are trying to [313:15] fetch secure files. So we need to ensure [313:19] that everything is secure so that we can [313:21] rest easy and don't worry about [313:24] something being stolen from your web [313:26] server. In the next chapter, we are [313:28] going to talk about vulnerability [313:31] scanning. I will show you how to test [313:34] your own defenses. I'm going to show you [313:37] how to update the engineext [313:39] configuration to block access to files [313:43] like envit [313:46] something like that so that bots [313:48] automated bots cannot reach them. I will [313:51] show you how to scan uh your web server [313:55] for open ports using a tool called end [313:57] mapap. This will uh with this we can uh [314:00] verify that only the minimum amount of [314:04] ports are open. Uh we want ports for [314:08] HTTP, HTTPS and SSH. These are the only [314:11] ports that we want to open to the entire [314:14] internet. But the other ports like uh [314:18] your applications back end or me search [314:21] or other services services those needs [314:23] to be uh closed and if we want to use [314:26] them we need to use SSH tunneling like [314:29] we did with the previous tutorials. This [314:31] is what we are going to focus on uh in [314:34] the next tutorial. See you next time. [314:37] Welcome back. In this tutorial we are [314:40] going to focus on vulnerability [314:43] scanning. You are going to learn about a [314:46] concept called soft 404. We are going to [314:50] fix it because this is present in our [314:52] application. You are going to configure [314:54] engine X to block malicious bots that [314:57] are scanning your website 24/7. [315:01] And finally, you will use a tool called [315:03] N MAPAP to scan your firewall from the [315:07] outside. Let's talk about this trap that [315:10] we call soft 404s. I have this diagram [315:14] that explains it. Here we don't have a I [315:17] mean we could have a regular user but [315:19] mostly these are bots. They will try to [315:22] get files that contains um valuable [315:26] information like your env files. So this [315:31] request will go through cloudflare and [315:33] will reach your application. Now, [315:36] engineext doesn't know about these files [315:38] because the location blocks that we [315:40] added are serving just our back end and [315:44] the built files uh for view. So, env is [315:48] not known. So, engineext will just fall [315:51] back to serving index.html [315:54] and this query or this request will [315:57] succeed. But this is a false success [316:00] because the intent of the bot is to get [316:04] not index.html. [316:06] Why this is a false success? Because the [316:08] bot will and will think that it was able [316:12] to get thev file. So it's going to [316:15] always come to your web server and it [316:18] will hammer it day and night. It's going [316:20] to keep asking it for the same file. But [316:22] if it gets A44 not found, it's going to [316:27] skip this. And this is useful for [316:30] malicious bots and also crawlers like [316:33] Google's crawlers, for example. If they [316:35] see that this path doesn't exist on your [316:37] website, it's going to stop uh accessing [316:40] it. And this this is b this is good for [316:43] you and for them because you will uh you [316:46] will not start wasting bandwidth on [316:49] requests that are uh going to give 404 [316:53] and uh in this case bot will not will [316:57] see a signal that will tell them you are [316:59] not going to get this file anyway. So [317:00] they are going to back up. This is the [317:03] trap that we are uh that we have right [317:06] now in our application and we can test [317:08] it here. Uh this is uh the new domain. I [317:12] can come here. So we can see that we [317:14] have just slash that will go to [317:16] index.html but I can add dot /.Nv. As [317:20] you can see uh the request was [317:22] successful. I was a I am still on the [317:25] website. I don't see any error and my [317:28] application uh I configured my [317:30] application to load a 404 page because [317:34] yeah this is not known. there is no [317:35] route that will there is no page that [317:37] will uh be loaded after accessing this [317:40] URL. So I have a custom page but this [317:44] gives 200 as a uh HTTP status code. Uh [317:49] but we don't want to do this. Now if I [317:51] go to my production server and type the [317:56] same query I am I will get 404 not found [317:59] because I am blocking these uh most of [318:02] these files that uh automated boss try [318:05] to check and now the response is 404 [318:08] it's not 200. If I go here and click on [318:13] this request we get 404. If we go back [318:17] to this website where we have the soft [318:19] 404 problem, you can see that here we [318:22] are receiving 200. Okay, let's see how [318:26] to solve this problem. Again, you can go [318:29] back to my GitHub repository. You have [318:32] everything in this file. If you scroll [318:34] down a little bit, you have the [318:36] vulnerability scanning section. And [318:40] here, this is what we need to add. You [318:43] can see that here we are blocking a lot [318:45] of files um so that we are blocking [318:48] mainly anything that starts with dot we [318:51] have SQL all yeah there are plenty of [318:54] things that that we don't want to uh [318:57] serve. I will copy this and I will paste [318:59] it uh in uh my engineext configuration. [319:03] Now let's go to my server. I am still [319:06] logged in. I will use nano and let's add [319:10] this uh you can add it u here or below [319:14] it doesn't matter let me explain to you [319:17] the second location block here we are [319:20] blocking uh a lot of stuff even though [319:22] we are not using PHP in this application [319:26] you if you inspect the logs you will see [319:28] that they are trying to access some [319:30] admin panels PHP files stuff like that [319:34] so I am just I just have this [319:36] application block that will check all [319:38] these um file extensions and uh will [319:41] block them. Here I as you can see I am [319:44] returning 404 so that I can display that [319:47] page and access log and log not found [319:51] are set to off because I don't want to [319:53] store um yeah these failed requests uh [319:57] so that I don't consume a lot of disk [319:59] space because as I mentioned you will be [320:02] receiving thousands of requests per day [320:05] just um from bots that are trying to [320:08] access these files and here we have [320:10] something called well-known phone. Uh [320:12] you can go to this page on Wikipedia and [320:16] read more about them. Uh this is [320:18] important. We don't need to block this. [320:22] If you do, search bot will not be able [320:25] to verify your domain and your SSL [320:28] certificates. And if that happens, um [320:32] yeah, uh s will not be able to renew [320:35] your certificates and that will break [320:38] after 90 days. we are so here again we [320:42] are using uh an expression that will [320:44] capture most files like envit stuff like [320:49] that but uh we have one exception which [320:52] is the the wellknown uh directory we [320:56] don't want uh we don't want to block [320:59] that we want to keep it for all files we [321:02] are going to deny that request and again [321:04] we are going to return 404 and we don't [321:06] want to add anything or yeah we don't [321:09] want to add noise to the log files. [321:11] Okay, now let's save the file. Let's [321:14] verify that the configuration is [321:16] correct. Yes, it is. And let's reload. [321:20] Now, let's go back to the browser. [321:23] Let's see what we have here. Okay, so [321:26] remember this is my new domain. This is [321:28] the response that I re that I got before [321:31] this update to the configuration. Now, [321:35] let's refresh the page. And voila, we [321:39] get 404. Now I cannot get um [321:44] or what were the other stuff. Uh yeah, [321:47] let's uh check the logs. So they were [321:50] getting some PHP stuff. I'm not sure [321:53] what about uh what they were trying to [321:55] get. Yeah, you can see that they are [321:57] trying to go to some admin thing. Uh [321:59] yeah, I'm not familiar with PHP, but we [322:02] can uh take this and see if that will [322:05] get blocked or not. [322:07] put it here and yeah that worked because [322:10] we have PHP at the end and um yeah we [322:14] can also I think we are also capturing [322:16] some Python stuff let's see mainph [322:20] that also gets blocked but let's try [322:23] with Python yeah that works amazing so [322:26] yeah the list can be can be big uh I [322:29] just try to you can analyze the the logs [322:33] to see uh the files that the bots are [322:37] trying to access or you could search [322:39] online. There are many blacklists that [322:42] you can find to add to your engineext [322:44] configuration. If you want, you can uh [322:47] use one of these GitHub repositories to [322:51] uh block uh bad bots. And here uh so [322:55] yeah, this is a repository. They give [322:58] you some text files that you can just [323:00] put on your engineext configuration. Uh [323:03] so yeah they had a few stuff here that [323:06] uh that you can just include in your [323:08] configuration. You can read more about [323:10] this and you can also uh come here on [323:13] this other repository and use it. They [323:16] have plenty of uh files here that you [323:18] can check. But this is something that [323:21] you can also configure yourself just uh [323:24] by reading the yeah by reading the logs. [323:27] You can also use LLMs. they will be very [323:30] helpful in this uh step. You can explain [323:33] to them your stack so that they don't so [323:37] that they can help you uh block [323:39] everything but u yeah don't touch some [323:43] some things that will make your [323:45] application not load or break. Now we [323:48] have one more thing that I want you to [323:50] learn and that will involve this tool N [323:55] map the network mapper. It's a free [323:58] utility that we can install uh on Ubuntu [324:02] and other Linux dros. You just run um [324:07] sudo app install end mapap and you will [324:10] get it. Uh I already have it so there is [324:12] no need to download it. And uh let's [324:15] verify the version. We have 7.95. [324:19] That's good. And to use it, we are going [324:22] to run this command here. I will I'll [324:26] type my domain name. And let's see what [324:29] we have. Now end mapap is going to [324:32] start. The f flag will run a fast scan. [324:36] And this is going to check just 100 [324:39] ports. And uh one one thing that I want [324:43] to mention is that end mapap you should [324:45] run this command on your local computer [324:48] uh not uh on the on the virtual machine. [324:52] Uh you can see that I have opened [324:54] another terminal tab and let's see what [324:56] we have here. So we have 97 and here we [325:00] have another three ports. The the sum [325:02] gives us 100 which confirms the theory [325:05] um or what I explained about this F [325:08] flag. Now let's analyze the the output. [325:10] You can see that here uh we have not [325:13] shown 97. These are other ports that are [325:17] not available or that are not accessible [325:19] from the out from the basically the [325:22] public internet. So that's that's a good [325:25] sign. And after that we have three ports [325:28] which are this is what I explained uh in [325:30] the beginning. We want to allow just [325:33] HTTP, HTTPS and SSH. We don't want to [325:37] open port 8080 or 7,700. These are ports [325:41] that we are using uh in that I am using [325:44] in my applications uh to basically [325:47] access M search fast API and also uh [325:52] view but those are running uh but they [325:55] are p uh they are they're not accessible [325:58] from uh the outside and uh as I showed [326:01] you in the previous tutorials if you [326:04] want to access those you don't open a [326:07] new hole in the firewall instead Use SSH [326:11] tunneling to open a temporary tunnel [326:14] between your local setup and the virtual [326:17] machine. Do your work and then close [326:20] that uh tunnel. At this point, you were [326:24] able to deploy your application [326:27] manually. You have u learned to how to [326:31] deal with uh bad actors that are trying [326:34] to access your website. you learn how to [326:36] optimize the load time and plenty of [326:39] other skills that are going to be very [326:42] um beneficial. [326:44] We saw you saw that we are using digital [326:47] ocean but you can take the concept that [326:49] you have learned here and use those with [326:52] other cloud providers. But this is the [326:55] first step. Now we are going to enter [326:58] another phase where we are going to [327:00] automate a lot of stuff. We are going to [327:04] create an automation pipeline that will [327:07] allow you to make changes to your [327:10] application locally. Maybe develop a new [327:13] feature uh fix a security problem and [327:17] instead of uh taking the source code, [327:21] copying it from your local computer to [327:24] the virtual machine, do the build [327:26] process and um install the dependencies [327:30] and then serve your application. We you [327:33] are going to automate this process. You [327:35] are going to learn about CICD which [327:39] stands for continuous integration and [327:42] delivery which will allow you to click [327:44] on a few buttons and those buttons will [327:48] run jobs uh on your behalf. Those jobs [327:52] will check for security problems. They [327:55] will check if uh dep the dependencies [327:58] that you are relying on are secure. They [328:00] will build your front end. They will [328:03] shut your backend application, reload [328:06] the new changes if there are any and um [328:10] other other kinds of job and you will [328:12] just sit there and watch uh your new [328:15] changes uh live become live on your [328:18] website and that will take you a few [328:21] minutes to do instead of hours of manual [328:24] labor. In the in that next module, we [328:27] are going to start slow. In the first [328:30] sub chapter, I will show you how to put [328:33] this code on GitHub. I assume that you [328:36] already have one or you can also use [328:38] other websites like GitLab or uh [328:42] Codeberg or yeah any any g compatible [328:45] client. Uh we'll show you how to protect [328:49] your main branch so that you don't push [328:52] code accidentally to it. We need to run [328:55] that pipeline to ensure that everything [328:56] is secure, everything is correct before [328:59] merging it to main. And after merging, I [329:01] will show you how to do the deployment. [329:04] But that will come uh in the next sub [329:06] chapters. So here we just want to focus [329:08] on the code quality. I'll show you how [329:10] to use the rough library because here we [329:13] are using Python to uh standardize to [329:17] run linting and formatting in order to [329:19] standardize the code that we write. And [329:23] on the front end side you will use [329:25] eslint which is the the default one. [329:28] Okay we will uh focus on formatting the [329:31] code and catching linting errors in the [329:33] next uh tutorial. See you next time. [329:37] Welcome to module five titled the [329:40] automation pipeline. In the previous [329:43] modules you were able to deploy the web [329:47] application manually. We have started by [329:50] copying the source code from your local [329:54] machine to the VM. You have been able to [329:58] install the dependencies, build the [330:00] front end and other stuff. This is [330:02] great. It's important to do things [330:05] manually, but you should not continue [330:07] like this because if you add features to [330:10] your application, you want to do the [330:12] deployment automatically. you you want [330:15] to focus on writing the code and let the [330:18] this automation that we are going to [330:20] build in this module do the work for [330:22] you. Here is what we are going to talk [330:25] about in this sub chapter. First I will [330:28] show you how to protect the main branch [330:31] using GitHub rule sets. In this tutorial [330:35] I am using GitHub but you can do the [330:37] same thing if you are using GitLab or [330:40] other uh websites. I will show you how [330:43] to automate local checks using [330:46] pre-commit. And for Python, we are going [330:49] to use the rough project to do linting [330:52] and formatting. And for JavaScript, we [330:55] will be using ESLint with prettier. If [330:58] you use other other programming [331:00] languages, look for the projects that [331:03] are there and use them. The concepts are [331:05] the same, but we are using different [331:07] technologies. [331:08] This is the repository that I was [331:11] talking about a lot during the previous [331:13] modules. Here you can find everything [331:15] related to the to the book that I am [331:17] writing and these the slides everything [331:20] is inside this repository and I want to [331:23] show you this is my this is my other [331:26] GitHub repository where I host my where [331:29] the source code of my personal website [331:31] and I want to show you what will what we [331:34] will achieve by the end of this module. [331:36] You can see that here is the automation [331:39] that I was talking about. You have [331:41] certain jobs that you run here on [331:44] GitHub. They run whenever you push code [331:48] and they do a lot of things for you. For [331:50] example, here this job will check my [331:53] dependencies. It will look at the Python [331:55] libraries. If it sees that there is some [331:59] security vulnerabilities, it's going to [332:01] stop the code that I pushed and it's not [332:03] going to be merged to the main branch. [332:06] This will protect you from deploying [332:09] code that is not secure. Uh here we are [332:12] performing linting and formatting. In [332:15] this video we are going to do this [332:16] locally but we can also do it here on [332:19] GitHub actions. You can run tests. We [332:23] have different kind of tests here. Here [332:25] you can see that we have unit tests, [332:27] integration tests and end to end test. [332:30] Finally, if everything goes well, we [332:33] will build the front end and the [332:35] pipeline succeeds. And we we can add [332:37] other actions that will allow us to [332:40] build the front end and deploy it. For [332:43] example, if I look at this one here, as [332:46] you can see, I have uh plenty of jobs or [332:50] workflows. And here I have this one, [332:52] manual deploy. after merging a pull [332:55] request to the to the main branch. I [332:58] come here and run the manual deploy [333:01] because here as you can see we just [333:03] built the front end because we are 100% [333:06] sure that there is no problem with the [333:07] code because it uh it passed all those [333:10] checks. Now I come here I run the job [333:13] and here it notifies me sends an email [333:15] telling me hey do you want to deploy? I [333:18] just click on a button to approve that. [333:21] And as you can see it says deploy and I [333:23] can check the website. This is what we [333:26] want to do. And we don't need to SSH to [333:29] the machine, run commands manually. [333:31] Everything is going to be done for us uh [333:34] in this interface. And what we do is we [333:38] just create a pull request. We check the [333:40] code. And here you have the checks. So [333:43] for example, here it says all 17 checks [333:46] have passed. If that's the case, we [333:48] merge. But if a check fails, it's not [333:51] going to be it's not going to to let me [333:53] merge until I fix it. But for now, let's [333:56] just focus on this branch. It's brand [333:58] new. I haven't set up anything. We are [334:01] going to do this uh together. The first [334:04] thing we want to do is to secure the [334:06] main branch. I also have my code editor. [334:09] Here it is. And as you can see, if I [334:12] click on this icon, it shows me the [334:15] changes that I have here. You can see I [334:17] I can come here uh add this and push. [334:21] Nothing is go no one is going to stop [334:23] me. But what what if I what if I don't [334:26] want that? Let's say that you are [334:27] working with a team of people and you [334:30] want to do a pull request before [334:32] merging. In that case, we want to [334:34] protect this so that you don't push [334:36] domain accidentally. But to fix this, we [334:39] come here, we click on settings. Let me [334:42] zoom in a little bit. Here we click on [334:45] branches and we add a branch rule set. [334:49] Here we need to give the rule set a [334:51] name. Let's call it main branch [334:54] protection. Here it currently it's [334:57] disabled. We want to enable it. Let's [335:00] choose active. Let's scroll to the [335:03] target branches section. Click on add [335:06] target. Let's select include default [335:10] branch. Let's go back to the repository. [335:13] As you can see, my default branch is [335:15] main, but in some cases it could be [335:17] master. That's why here it shows default [335:20] instead of uh main or master. And here [335:24] we have some rules that we can apply. [335:26] You can read through them. And here for [335:29] example, let's scroll down to restrict [335:33] deletions. This will prevent anyone from [335:36] deleting the main branch. That's why we [335:39] want to restrict it. Here we have [335:41] another rule that we need to keep. It [335:43] says block force pushes. This will [335:46] prevent you from writing from using git [335:49] push-force [335:51] because when you use force you rewrite [335:53] history and that can lead to problems [335:56] here. These two are uh checks by [335:58] default. We want to add this third one. [336:01] Require a pull request before merging. [336:05] If you are working with a team, required [336:08] approval should be either one or two. It [336:10] depends on the size of the team. But if [336:12] you are working as a solo developer, [336:15] select keep this at zero. I am I am [336:18] doing it on my other project. Why? [336:21] Because it builds the habit of uh [336:23] creating a branch for each feature you [336:25] work on. And if you here you have to do [336:28] zero because you are the only one who is [336:30] working on this project. And it is very [336:32] helpful. is going to make you think of [336:36] always working with making small [336:38] changes, creating a pull request and [336:41] merge it to main like I do with my other [336:44] projects. We will come back to this uh [336:47] in the upcoming sub chapters because we [336:50] don't want to do everything in this in [336:51] this first video, but we just want to [336:53] get up and running. Now, let's click on [336:56] create. Here it's going to tell you you [336:58] need to sign in. I will do this and I'll [337:00] come back. I am back. You can see it [337:03] says rule set created and here it is and [337:06] it it says applies to one target which [337:09] is the main branch. This is what we want [337:12] to see. Now if I go back to the [337:15] repository you can see that this is our [337:18] main branch. If we click on view all [337:20] branches and here you can see that we [337:23] have we can also view the rules from [337:25] this button. And yeah we have three [337:28] branch rules. That's correct. But let's [337:30] go back to VS Code and let's see if we [337:32] can push something to this codebase from [337:36] main. Here is the repository. Let's make [337:39] some change to this file. I will come [337:41] here and let's just add an empty line. [337:45] You can see that we we have modified the [337:48] file. I will stage it and let's create a [337:51] commit. Get commit- m let's call it [337:56] test. Hit enter. Okay, that worked. But [337:59] now let's try to push this get push. As [338:03] you can see I was not able to do that. [338:06] Let's see what is the error. You can see [338:08] that here it says repository rule [338:10] violations found on the main branch. [338:14] Here it says changes must be made [338:16] through a pull request. This is what we [338:18] want to see. So remote rejected our push [338:22] operation. Amazing. And to undo this [338:25] commit because it was just test, you can [338:27] run git reset add. And here we just want [338:31] to reset the latest commit. Let's do it. [338:34] And as you can see, uh, it was removed. [338:37] And it shows me that this is the last [338:38] modification I made to this file. Great, [338:41] it's working. So, let me just undo this [338:43] because I don't want to make that [338:45] change. [338:46] You have successfully protected the main [338:49] branch. But we still need to check the [338:52] commits that we are pushing. What do I [338:55] mean when I say we need to check [338:56] commits? Sometimes you might be working [338:59] on your project and you might [339:01] accidentally commit let's say a secret [339:03] key. That's that's a problem because [339:06] that will that will sit on the git [339:08] history and everyone can see it unless [339:11] you rotate the key. With something [339:14] called pre-commit, we are able to catch [339:16] this problem before pushing the commit [339:18] to the remote repository. We can do [339:22] other checks. We can check if the [339:24] linting is correct, if the code is [339:26] formatted correctly and other stuff. [339:29] This concept is called get hooks and we [339:32] are going to target pre-commit hook [339:35] specifically. So when you type get [339:38] commit, this hook will intercept uh your [339:42] request. Basically, it's going to see [339:44] what you are what you are committing and [339:46] it's going to run a series of checks and [339:49] you can add uh as many checks as you [339:52] want. If everything passes, great. The [339:55] request will go through and your commit [339:57] will be deployed to the cloud or yeah to [340:01] the cloud repository. But if something [340:04] if a check fails the the hook will going [340:08] to block you unless you fix the problem. [340:11] Okay. So this is what we what we are [340:12] going to do uh now and we are going to [340:15] use this amazing project called [340:17] recommit. Let's go back to VS Code to [340:20] show you how to use this. Here you can [340:23] see that I have some changes made to [340:26] this get ignore. I'll try to stash that [340:29] because I don't need it. Now I need to [340:33] add this specific file [340:35] pre-commit-config.yml. [340:39] Here we are going to add the checks that [340:41] we want to perform. Let's now start with [340:44] the back end. You can see that here we [340:46] don't have any Python code but we will [340:48] add one to test this. Let's just add [340:51] main.py and let's create a a function [340:56] that will print hello world. We can run [340:59] this and it should work. So if we do [341:03] main.py, [341:05] as you can see, we get hello world. We [341:07] are doing this just to test this hook. [341:10] We have two kinds of texts that we can [341:13] use. You have remote repositories and [341:16] you have local custom hooks that you can [341:19] create on your own. For example, here if [341:22] I click on this link, it takes me to [341:25] GitHub. You can see that people who are [341:28] working on rough have created this [341:31] pre-commit that I can just install by [341:35] just putting the link of the repository [341:38] here. So this means when I run install, [341:41] it's going to go to this repository. [341:44] It's going to bring this version. How do [341:47] I know if this is the latest version? [341:49] It's simple. Just go back to the [341:52] repository. Scroll down and here click [341:55] on releases and that will show you the [341:57] latest release. Currently we uh in here [342:01] in rough the latest version is 0.15.12 [342:05] and this is exactly what I have [342:07] specified here in this field and here [342:10] you put your hooks what you want to run [342:13] here I have two kind of checks I have a [342:17] linting step and a formatting step. [342:20] Let's focus on the first hook. You can [342:23] see that the ID is rough and I give it a [342:26] name and I specify that I want to [342:27] perform LinkedIn. And here I have I am [342:30] passing additional arguments. And here I [342:32] pass d-fix because sometimes there are [342:35] some some things that we can fix in the [342:38] code. And here I am targeting everything [342:40] inside the backend folder. So let's do [342:44] that. Let's let's create a backend [342:46] folder. And let's drag this file and put [342:49] it there. Otherwise, this hook will not [342:52] find anything. It's going it's going to [342:53] succeed. But we want to target only u [342:58] Python files that are inside the backend [343:00] folder. In the second hook, we are just [343:02] performing formatting. Uh it's simple. [343:05] This file is is good. There is no [343:07] problem in it. Now, if I try to use this [343:11] hook, it's not going to fail. Let's see [343:13] how to install this. You have different [343:15] options of installing it. You can use [343:17] Anaconda, you can use poetry, uv or any [343:21] package manager in Python or you can [343:24] simply use the ven module. Uh let's try [343:28] let's try that. Then ven as you can see [343:31] we have our virtual environment and [343:34] let's activate it. Source van bin [343:38] /activate. [343:40] As you can see here I have van and I am [343:42] using python 3.12 here we need to add [343:46] dot getit ignore so that we don't push [343:48] what we have here and let's put van here [343:52] okay good now we have just three [343:53] modifications now let's install precomit [343:56] to do that make sure that the [343:58] environment is activated and run pip [344:01] install precomit [344:04] great that worked and now we need to run [344:06] precomit install you can see that it [344:09] says pre-commit installed at this [344:11] location. Let's controlclick on this [344:13] file just to see what we have here. You [344:16] can see that this file was autogenerated [344:18] by pre-comit and here we have this [344:20] variable install python that points to [344:23] the virtual environment. The problem [344:25] with this approach is that you can see [344:27] that we have installed precomit on this [344:31] project. So if I move if I delete this [344:33] virtual environment, the pre-commit hook [344:36] will fail. It's not going to find this [344:38] path and it's not going to work. But if [344:41] you stay here, it's going to work [344:43] without any issue. But we have other [344:45] options that we can try to fix this [344:48] problem because ideally we want to [344:50] install pre-commit globally so that we [344:52] can use it in any Python pro in any [344:55] projects not just Python and that way we [344:58] will fix these issues. So using a [345:01] virtual environment is perfectly fine. [345:03] The only issue that you will face is [345:06] when you decide to delete the virtual [345:08] environment. But if you don't then you [345:11] can keep using this or we have another [345:14] option which is to install pre-commit [345:16] globally with pep x. This is another [345:20] python library that we can use and we [345:22] have plenty of options to download it. [345:25] Here I am using because I am on iuntu I [345:27] am using apt to install this from the [345:30] repository. Uh I have already installed [345:33] it. So, it's not going to add anything, [345:35] but I just want you to go to GitHub, you [345:38] will find the pipex there. As you can [345:40] see from the about section, it says [345:43] install and run Python applications in [345:47] isolated environments. And this is what [345:49] we want to do. I am not using the [345:52] regular pip because I don't want to [345:54] install this globally because it's not [345:57] it's a bad idea. That's why we use [345:59] virtual environments. But PEX solved [346:03] this issue for us. And if you scroll [346:05] down a little bit, you can see here in [346:08] the image, it shows you that you can [346:10] install libraries and run them in [346:13] isolation. And you have plenty of [346:15] options to install this here on Linux. [346:18] You can use pip if you want. And uh I [346:21] decided to use apt. But I just want you [346:24] to go here to in to look at your current [346:27] OS to uh get the perfect command for [346:31] your system. After installing pipex, you [346:34] need to run pipex insure path. And [346:37] that's that's the second command here. [346:40] Let's do it. As you can see, it says [346:42] that it already found the path. So [346:45] that's fine. And if you see this [346:47] warning, no problem. uh the the message [346:50] that let tells us that if it is working [346:52] is this otherwise pipex is ready to go. [346:55] If you see this, you are good to go. [346:57] Let's clear the page. And now we run pip [347:00] x install pcomit. As you can see, I have [347:04] already installed it before. That's why [347:06] it didn't work. So it tells me that it [347:08] seems that this library is already [347:11] installed. So that's fine. And as you [347:13] can see here is the location of that [347:17] specific library. Now all we need to do [347:20] is to go back to VS Code. You can see [347:24] that here I have activated this virtual [347:26] environment. Now I need to deactivate. [347:29] Let's type deactivate. Good. The virtual [347:33] environment is is no longer there. And [347:35] we can also delete it because we uh I [347:38] want to show you that this is going to [347:39] work without a virtual environment. You [347:43] can see we have removed that folder. And [347:46] I can also remove the changes from git [347:49] ignore. We need to do the same command [347:52] pre-commit install. I'll hit enter. As [347:55] you can see, this worked. And in this [347:57] file, you can see that the path has [347:59] changed. Now the path is pointing to the [348:03] pipex environments. This is what we [348:05] want. Now, as you can see, I don't have [348:07] a virtual environment. That's fine [348:09] because this is installed globally. Now, [348:12] I can open another project, run [348:14] pre-commit install, and it's going to [348:17] use the same virtual environment. That's [348:19] great. And this solves the issue with [348:22] local virtual environments. Great. Now [348:24] we can continue. First, we need to [348:27] commit this file. This is the first [348:29] operation that we need to do. So, let me [348:32] commit this. I'll click on the commit [348:34] button. Great. It works. The code uh I [348:38] have I have been able to create the [348:39] commit. Nothing has stopped me. Now, [348:41] because we have this specific hook that [348:44] targets Python files, if I try to commit [348:48] this, it this these hooks are going to [348:51] check the file. And I'm going to do this [348:53] in the in the terminal because it's [348:55] going to show us what is happening. [348:57] Okay, let's do get commit dash m and [349:01] let's say test python hooks and let's [349:06] hit enter. As you can see the first [349:10] check passed. So linting was good but [349:13] formatting was not good. So here it says [349:17] failed and we as you can see it says one [349:20] file reformatted and the commit is not [349:23] going to go. The file has been changed [349:26] and as you can see here is the change [349:27] that was made and we are using this to [349:30] make sure that formatting is consistent. [349:33] If you work with a team, you want the [349:35] formatting to stay consistent because [349:38] otherwise it's going to be very hard to [349:40] review put requests because they are [349:42] going to make changes with different [349:44] formatting and and that's not a pleasant [349:47] uh situation to be in. This is why we [349:49] use pre-commit hooks. And as you can [349:51] see, we were not able to push this. And [349:53] if everyone is using the same set of [349:56] rules, it ensures that the code uses the [350:00] same formatting rules and everything [350:02] stays the same. The only thing I need to [350:04] do is to add this again to add the [350:08] modifications. Let's try to commit. And [350:11] voila, it worked. You can see that uh [350:14] both checks have passed and I was able [350:17] to create the commit without any [350:20] problem. As you can see, I can click on [350:21] the button to push these two commits. [350:24] But this is not going to work because I [350:26] am on the main branch. We can test it [350:29] again. It's fun to test stuff like this. [350:32] So, let's do get push. And it was [350:35] rejected. Great. So, this confirms that [350:37] it it's working. Awesome. So, this was [350:40] just an example. But now let's go back [350:42] to my other repository in order to show [350:45] you the pre-commit hook, the pre-commit [350:48] configuration that I have there. It's [350:51] totally different from this one because [350:53] we have JavaScript and Python. Here is [350:57] my other project. And if you remember, [350:59] this is exactly the code that I told you [351:02] to download if you didn't have your own [351:05] projects to work with. And here here is [351:08] my configuration. We have plenty of [351:10] things but you can see that the first [351:12] part is similar to what I have showed [351:15] you in the other project. We are using [351:18] rough and we are we have two hooks one [351:22] for linting and the other one for [351:24] formatting. Let me collapse this because [351:26] we have seen it here. I have another [351:29] repository and this is used for [351:32] JavaScript. Here we are using eslint [351:35] because we want to do linting. We want [351:37] to verify that the syntax in our [351:40] JavaScript code is correct and that [351:42] there are no problems. As you can see [351:45] here, we have just one hook and uh I've [351:48] called it yes link front end. Here I [351:51] have files. It's going to target [351:53] everything inside the front end folder. [351:57] This is a regular expression that will [352:00] target only view and JS files inside the [352:04] front end directory. If I open source [352:07] you can see that here view files end [352:10] with dot view that's why I have it here [352:13] and I also have js here I am not using [352:16] typescript if I was using it I will [352:18] change js to ts so the types that uh I'm [352:22] looking for files and here I also add [352:25] arguments so that I can fix problems if [352:28] I find them and I also point the conf to [352:31] the configuration file because yes lint [352:33] can be configured and here is a [352:35] configuration file. So here if you want [352:38] to disable some rules, you can uh come [352:40] here and modify the rules that uh you [352:44] want your hook to check for uh or to [352:48] skip in this case. And here I have [352:50] additional dependencies. These are these [352:53] will change. It depends on the project [352:55] that you are working on. If you aren't [352:57] working with Vue, you won't you won't [352:59] add this. So yeah, these are just [353:01] additional dependencies that will be [353:02] installed in order to run this hook. So [353:05] this is the front end part and we also [353:08] have some local repositories or local [353:11] hooks that we create on ourselves. This [353:14] is what why I mentioned before that you [353:17] can use either hooks created by the [353:20] community or use your own. For example, [353:23] here I have two local ones and here I [353:26] will you can see that I I call this one [353:28] prettier because with prettier I can [353:30] check many files. I can check JS view, [353:34] CSS and other file extensions and uh [353:37] same if I detect that some format [353:40] something is not formatted well I I add [353:43] the dash dash write to change the file [353:46] and uh after that I can just come here [353:48] and add click on the plus button to [353:51] accept the change and finally I have [353:53] this other hook that will automatically [353:56] generate the site map for my website and [353:59] this will run whenever I commit and here [354:01] you can see that I just have a simple [354:03] bash script. I go inside the front end, [354:06] run the sitemap. So this is a script [354:09] that I have and it's going to generate [354:11] this XML file that I will stage. Using [354:15] pre-commit is really easy and it helps [354:18] you catch a lot of problems locally in [354:22] before deploying or publishing the the [354:25] code changes. And we can also do a [354:28] simple test here. Let's let's open app [354:31] doview view and let me add some spaces [354:34] some new lines here that will trigger [354:37] prettier and it's going to block this [354:38] operation. Let me open my terminal and [354:41] let's check the status just to confirm [354:44] that everything is all right. Let's add [354:47] this. Okay, now we can write the commit [354:50] message. Let's say uh let's hit enter. [354:56] As you can see that works here. rough is [354:59] not failing. It is skipped because we [355:01] haven't changed anything uh in the back [355:03] end but uh we have changed something in [355:07] the front end directory. Yes, lint [355:09] worked. That's not a problem. But [355:11] prettier didn't like the changes that we [355:13] have made. And because we have set [355:16] rights, you can see that it changed the [355:17] file. So, it removed those new lines for [355:20] us. That's great because I can just come [355:22] here verify the new changes and click on [355:25] add. And yeah, I can come let me also [355:30] you can see that sitemap was generated [355:32] for us automatically. That's good. I can [355:36] come back and uh run it. And as you can [355:38] see this time it worked because we fixed [355:41] all the issues. And that's it for this [355:44] video. Your local environment is now [355:46] fully secure. You have protected the [355:49] main branch and you have created [355:52] pre-commit hooks to check commits before [355:56] pushing them. In the next sub chapter, [355:59] we are going to talk about continuous [356:02] integration. We are going to perform [356:04] automation using GitHub actions that [356:07] will allow us to write workflows that we [356:10] can that we can run after let's say [356:14] creating a pull request that will allow [356:16] you to do something similar to what I [356:18] have here. You can define your jobs and [356:23] uh put them in a pipeline that will run [356:26] uh automatically and if everything [356:27] succeeds you can merge your pull [356:29] request. This is going to take us a lot [356:32] of time to develop. But I'll try to go [356:34] uh step by step. I'll try to go slowly [356:37] so that you can understand everything. [356:39] Uh once you understand this, once you [356:41] make one or two jobs, then the rest is [356:43] going to be repetitive. You will uh then [356:46] you will know how to add uh anything. [356:48] And you might be asking, do I need to [356:51] follow a specific rule? You can. No, you [356:54] can design this however you like. Just [356:57] make sure that you catch the things that [356:59] you want to focus on. For example, if [357:01] you want to do security, you need to add [357:03] some jobs that will test if your app is [357:06] secure. If you want to check LinkedIn [357:08] and formatting, you need to add jobs [357:09] that will check that. So each job is [357:12] going to check one thing for you. And by [357:14] building uh the these sets of jobs, you [357:17] will ensure that uh the code that you [357:20] push to production is secure. Uh there [357:23] are and there are no bugs no bugs in it. [357:26] See you in the next video. Welcome back. [357:29] In this video we are going to talk about [357:32] continuous integration. In the previous [357:35] video you have seen how to perform [357:38] checks locally to stop bugs or [357:42] unformatted code from being synced with [357:46] the remote repository. Today we are [357:49] going to do the same thing but on GitHub [357:52] or you can do the same thing on GitLab [357:54] or other platforms. The same concept [357:57] applies but there are some uh things [357:59] that might change from platform to [358:01] platform. Here is what we are going to [358:03] talk about in this video. We are going [358:06] to learn about continuous integration. [358:08] You are going to build modular and [358:11] reusable workflows to automate linting [358:15] formatting and to perform unit tests. [358:19] And of course, we want to block bad code [358:21] from being merged to the main branch by [358:25] using GitHub status checks. In this [358:28] session, we are going to build a small [358:30] pipeline that will do the following. You [358:34] can see that here. This is specific to [358:36] GitHub but you can find the same concept [358:39] on GitLab also or other platforms. Here [358:42] we have five jobs. You can see that we [358:45] have two jobs that we run to test the [358:49] back end and we have the same that but [358:52] for the front end. The first step is to [358:55] perform linting and formatting and after [358:58] that we perform unit tests. And you can [359:01] see by default GitHub will run jobs in [359:05] parallel but some jobs can depend on [359:08] each other. For example, before I run [359:11] the unit tests in the back end, I need [359:14] to make sure that there is no syntax [359:16] error and the code is formatted [359:18] correctly. If this is not the case, this [359:21] job will be skipped. It's not going to [359:23] run. The same the same goes for the [359:25] front end. Finally, we have this [359:27] pipeline success which is going to be [359:30] used to allow us to merge the pull [359:35] request or to uh block it. You can see [359:38] that in this case everything passed that [359:40] mean that's why I get a green check mark [359:43] which means that I can merge this pull [359:46] request. If let's say the lint failed [359:48] this is going to turn red and it's going [359:50] to block the mer the pull request. We [359:52] will keep adding more jobs to this [359:54] pipeline in the upcoming videos. But for [359:57] now, let's just start with these two. [359:59] This is what we have performed locally [360:01] in the previous video. But on top of [360:03] that, we are going to add unit tests. I [360:06] am now on GitHub and I I switched to my [360:10] secondary account. Here I have forked [360:13] the source code that we used to deploy [360:17] the website on digital ocean because I [360:20] wanted to experiment here uh without any [360:23] problem. So I have forked this and now I [360:25] will create I've already created a pull [360:28] request but we will start from scratch [360:30] and I will show you how this works and [360:33] how to build a GitHub action. First [360:37] let's start by creating a branch. Here [360:40] we are using VS code. I will just click [360:43] on this button that will allow me to [360:47] create a new branch from a specific [360:50] branch. And here I am creating it from [360:52] master. I will call this test number [360:55] two. I will hit enter and I will select [360:58] head. I will switch to that branch [361:01] because remember the master branch is [361:04] protected. I cannot push directly to [361:08] that branch. If you are using GitHub, [361:10] you start by creating agitub [361:14] folder. You can see I already have it [361:16] here and we have plenty of things. But [361:19] to create a pipeline, we put our [361:21] workflows inside the workflows folder. [361:25] Let's open it. You can see that I have a [361:28] lot of things. Don't worry about these. [361:30] Don't worry about this. Uh what you are [361:32] seeing here is the final product. Uh we [361:35] are going to build these step by step. [361:38] I'll go through these and remove them [361:40] and I will keep just the ones that we [361:41] need for this video. I am back. As you [361:44] can see, I have removed a lot of files [361:47] and let's start by working on linting [361:49] and formatting in the back end. If you [361:51] want to learn more about this, you can [361:54] search GitHub actions. As you can see, [361:57] you can search GitHub actions and you'll [361:59] find you will find them here. Their [362:02] documentation is great. So if you want [362:05] to dive deeper, you can use this or you [362:08] can just take the documentation and uh [362:12] use that with your preferred large [362:14] language model and they will help you uh [362:16] understand the concept. But I'll try to [362:18] also do my best to explain to you how [362:20] this works. Our goal is to build a [362:24] modular and robust pipeline. Which means [362:27] that we want each job to be contained in [362:30] its own file. And we call these reusable [362:34] workflows. For example, here I am [362:37] hovering over the backend lint and [362:39] format. This is going to be one file [362:42] which is this one. The code that we will [362:45] put here will run at this step. And [362:48] after that this file ci will be the [362:52] orchestrator. It's going to call these [362:54] uh workflows independently one after the [362:57] other. And we don't want to put [362:58] everything inside one giant file because [363:02] if you get an error, it's going to be [363:04] very hard to debug. But by putting [363:07] everything inside a its own file, it's [363:11] going to make life easier and you will [363:13] see how the the orchestrator is going to [363:16] be very minimal. It's going to be very [363:18] easy to read and debugging is also going [363:20] to be fun. To create a workflow, you [363:23] start by giving it a name. Here you can [363:26] put anything and it's going to be easier [363:28] for you to locate. For example, if we [363:32] let me go back to GitHub, you can see [363:35] that here we have this actions tab. If I [363:38] click on it, uh you can see that we have [363:40] plenty of workflows and you can see that [363:42] they're named uh they are named. Let's [363:44] look at this one. You can see that it's [363:46] called backend lint and format. Let's go [363:50] back to GitHub and we we find it here. [363:53] back end lint and format. You can click [363:56] on it, you will see the how many times [363:58] it we run it, etc. Uh, okay. And [364:02] sometimes you can even run them manually [364:04] if you want to. Let's look at the [364:05] workflow or the orchestrator. You can [364:08] see that I can come here and run it [364:10] manually if I want to. We will I will [364:12] show you how to implement that. But this [364:14] is why we give them names so that it [364:16] becomes easier for us to uh find them [364:19] here in the actions tab. After that we [364:22] add the onb block and here we specify [364:26] workflow call. This tells GitHub that [364:30] this file is a reusable template not a [364:34] standalone script which means we cannot [364:35] run it on its own. We need to call it [364:38] from the orchestrator. After that we [364:41] create a series of jobs that we want to [364:43] run inside this specific workflow. And [364:47] here we want to perform just lint and [364:49] formatting. That's why we you can see [364:52] that here we have jobs and here you can [364:54] specify your jobs and here we are going [364:56] to run one one specific job. We are not [364:59] going to run more than that. Inside the [365:02] lint and format job we can also give it [365:06] a name and runs on is very important. [365:09] Here you can choose different operating [365:11] systems. Here I am using iuntu and [365:14] latest will take the la the the recent [365:18] version of Ubuntu. Why are we doing [365:20] this? Because remember we are going to [365:22] run this on the cloud. It's not going [365:25] the these checks are going to be [365:27] executed on GitHub's own machines. [365:31] That's why we need to specify a an [365:34] operating system. After that we add [365:37] defaults run and we specify the working [365:40] directory. We are doing this to make our [365:42] life easier because if we don't add this [365:45] we have to always type cd back end [365:49] whenever if we want to perform if you [365:51] want to perform something inside the [365:53] backend folder we need to always do that [365:56] but by putting this GitHub will [365:58] automatically move us to the backend [366:01] folder and we can run the commands [366:03] without always repeating cd space back [366:07] end and finally we put the steps A job [366:11] is a series of steps and here you can [366:14] you see the the template is the same. [366:16] Everything has a name. You can see that [366:18] here we have name checkout repository. [366:22] This is going to be very helpful. If you [366:25] go to your pipeline if you click on this [366:29] backend lint and format you see here are [366:31] the steps and let me split the screen [366:34] and let's look at let's look at this. [366:36] You can see that here we have checkout [366:39] repository. This is exactly the name [366:41] that I have here. Here we have setup [366:43] Python. We also find this here. And why [366:46] is this important? Because let's say [366:47] that this step has failed. You can [366:50] quickly go back to your YAML file and [366:54] find that specific step. It's going to [366:56] make your it's going to make debugging [366:59] very easy. And as you can see, each step [367:01] is going to run. And here they're going [367:03] to run sequentially one after the other. [367:05] And if a step failed, the other ones are [367:08] going to be skipped. So let's look at [367:10] what we have here in this specific job. [367:13] First we need to check out the [367:14] repository. Because we are running on a [367:17] virtual machine, we need to download the [367:21] source code. And here we have something [367:24] that we call actions. You can see that [367:26] here we have two different keywords that [367:29] we are using. Here we have users and [367:31] down below we have run. Basically this [367:34] keyword is going to use scripts that the [367:38] community has made. Let me show you what [367:41] I am talking about. If I copy this name [367:44] and search for it, let's see what we are [367:47] going to get. You will find that there [367:49] is a repository that contains the code [367:52] for that specific action that we want to [367:54] run. And uh you can see that here here [367:56] is here it is actions/checkout. [368:00] If I go back, it's the same name. and [368:03] let's verify the version. How do we do [368:05] that? It's simple. You click on the [368:07] releases page and here you find the [368:10] versions. So the latest version is 6.0.2 [368:14] but I am using version 6. And this is [368:17] important because sometimes these [368:19] actions change and uh if they introduce [368:22] breaking changes you can read more about [368:25] them here on GitHub. But if you use run [368:28] this means you will provide uh commands [368:32] that the uh that Ubuntu will run. This [368:36] is the difference between users and run. [368:38] Okay. So when I do this, let's look at [368:41] here on uh this job. Let's open this. [368:45] You can see that we this action has been [368:48] executed and now it is syncing the [368:50] repository. And yeah, you can open this [368:53] and read more about this. You can see [368:55] that they are initializing the [368:56] repository and fetching the repository. [368:59] This is what we want to see. After this [369:02] step, the source code is on this Ubuntu [369:04] machine. And now we can run commands in [369:06] it. Setup python is going to download [369:09] Python for us. And uh let's uh let's see [369:13] here. We install rough. You can see that [369:15] we use pip to install that library. And [369:18] it worked. After that, we run the [369:21] llinter and formatter. We can see that [369:24] everything is great. Every there is no [369:26] problem. And here these are jobs that [369:30] GitHub will run. We can see that we [369:32] don't have these uh post setup Python. [369:34] It's not here. But the this these are [369:36] just steps that are uh run to clean the [369:40] machine because we are done with it and [369:42] uh there is no need to keep this there. [369:45] And you might be wondering how many [369:47] steps do I need to put here. You can put [369:50] as many steps as you want. Uh you feel [369:52] free to customize your jobs however you [369:56] like and um yeah there is no constraint [369:59] know the difference between users and [370:01] run and sometimes it's going it's easier [370:03] to just use pre-made scripts. Uh so for [370:08] example here if we wanted to install if [370:10] we wanted to use rough you can we can [370:14] search for it and uh use it maybe [370:16] instead of doing all these steps. Uh [370:19] let's do this just uh to show you how [370:22] how to search this. So you type rough [370:25] GitHub action and you search for it. As [370:28] you can see there is one. So instead of [370:30] repeating instead of doing these these [370:33] steps manually we can reuse this rough [370:36] action and that will make our life [370:38] easier. It depends sometimes you might [370:41] you don't find actions for the specific [370:44] uh library that you are using. In that [370:46] case you you use uh this approach and [370:50] here I wanted to show you both so that [370:52] you so that you don't get stuck if you [370:54] don't find a a an action for a specific [370:58] library that you are using. We have [371:00] spent a lot of time in this specific [371:04] file. So now let's go through the other [371:06] steps quickly and let's see the [371:09] difference between the back end and [371:11] front end. Okay. Now let me go back to [371:14] the CI pipeline. uh let's click on it [371:16] again and let this time let's focus on [371:19] uh linting and formatting the front end [371:22] and here I will also open that file here [371:24] it is front end lint and format okay [371:27] let's see what we have here uh we start [371:30] same as before we define the name and uh [371:33] here we use workflow call to let GitHub [371:36] know that this is a reusable template [371:39] and here we have again just one job here [371:42] let's see this time the working [371:44] directory is front end. We are using [371:47] checkout. And here we use, as you can [371:50] see, this time we didn't install PMP [371:53] manually, but we used an an action that [371:57] we can find on GitHub. And we specify [372:00] that we want to install PNPM version 10. [372:03] But same thing with node, we are using [372:07] an action to install node and we target [372:10] this specific version. And here I forgot [372:13] to talk about caching. We are doing that [372:16] here. And on pi the the python side you [372:21] can see that here we also cache pip. Why [372:23] are we using caching? Basically if your [372:26] dependencies don't change you don't need [372:29] to download them every single time. So [372:31] caching will make sure to save uh the [372:35] dependencies so that uh the next job [372:38] will just read them instead of [372:39] downloading them each time. This is [372:41] going to make your pipeline faster and [372:44] it's uh yeah it's great uh it's a great [372:46] thing to add. After that we install the [372:49] dependencies. We run eslint to lint the [372:53] frontend files and we use prettier for [372:56] formatting and uh as you can see we have [372:59] the these steps are are shown here on [373:02] GitHub. Here we have check out [373:04] repository, set up pmppm and nodejs, you [373:09] install the dependencies and you run the [373:12] linting and formatting steps. Great. Uh [373:15] these are the two things that we have [373:17] done in the previous video. We did them [373:20] locally um before accepting the commit [373:24] and now we replicated them here on [373:27] GitHub using GitHub actions. Now we are [373:30] going to add one more job. uh we are [373:33] going to perform unit testing and let me [373:36] show you how that looks like. Let's open [373:38] backend unit tests. It's going to be uh [373:41] similar but uh we will change some steps [373:44] and I want to emphasize that in this [373:47] course I am not going to show you how to [373:49] write unit tests because this is outside [373:52] our scope. If you want to do this that [373:54] will that will take a lot of time and it [373:57] depends on what programming language we [373:59] are using. So we are going to consider [374:02] that as a blackbox. And if you are using [374:05] let's say go or another backend language [374:09] you can swap the commands that I have [374:11] here with the ones that are appropriate [374:14] for the specific stack that you are [374:16] using. Okay. So let's get started. Here [374:20] the name is backend unit tests. And here [374:24] we have again one job which I called [374:27] tests. Let's go inside. First, we [374:30] download the source code. We set up [374:32] Python. We install the dependencies. And [374:35] here we run the unit tests. Okay. And [374:39] let's look. You can see that here we are [374:41] using piest. This is the library that [374:45] that we are using to test our backend [374:47] code. And after that, this is not [374:49] important. Basically, here we are using [374:51] code coverage. Why? So that we can see [374:54] the code coverage. You can see that here [374:57] in the readme I showed that okay uh the [375:00] tests cover 95% of the back end and 85% [375:04] of the front end and uh yeah you if you [375:07] want you can add that and for that we [375:09] use the code coverage package or yeah [375:13] you can search for it code cover this is [375:16] the the service that we are using they [375:18] have a free plan that uh that you can [375:21] benefit from okay and so this step is [375:24] not important but what we have here is [375:27] is what we need. Yeah, it's it's the [375:29] same. So, if I open this and look at run [375:32] unit tests, you can see that here are [375:35] the tests that we have. The everything [375:38] went well. So, uh so we have 9 59 tests [375:42] and they all pass without any problem. [375:45] But if let's if one of them failed, this [375:47] step is going to fail which means that [375:50] this pipeline will not succeed. And we [375:52] are doing the same thing uh in the front [375:55] end. Let's open that front end unit [375:58] tests. Let's go down. So here after [376:00] installing the dependencies we have this [376:04] test script and on in the front end we [376:08] put those in the in your package.json [376:11] file. So here in this scripts object you [376:16] let's find test. Here it is. We are [376:18] using v test for that and we have other [376:21] commands. So if you want to run end to [376:23] end tests, we are going to look at this [376:25] in the upcoming tutorials. But for now, [376:28] we just run unit tests. And again, we [376:31] use code coverage to get the code [376:34] coverage for the front end. Great. So [376:36] now we created these four individual [376:39] components. Now let's see how to put [376:41] them in CI.YAML [376:44] so that they run so that it orchestrates [376:47] all of them. Here is CI.YML. I have [376:50] removed uh some stuff so that um yeah so [376:53] that you don't get overwhelmed. First [376:55] the name is CI pipeline and here we have [376:59] something different. So here we put on [377:02] push on pull request and on workflow [377:05] dispatch. Let's start with this one. So [377:08] on push will run this pipeline whenever [377:11] we push something to the master branch. [377:13] We will also run it if we create a pull [377:16] request that targets the master branch. [377:19] This is what I have here. If I click on [377:22] pull requests. Let's look at this one. [377:24] It is targeting the master branch which [377:27] means it's going to trigger this [377:29] workflow to run. If I scroll down, uh [377:32] you can see that I have the workflow [377:34] here. Here I have my four steps and you [377:38] can see I cannot merge because we have [377:41] some failures. Some jobs have failed. [377:43] And finally we have workflow dispatch. [377:46] and workflow dispatch allows us to look [377:50] at the workflows. Let's look at CI [377:52] pipeline. And I can come here and run it [377:55] from this page. This is useful [377:59] especially if you want to uh if [378:01] something failed and you want to rerun [378:04] the job manually, you can do it from [378:07] this uh page. Let's start with the front [378:10] end link format check. First we give it [378:13] a name and here users. This time we [378:16] already have a workflow. We are going to [378:19] reuse it here. And as you can see now it [378:21] looks clean. Imagine taking all all [378:24] everything inside this file and putting [378:26] it here. And then do the same thing for [378:29] these three. This file will be very [378:31] long. Debugging. If you get a problem [378:34] here uh in your workflow, it's going to [378:37] be challenging to read the whole file, [378:41] find the bug, and solve it. But now it [378:44] looks clean. I just use this keyword to [378:47] let GitHub know that the implementation [378:49] is inside this specific file. And I do [378:54] the same thing for backend lind, front [378:56] end unit tests, and backend unit tests. [378:59] The only thing that that we changed here [379:02] is that we added the needs keyword. Why [379:05] did we do that? Let's go back again to [379:08] the pipeline. Let's collapse these two [379:11] and let's look at just uh let's look at [379:14] just these two jobs. By default, GitHub [379:18] will run your jobs in parallel. Here we [379:21] are performing linting and formatting in [379:23] the front end and back end. And because [379:25] these jobs are independent, we allow [379:28] GitHub to run them in parallel. But if I [379:31] want to run unit tests, I need to ensure [379:33] that the code is clean. There is no [379:35] syntax error. There is no problem in the [379:37] code. If that's the case, I can run [379:39] this. So if you have a dependency, you [379:42] add the needs keyword. So this job needs [379:46] front end lint format check to succeed. [379:50] And this creates this dependency which [379:52] will force GitHub to run them in [379:54] sequence. We do the same thing for [379:56] backend. It's going to wait for backend [379:59] lint format check. If backend lint [380:02] format fails, this job will not run. [380:05] GitHub will skip it. And finally, we [380:07] have pipeline success. For the pipeline [380:10] to succeed, we want every job to [380:13] succeed. Okay, that's why we put [380:15] everything here. again runs on Abuntu [380:18] latest and if always is important [380:21] because if something failed we want to [380:24] run this anyways we don't want to skip [380:26] it because this is what's going to give [380:28] us that status at the end the status is [380:32] going to be either uh success or failure [380:35] let's look at steps because this is [380:37] where we have that logic let me expand [380:40] this page so that we can see this better [380:43] and let me also decrease the font size [380:45] First here we have two steps. The [380:48] pipeline can either succeed or fail. How [380:52] do we know if that's the case? We use [380:55] this function contains and here needs so [381:00] this variable result is going to give us [381:02] the results for all these jobs. This [381:06] condition will check if we don't have [381:08] any failures and if no job has been [381:11] cancelled we will show check all checks [381:14] passed successfully. But if we find a [381:18] failure or something has been cancelled [381:21] we will show some checks failed or were [381:24] cancelled and we exit with number one. [381:28] When we exit with one, we tell the [381:30] runner that there was an issue and [381:33] GitHub will display a uh let me show [381:36] you. It's going to tell you that yeah [381:38] that the test has failed. We can compare [381:41] both of these tests to uh understand [381:44] what's happening here. So the first test [381:47] you can see it shows us it is showing us [381:50] a failure. Why? because that specific uh [381:55] condition this one contains failure is [381:59] true. Why? Because we have one failure [382:01] here. It is and this one was skipped [382:03] that this will make pipeline success [382:06] exit with one and the pipeline has [382:10] failed. So it's which means that I won't [382:12] be able to merge. But in this other [382:15] case, you can see that it succeeded [382:17] because um there is no failure and [382:20] nothing has been skipped. Great. And now [382:22] if I save this, you can see that we have [382:25] we are in this branch and uh push the [382:28] code, we will we will see that this will [382:31] run. Okay. Now let me remove the [382:34] splitting to give us some room to play [382:36] with. Let's push all of this. Uh let's [382:39] just type test because we are we are not [382:42] doing something serious now. And but [382:44] remember to type uh good com commit [382:47] messages. I'll commit and push. Let's go [382:51] back to GitHub. You can see that we have [382:53] this new branch. And uh let's click on [382:56] new pull requests. And here I don't want [382:59] to merge with the upstream repository. I [383:02] want to merge with my fork. And uh let's [383:06] see. So I want to merge with the master. [383:08] But here I need to select test number [383:11] two. Okay. Okay, it shows me these are [383:13] the changes that I have made. Click on [383:15] create pull request. Let's keep [383:17] everything like that and hit create pull [383:20] request. Okay, let's scroll down and as [383:23] you can see here are the checks. But [383:25] notice that we have a problem. I can [383:27] this button uh it's it's not grayed out [383:30] like it is still active that which means [383:33] I can still click on it. That's a [383:36] problem and I am going to show you why. [383:39] Because we are here we are not waiting [383:42] for the final stage to finish which is [383:46] pipeline success. But the solution is [383:50] easy. Let's go to settings and we need [383:53] to update the rule sets. This is the [383:56] rule that we created before. Open it. [383:59] Scroll down and here uh make sure to [384:02] activate require status checks to pass. [384:06] here. Let's click on add checks and [384:08] start typing pipeline success and it [384:11] should appear. Yeah, here it is. Select [384:14] it and just like that save the changes. [384:18] Now let's go back to that pull request [384:21] and let's see what will happen. Uh as [384:23] you can see I cannot now click on it. [384:25] And if I hover the message says merging [384:28] is blocked due to pending merge [384:31] requests. And here is the pending uh [384:34] job. It's pipeline success. And this one [384:37] is going to wait for the other ones to [384:39] finish before running. And let's see. [384:42] It's going to run. Let's click on it [384:45] just to see what's happening. So, [384:46] pipeline succeeded. This is the message [384:49] that we have uh that we wrote here. Echo [384:53] all checks passed successfully. This is [384:55] what we see here. Great. So, that [384:57] worked. And now I can merge because [385:00] there is no failure and nothing has been [385:03] skipped. Amazing. But let's uh let's try [385:07] to break this just to show you that we [385:10] won't be able to merge broken code. [385:13] Okay, let's open let's uh let's say [385:16] main.js. [385:19] Yeah, let's let's remove import and [385:21] let's just put Axios from Axios. This is [385:24] not correct. And here you can see that [385:26] I'm not running this locally. I am [385:28] running it here on yeah on GitHub just [385:32] to make modifications quickly which [385:33] means this is not going to be catched by [385:36] the pre-commit hooks that we defined [385:37] earlier. Normally this will will catch [385:41] that but this is the limitation that I [385:43] mentioned in the beginning of this video [385:45] is that yes it's good to have those [385:47] pre-commit hooks locally to stop the uh [385:52] some bugs like these but people can also [385:56] skip them. Uh this is why we create this [385:58] pipeline to be the gatekeeper. It's not [386:00] going to allow anything in unless it's [386:03] good and yeah let's verify that. So [386:06] let's just have another test commit [386:08] commit and push. Go back to here. Okay [386:12] is blocked. And now um the the linking [386:17] and formatting step in the front end is [386:19] going to fail. Let's click on it and [386:22] let's see what's happening. So now we [386:24] are set up in NodeJS. [386:26] Good. And let's read the message. You [386:29] can see that here it says cache restored [386:31] successfully, which means it it skipped [386:34] installing NodeJS and it just took it [386:38] took it restored it from the cache which [386:41] is great. It confirmed that the caching [386:43] is working. But let's scroll down a [386:46] little bit. Ah, you can see that ESLint [386:48] is not happy with this. It's saying that [386:50] error parsing uh this token. So it it [386:53] couldn't find the import and it uh [386:56] exited with code one. Now let's go back. [386:59] You can see that because this job has [387:03] failed uh we we weren't able to run the [387:07] unit the tests because this depends on [387:11] the first job and pipeline success [387:16] failed two because we have one failed [387:19] job and one cancelled job and now I [387:22] cannot merge this. So I have to fix uh [387:25] the problem and they cannot also push to [387:28] master because it is protected. You see [387:30] the beauty of having this kind of [387:32] pipeline in your repository, it's going [387:34] to be very helpful to catch problems and [387:38] whenever you merge something to master, [387:40] it's it is guaranteed that there are no [387:42] bugs in it and you can merge without any [387:46] problem and even deploy to your [387:49] production server. We have reached the [387:52] end of this tutorial. I hope that you [387:54] found it interesting and uh you can find [387:57] the documentation here on this GitHub [387:59] repository. You will find the link in [388:02] the video description. And what is next? [388:06] So we still have two more sub chapters [388:09] in module 5 and in this one we are going [388:12] to talk about automated security [388:15] scanning. We are going to scan [388:17] vulnerabilities in the libraries that [388:20] you use in your stack. This is very [388:22] important. We don't want to push [388:24] vulnerable code to a production system [388:27] because you could lose data. There are [388:30] many problems that can happen if that if [388:33] you push vulnerable uh libraries and we [388:36] will perform also this specific test to [388:40] yeah ensure that uh there are plenty of [388:44] tests that we can run and I'm going to [388:47] show you some of them in the upcoming uh [388:50] tutorial. [388:52] Welcome back. So far you have built a [388:55] pipeline that guarantees your code is [388:59] formatted correctly and your logic works [389:02] as expected. [389:04] Today we will focus on automating [389:08] security scanning. [389:10] Here is what we are going to talk about [389:12] in this sub chapter. I will try to [389:14] explain to you the shift left security [389:18] model. You are going to run software [389:21] composition analysis. We will also add [389:25] static application security testing [389:28] and finally we will implement a fail [389:31] fast strategy. CSA and SAS are the two [389:35] types of tests that we are going to add [389:38] because that they look for specific [389:41] security issues in different areas. And [389:46] the fail fast strategy basically means [389:48] if the security scan fails, we stop the [389:52] pipeline. We don't need to run the [389:55] subsequent jobs because it's just going [389:57] to be a waste of time because we know [389:59] that we have a problem in the [390:01] application. We should never deploy it. [390:03] And in terms of the shifting security [390:06] left concept, here it is. I have made [390:09] this diagram to explain this to you. We [390:11] have two ways of deploying an [390:14] application or working and uh we have to [390:18] work on security. This is very [390:20] important. But before or traditionally [390:22] we used to do it this way. First you can [390:25] see that this is the development cycle. [390:28] We plan, we design the application, we [390:31] start coding, we build to make sure that [390:33] everything is is correct. we have some [390:36] tests that should pass and right after [390:39] that we move to security. The problem [390:42] with this approach is that if you work [390:44] on a giant application and at the end [390:47] you realize that you have a lot of [390:50] problems that will make this development [390:54] cycle take a lot of time and you will [390:58] you will be fighting a lot of problems [391:00] and that will push the release cycle [391:03] further. uh this is very problematic [391:06] especially with bigger applications but [391:09] the shifting left security model fixes [391:13] this issue by integrating the security [391:16] checks during this inear cycle let's say [391:19] so after designing the application now [391:23] you start coding but while you are [391:25] coding you are making sure that you are [391:28] doing the best practices and so you keep [391:30] thinking about security if you have for [391:33] example input fields you think about uh [391:36] sanitizing the inputs. Try to test the [391:39] application. Can people do SQL injection [391:41] or other types of attacks while in this [391:45] tight loop? And you do it before you [391:48] build you make sure that the feature [391:50] that you are working on is secure. And [391:53] if you do this, you ensure that even if [391:56] you work on large applications, [391:57] integrating security is going to be an [392:00] easy task. It's like translation or [392:03] adding themes in your application. If [392:06] you do this at the end, it becomes hard. [392:09] But if you do it in the beginning of the [392:11] project, it's going to be very easy to [392:14] uh to integrate. And finally, once uh [392:17] once you are done with a specific [392:19] feature, you release and the cycle [392:22] repeats. Uh this is my GitHub profile [392:25] and last time you can see that I showed [392:28] you how to block merging if some of the [392:32] tests fails. That's good. Now what I'll [392:36] try to do I will fix these problems and [392:38] I will come back because we want to [392:41] ensure that these steps are fine but uh [392:44] we want uh to add on top of this the [392:47] security jobs. I am back and as you can [392:50] see every job has run successfully [392:53] without any issue. Now let's start [392:56] auditing both the front end and back [392:59] end. This is the software composition [393:02] analysis step and we use it to basically [393:06] audit the third party libraries that we [393:09] are including in our projects. For [393:11] example, in the front end folder we have [393:15] package.json. [393:16] It contains all the dependencies that I [393:19] am using for this project. You can see [393:21] that we have axios view router and many [393:26] more. The problem that might happen is [393:29] that let's say axio might be attacked. [393:33] Attackers might find a vulnerability in [393:36] this package which they can exploit. [393:39] Your application will run without any [393:41] problem. because that vulnerability is [393:44] invisible to the checks that we have [393:47] here. So by adding another job that goes [393:51] through all your dependencies, checks if [393:54] there is a vulnerability. When we do [393:56] this, we can uh successfully block the [394:00] pipeline. We will not allow deploying a [394:04] vulnerability a vulnerable package to a [394:08] to the production environment. we will [394:10] wait for the maintainers of that [394:13] specific project to fix the problem [394:16] before updating to the newest version. [394:19] And the same story for the back end. If [394:21] I go back, if I go to the backend [394:23] folder, you can see that here I am still [394:26] using the requirements.txt. [394:28] People nowadays use uv or other package [394:31] managers. Uh there is nothing wrong with [394:34] requirements. [394:35] We define the dependencies that we are [394:38] relying on and we need a tool that will [394:41] go through all of them, check the the [394:43] database, see if there are [394:45] vulnerabilities and block the pipeline [394:48] if it finds any. Okay, let's now go back [394:51] to VS code. You can see that I am still [394:54] using the web version because these are [394:57] changes that are simple. Now let's go to [395:02] the GitHub folder. open workflows and [395:06] here I need to add two more jobs one for [395:09] double check the dependencies in the [395:11] front end and the other one will do the [395:13] same but for the back end let's start [395:16] with the front end I will zoom a little [395:18] bit you can see that I have added this [395:20] job front end vulnerability check this [395:23] is the name that I've used here this is [395:26] a reusable workflow we have explained [395:29] this in the previous sub chapter and [395:32] here what are we doing so We are [395:34] checking the repository here. We are [395:37] using PMPM version 11. I have changed [395:41] this from the previous sub chapter. Why? [395:44] Because if if you want to learn more [395:46] about this specific version of PMP, make [395:49] sure to go to Google and search for [395:52] PMP1. [395:54] You will land on this page. You can read [395:57] more about it here. But basically uh by [396:00] default PNPM is more secure than npm. [396:05] Recently there were there were a lot of [396:07] attacks on very popular npm packages and [396:11] people sometimes they just download the [396:14] latest version and that uh gets them [396:16] hacked. PNPM solved these problems by [396:20] blocking recent updates. So if an update [396:23] is less than 7 days, it's not going to [396:25] be installed automatically. you will you [396:27] will get a warning or an error and you [396:30] had to decide if you want to download it [396:32] or not and it comes with other security [396:35] features by default. This is why I've [396:38] moved to version 11 to also help us in [396:42] um blocking the download of malicious uh [396:46] code in specific packages. And since we [396:50] moved to PMPM version 11, that requires [396:54] node version 22 and up. You cannot, as [396:57] you can see, this is a requirement. You [397:00] cannot use node 18, 19, 20 or any [397:03] version below 22. And after that, we [397:06] install the dependencies and again we [397:08] are using pmppm to audit the third party [397:12] packages. If everything is good, this [397:15] will not fail. But if we if PMPM finds a [397:19] vulnerability in a specific package uh [397:22] then this step will fail. Now let's do [397:25] the same thing for the back end. Uh the [397:28] job is called backend vulnerability [397:31] check. We are doing the same steps [397:33] checking the repository. We installed [397:36] Python version 3.13. [397:39] And here we are using a specific GitHub [397:41] action that uses the pip audit library. [397:46] And here as input we give it [397:49] requirements.txt because here we define [397:52] the dependencies. If you were using uv [397:56] or other py uh other package managers [397:59] those they they generate a file called [398:03] hi project.toml. [398:05] If this is the case, you just change the [398:09] input from requirements.txt [398:12] to pip projects.totml. [398:14] And you can read more. You can find more [398:16] examples in their GitHub repository. [398:19] Always this is what how how you should [398:22] reuse these uh specific GitHub actions. [398:25] Just look for their GitHub repository [398:29] and read their readme. From the readme, [398:32] they state that this action uses pip [398:35] audit which is the default library that [398:38] we use in Python to audit the [398:40] dependencies. And uh if you want to [398:43] learn more about the usage, just go to [398:45] the usage section. And as you can see [398:48] here uh for example, here they are [398:51] pointing to the environment folder. Uh [398:53] here is where how you can point to [398:56] requirements.txt. [398:58] And here you can point to more than one [399:00] file. For example, if you have if you [399:02] split the dependencies that are used by [399:06] your application and those that are used [399:08] just for development, you can provide [399:11] both files. And let me show you where is [399:14] the TOML one. Here it is. So if you read [399:17] this, they say that to audit a project [399:19] that uses pi project.toml, [399:22] you use this specific uh notation. You [399:25] just point to the directory where this [399:28] file lives. Now let's go back to VS code [399:33] and voila. You can see that it's very [399:36] easy. Now we just need to go back to the [399:39] orchestrator and add these jobs here. I [399:43] have changed the jobs list. Now I start [399:46] with the vulnerability checks. we what [399:49] we have the front end and back end and [399:52] we point to those two files that we have [399:55] created in this session and after that I [399:59] update the front end LinkedIn and [400:02] backend LinkedIn jobs to depend on these [400:06] vulnerability checks. You can see that [400:08] here we added needs and this front end [400:12] link job needs the vulnerability check [400:15] to succeed before running. At the end in [400:19] the pipeline success job I've also [400:21] updated the needs list by adding the two [400:25] security jobs. Now we can commit this. [400:29] Let's say CI add security jobs and let's [400:34] commit and push. Now we can go back here [400:38] and uh you can see the pipeline will run [400:40] again and this time we can go we can [400:44] click on view details click on summary [400:48] and here is our graph. Let me zoom in. [400:51] You can see that we have the backend [400:54] audit dependencies job and the front end [400:58] vulnerability job. Both of them run in [401:01] parallel and as you can see the backend [401:04] lint and front end lint depend on these [401:09] two jobs to succeed. If they fail the [401:11] pipeline will stop. And as you can see [401:15] the front end has failed. We will see [401:19] why in a minute. And the back end also [401:22] has failed which means that the pipeline [401:26] success job will fail too. Good. And now [401:31] if we go back to the food request tab, [401:35] you can see that I cannot merge. This is [401:38] blocked. And uh yeah, I need to solve [401:41] these problems. Now let me let's verify [401:44] what is happening. I will open this in a [401:46] new tab and I'll open the backend job [401:50] here. This is the result that we get [401:52] from pip audit. And here we have a [401:55] table. So name is basically the name of [401:58] the package. Here we have flask cores [402:01] and pi jwt. [402:04] We have the current version the ID of [402:07] the security problems. If you want you [402:10] can take this ID open a new tab and [402:13] search for it. You should be able to [402:15] find the details about that [402:17] vulnerability. And the severity is a I [402:21] think it ranges from 0 to 10 if I [402:24] remember correctly. And if it is closer [402:27] to 10, that's very problematic. And as [402:29] you can see, for this one, it's very [402:31] critical. When you see this, it means [402:33] that the you should update that package [402:36] as soon as possible. But sometimes it's [402:38] just medium or low. That's fine. Uh it's [402:41] not very problematic. Okay. Now, let's [402:45] look at the front end job. Ah, okay. So [402:49] here because we are using PMPM version [402:51] 11, we need to yeah migrate from uh the [402:56] previous version. So we need to first in [402:58] uh try to update the the pmppm lock [403:02] file. Let's take this command. Let's go [403:05] back to VS code. I've opened the [403:07] terminal. Let me let me go through the [403:10] front end folder and let's run that [403:12] command. Okay. So we have we have the [403:15] dependencies but here I think yeah we [403:17] are using pmppm version 10. So we need [403:20] to update it. So to update they give us [403:24] this command to run. Let's run it. Okay. [403:28] So let's run pmppm setup to fix that [403:32] issue. Let's run this command. Okay. [403:36] Good. Now let's try to update it again. [403:40] Okay. So we need to use pmppm self [403:42] update. That's the new uh method. And [403:45] now we can run pmpm- [403:48] version. Still says 10.32.1. [403:52] Maybe I need to close and open a new [403:54] terminal window. So pmppm- version. [403:58] Okay. Yeah, that now that worked. So we [404:00] have version 11.1.3. [404:03] Let's go back to front end. Let's copy [404:07] this command again. Let's past space [404:10] here and hit enter. Let's say yes. And [404:14] now you can see that PMP is verifying [404:16] the log file for any supply chain [404:19] problems. Everything went well. But here [404:22] we have a problem. By default, PMPM is [404:26] going to ignore ES build. So if we want [404:29] to fix this, we need to run PMP approve [404:32] builds or so. We can either do this or [404:35] come back here because this is going to [404:37] work locally. But if we move to CI, it's [404:40] still going to fail and we cannot uh run [404:43] this in the CI interactively. Now with [404:46] PMP version 11, we have a new file [404:50] called PMP workspace that we can use to [404:54] modify or to update the settings. Here [404:56] it is. You can read more about it. And [404:58] if you want to search for uh specific [405:02] values within it, you you will find you [405:05] go to this [clears throat] link pm io/ [405:08] settings and you can see they are [405:11] talking about this specific file and [405:14] what we need to do let's go back to this [405:17] tab. So here they are talking about [405:19] approved builds. We can search for it. [405:24] Okay, now it's approved builds. Yeah, [405:27] it's this one. And as you can see, we [405:29] need to add this to PMPM workspace and [405:33] we need to set AS build to true. Let's [405:36] do that. I will go back to this tab. We [405:40] are in the front end folder. Let's add [405:44] PNPM [405:46] workspacey. [405:49] Ah, we already have it. Yeah, we already [405:51] have it. Good. Ah, okay. Okay. So when [405:54] you run the command, PMP PM will try to [405:56] help you. Basically, as you can see, [405:58] it's created the file for us and it's [406:01] telling us set this to true or false. [406:04] Let's set it to true. Let's open the [406:07] terminal again. Yeah, let's try [406:10] installing this. Copy the command. Paste [406:13] it. Hit enter. Okay, it worked. Okay, [406:17] let's commit these two. Let's commit [406:20] this. Let's go back. [406:23] to the pipeline. Ah, I need to sync the [406:26] changes. Let's focus specifically on the [406:28] front end job because the back end will [406:31] fail. We haven't fixed that yet. Here we [406:34] are setting PNPM. [406:36] That's good. Can see that here it [406:39] switched to version 11. That's what we [406:41] want to see. We installed node and here [406:44] we are installing the dependencies. [406:46] Okay. So, that failed because we have 17 [406:50] vulnerabilities. [406:52] Okay. And now let's go back to PMPM [406:55] workspace again. We need to let's look [406:59] at npmrc [407:00] or sorry package.json. [407:03] You can see that here before I used to [407:07] to resolve those problems and to do that [407:10] I added the overrides object because [407:13] these are transitive dependencies. I [407:15] don't depend on these libraries [407:18] directly. But some of my primary [407:21] thirdparty libraries are using them. [407:24] That's why we added overrides. But now [407:27] with pmpm11, we need to move these to [407:31] this pmpm workspace. And uh we can [407:35] search for that here. Let's see. Okay. [407:38] So we click on overrides. And as you can [407:41] see here is how we do it. We can copy [407:43] this format. We can let's put it here. [407:47] And yeah, I just need to copy all of [407:51] this and put it here. Okay, let's see. [407:54] We don't need these commas. We can [407:57] remove them. Yeah, I think this is the [408:00] format. Now, let's remove all of that. [408:02] Let's remove this from package.json. We [408:06] no longer need that. And let's save. And [408:08] if you are wondering how do I know all [408:10] of this, you can read the migrate v uh [408:14] v12 to v11 empm guide. So they have a [408:19] guide that will help you migrate to this [408:22] new version. Okay. Now let's save this. [408:25] Let's type fix move overrides to pmppm [408:32] work space commit [408:35] and sync the changes. Let's open the [408:38] front end job again. Remember before we [408:40] got 17 vulnerabilities. [408:43] Let's see how many we will get or if [408:46] this change will solve will resolve all [408:49] of them. Okay, we have the same problem. [408:52] I think because we have moved those from [408:54] package.json, we need to uh again update [408:58] the log file. Let me do it again. pmppm [409:02] install. Okay, so that changed the log [409:05] file. Good. So this is how how we work [409:08] with uh GitHub actions. You start in [409:13] with something, you test it doesn't [409:15] work, it gives you the error code, you [409:18] fix it, and you keep iterating until you [409:21] get something that works. And in the [409:23] future, let's say that we we have fixed [409:26] this. In the future, this job might [409:28] fail. So you open it, you look at some [409:32] the the step that failed, you read the [409:34] error and you go fix it. And this is [409:38] going this is super useful to ensure [409:42] that the code that you are deploying is [409:45] secure and that there are no problems in [409:47] it. Okay. Uh we are the job has failed [409:51] once more and now we have 14 [409:55] vulnerabilities [409:56] and most of them are coming from Axios. [409:59] You can see that all of them are coming [410:01] from Axios. Uh let's see. Yeah, all of [410:04] them are coming from Axios and you can [410:06] see that here the vulnerabilities they [410:08] show you the range where that was found [410:10] and the patched version. You can also [410:14] go to the GitHub repository of that [410:17] specific project, open their releases [410:21] uh page and here you can read about the [410:24] release. Sometimes they might have a [410:26] security section in the release where [410:30] they talk about the specific [410:32] vulnerability that was found. So now we [410:35] can install version 16.1. [410:38] Uh it's the latest version. uh it was [410:41] released for more than a week. So that's [410:44] fine. PMPM will be able to install it. [410:47] Okay, let's take this version. Let's go [410:50] back to VS Code. We no longer need this, [410:53] but we need to search for Axios here. [410:56] Let's change the version to 16.1. [410:59] Let's also again run pmppm install. [411:04] Okay, good. So the the log file has [411:06] changed. Let's commit and let's push the [411:09] changes and hopefully this time this job [411:12] will succeed. Let's go back to the pull [411:14] request. Open the job in a new tab and [411:18] uh let's wait. Awesome. We were we were [411:21] too close. Uh but now we have one. So [411:24] this this library we are not depending [411:27] on it. But there is uh some third party [411:30] library that uses that uses it under the [411:33] hood. But you can see we went from 17 [411:35] vulnerabilities to only one. And um [411:40] yeah, we can add this to the transitive. [411:43] Let's see. So, copy this. Go back to [411:46] PNPM workspace where it is here. Here we [411:50] have that version. And now we can add [411:53] what was the package name? I think it [411:55] was WS. Here we put this and and I think [412:01] the the last version is 8.20.1. [412:05] So we can fix it to that specific [412:08] version. You can always verify. You can [412:10] either type the package name and then [412:13] add npm or you can add GitHub to the [412:16] end. As you can see the last version is [412:19] 820.1 and it was published 9 days ago. [412:23] That's good. If it was updated just a [412:25] day ago, the MPM will not allow you to [412:27] download it because by default it's I [412:30] think you can change the value. If you [412:33] go to settings, you can override that. [412:36] Uh you can set it to one day, a week or [412:40] more. It depends on the severity. If a [412:43] package is very popular, uh it's it's [412:46] good to wait for a while before [412:47] downloading the last version. Okay, [412:50] let's now again run pmppm install and [412:53] let's see if the log file will change. [412:55] And yes, it did. Commit. Save the [412:58] changes and let's push. This time it [413:02] should pass. We are installing the [413:05] dependencies and yes it worked. It took [413:09] it took a while but we have solved the [413:11] problem. Now you know how to iterate uh [413:15] over this. Now we are using PMP version [413:17] 11 which is strict. It uh as you saw it [413:22] blocks a lot of things and we needed to [413:24] do uh some migration but it's worth it [413:27] once you do this. You do this once it [413:29] takes some time but later it's uh you [413:32] don't think about it. Awesome. Now we [413:34] can uh focus on the backend job. Let's [413:39] go back to the audit section and let's [413:42] see what we have here. Okay. So we have [413:44] two packages. We have flask cords and pi [413:48] JWT. The first package is problematic [413:51] because they don't have a new version. [413:53] So this is the last version that they [413:55] have. And pi JWT I'm not sure about that [413:59] one. So let's search for it. Uh let's [414:02] click here. Let's look at releases. Is [414:05] it an active project? Yes, it is still [414:07] active. And the last version is 2121 [414:11] which is again we have the same problem [414:13] with PI JWT. We don't have a new version [414:16] that addresses this um vulnerability. [414:20] How do you handle this case? Well, it [414:24] depends. For example, in my project, I [414:27] am not using Flask. I am using fast API. [414:31] And I'm not sure why this is here. But [414:33] if you dig deeper, you will find that a [414:36] dependency that I'm using just for [414:37] development is relying on this package. [414:41] So I can move that specific package to a [414:45] to requirements dev and I can scan only [414:48] requirements. That's that's problematic [414:52] too. Yeah, you solve the problem with [414:54] production. So the code that you will [414:55] put in production is going to be safe [414:57] because we will not use this library. [414:59] But if uh you you have if you run this [415:02] [clears throat] locally, you might also [415:04] get infected. [415:06] There is another solution which is to [415:08] basically remove that package if you [415:10] don't need it. For example, if you used [415:12] it only once. Yeah, you can remove it [415:15] then. It's not it's not necessary. And [415:18] as we saw in this package, the [415:20] vulnerability is critical. Maybe if it [415:22] was low or medium, we could skip it. uh [415:26] or you could read it. Uh if it requires [415:29] user input or something like that. Uh if [415:32] you run it locally, you are safe. There [415:34] is no going to be any issue. But yeah, [415:37] this one has a critical problem. And uh [415:40] so it allows SQL injection. Can see that [415:43] here they say this vulnerability can [415:45] lead to unauthorized data manipulation, [415:47] data exfiltration, DOSs and yeah a lot [415:51] of problems. But most of them so SQL [415:54] injection requires an application that [415:56] users interact with. So that's the [415:58] problem. And here we we are not we are [416:01] not we are not using this. We are using [416:03] fast API for the front end. But it's [416:06] it's very tricky. Sometimes you have to [416:08] wait for a specific package to release a [416:10] fix uh for the that specific problem. [416:15] What I propose to do is to divide the [416:18] requirements file to two files. one for [416:22] production and the other one for local [416:24] development. Let's go back to GitHub. [416:27] Let's remove or let me keep that [416:30] specific workflow because we need to [416:32] update it. Backend vulnerability check. [416:35] I'll keep that one here. I'll go to back [416:37] end here. This is what we currently [416:40] have. I'll copy this file. Paste it. [416:43] Okay. So, it's not allowing me to copy [416:45] the file. Okay, that's not an issue. We [416:48] will create another one. [416:49] requirements-dev.txt. [416:53] Let's copy all of this. I already uh [416:57] have them here. So, let's put them in [417:00] this place. How do how do we split the [417:03] the file into dev and one the other one [417:06] for production. Basically, these [417:08] libraries are used for development. For [417:10] example, for pi test, we use that to run [417:12] tests. We're not going we don't need [417:15] this in production. So, that we can [417:17] drop. We can also drop the other with [417:20] the other packages by this facing kio [417:22] and the coverage pre-commit that's [417:25] another local one we have explained this [417:27] in the previous sub chapters locust this [417:30] is we this is new we haven't talked [417:33] about this uh but we will see it in the [417:36] upcoming chapter so I will not explain [417:39] this now but we will see it later uh the [417:42] other packages are re are uh important [417:46] python TNV is used to load environment [417:49] variables. Fast API as our back end. We [417:53] use Pantic to validate input. UVON is [417:57] important. Yeah. So everything else is [418:00] is required by our application. So this [418:03] is the first thing we will do. Uh now we [418:07] have two separate requirements files. So [418:10] let's do this here. Let's commit this. [418:13] Now let's go back here. Here you can see [418:16] that we are still pointing to [418:18] requirements.txt. [418:20] That's good. Uh let's just sync the [418:23] changes. Let's see what will happen on [418:25] GitHub. Uh let's click on backend [418:28] vulnerability. It should fail but I at [418:32] least we will have only one [418:34] vulnerability. Yeah, we have just one [418:37] which is PI JWT. This one is problematic [418:41] because if I did uh more research and I [418:45] found that docker are also flagging this [418:49] library and here the score is seven. [418:51] It's not uh like the other site that we [418:54] saw before. Let me explain to you how do [418:58] we get these CVE. Basically security [419:02] researchers they go through packages and [419:04] they look at the code to see if there is [419:06] some something that is that should be [419:08] fixed and in pjwt they say that it has [419:13] weak encryption but the people that were [419:16] that work on this project claim that no [419:18] the the project is fine there is no [419:21] issue and the problem that those [419:23] researchers have pointed is that the [419:25] length of the key uh is not very large [419:29] so it can be small. So in this case [419:32] where the maintainers don't want to [419:35] address the CVE, we can tell this is why [419:39] I kept this here. We can tell pip audit [419:41] to ignore that specific CVE because it's [419:45] just noise and the maintainers they are [419:47] the ones who understand their project [419:50] and they have the freedom to either [419:52] accept the claim the CVE or reject it. [419:57] because they have rejected it. We will [419:59] add ignore vulnerabilities and we [420:02] specify the name of that vulnerability [420:05] and uh you can find that here uh it's in [420:09] this in the ID column. Just copy this [420:12] and paste it in this YAML file. Let's [420:16] commit this. Let's think and hopefully [420:20] this time this job should should not [420:23] fail. Let's click on it again. Okay, we [420:26] are checking the repository. We install [420:28] Python. It should be cached. And [420:32] let's wait for a moment. And as you can [420:35] see, it says no known vulnerabilities [420:38] found. And it also tells us that we [420:41] ignored one vulnerability. And now if we [420:45] go to summary, everything will run. So [420:48] now because these security jobs has s [420:53] succeeded, we now can run LinkedIn. [420:54] adapt that we run unit tests. But if one [420:58] step fails, the other one will not run. [421:01] And here we have a problem with [421:03] LinkedIn. Why? And we have a problem [421:06] with unit tests. Interesting. Here we [421:09] are installing pip install-r [421:11] requirements.txt. [421:13] But now we need to also we need to [421:15] install from both of them. We keep [421:17] requirements txt and we add requirements [421:19] there. There is another option where you [421:22] put - r requirements txt within [421:26] uh requirementsdev.txt [421:28] but we will keep it simple like yeah [421:30] it's explicit it's uh it's not going to [421:32] hide that uh specific part of the [421:34] command commit and let's sync the [421:37] changes. We also have a problem with [421:39] linting that's uh that's weird. So let's [421:42] see what is happening there. Well, we [421:45] can just come let's come here and let's [421:47] uh run pmppm lint and pmppm format. [421:52] Okay, so we needed to format workspace. [421:54] That's the issue. Okay. Uh let's type [421:57] style apply formatting. I think now [422:00] everything should pass hopefully. Let's [422:03] close this tab and this one. Let's go [422:06] back here. Let's click on view details [422:10] summary and let's wait. Stage one [422:13] completed successfully. Let's move to [422:16] stage two. We have no problem with stage [422:19] two. Now we can move to stage three. And [422:22] the unit test has without any problem. [422:26] Amazing. And the pipeline succeeded. [422:29] Great. Now we can go back to the pull [422:30] request and if we are happy we can merge [422:33] the change the features that we are [422:35] developing or the bug fix that we that [422:38] we resolved. Amazing. You can see that [422:40] uh it took us a while. Uh we had a lot [422:43] of problems. We fixed an issue here. It [422:45] generated other problems there. But I [422:48] wanted to show you the full um the yeah [422:52] the row experience. That's how you [422:54] develop this. And uh it's it's it's [422:57] easy. Yes. But you get a lot of errors [423:00] along the way. And uh now I I think you [423:03] you have a better idea of how to work [423:05] with GitHub actions and how to resolve [423:08] any problem that you might face. But we [423:11] are not done yet. We have uh this is [423:13] this was the CSA software composition [423:17] analysis. Now we need to do static [423:20] application security testing. What is [423:23] the difference? Basically SCA doesn't [423:26] look at your source code. It checks the [423:29] third-party dependencies but static [423:32] application security testing which is [423:34] which we call SAS for short will go read [423:38] your Python code and see if there are [423:40] problems there. If you have password [423:43] within the source code if you are using [423:46] weak uh deprecated methods for example [423:50] if you are using the execute function [423:53] within your code that's a risky one. So [423:56] this is going to check for these [423:57] patterns and it's going to let you know [423:59] if it found something. Let's go to VS [424:03] code and let's add another job here. We [424:07] will call this one backend stat check [424:11] and we will put this code inside. Let's [424:14] scroll down. What we have here is this [424:17] is uh similar to what we have done [424:19] before. The only thing that changed is [424:22] bandit. Here we are using a library [424:25] called Bandit. You can search for it on [424:28] GitHub. And as you can see, it is a tool [424:31] designed to find common security issues [424:34] in Python code. That's the difference [424:37] between this and uh the other two jobs [424:40] that we worked on earlier. It is going [424:43] to read your code. And here we are [424:45] bandit with these specific arguments. Uh [424:48] basically dash r means recursive. Here [424:51] we are. So the working directory is [424:54] backend. It's going to go in the inside [424:56] the backend folder and go through all [424:58] the Python files. Dash LL means so here [425:03] we have two L's. This means that this is [425:05] going to flag only medium or high [425:09] issues. It is going to skip low [425:13] problems. We will see this in the logs. [425:16] If you if you don't do this, you will [425:18] have a lot of issues and they are just [425:20] false positives like uh what we have [425:23] done with PI JWT that CVE was blocking [425:27] us. But if you analyze it well, you see [425:30] that in the end it's not an issue. [425:32] That's why we are skipping low problems. [425:34] We want to focus just on medium or high [425:38] issues. And if you want to find uh if [425:41] you want to learn more about this [425:43] library, you can always go to the source [425:45] repository and read their documentation. [425:48] There you will find everything. Okay. [425:51] Now we need to go back to CI and we need [425:54] to add that after the backend [425:56] vulnerability check and I want let me [426:00] take this. I want backend linting and [426:02] formatting to run after both of these [426:06] tests. Okay. And uh I will add that here [426:10] also. I think that's all we need to do. [426:12] Now we can we commit this and let's [426:16] think the changes. Let's go back to [426:18] GitHub. I think it should pass because [426:21] uh I am not using any unsecure code in [426:24] my back end. But let's verify or refresh [426:27] the page. Sometimes it gets stuck. Uh I [426:30] have been refreshing the page uh for a [426:32] few minutes but it is still stuck. And [426:36] when that happens, sadly, GitHub, they [426:38] don't show you that there is a syntax [426:40] error. But here you can see that I have [426:43] missed a comma. That's the problem. And [426:45] that's why the pipeline didn't start [426:47] right away. Okay. And now it's working. [426:51] Yeah, that they should they should fix [426:53] this uh somehow because if you don't [426:56] know that, they should tell you that it [426:59] failed because of a syntax error. But [427:01] anyways, now we know if something gets [427:03] stuck uh that means that you have a [427:06] syntax error somewhere. And here we are [427:09] running those jobs in parallel. You can [427:11] see that backend Linton is uh requires [427:16] both of these jobs to succeed before [427:19] running. And as I told you, Bandit will [427:21] not fail. But let's look at the logs [427:23] because I want to show you that we have [427:26] so Banditis found 1,900 lines of code. [427:29] Here are the metrics. So you can see [427:30] that we have 159 low issues. If we [427:34] didn't skip low issues, you can see that [427:38] it will be a nightmare to look at all of [427:40] these problems. And most of them are [427:43] going to be false positive. These aren't [427:45] real problems in your code. But if we [427:48] had something in medium or high, now we [427:50] should look at those seriously. And here [427:52] you might be conf So don't confuse [427:55] issues and confidence. So here they have [427:58] total issues by severity. So these are [428:00] basically if we have an issue is it [428:02] severe or not in our case no they are [428:05] low and confidence is that here it is [428:09] set to high. This doesn't mean that all [428:10] of these problems are of a high [428:13] severity. No it's just the confidence [428:15] level. So here this means that bandit is [428:18] confident that all of these are low [428:20] problems that we that we should skip and [428:23] don't care about. It seems that this is [428:25] working. The pipeline has succeeded. [428:28] Everything has passed and now we can [428:29] merge without any problem. But before we [428:32] end this tutorial, I would like to show [428:33] you how banditis fails. That's [428:36] important. Um let's open let me open [428:38] main.py. I have added this because it [428:41] uses an outdated cryptography module. Um [428:45] I've just added something just to [428:47] trigger banditis. So let's see. Sure [428:51] trigger banditis, but I'm going to [428:53] remove it right away. I just want to [428:55] show you what happens when Bandit fails. [428:58] This will Yeah. Okay. So, let's wait for [429:01] this. Okay. So, it is it is running. Uh [429:05] let's let's click on Bandit. That's the [429:07] one we care about. Did it fail? Yes, it [429:10] did. So, the exit code is one. And now [429:13] we have one high problem. And if you [429:16] look closely, okay, so it is running [429:19] here. It says issue. Uh so it it found [429:22] that we used the MDA hash and it's uh [429:26] yeah it's not uh secure. This is an [429:29] outdated module that we should not use. [429:32] So here they show you the where where it [429:35] found the problem. Here is the location. [429:37] It's online 18. And you can see that now [429:40] we have one in the high severity [429:43] category. That confirms that bandit is [429:45] working and uh it it is reading the [429:48] code. [429:49] Uh it's not like the other the other [429:51] ones. So now we can remove this safely. [429:54] Uh we've confirmed that Bandis is [429:56] reading the code and uh let's wait for [430:00] this job to finish one more time. I want [430:03] to ensure that every every job passes [430:06] and that I can merge without any issue. [430:08] I am back. The pipeline took a minute [430:12] and 37 seconds. It succeeded which is a [430:16] great sign. And uh and yeah, we have [430:19] reached the end of this tutorial. You [430:22] know, I've really enjoyed making this [430:24] video. Uh it was challenging. Uh we have [430:27] hit a lot of problems and we fixed them [430:30] uh interactively. I hope that you [430:33] enjoyed it too and that you have learned [430:35] how to deal with securing your [430:38] application. You can see that the [430:39] pipeline is starting to get bigger and [430:41] bigger. We are adding we are making sure [430:44] that we develop a robust pipeline and by [430:48] the end of the of this module you will [430:52] get something that uh that is really [430:54] great and that will help you uh find [430:57] bugs easily and uh ensure that when you [431:00] hit deploy that your application will [431:02] run without any issue. In the next sub [431:05] chapter which is titled advanced testing [431:09] and dust. Dust is you can see that here [431:12] we have sust and dust is dynamic. So we [431:15] will add that one too. Uh we will still [431:17] continue working on testing and [431:20] security. So far we have just unit tests [431:23] but we will add integration tests and [431:26] end to end tests. Uh we will we will see [431:29] we will see that in details but I just [431:31] wanted you to know what is the next step [431:34] in our course. Thank you again for [431:36] watching. See you in the next sub [431:39] chapter. In this sub chapter we are [431:42] going to talk about advanced testing. [431:46] Here we are going to add both [431:48] integration and end to end tests and I'm [431:51] going to show you something called dust. [431:54] Here is the agenda that we will follow. [431:56] As I said, we will start by writing [431:58] integration tests. And here we are going [432:02] to test a live me search instance [432:06] because this is the search engine that I [432:08] am using on my personal project. We are [432:11] going to perform end to end tests using [432:14] the the playright tool in order to [432:17] simulate real users. And finally, we are [432:21] going to look at dynamic application [432:23] security testing. And we're going to do [432:26] that with this tool OASP zap. If you [432:30] want to follow with me, every command [432:33] will be here. Just go to this GitHub [432:35] repository, open the book folder and [432:38] here we are in chapter number five. Just [432:42] search DOS and you will find the this [432:45] new sub chapter advanced testing and [432:47] dust. And here you have all the commands [432:50] that I will run here. And this is [432:53] explained uh in detail. I will now close [432:56] this tab because we no longer need it. [432:58] And here you can see that I haven't [433:00] merged this pull request yet. We will [433:03] keep using it for uh the upcoming [433:05] videos. And as you can see at the end uh [433:09] if I click on this icon, you can see [433:12] that we have all these tests. We can [433:14] click on details here. Let's click on [433:16] summary. And currently this is what we [433:19] have in this video. We are going to add [433:21] integration tests and end to end tests [433:25] before the final pipeline success step. [433:29] Let's start doing this. Open the GitHub [433:33] folder. Open workflows. And here let's [433:36] you can see that we have backend [433:38] integration tests. This is what we want. [433:41] But before we do that, let's change the [433:45] branch to what did I call it? Test [433:48] number two. Okay, let's do it here. Test [433:51] number two. Okay, now we don't have that [433:54] file. So, we need to create it. Click on [433:56] this icon and add this file backend [434:00] integration tests. I will copy and paste [434:03] the code. And here, let me explain to [434:05] you what we have. There are some things [434:07] that we have seen before like checking [434:10] out the repository. I will collapse [434:13] that. Installing Python, installing the [434:16] dependencies. All of these steps are [434:19] known and uh we will skip them in this [434:23] uh sub chapter. We won't we don't want [434:25] to waste time here. The new stuff is the [434:28] main search environment variables and [434:30] the steps to launch that service. And [434:34] let me explain to you why do we need [434:36] integration tests. If I go back to [434:39] GitHub and open the pipeline, as you can [434:43] see, we have some tests already. We call [434:46] them unit tests. These are important [434:49] tests and they test few things in the [434:52] application in isolation. For example, [434:55] if I have a button or a function in the [434:58] back end that converts time into a [435:00] specific format, I can test that [435:02] function in isolation. And let's say if [435:05] I have if I am making requests to m [435:08] search, I am going to mock that. I am [435:11] going to uh not talk to M searcharch in [435:15] unit tests because here we are just [435:18] testing the individual functions or [435:21] functionalities. Integration tests are a [435:24] bit different. They test integrating [435:28] external services like me search or if [435:31] you have let's say a postgress SQL [435:34] database or other services you test them [435:37] in integration tests. And here when if I [435:40] want to make a let's say a search query [435:43] to me search I am going I need an [435:46] instance of me search running to be able [435:48] to do that that's why here in this uh [435:52] workflow we are going to set up m search [435:56] so that it runs in this job and so that [436:00] we can make requests to it. Here we have [436:02] a few environment variables. Uh these [436:05] are specific to me search. So if you are [436:07] using other services you need to change [436:09] this part but we have a master key a URL [436:13] and a name index name. Here we will [436:15] store documents. I will explain these [436:18] two environment variables down below [436:20] because they are important. Okay let's [436:22] start. Here we have the download me [436:25] search binary step. So after installing [436:27] the dependencies, Python and checking [436:30] out the repository, we need to install [436:32] me search and it is a binary file. So we [436:35] can download it easily using this curl [436:38] command. And uh this this also needs to [436:42] change if you use other services. And [436:45] here we have these two variables max [436:48] retries and retry delay. Sometimes in [436:52] the GitHub actions this step will fail [436:55] and you won't be able to download M [436:58] search and uh you might say oops the the [437:02] integration tests have failed but [437:04] actually the problem was just network [437:07] and uh we just need to retry this step [437:10] multiple times until we download mainly [437:12] search. So we are here uh redefining the [437:16] variables max retries and retry delay. [437:18] And here we are using a for loop to uh [437:21] try downloading me search max retries [437:25] times and in this case it is set to [437:27] three but you can change the value to [437:30] something higher. If this keeps failing [437:32] and if this step succeeds we will show [437:36] my search was downloaded and we will [437:39] break out of the loop. Otherwise we will [437:42] exit with one and uh tell GitHub that [437:45] this step has failed. Okay. So here we [437:48] download me search. After that we need [437:51] to start it. And in order to start it we [437:53] just run the binary and we pass the [437:56] master key. And here notice what I have [437:58] what I have added at the end. I have [438:00] this amperand. And this is very [438:03] important. If you don't add this, this [438:06] call will block this job or this step [438:09] and we won't be able to run the next [438:11] steps. But this by adding an amperand, [438:14] it will run this as a back as a [438:17] background job and it will allow the [438:19] next steps to run without being blocked. [438:22] And here when we run medarch, we need to [438:24] ensure that the server has started. And [438:27] the way we do that is we call the health [438:30] endpoint. If this endpoint replies it [438:34] means that M search is ready to accept [438:36] requests. That's this is what we are [438:39] doing here. And again we are using a for [438:41] loop so that we keep pinging this [438:44] endpoint. If we detect available great [438:47] we are going to show this in the logs [438:50] and we will break out. Otherwise we will [438:53] sleep a little bit to uh retry. We don't [438:55] want to do this quickly. And here uh at [438:58] the end we just if this uh doesn't uh [439:01] work we will show that the me search [439:04] fails to start after the number of [439:06] attempts that we have here. If this [439:08] succeeds we will run the integration [439:11] tests and uh again we will use piest and [439:15] in my case I I am storing you can see [439:18] that I try to group tests by their [439:21] nature. Here I have all of my unit [439:23] tests. Here I have integration tests and [439:26] I also have load testing. We might see [439:28] this in upcoming uh subchapters. And as [439:32] you can see here I am using tbhore so [439:33] that the exceptions are not very verbose [439:37] and I passed the environment variables [439:39] because we need them inside the tests. [439:42] So the only thing that I want you to [439:44] take from this workflow is that in [439:47] integration tests you need to start your [439:50] services. Uh if you have me search this [439:52] is how you do it. If you have Postgress [439:54] SQL, you need to do it differently. If [439:58] you are having services in Docker, you [440:00] need to run the containers, stuff like [440:03] that. And after running all of your [440:05] services, you can run your tests. And [440:08] the beauty is that here if something [440:10] broke, it's going to uh be uh very [440:13] informative because you are testing the [440:16] live services. Awesome. Now, we can go [440:20] back to CI.YAML YAML and add this [440:24] integration test there commit and we can [440:27] see if that will fail or not. Let's open [440:30] CI.YAML [440:32] and here let's add that job. We have [440:35] backend integration tests. I want to run [440:38] it after unit tests. And uh here is the [440:42] file. I need to copy this and also add [440:45] it here below backend unit tests. And I [440:49] guess that's it. Let's commit this. I [440:52] have pushed the code. Now let's go back [440:54] to GitHub. Let's refresh the page and [440:57] let's see if this is going to uh fail or [441:01] not. Let's open the details page and [441:04] let's see what we have. I want to go [441:07] back to summary. I want to see how this [441:09] looks like. And yeah, everything is [441:11] good. So we have unit tests and after [441:13] that we will run the integration tests. [441:16] I will wait for all these steps to [441:17] finish and I will come back. And as you [441:19] can see here, we have a problem with [441:21] audits. Probably something in the front [441:23] end changed. But this is not a problem. [441:25] We now know how to solve this. Uh I will [441:27] just focus on uh the integration tests [441:30] and then maybe we will we can come back [441:32] here and solve the problems that we [441:34] have. Now we are in the unit tests [441:37] phase. [snorts] Let's wait for it. Okay. [441:39] Uh let's click on backend integration [441:42] tests because I want to see the logs. [441:44] Okay. So far so good. These are the [441:47] steps that we that we have in every job. [441:50] Okay, great. Now we are downloading [441:51] Millie search and as you can see this [441:54] time it worked. Here we we are showing [441:57] attempt one of three and yeah we are [442:00] still trying to download it and that [442:02] that worked great and you can see that [442:05] they show us how to run it. Now here in [442:08] this step we are running it and as you [442:11] can see uh these are the logs and [442:14] awesome that worked. So far so good. Uh [442:17] let's collapse all of these steps. Okay, [442:20] but I think yeah, we have a problem [442:22] here. Import error while loading conf [442:25] test. Okay, so it couldn't import [442:27] something because it doesn't understand [442:30] fast API. Interesting. So I guess we [442:32] need to fix something when we in when we [442:36] are installing the dependencies. Okay, [442:39] let's go back here. So we are installing [442:42] requirements dev and uh what happens [442:46] what what's the problem? So if we go [442:48] back to back end we have divided the [442:50] requirements into two files. Ah I see so [442:54] I forgot to add - r. Yeah the difference [442:57] here is that we need to also do this. So [443:01] we need to do - r requirements [443:04] txt or we can do it here. So we do r [443:09] requirements.txt. txt space-ash r [443:12] requirementsdev.txt [443:14] txt and u yeah this time I will do it [443:17] this way pip will go inside requirements [443:21] dev it's going to find this it's going [443:24] to open requirements txt and it's going [443:26] to install fast API okay let's commit [443:29] this and let's push let's go back to [443:33] GitHub the pipeline has started let's [443:37] let's see the details let's open the [443:40] integration tests and I think this time [443:44] it should work. Let's collapse all of [443:46] these steps. We've installed M [443:49] searcharch successfully. We have started [443:52] it. Okay. And now we are running [443:55] integration tests and as you can see we [443:57] didn't have we didn't get the same error [443:59] and they succeeded. Great. So that [444:02] worked. Let's go back here to summary. [444:05] As you can see, both tests [444:09] uh run successfully, but the pipeline [444:11] didn't succeed because we have a problem [444:14] in the audit dependencies job. And as [444:18] you can see here, we have a problem with [444:20] JS cookie vest. So these are the two [444:24] libraries that we need to fix. Let's do [444:26] that. While we are here, remember that [444:29] we have PNPM workspace and here we can [444:32] add the libraries. uh we need to [444:35] override them because in my case I am [444:37] not using JS cookie that's why I need to [444:41] put it here in the overrides object and [444:44] what was the the version 306 okay so [444:48] let's go back here and let's put 306 [444:52] that's the first one and what what else [444:55] we have test so for test I am using it [444:59] so if we open package JSON You can see [445:04] that I have a V test. Here it is. And [445:07] the here the version is 4014. [445:11] Okay, that's why. Yeah, that's why it [445:13] will it is complaining. So, we need to [445:15] upgrade that to for the 1.0. [445:19] Let's go inside the front end folder. [445:21] And here, let's try to install the [445:25] newest version. Here is the full [445:28] command. Let me open package.json JSON [445:31] again just to verify that we have V [445:34] test. Yes, it is in the dev [445:36] dependencies. This is why we are adding [445:38] dash D and yeah let's install the newest [445:43] version. Okay, it is installing and we [445:46] are using PMP [445:49] version 10 because this is a a fresh [445:52] virtual machine. So that's fine. You can [445:54] see that vest uh we were able to update [445:57] it upgrade it to 4.1 and uh yeah let's [446:01] see what we have here package.json yeah [446:04] that worked. I have updated pmppm to [446:07] version 11 because uh I don't want to [446:09] see any error in uh this PCI pipeline. [446:14] Let's run pmp install just to ensure [446:17] that nothing uh oh sorry I need to go to [446:20] front end and let's run pmppm install. [446:24] Uh let's say yes. Okay, we are good to [446:28] go. Let me close the terminal and let's [446:32] say sure update dependencies. Let's sync [446:36] the changes and hopefully this time the [446:38] pipeline will succeed. We will let that [446:41] one running and I will come back to it. [446:43] But now we need to add let's see we need [446:46] to add end to end tests. Let's open [446:48] workflows again and let's create this [446:51] file end to end tests. I will copy the [446:55] code and I will paste it here. Uh let me [446:59] collapse this and let's look at this [447:01] environment variables. This is identical [447:04] to what we have there. And in end to end [447:06] test we need to have everything running. [447:08] We need to have the front end, back end, [447:10] and all services. So here we have all of [447:14] these environment variables for me [447:15] search because we need to start the mele [447:17] search service. And let's see what we [447:20] have here. Here we added timeout minutes [447:22] 30 minutes. Uh we don't want so if end [447:25] to end test for some reason take more [447:27] than 30 minutes, we want to time out. [447:29] This depends on how many tests you have [447:31] and how and the complexity, but for me [447:34] 30 minutes is more than enough. Here we [447:36] are checking the repository. We install [447:38] Python here. Let's see. So we want to [447:42] install [447:44] R requirements and here let me add [447:46] dev.txt [447:48] because we want to run uh maybe this is [447:51] not important here. Yeah, it's not [447:53] important because we won't run it. We [447:56] will be right we will be using the [447:58] playright tool and that one runs in the [448:00] front end. That's why we just need the [448:03] production dependencies to run the fast [448:05] API service or server. Here we download [448:09] me search. This is identical to what we [448:11] have in the integration tests. We start [448:14] search and here we start the backend [448:16] server. We are using uicorn and again we [448:20] have the same for loop. We ping the [448:23] health endpoint and if it returns okay [448:25] that's great. We know that the backend [448:27] has started otherwise we exit. And here [448:31] we have a step that I called see the end [448:34] to end test data. When we create when we [448:37] download search and start it, it's going [448:40] to be empty. We won't have any data to [448:42] play with. In end to end tests, we need [448:45] to have data in the browser so that we [448:47] can interact with uh articles or [448:50] products or any it depends on the on the [448:53] project that you are working on. And in [448:55] my case, I want to put data in my [448:58] research. So I have created a a specific [449:01] file that I called test data that will [449:04] insert documents to many search. You can [449:07] see that here is the index that I am [449:09] using for tests. It's called articles in [449:13] test here and here here are all the [449:16] documents that I will inject just for [449:19] end to end tests. If you have Postgress [449:22] or other databases, you can do the same [449:24] something similar where you seed your uh [449:27] where you seed the test data and after [449:30] that we install here we have we use PNPM [449:33] version 11 we install node we install [449:36] the frontend dependencies and now we [449:39] arrived at the playright tool. Okay. So [449:42] we install it we check the version the [449:45] version is coming from again [449:47] package.json. [449:48] Uh if we look at playright test here it [449:51] is we take this specific version and [449:55] display it in the logs. Okay. After that [449:57] we cache play right because here we in [450:00] we need to download browsers and uh this [450:03] depends on your configuration. If you [450:06] some if you want to run the test only on [450:09] chrome this step will download Chrome. [450:11] If you want to also add Firefox it's [450:13] going to download Firefox as well. If [450:15] you want to to do it on Safari, you will [450:17] install Safari. So, and if the browsers [450:20] are the same, we need to cache them so [450:22] that next time when we run end to end [450:24] tests, they are going to be there and we [450:26] are just going to use them. And for [450:27] that, we are using uh an action called [450:30] cache, which is a very handy uh GitHub [450:34] action that allows us to cache stuff and [450:37] uh as you can see the continuation is [450:39] here. So here we basically define the [450:41] keys where we want to store um the cache [450:44] for playright and here we just check. So [450:48] here if there is no cache we will [450:50] execute this function that will install [450:53] here I want to run the tests on Chrome [450:56] and Firefox I don't want to use Safari [450:59] otherwise if we if we have downloaded [451:02] the the browsers previously we will just [451:05] uh go here and as you can see we have an [451:06] if block so if we have the cache so we [451:10] are checking if it is true that means we [451:12] can just reuse that and we don't need so [451:15] if we look at the difference between the [451:17] You can see that here we have with [451:19] dependencies [451:20] and this takes more time. Uh but if we [451:23] have things in the cache that will make [451:25] things faster. So after all of these [451:28] steps you can see that this is why I [451:30] said that end to end tests uh this [451:32] workflow is going to involve a lot of uh [451:35] work but we are just reusing what we [451:37] already have in the other workflows. So [451:40] after setting up all of these s servers [451:44] we can run the end to end tests and uh [451:47] in package.json JSON I have this script [451:50] that will run end to end tests and here [451:53] I pass environment CI to true to [451:56] differentiate between because you can [451:58] run end to end tests locally on your [452:00] laptop and or in this case we run it on [452:04] G using a GitHub runner and finally [452:07] after the the end of uh the tests we [452:11] will upload the report we will use the [452:13] upload artifact action and that will [452:17] allow us to download the report and [452:19] reports are amazing. Locally when the [452:23] tests finish uh playright will open a [452:26] new tab and show you the report. But [452:28] here since we are running this on GitHub [452:31] we can't open a new tab. So we need to [452:33] download a file that will that contains [452:36] the report. We can check it later. We [452:38] can see screenshots. Yeah, it's uh it's [452:41] really handy. Okay. So this is the end [452:43] to end tests and this will run as I said [452:47] on the cloud using GitHub actions. But [452:50] what if you want to run this locally? [452:52] For that I mean here on the repository [452:54] everything is set up already. Uh you can [452:57] see that we have this playright [452:59] configuration file that will come when [453:01] you install playright. And the way you [453:03] do it is simple. You just you just run [453:07] pnpm create playrite and this will [453:10] install playright for you. When you do [453:12] this the playright configuration JSON [453:15] will be created with some default [453:16] values. Maybe there um this command will [453:20] also create a uh some tests so that you [453:23] can just uh make sure that playright is [453:26] running but you can delete all of that. [453:29] Let's run the command together and let's [453:30] see what happens. You can see that here [453:32] it says do you want to use typescript or [453:35] javascript. In my case this project was [453:37] using javascript that's why I kept [453:39] javascript but if you rely on typescript [453:43] uh change that to typescript and here [453:45] they tell you where to put your end [453:47] toend tests. Uh in my case if I look at [453:49] the front end I always try to group my [453:51] tests so they are inside the end to end [453:54] tests. So here the default value is end [453:56] to end. Okay let's just keep it as end [453:59] to end test. uh add a GitHub actions [454:01] workflow. I mean no because we have we [454:04] did it ourselves. And here this step [454:08] tells you do you want to install [454:10] playright browsers? This can be done [454:12] manually via pmppm execute playright [454:15] install. And this is the exact command [454:19] that we have here. As you can see pmppm [454:21] exe execute playright. Let's install [454:24] them because in this machine we don't [454:26] have them. install play operating system [454:28] dependencies. Why not? And you do this [454:32] once and later you don't need to uh [454:35] install the browsers every single time. [454:37] Okay, so this finished. There was [454:40] something that was added to uh the logs [454:43] to the log files, but I I won't change [454:46] this. It's not important. So I'll just [454:48] undo this. I just wanted to show you the [454:50] steps that you need to run. And uh okay, [454:53] let's see what we have here. Okay. So, [454:56] we didn't override the file because it [454:58] found it. But uh in your case, if you [455:01] don't have this file, this is going to [455:02] be created for you. So, let's go through [455:04] it together in order to understand what [455:06] we have here. First of all, we tell [455:09] playright that our end to end tests are [455:12] located inside this folder. We want to [455:15] run things in parallel so that tests uh [455:19] finish quickly. Here we set the number [455:22] of retries. So if we are on CI sometimes [455:25] tests fail randomly so we want to retry [455:28] them but if you are locally we won't [455:30] retry anything here forbid only is [455:34] sometimes when you run end to end tests [455:37] you can tell player to run just one but [455:40] in our CI job we want to run everything [455:43] so this will fail if you tell player to [455:45] run one one test only because it's going [455:48] to skip the other ones and we don't want [455:50] to allow that. Here we have reporter. [455:53] This is also important because when you [455:55] run uh tests locally as I said you can [455:58] get an HTML page that will show you what [456:02] failed, what succeeded. Uh but if we are [456:05] in CI we are running this on GitHub. So [456:07] we need to to tell play that please show [456:09] us nice logs on GitHub. Here we define [456:12] the projects. You can see that I I am [456:14] using Chromium and Firefox. If you want [456:17] you can also add Safari down below. Uh [456:19] you can change this however you like. [456:21] And finally we have something very [456:23] interesting which is web server. This is [456:26] this will allow playright to run the [456:28] server the front end for you. So if you [456:30] forget to add a step to run the front [456:33] end playright will use this command to [456:36] start it for you and you tell it the URL [456:39] or and here the use existing server is [456:42] uh is interesting. So if uh we started [456:46] the server manually play will see that [456:49] oh the user has started the server so I [456:52] don't need to run this command. This is [456:54] why we use uh reuse existing server and [456:56] again we have timeout. Awesome. So now [457:00] we can close this. Let's open uh ci.yml [457:04] and let's add end to end tests there. [457:08] Okay we have integration tests. Let's [457:11] add end to end test underneath it. Copy [457:14] this and let's paste it here and let's [457:18] see what did I do in the needs um list. [457:23] So I am waiting for the integration [457:26] tests and also unit tests uh in the [457:29] front from the front end. Okay, let's [457:31] now commit this add end to end tests [457:34] commit and push. Okay, uh the tests are [457:40] running. Let's look at the details page. [457:43] Let's go back to summary and let's [457:45] ensure that the end to end test is [457:48] correctly placed. Yes, it is. You can [457:50] see that it is waiting for the front end [457:53] unit tests and the backend integration [457:56] tests. I will wait for these jobs to [457:59] finish and I will come back once the end [458:01] to end tests uh start. I am back and as [458:04] you can see so far everything is green. [458:08] Uh and wait yes the end to end tests [458:10] started because uh front the unit tests [458:14] in front end and integration tests in [458:16] back end didn't fail. Let's click here [458:18] and let's see what is happening. We are [458:20] now installing the backend dependencies. [458:23] We are downloading millarch. Good. That [458:26] worked. Now we are starting the mill [458:28] search server. That worked also. Let's [458:32] now start the backend server using [458:34] uicorn. And as you can see, we were able [458:36] to ping the health endpoint. We seeded [458:40] the M search with 15 documents. We've [458:44] installed PNPM node. And now we are [458:48] installing the playright browsers. And [458:49] as you can see, this takes time because [458:52] now as you can see, we are in the if no [458:54] cache state in the upcoming commits. Uh [458:57] this will not run and it's instead it's [459:00] going to to download to take the [459:02] browsers from the cache. Okay. Yeah, as [459:05] I said, this takes some time. Where are [459:08] we at? Okay, so it took 30 seconds. [459:11] Okay, and now we are you can see that [459:13] this the other the the other step was [459:15] skipped because we don't have any cache [459:18] and now we are running the end to end [459:20] tests and as you can see we have 178 [459:22] tests and here we are using two workers. [459:25] So they are going to run the tests in [459:27] parallel. You can see that the tests are [459:30] uh so here we are running the first one, [459:32] second one, third one but now they start [459:34] randomly because we are running them uh [459:36] not in a sequential order but in [459:38] parallel. Okay, end to end tests usually [459:42] take minutes to finish and it depends on [459:44] how you uh design them. Mine uh mine [459:48] will take around 3 minutes to finish. So [459:51] I will stop the video here and I will [459:54] come back. We are almost done. Uh we [459:57] have 178 tests. Let's see. So they that [460:01] should finish now. And uh all of them [460:03] succeeded. You can see it took 3 [460:05] minutes. And here, okay, we are trying [460:08] to upload the report. And awesome. [460:12] Everything works. Let's go back to [460:14] summary. Amazing. You can see that the [460:17] pi the whole pipeline took almost 7 [460:20] minutes. Okay, let's wait for the last [460:23] or let's go back. Let's go back here. Uh [460:26] yeah, everything passed. Amazing. Now [460:29] what I would like to do is I want to [460:31] make a small change just to show you [460:33] that the cache is working. Let's see [460:36] what should I do here. I will just [460:39] remove this comma from this file cuz [460:43] it's not needed. So sure, remove extra [460:48] comma and let's push this. Let's go back [460:51] to GitHub. Okay, everything is uh okay. [460:57] Let's click on summary and let's wait [461:00] for end to end tests to start and [461:02] hopefully this time we should see that [461:04] we will be uh we will uh run the if [461:08] cache uh step. The end to end tests job [461:12] has started. Let's wait for Okay, so we [461:16] are now in the front end area. Okay, we [461:19] are installing the dependencies. [461:22] Let's see what we have here. So we are [461:24] looking at the cache and as you can see [461:26] that worked. Uh let me collapse this. [461:29] Before we ran this step if no cache but [461:33] now we run we are running this other [461:36] step if cash hit. Before this step took [461:40] 30 seconds but now this one took almost [461:43] half of that. It's just 15 seconds but [461:46] it's it's working. It's better than [461:48] nothing. And yeah, the end to end test [461:51] will succeed and uh uh yeah, everything [461:53] will work. Let's create a new workflow [461:57] called dast scan. I will copy the code [462:01] and paste it here and let's see what it [462:04] does. First of all, you can see that we [462:07] are we also need my search because here [462:10] we are going to scan the API. First of [462:12] all, we are checking the repository. We [462:15] set up Python. We install the [462:17] dependencies. We download me search. We [462:21] run it. Here we see the data because we [462:23] want to have something in me search. We [462:26] start the back end. And uh finally this [462:29] is the new step uh that will use the zap [462:34] project. And if you are not familiar [462:37] with zap here it is. You can check their [462:40] website. Uh it's an amazing project. [462:42] It's free and open source. And uh we you [462:45] can use it to scan your applications. [462:48] And they have good documentation as [462:50] well. We will use this in this video [462:52] because by default this scan is going to [462:57] flag a lot of things that are uh that [463:01] are not problematic because we are not [463:03] in production. Well, I will show you I [463:06] will show you that. So let's look at [463:08] this step. You can see that we are using [463:10] a GitHub action provided by the Zap [463:13] team. And here the target is the open [463:16] API endpoint. Because we are using fast [463:20] API by default it gives us all the [463:23] endpoints in our back end. And this is [463:26] useful because Zap will use this in [463:29] order to know which endpoints to target. [463:32] The format is open API. And here you can [463:36] see that I'm referring to a file uh [463:40] called zap rules. Right now I will just [463:42] delete this one because we don't have it [463:44] and I will I want to do this so that you [463:47] understand the reason why we add this [463:49] specific file. I have deleted that line. [463:53] Now open ci.l because we need to add [463:55] this scan here in the main pipeline. I [463:59] will scroll up. Let's look at backend [464:02] length and format. I want to put it uh [464:06] or I want to run it after LinkedIn. So, [464:09] let me let's go down below. Let's paste [464:12] it here. And as you can see uh it's [464:16] going to run after the l after the lint [464:18] job. Now, copy it and paste it here. [464:23] Now, let's commit this. Add dust and [464:26] let's push it to the repository. Let's [464:30] go back. Let's click on the the pull [464:32] request. Scroll down and let's see. We [464:36] should see the pipeline running. Okay, [464:39] that's good. Let's try to see to look at [464:42] the details. Click on summary and let's [464:45] wait for dust to run. As you can see, it [464:49] should be right after. Yeah, it is after [464:53] uh backend link and format. You can see [464:55] that right now we have two kinds of [464:57] tests. We have sust and dust. The [465:01] difference is that SAS checks your row [465:04] source code for security flows. But some [465:07] security problems only appear while [465:10] interacting with with your application. [465:12] This is where Dusk comes in and this is [465:15] why we started mini search and the back [465:17] end and we gave it as the target the [465:20] open API spec so that when it looks at [465:23] that at those endpoints it's going to [465:25] attack them like any uh attacker would [465:28] do uh in order to find vulnerabilities [465:31] in your application. And as you can see [465:34] right now it failed. Okay. So why did it [465:36] fail? It failed because it couldn't [465:39] download me search. You can see that [465:41] even though we did three attempts, so [465:44] the first one failed, the second one [465:47] failed and the third one also failed. Uh [465:49] if this happens for you, no problem. You [465:52] can just go back and rerun the job. [465:55] Okay, so I can run rerun failed jobs. Uh [465:59] it gives me all of these. So I'll hit [466:01] rerun. Okay, let's wait for now. Let's [466:05] click again on this job. Hopefully this [466:08] time we will be able to download Mary [466:10] search. Sadly it failed again for some [466:13] reason. I'm not sure why. I guess we [466:15] just need to wait a little bit and [466:17] retry. They tell us to download this [466:20] manually from GitHub. But yeah, we [466:23] cannot do that here in the pipeline. We [466:26] we will need to change that specific [466:28] step. Okay, I will just I'll try to [466:31] rerun it one more time. If it fails [466:33] again, I will wait for a few minutes and [466:36] come back. I did some research and I [466:38] think the problem that we have here is [466:40] that if we look at their latest version, [466:43] you can see that it was released 14 [466:46] minutes ago. But creating the assets [466:49] take time. You can see that for example [466:51] this one was just created a minute ago [466:53] even though the release was pushed 15 [466:56] minutes ago. So that's why we are [466:59] getting 404s because um the installer [467:03] script is trying to download something [467:06] from GitHub that doesn't exist yet. This [467:09] is why we are getting uh 404 not found. [467:13] So we have two options here. We need to [467:17] wait a little bit to get the new file [467:21] and then we can run the script again or [467:24] in the future we need to update this to [467:28] download a specific version because here [467:31] when we run the installer script we try [467:34] to download the latest version or the [467:37] latest tag from GitHub and in this case [467:40] it is [467:42] 146.1 [467:43] right now I will keep the script like [467:45] this. But uh right after uh explaining [467:49] the dust scan, I will go back and I will [467:51] show you how to update only this [467:54] specific step to uh download a specific [467:58] version. And uh yeah, we will remove [468:00] this line because it downloads all it [468:03] tries to download the latest version [468:05] always. Okay, I rerun the script and uh [468:09] as you can see it worked because now we [468:11] we have uh the assets were created and [468:15] uh the installer script was able to get [468:18] it from GitHub. Awesome. So now let's [468:20] see as you can see the zap scan step is [468:23] running and uh okay so here let's see [468:27] what we have. We see a deprecation [468:30] warning that's fine. We can uh avoid [468:33] that. You can see that here it found 15 [468:35] URLs and it was able to find them [468:38] because we provided the open API spec [468:41] and uh here I want to mention that the [468:44] dust scan that we are using is [468:46] lightweight because it is targeting just [468:48] the back end but we can also have a [468:52] serious one that will check the front [468:53] end as well. That one takes a lot of [468:56] time. This is why I didn't include it in [468:58] the main pipeline. Instead, we will [469:01] create another file that will uh have [469:04] the full dust scan and we will run it [469:07] daily um outside of the the pipeline [469:10] because we don't want to block [469:12] everything just to perform that heavy [469:15] scan and as as you can see this thing [469:18] has failed and uh I was expecting that. [469:21] Uh let's see what we have here. You can [469:23] see that it checks a lot of things and [469:26] most of them passed. As you can see, [469:28] everything passed except some problems. [469:32] And as you can see here, it says X [469:34] content type options header is missing. [469:36] Uh let's see what we have. We also have [469:38] cross origin resource policy missing. [469:42] And as I told you, we will have these [469:44] problems because we are running this in [469:46] a CI environment. It's not running in [469:50] production. And if you remember in the [469:52] past we have we have resolved these [469:56] problems and in our application that we [469:58] deployed we are using engineext we added [470:01] the HTTP security headers so everything [470:04] is set up already and zap is telling us [470:07] that these are missings that's that's [470:09] correct because we are running on a CI [470:11] environment this is why we add that [470:15] specific file that I was talking about [470:18] because it allows us to tell zap to skip [470:22] those specific tests. Here I will just [470:25] add rules file name and we need to [470:28] create this one in the GitHub folder. Uh [470:32] let's create it here. Uh it's already [470:34] there. Okay. So let's look at it. Here [470:37] is the zap rules file and this is [470:40] specific to my codebase. So if you add [470:44] dust to your codebase, you might have [470:47] different warnings. So in that case you [470:50] need to change this list. So how do we [470:52] do this? If I open let let me show you. [470:55] So here we have ids and the format is [470:59] simple. You have the ID of the rule. [471:01] It's the first column. You have the [471:03] action and you have a comment. So here [471:07] we tell zap to ignore all of these rules [471:12] because here the action is ignore. And [471:14] this comment is for us so that we can [471:16] understand what is happening here. How [471:18] do we get these rule ids? You can get [471:21] them from their documentation. You can [471:24] see that here. For example, if we open [471:26] this active scan rules, you have the ids [471:30] here. So they they show you the rule, [471:33] they explain it and they also give you [471:35] the ID. There is also this website that [471:38] I will make sure to drop in the video [471:41] description where you can find uh every [471:45] every rule. And here if we search for [471:47] cross origin which is basically this [471:50] error that we got here. Now you can see [471:53] the ID is 90,000 and4. Here it is. You [471:57] can see that we have uh uh a lot of them [472:00] here. And yeah you can use this website [472:03] to uh learn more about uh these specific [472:07] rules. And if you want to read it you [472:10] can click on it and you will find here a [472:12] summary solution etc. Okay. And uh we [472:15] can you can also find the ids here [472:17] directly from the logs. Uh so for [472:19] example here I can take this id and uh [472:23] paste it here in the rules. Uh here it [472:26] is. You can see 10,21 [472:29] uh and I have added it here. Once you do [472:31] this the errors are going to go away. [472:34] But make sure to not skip everything. [472:37] Skip only the things that are related to [472:39] a production environment. And as you can [472:41] see here, many things are handled by [472:44] EngineX. That's why I've added them [472:46] here. And for example, this one is [472:48] failing because we are running this as [472:51] HTTP. We are not using HTTPS. And again, [472:54] because we are not in production. This [472:56] is fine. There is no problem with that. [472:59] Here we have a problem with timestamps [473:01] because M search uses Unix timestamps. [473:04] It doesn't use other formats. And here [473:07] uh Zap is thinking that this is a [473:09] problem. But in fact that's how med [473:12] search works. Okay. Now what I will try [473:15] to update this uh so let's say CI add [473:21] zap rules commit this. Let's sync the [473:24] changes and let's see what will happen [473:27] this time. I'll go back to the pull [473:30] requests. Let's look at the pipeline in [473:33] detail. Go to summary. Let's click on [473:37] OASP zap API scan. Uh we need to wait [473:40] for it first to start. Okay, let's see [473:44] the now we are auditing the backend [473:46] dependencies. This should pass without [473:49] any problem. Great, that worked. Now [473:52] let's go to LinkedIn and formatting. And [473:55] now let's look at the dust scan. The [473:59] scan has started again. It found 15 [474:03] URLs. And as you can see, it is using [474:06] our zap rules file. Okay, let's see now [474:09] what will happen. Hopefully, it's uh [474:12] it's not going to fail. And yes, it [474:14] didn't. Let's see what we have here. [474:17] Okay, you can see that they tested 197 [474:22] uh rules or URLs. [474:24] Um so all of them passed and again we [474:27] have those problems with timestamp, [474:29] cross origin, etc. because we have added [474:32] them to zap rules the zap project or the [474:37] zap tool is not failing it is skipping [474:40] them and u that's why uh the pipeline [474:44] has passed something to note is that [474:47] over time this file will stabilize I'm [474:50] talking about zap rules because [474:52] especially if you keep adding more [474:54] endpoints if you keep expanding your [474:56] backend application yeah zap is going to [474:59] find some uh small issues that you can [475:02] skip. After you do that job once, when [475:06] you get an error in your CI pipeline, it [475:09] means that you have a real security [475:12] issue that needs your immediate [475:14] attention. Right now, uh you can see [475:16] that we got a few errors because uh we [475:19] are just setting up D for the first time [475:22] and that's normal because uh by default [475:25] these tools are trying to find anything. [475:28] So that's why you get false alarms. But [475:31] once you handle those, you you are good [475:34] to go. And uh when you get an error, [475:37] that's a good that's a good signal that [475:39] something bad is happening. Great. And [475:41] as you can see, the other steps are are [475:43] also running without any problem. So we [475:46] were able to run integration tests. End [475:48] to end test should also pass. And [475:50] finally, we will uh the pipeline will [475:52] succeed. Awesome. We have done [475:54] everything uh in this sub chapter. But [475:58] uh let's let me also show you how to [476:01] update the download step so that you can [476:03] install a specific version. I'm not [476:06] going to apply that to the to all files [476:08] because I like to download the last [476:10] version and I don't want to go update [476:13] the the file and then commit to be able [476:16] to download the last version. But if you [476:18] are curious about how we can accomplish [476:20] this, here is how to do this. I will uh [476:23] remove all of this and I will paste [476:26] this. Okay. And now what do we need to [476:28] do? We need to go to GitHub. Click on [476:31] releases. And here the last one is [476:34] 146.1. [476:36] I will take this. I will take this one. [476:39] So here we need to change this variable [476:42] to uh target this specific tag. Here we [476:46] are just echoing this message. And uh [476:50] here in the curl command we have uh this [476:53] handy arguments that we can use. Before [476:57] we uh we had to add these environment [476:59] variables. Yeah. Because we had an [477:02] external loop. But here we can do that. [477:04] We can do it directly with curl. So if [477:08] uh the command fails we retry three [477:11] times and between each retry we wait for [477:13] 5 seconds. And we are we go to the [477:16] releases page download version and we [477:19] download this specific binary. After we [477:22] do that we uh make it executable so that [477:25] we can start it. And uh you can see it [477:28] is smaller than the previous solution. [477:31] But this one requires you to always open [477:33] the file, update this commit, push and [477:37] wait for the pipeline to finish. That's [477:39] the the difference between the two. But [477:41] if you prefer this, you can keep it. Uh [477:44] in my case, as I said, I will not keep [477:46] that one. I I like to always download [477:48] the last version. We have arrived at the [477:51] end of this sub chapter. I really hope [477:52] that you enjoyed it. In my case, I I [477:55] did. And uh it's always great to [477:58] struggle to build a pipeline because [478:00] once you do, uh you you see this uh [478:03] amazing view where everything is green. [478:06] And um yeah, it's super super useful. [478:10] And now we finished the continuation the [478:12] continuous integration pipeline. In the [478:15] next uh subchapter we will work on [478:18] continuous delivery which is also [478:20] amazing. Uh we will connect this [478:23] pipeline to our virtual machine on [478:26] digital ocean. to that if uh this step [478:29] which I'm I'm referring to continuous [478:31] integration passes we will click on a [478:34] button to deploy the built application [478:38] directly without having to SSH to the VM [478:42] copy the code run some some commands we [478:46] will create a a workflow that will do [478:48] this for us and we will just hit the [478:51] button wait for a minute or two and see [478:54] the new version live on the website. We [478:57] will also update this pipeline because [479:00] you can see that pipeline success [479:01] depends on every job before it. So the [479:06] so the graph looks a little bit ugly. [479:08] But we just need to look at the last [479:12] job. We we need to wait for end to end [479:14] tests because if we wait for this job, [479:17] we are waiting for this one. This one we [479:20] are waiting for every previous job. So [479:22] we don't need to uh make this uh let's [479:27] go back to CI we can simplify this needs [479:30] to just uh the end to end tests we know [479:33] we don't need to have all of them [479:36] because implicit they are implicitly uh [479:39] within the end to end tests if we are [479:41] waiting for that specific job. So we [479:43] will do some optimizations to make the [479:45] pipeline uh cleaner and uh yeah that sub [479:49] chapter will be the the last one in [479:51] module 5. After that we will move to [479:54] module six which is the last module in [479:56] our course. See you in the next sub [479:59] chapter. We have arrived at continuous [480:02] delivery. This is the last subchapter in [480:06] module 5 and here are the things that we [480:09] are going to talk about. I will show you [480:11] how to compile and upload the static [480:14] view front end. You will trigger [480:16] automated database and search engine [480:19] backups. Before deploying, we are going [480:22] to require a human review using GitHub [480:26] environments. And finally, we will [480:28] execute an atomic virtual environment [480:31] roll back strategy. So if something bad [480:34] happened, we can go back to the previous [480:37] working version. In the [clears throat] [480:38] last sub chapter, we have built this [480:42] pipeline and the last step is end to end [480:47] tests. What are we going to do before [480:49] moving to continuous delivery is to also [480:51] add front end build. So we will try to [480:54] build the front end to make sure that [480:55] there is no syntax error that there is [480:57] no problem in the dependencies that we [481:00] are using before finishing the [481:02] continuous integration pipeline. I have [481:05] I forgot to add this in the previous sub [481:07] chapter but we are going to add it now. [481:09] Open the GitHub folder. Open workflows [481:13] and add this new file content build.yml. [481:18] But before I do that I need to change to [481:21] test number two because this is the [481:23] branch that we were using. Okay, let's [481:25] now try to create the file. Okay, now we [481:28] are able to create it. I will copy and [481:31] paste the code here. It is similar to [481:34] what we have done before. The name here [481:36] is build front end. First we check the [481:38] repository. We use pmppm version 11. We [481:43] install the dependencies and after that [481:45] we just run pmppm build. If everything [481:47] goes all right, that's what we want. But [481:49] if something bad happens, this job will [481:52] fail. After that, we are going to upload [481:54] the artifact so that if we want, we can [481:57] reuse the build files in a next job or [482:01] something like that. And the retention [482:03] days is set to one because because if [482:05] you add another feature, we are not [482:08] going to use this uh cache. So there is [482:11] no need for it. It's going to become old [482:13] and it's going to be uh removed. Now [482:15] let's open ci.l and let's add this job [482:19] there. Here is how we do this. I'm going [482:21] to come here and I want this to depend [482:25] on end to end tests because if this [482:28] succeeds I know that at least there is [482:30] no problem in the code so we can build [482:33] and here I will copy the job name and [482:38] let's put it below dust check and that's [482:41] all we need to add now commit all of [482:44] this let's go back to GitHub let's look [482:47] at the put requests the pipeline has [482:49] started I will Wait for the pipeline to [482:52] finish and I will come back. I am back [482:54] and as you can see the front end build [482:57] job has started. Let's click on it to [483:00] make sure that it works. Okay. So now we [483:02] are building the front end as you can [483:04] see and it should work. Uh this is just [483:08] a step that I wanted to add to ensure [483:10] that building the front end succeeds. [483:13] Great. And as you can see we get a green [483:15] check mark. Amazing. Now let's go back [483:17] to the pull request. Let's click on pull [483:20] requests here. Let's click on this one. [483:24] And let's make sure that everything is [483:26] green. Awesome. So now we have finished [483:28] working on continuous integration. Now [483:31] we will work on continuous delivery. [483:34] Before we start deploying the new built [483:39] files to digital ocean, I would like to [483:42] start with uh create working on two [483:45] things. First of all, a backup [483:48] management system. This is going to [483:50] ensure that the servers disk does not [483:53] fill up because if we keep creating [483:56] backups for let's say mail search uh and [483:59] maybe databases if we have them that [484:02] will eat a lot of storage and because we [484:04] are using a cheap digital ocean uh [484:07] virtual machine that's we don't have the [484:10] luxury to uh waste space. So we are [484:12] going to create a script that will clean [484:14] the backups for us and keep only the [484:17] last five. That means we need to SSH to [484:21] the server. Let me do that quickly. I am [484:24] back and as you can see I am logged in. [484:27] Now what I will try to do is the [484:30] following. I will create this clean [484:32] backups script in this location. Now I [484:35] need to provide my password. Great. Now [484:38] I will copy and paste the following code [484:40] and I will explain it to you. First we [484:43] have this keep count variable. Basically [484:46] this this is going to uh control how [484:49] many backups we want to keep. And by [484:51] default it's it is set to five. I will [484:54] keep it that like that but if we want we [484:56] can change it. Here we have a we have an [484:58] offset variable that will add one to [485:02] keep count. We are doing this because of [485:05] this uh huge command that we are running [485:07] here. So I will explain what we are [485:09] doing here. Basically this is a function [485:11] that we are calling here on two folders. [485:14] In my case in this application that I am [485:17] deploying I have mainly search and I [485:20] also have a SQLite database. So for me [485:24] search the backup is going to be stored [485:26] in the dumps folder and for SQLite I [485:30] will store that in the backups folder [485:32] inside the web application. So what I [485:35] want is that I want to give this path to [485:38] this function and keep only the last [485:41] five files inside each folder and this [485:45] is how we do it. So the target is [485:48] basically going to be the path that we [485:50] are passing to the function. If if the [485:52] path exists, we are going to show in the [485:56] screen that we are cleaning it. And um [485:59] after that here here is what we are [486:01] doing. First of all, we are using ls-tp [486:05] so that we can sort the files by time. [486:08] Here we use grap-v [486:10] and here we are inverting the search so [486:13] that we uh skip folders. This command [486:16] will only keep files. And here we have [486:19] tail dash n and we pass the offset here [486:22] and the plus that we are passing here is [486:25] going to basically give us all the files [486:28] that are older than uh offsets. So for [486:31] example if offset is set to six we are [486:33] going to start from there and it's going [486:36] to give me the backup that was from day [486:38] six day seven etc. And we use X arcs to [486:42] pass each file to the remove command. [486:45] It's going to be put here so that we can [486:47] remove it. And this is how this function [486:50] works. And if you want you can play with [486:53] this command on your own. You can copy [486:55] copy paste it and see how it how it [486:57] works. So if you now I can save the [487:00] file. I will exit. Let me show you. [487:03] Let's go back to web app. And as you can [487:06] see so far I don't have any backups [487:08] because I haven't deployed from GitHub. [487:10] But once we start doing that the backup [487:13] folder will be created and uh in terms [487:16] of med search we will have the backups [487:19] in the dumps folder. Now we need to make [487:22] the script executable using sudo chode [487:25] plus x. That's done. Great. Now we need [487:29] to do something interesting. So we need [487:32] to run sudo visudo. This is going to [487:35] allow us to run commands that typically [487:39] require us to enter the password because [487:41] we need to run them using pseudo. But we [487:43] are going to skip that. We are going to [487:46] run them without a password. Why do we [487:48] need to do that? Because uh in g on [487:51] GitHub, we are going to have a job that [487:54] will tell the virtual machine let's say [487:57] to restart engineext or to restart your [488:00] uh backend application. And for that we [488:02] need to run systemctl reload engineext [488:05] and but this requires us to enter sudo. [488:08] We are going to run this command. We go [488:11] at the end of the file and here we need [488:13] to add the following. I will paste this. [488:16] I'll go back to the beginning and I will [488:18] change the username to mine. So for [488:21] example my username is IAD but for make [488:25] sure to change this one. So here what do [488:27] we have? You can see that here we [488:29] specify that we want to run these [488:31] commands without a password. And the [488:33] commands that we that we want to allow [488:37] are systemctl reload engine x supervisor [488:40] ctl restart all and the cleanup scripts. [488:44] Other than that we the other commands [488:47] are going to require a password. Never [488:50] never never uh remove password because [488:53] that's insecure. Let's say that someone [488:56] was able to steal your SSH keys somehow [489:00] and they were able to log into the [489:01] virtual machine. The only thing they can [489:03] do is either reload EngineX, reload, [489:06] restart supervisor and uh run the [489:09] cleanup com backup script. If they want [489:12] to, let's say remove something from a a [489:15] directory owned by sudo by the root [489:18] user, they won't be able to because uh [489:20] Linux will prompt them for the password. [489:23] Okay. Now save the file and exit. Now I [489:26] would like to also explain to you this [489:28] specific file. Let's go. Let's use cat [489:33] and let's call that specific file rs [489:36] sync exclude. Here is what we have. What [489:39] why do we need to do this? Because we [489:41] want to copy the source code from GitHub [489:45] and put it here on digital ocean in this [489:48] specific virtual machine. And the way uh [489:51] we are going to accomplish this is with [489:53] a command called r sync. This is an [489:56] amazing tool that allows you to copy [489:58] files. But the problem is that it's [490:01] going to make sure that what we have [490:04] here on the virtual machine matches what [490:06] we have on GitHub. And the this causes a [490:09] problem because let's say that I added a [490:11] file that I want to have here on the [490:14] virtual machine but that I don't want to [490:16] push on GitHub. The problem is that [490:18] rsync is going to delete this specific [490:20] file that I added here on the virtual [490:23] machine. This is why I have added all of [490:26] this here. You can see that in rs sync [490:28] exclude I tell rync to never delete [490:33] these files from the virtual machine. [490:36] Keep them and uh keep them on top of [490:39] what we have on GitHub. Okay. So for [490:42] example we are keeping backups, we keep [490:44] the same file, we keep node modules etc. [490:47] These are things that I want to keep [490:49] here when I run the rs sync command. I [490:52] am showing you this because now we are [490:54] just preparing. We haven't yet wrote [490:56] wrote the deploy job but we will see [490:59] these things in that yl file. So don't [491:02] worry about this. We are just preparing [491:04] for that step. Great. So now we need to [491:06] go to go to GitHub because we need to [491:08] create an environment. I will show you [491:10] how to do this. Click on settings. Click [491:13] on environments. Click on new [491:16] environment. Here let's call it [491:18] production and click on configure [491:21] environment. Now click on required [491:24] reviewers. We are doing this so that as [491:26] I said when before deploying we will [491:29] have certain number of people that will [491:31] receive an email so that if they want [491:34] they can deploy the application. We [491:36] don't want the deployment to be [491:38] automatic. So that we just have one [491:40] layer of confirmation before confirming. [491:44] Here I will put my own account but if I [491:46] have other team members I can add them [491:49] here and just click on save protection [491:51] rules. Now we need to create a few [491:54] secrets. For that we need to go here [491:57] click on secrets and variables and click [492:00] on actions. These are secrets that are [492:03] going to be used in the deploy job. For [492:05] example, to connect to the virtual [492:07] machine, we need an SSH key. So that [492:09] this is why we need to create these [492:11] secrets inside the repository secrets [492:15] section. We click on this button. I will [492:17] add the following keys. So we need to [492:20] have four of them. We need to have the [492:23] IP address of the virtual machine, my [492:26] user, and uh we need the SSH key and [492:30] finally the mainly search master key. [492:32] These are all the keys that we need to [492:34] create. Let's start with adding the SSH [492:38] key. Now I will open my terminal. I will [492:41] create this a key using this command. [492:44] And here I will just add test test to [492:48] make it obvious. Yeah, let's hit enter. [492:51] So now we are generating the public and [492:54] private keys. And here the the command [492:57] is asking us to add a passphrase. We [492:59] should not add a passphrase. So just hit [493:02] enter. Hit enter again and the uh the [493:04] keys were generated. Why don't why we [493:07] shouldn't add a passphrase? Because when [493:10] we run the deploy job on GitHub, the the [493:12] job should run without being asked for a [493:15] passphrase. That's why we created a new [493:17] key just so that we can use it on GitHub [493:20] to connect to the virtual machine. Uh [493:23] use the cat command to print the public [493:26] key to screen and then copy it. I will [493:29] do that and I will come back. After [493:31] copying the key, go back to the virtual [493:34] machine. Run nano SSH authorized keys. [493:39] After you run this, paste the key in a [493:42] separate line. I will do that and I will [493:44] come back. My public key is now inside [493:48] the virtual machine. Now I need to go [493:50] back to my to the local terminal or to [493:54] the local machine and this time copy the [493:58] private key. Let me do this quickly. So [494:01] I'll just remove public and I'll run the [494:04] command. And here I have the WL copy [494:07] program that allows me to take the [494:09] output of this command and just uh copy [494:12] it to my clipboard. Now go back to [494:15] GitHub and here you need to paste the [494:18] private key in the secret input field [494:21] and use digital ocean SSH key as the [494:24] name. I will do that and and I will come [494:26] back. Awesome. So we are we were able to [494:29] create our first repository secret. [494:32] Later we can either update it or delete [494:35] it. Now let me add Digital Ocean [494:38] username. In this case it's a Mac. Let [494:41] me also add the host IP. And finally, I [494:45] need to add me search master key. If you [494:48] don't remember where we stored the me [494:52] search API key, run this command. sudo [494:56] nano etc me search environment. Copy it [494:59] from there. Paste it here and click on [495:02] add secret. As you can see, I have [495:04] created the four secrets that I will use [495:07] in the deploy job. We are now ready to [495:11] create the deploy job. First, open the [495:15] workflows folder and create a file [495:19] called deploy.yaml. [495:21] Now, let's look at the content of this [495:23] file. The name is manual deploy because [495:26] after merging a boot request, we want to [495:29] have the possibility to click on a [495:32] button manually to deploy the [495:34] application. The way we do that is by [495:36] adding workflow dispatch. We will see [495:39] this later when we merge this put [495:41] request on GitHub. We will have a button [495:43] that will allow us to manually run this [495:46] particular job. And here under jobs, we [495:49] have two. We have build and deploy. [495:53] Build is reusing the work the front- end [495:56] build workflow that we have created uh [495:59] in the CI pipeline. Now under deploy we [496:03] have the name is deploy. This needs to [496:06] run after the build process. And here [496:09] under environment, we have production [496:12] and we point it at the URL. If you have [496:15] different environments, make sure to [496:16] change this. But in our case, we have [496:19] just production. And now here are the [496:21] steps. We will start with the same steps [496:24] that we have seen in the previous [496:26] workflows. First, we check the [496:28] repository. We download the front end [496:31] artifacts because now in the in the [496:34] deploy job we are going to take the [496:37] files that were built in this job and we [496:39] are going to put them in digital ocean. [496:43] This is why we are using this GitHub [496:45] action to download the front- end build [496:48] from this specific path. Up to now as I [496:51] said these are the these are the steps [496:53] that we have been following before and [496:56] the new step which is related to the to [496:58] the deploy job is create backups. What I [497:02] will try to do is first I will just [497:04] commit this and then I will add the [497:06] other step. So let me try to do this. So [497:10] here because we are u this is something [497:13] new I will add add deploy job and I will [497:17] commit this. Now let me publish this [497:20] branch and I want to publish it to [497:24] basically my own fork. I don't want to [497:26] push it to upstream because this is just [497:29] a test repository to show you how things [497:32] work. And now on GitHub I can click on [497:34] this button to create the pull request. [497:37] And here okay so let's see. No I don't [497:39] want to I want to merge it with my own [497:41] branch or yeah and here. So yeah, let's [497:44] just keep things simple and let's just [497:47] hit create put request. Now the first [497:50] thing that will run is basically the CI [497:53] job and then if everything passes, we [497:56] can uh yeah, we can merge this and try [497:59] the deploy job. But while this is [498:02] running, let's go back to here and let [498:05] me undo the changes to show you the [498:07] first step that we are going to perform [498:10] in this deploy job. As you can see this [498:13] specific step is called create backups. [498:17] And here I am going to create backups [498:19] for both SQL light and mele search. But [498:23] you can adapt this to create backups for [498:25] other databases if you are using them. [498:28] And we are using an new uh GitHub action [498:32] called SSH action. This is an amazing [498:36] GitHub action that you can use to [498:38] connect using SSH. Here it is. If you [498:42] want to read more about it and they have [498:44] a few examples here in their GitHub [498:47] repository, if you scroll down a little [498:49] bit, you will find there this quick [498:51] start example that you can just copy and [498:55] uh paste in your workflow and you can [498:57] just tweak a few settings there. And as [499:00] you can see here for example in order to [499:01] connect they are using a username and a [499:04] password. But if you look at my own [499:07] version I am not doing that because we [499:11] have disabled the ability to connect [499:13] using a password and instead we are [499:16] using an SSH key because as we said this [499:19] is more secure than using a password. [499:23] Okay. And here we are using version [499:25] 1.2.4. [499:27] So if you find that there is a new [499:29] version just go to GitHub look at the [499:31] releases tab and see if there is a new [499:33] release. If that's the case then updates [499:35] the version. Here under environment we [499:38] are specifying so we are adding mail [499:41] search uh and we take that from the [499:44] secrets and under with we have the host [499:48] which is basically the IP address of our [499:50] virtual machine. We have the username [499:52] key and here we pass the environments. [499:55] So here we have me search. We need this [499:57] because we are going to communicate with [500:00] the me search service or server in order [500:03] to create the backups and under script. [500:06] So this is the what I like about this [500:08] GitHub action. This allows me to run a [500:10] script inside the virtual machine. So if [500:13] I collapse this, you can see that's the [500:15] only thing left in this step. Let's [500:17] let's uh go through it step by step. [500:20] First we are creating some variables. We [500:24] have backend directory, backup directory [500:28] and how many backups we want to create. [500:30] So here the maximum amount of backups is [500:33] going to be five. But if you want you [500:35] can change this. And here we create a [500:37] time stamp. We are going to use this to [500:40] create uh when we are going to create [500:42] the backup. So that's the file names are [500:44] going to be different. The the first [500:46] thing we need to do is basically to [500:48] create the backup directory because if [500:51] we go to the virtual machine you will [500:53] find that we don't have a backups folder [500:56] inside web app. This is something new [500:58] and using the make directory command we [501:01] will create it and here you can see that [501:04] we are using this option dashp. It's [501:08] amazing because uh what it does that if [501:11] the back if the backups folder doesn't [501:13] exist, it's going to create it. But if [501:16] it if it finds that we already have a [501:19] backups folder, it's going to skip this [501:21] operation. And after creating the [501:23] backups directory, we are going to [501:26] start. So the first thing here is uh so [501:29] this block is going to is related to [501:32] SQLite. You can see that in my virtual [501:35] machine or in my project I have this [501:38] database inside the backend directory. [501:41] Let me go back to the virtual machine to [501:43] show you how it looks like. As you can [501:45] see I am now inside the virtual machine. [501:48] Let me zoom in a little bit so that we [501:50] can see this better. Let's type ls-la. [501:54] Now let's go inside backend and let's [501:57] see what we have. You can see that we [502:00] have the visitors database. This is a [502:03] SQLite database that I am using uh in my [502:06] application. This is what we want to [502:08] back up uh when we deploy. Let's go back [502:11] to VS Code and let's see. So here we [502:15] have an if block. We are checking if [502:18] this file exists. If it does, we are [502:21] going to move we are going to copy that [502:23] file and we are going to create a uh [502:26] this new file. And as you can see here [502:28] we are using the timestamp as a suffix. [502:32] So we have visitors underscore and we [502:34] put the timestamp. This is helpful [502:37] because after that we are going to sort [502:39] these files and remove the older ones if [502:42] we have more than five backups. Then we [502:46] echo basically that we we were able to [502:49] back up the database. Otherwise, if this [502:52] file doesn't exist, we are going to show [502:54] this warning to let us know that we [502:57] couldn't uh create a backup. That's it [503:00] for SQLite. It's simple. It's just a [503:02] file that we copy. What happens if you [503:05] want to do this with me search? It's a [503:07] little bit tricky because we need to do [503:09] more work. As you can see here, first we [503:12] check if we have the me search key. [503:14] Normally we should because as you can [503:16] see we have passed it to the [503:18] environments uh option here and if we [503:22] look inside we are using this option to [503:25] check if this environment variable does [503:27] exist. If it doesn't we are going to [503:30] exit and return an error. Now here is [503:33] how we this is how we are going to [503:36] basically make a backup. we are going to [503:39] uh make a post request to this specific [503:43] endpoint which is called dumps. And with [503:46] this we we are going to tell me search [503:49] to create a backup for us. But in order [503:52] to do that we need to pass the me search [503:54] key because if you remember we have [503:57] created this server we have created a a [504:00] API key so that no one can just make [504:04] requests to the server. they need to [504:05] have the API key first. After making the [504:08] post request, we add a header that will [504:10] pass that uh me search API key. And here [504:14] you can see that we are running curl [504:17] with the s option which is basically [504:19] silent. And here we what we are doing [504:22] here is that we are tell we are telling [504:24] curl to fetch the HTTP code. And if you [504:27] are not familiar with HTTP codes, you [504:30] can go and search for HTTPS HTTPS status [504:34] code and you can learn more about them. [504:37] You can go here to Mozilla. They have a [504:40] good documentation that explains how [504:43] this works. HTTP status codes will tell [504:47] you if if your request finished [504:49] successfully or not. In this case, uh [504:52] you can see here are the ranges. So we [504:54] have something between 100 to 200 etc. [504:58] and we are going to use this information [505:00] to uh basically check if what we have [505:03] done succeeded or not. Now let let me go [505:06] back. So after capturing the HTTP code [505:10] we will have all of that we will put all [505:12] of that response inside this variable [505:15] and from that we are going to get the [505:18] HTTP code. After getting the HTTP code, [505:22] we will uh compare it to see if it is [505:25] within this range. So if it is greater [505:27] than or equal to 200 and less than or [505:30] equal to 300, this means that we made [505:33] the request successfully. I can show you [505:36] this. If if this seems complex, I will [505:39] copy this and paste it in the virtual [505:42] machine and let's see what we will get. [505:44] First, let's try this um help endpoint [505:48] because it doesn't require us to pass a [505:51] an API key. You can see that we have uh [505:56] -x get. This is not necessary because by [505:58] default when you run a curl command, a [506:01] curly request, it's going to be get. [506:03] It's not we add the -x option when we [506:06] want to perform other actions such as [506:09] post. And here, let me just run curl [506:12] directly to see what we get. So for [506:14] example here we get status available but [506:17] what happens if I add the -w option [506:20] after curl let's add dashw [506:24] and let's see what we get you can see [506:26] that here let me retry the command you [506:28] can see that here we have 200 and this [506:31] is the the status code that we are [506:33] looking for and let's put this inside a [506:36] variable like we did uh in the deploy [506:39] job so you can see that I can wrap [506:41] everything around this Let's do it like [506:44] this and let's add the variable name. [506:48] And I need to close this like so. Now [506:51] let me hit enter and let's see what we [506:52] get. You can see that uh now we need to [506:55] echo the content of this variable to see [506:58] what it what it holds. Let me copy it [507:01] and paste it here. And as you can see we [507:03] are getting the same output. Here we get [507:05] the response from that endpoint telling [507:07] us that the M search server is up and [507:10] running and 200 is the response that we [507:13] got after making this HTTP request. Now [507:17] let me copy this second line. Let's take [507:21] HTTP code. Let's put it here and let's [507:24] again echo that variable. So let me run [507:28] echo and let's copy this and paste it [507:32] below. Let's see what we get. You can [507:33] see that we are getting 200. Why? [507:36] Because here we are taking this output. [507:39] We give it to tail and the tail will [507:41] take the last line which is in this case [507:44] is 200. And this is what we are getting [507:47] here. And the next uh action or command [507:52] is will get just so let me copy this. [507:55] It's going to get the status. It's uh [507:58] because here we are using the set [508:00] command. It's going to remove the status [508:03] code. It's going to take just the other [508:05] part which is the response body. Okay. [508:07] Uh as you can see the these commands are [508:10] simple. They look complex but if you run [508:12] them like what like uh I did uh you will [508:15] find that um they are easy to understand [508:19] and what they are getting is uh going to [508:21] be used here to basically check if the [508:24] request uh was made successfully or not. [508:27] So as you can see here we got 200 as the [508:31] status code. If we compare if we look at [508:34] this condition it's true because the [508:36] HTTP code is greater than or equal to [508:39] 200. So that means that we will show [508:42] this in the console showing that the [508:45] request was accepted. But in me search [508:48] when we call for example the dumps [508:51] endpoint it is not going to create the [508:53] endpoints directly because it depends on [508:56] the size of your database. So med search [508:59] will give you an ID so that you can [509:01] track the progress and uh this is what [509:05] we are getting here in the response body [509:07] we will try to look for the task UID. [509:11] Let me try to do this so that so that we [509:13] can see how this works. Instead of [509:16] hitting the health endpoint, I'll try to [509:18] hit the dumps endpoint to see the real [509:20] output that we will get from uh that uh [509:23] specific request. Let's run curl. Let's [509:26] add dash x because as I said here we are [509:30] going to perform a post request instead [509:33] of a get request. And let me add this [509:36] and let's see what we get. You can see [509:38] that now we weren't able to create to [509:40] make the post request because we are [509:42] missing the uh yeah the header. So we [509:46] need to add that. Here is the the other [509:48] missing part. So let me copy it and [509:51] paste it after this. Here is the full [509:55] command that we can run. Now it should [509:57] work because we are passing the API key. [510:00] Let's run this and let's see what we'll [510:02] get. Okay, here is the output and as you [510:04] can see I've told you that me search [510:06] will return the task UID and this is [510:10] what we will try to get in this nest [510:13] command. So if we go back here you can [510:15] see that we have this variable that will [510:18] take the response body and we'll search [510:20] for the task UID using this uh [510:24] expression and uh yeah if we print this [510:28] we will get the task u ID. So again [510:31] let's do the same thing. Let's store [510:33] that information inside dump dump [510:37] response and let's try to get the [510:38] response body. Let's put the response [510:41] here. Let me add this. Okay, that [510:44] worked. Now let's take the response body [510:49] and let's now run the task view ID. [510:52] Let's try to print the content of this [510:54] variable to see if we are getting the [510:56] correct view ID. Okay, that didn't work [510:59] because I forgot to add the other [511:01] options. So, here I have just d - x. So, [511:05] let me add the other ones dash s and [511:09] dash w. Let me add them here. Okay. And [511:12] now it should work. So, let's go back to [511:15] response body. And now let's go back to [511:18] task UID. Let's now try to echo. And as [511:23] you can see, we got the ID. This is what [511:26] we want to basically try to communicate [511:29] with Misarch to keep asking it did the [511:32] task finish or or not. Let's see how we [511:35] can do this. So after getting the task [511:38] ID here we are going to check if it [511:41] exists. So if it doesn't we are going to [511:43] throw an an error and uh basically fail [511:46] the deploy pipeline. Otherwise, if we [511:48] have the task ID, we are going to tell [511:51] mainly search uh by making this curl [511:54] command if the operation finished or [511:56] not. So, let's see what we will get if [511:58] we run this. So, let's do that. And [512:01] okay, so here they say that the [512:03] authorization header again is missing. [512:06] So, we need to add that one. Let's add [512:08] it like this. Let me clear the screen. [512:11] Let's add the authorization header and [512:13] let's see what we will get what we get. [512:15] Okay. So the output is we have a lot of [512:18] things but here you can see that they [512:20] confirm that the ID exists and here we [512:22] have the status which is succeeded. This [512:25] is what we want to see but in the [512:27] beginning it might be incued or [512:29] processed or sometimes it might fail. So [512:31] we are going to take this status [512:33] information and we are going to check uh [512:36] to check it inside the Y loop. Okay. So [512:38] after making the request we are going to [512:41] basically grab the status from the [512:44] output. If it says succeeded like in [512:48] this example, we are going to basically [512:50] show that the dump completed [512:53] successfully and we are going to break [512:55] out out of the while loop. Otherwise, if [512:58] it failed, we are going to fail the job. [513:02] But if it is still uh processing, we are [513:05] going to increase the attempt variable [513:08] and sleep for 2 seconds. And here with [513:11] the max uh amounts of the max retries is [513:14] set to 30 but you can change this. It [513:17] shouldn't take that much but this is how [513:19] we are uh this is the condition that we [513:22] have in the y loop. So if we reach 30 we [513:24] are going to uh break. Okay. So now here [513:28] we are inside this y loop. We can [513:30] collapse it. And after that if for some [513:33] reason we have exhausted these attempts [513:35] we are going to exit. And now we can [513:38] close this and we can focus on the else [513:41] block. So if the status code is not [513:44] within this range, we are going to exit [513:46] because something bad happened. And [513:48] finally we are going to run the this [513:52] specific script that we have created to [513:55] basically uh ensure that we keep at most [513:59] five backups for SQL light and mele [514:02] search. Okay. And uh yeah that's as you [514:06] can see uh that's all we are doing here [514:09] and this is this step is going to run [514:11] whenever we want to deploy our [514:13] application and this is very important [514:15] especially if you have sensitive data in [514:18] your database. Great. So this worked. [514:21] Let's go back to GitHub. And the CI [514:23] pipeline also worked without any [514:25] problem. Amazing. Uh let me commit this. [514:28] Let's say CI add backup job or step. Uh [514:35] let's commit it and let's sync the [514:37] changes. Let's go back again to the [514:39] virtual machine and let's ensure that we [514:42] don't have a backups folder. You can see [514:43] that there is no backups folder and if I [514:46] go to back end let's see so we have the [514:49] visitors DB. Great. Now let's I will try [514:53] to wait for the pipeline to finish and I [514:56] will come back because I need to wait [514:58] for this to finish uh in order to merge [515:02] the p request so that I can go to the [515:05] actions tab and uh click on the deploy [515:08] uh job so that I can deploy the [515:11] application. Okay, I'll wait for this to [515:13] finish and I will come back. I am back [515:16] and as you can see now we can merge the [515:18] pull request because all the tests have [515:21] passed successfully. I'll hit now [515:23] confirm merge and okay that's done. [515:27] Let's also go back here and uh let's go [515:31] to master to fetch the new changes. [515:34] Let's do that. Okay, great. So now we [515:37] have the new deploy file. Let's go back [515:42] to GitHub. Let's click here. And as you [515:45] can see, we have merged the third pull [515:47] request. Now let's now refresh the page. [515:51] Let's go to actions. Let's click on [515:56] manual deploy. This is what I explained [515:59] to you. So now we have merged. As you [516:02] can see, we have a CI step. It [516:04] succeeded. Now we have a second step [516:07] which is continuous delivery. And this [516:10] is how we do this. So as I said we will [516:14] do this manually. We click on run [516:16] workflow and here are the two jobs. [516:19] First of all we are going to build the [516:21] front end. Once this finishes we are [516:24] going to deploy the application. This is [516:26] not going to run automatically. No [516:30] GitHub will stop the deploy job and it's [516:33] going to basically send us a [516:35] notification to let us know if we should [516:38] let this continue or not. And this is [516:41] what we have configured uh earlier. You [516:45] can set as many people as you want so [516:47] that they can click on the deploy uh [516:50] confirmation. And this is what I was [516:52] talking about. You can see that we built [516:54] the front end without any issue. That's [516:56] great. But now we have uh you can see [516:59] that they requested a review from us. We [517:02] click on the button, we review. So we [517:04] want to deploy to production. We confirm [517:07] that and we hit approve. Let's click on [517:10] that and let's see what will happen. So [517:13] the deploy has started and you can see [517:15] that we have four steps. Let's click on [517:17] them to see what what is happening. So [517:19] now we are creating the backups. You can [517:21] see that everything went well. Nothing [517:24] bad happened. Uh and we can read through [517:27] the console. We can see these are the [517:28] messages that are um that we see because [517:31] we have uh we are using the echo [517:33] command. And let's read through this. [517:35] You can see that SQLite backup created [517:37] successfully. Here is the task ID. Here [517:40] is the HTTP status code that we got from [517:43] media search. And at the end, we say [517:46] that backup process finished [517:48] successfully. Amazing. That's what we [517:50] want to see. But let's confirm. Let's go [517:52] to the virtual machine to see if that is [517:54] correct. Uh before running the the [517:56] deploy job, we didn't have a backup [517:58] folder. But now, if I go back to uh web [518:02] app, let's run ls-la. [518:06] As you can see, we have the backups [518:07] folder. That's amazing. Now, let's go [518:10] inside that folder. Backups. Normally, [518:13] we should have just one file because [518:15] this is the first time we are running [518:17] this deploy job. And as you can see, [518:19] here is the time stamp. Now, what I will [518:22] try to do is I will copy this file and I [518:24] will paste it [518:26] four four times because I want to have [518:29] five files. when I will when I will [518:31] deploy the next time and we are not go [518:34] we are going to create the sixth file [518:35] but we are going to remove one of the [518:38] old ones because the script will keep at [518:41] most five files let's use the copy [518:44] command let's copy the file and here [518:47] I'll just change uh 12 to let's say 13 [518:52] let's change this again to 14 15 and 16 [518:57] let's see how many backups we have 1 2 3 [518:59] 4 and five. Now let's go back to GitHub [519:03] and I will try to rerun all jobs. Okay, [519:08] this will trigger the deploy job again [519:10] and it should create a new backup but [519:14] it's going to delete this one because it [519:17] is the oldest. Let's confirm this. We [519:19] have the output and uh after uh this [519:23] deploy job finishes I will run again ls [519:26] and we will compare um the two lists. [519:30] Let's wait for the build job to finish. [519:32] We have the artifact. Great. And now [519:36] let's hit deploy. Let's click on that [519:40] specific job. We are checking the [519:42] repository. We download the artifacts. [519:44] And here we are creating the backups. [519:47] And yeah, that finished. Let's see. So [519:50] here you can see that we have this [519:51] specific step. Starting cleanup [519:54] retention policy is to keep the last [519:56] five files. Well, let's confirm if [519:59] that's correct. So let's run ls. And as [520:02] you can see, we don't we no longer we no [520:05] longer have this specific database. We [520:08] have 13 up to up to 16. And now we have [520:11] a new upto-date backup of the database. [520:15] This is great. And if we want we can [520:18] also check the dumps folder to see what [520:21] we have there. Normally we should have [520:23] multiple uh backups. So if we go there, [520:26] okay, we don't have the permission, but [520:28] we can run sudo and pass the password. [520:31] And as you can see here, we also have a [520:33] few backups. Awesome. So this is working [520:36] really, really well. Let's keep going. [520:39] This is not the end of the deploy job. [520:41] We still have a few jobs to add. Let's [520:44] click on here and let's create a new [520:46] branch. Let's call it R sync. Okay, [520:50] because this is what we want to do. Let [520:52] me collapse these jobs. So I will [520:55] collapse the checkout step, the download [520:59] front end artifact and also uh the [521:02] creation of the backups. And now we will [521:05] add this new job that will sync the [521:09] files from the from GitHub to the [521:12] droplet. For that we are using SSH [521:15] deploy and rs sync is a command that [521:19] will sync the files. So basically it [521:22] will take whatever we add in GitHub and [521:25] make sure that we have that on the [521:27] virtual machine and uh by default it [521:30] will try to create a mirror. Basically, [521:32] whatever we have on GitHub should be on [521:35] the virtual machine. If we have new [521:36] files on the virtual machine, the rsync [521:38] command will delete them. But we don't [521:40] want that because as you can see, we [521:42] created a backups folder and this folder [521:45] doesn't exist on GitHub. So, if we run [521:48] this, we will risk deleting the backups [521:50] folder and any other file that doesn't [521:53] exist on GitHub. For example, thev file, [521:56] that's another problem. That's why we [521:58] created that rync exclude file to tell [522:01] rync yes try to create the mirror but if [522:04] you find something new in the virtual [522:06] machine please keep it. You can also [522:09] search for this new GitHub action that [522:11] I'm using here. We pass the private key. [522:14] We pass the host and user. So the host [522:18] is the IP address. The username that I [522:21] am using um in my virtual machine. Here [522:25] source will basically copy everything we [522:27] have here and the target is web app and [522:31] here are the arguments that we will pass [522:33] to the rs sync command. Uh so here by [522:36] default we will delete and but we [522:38] exclude these files and folders that are [522:40] inside the rsync exclude command. Okay, [522:43] you can see that this job is simple. So [522:46] let's try to commit it and let's test [522:48] it. Uh yeah, let's just call it ci add r [522:52] sync job. Let's commit this and let's [522:56] publish the branch. We want to publish [522:58] it in to origin. Uh let's go back to [523:01] GitHub. Let's click on pull requests. [523:04] Compare and pull requests. Let's target [523:08] my master branch. And yeah, let's just [523:11] uh create the pull request. I will wait [523:14] for the pipeline to finish and I will [523:15] come back. And like uh the creation of [523:18] the backups, I will deploy manually to [523:20] confirm that the step is working. This [523:22] is the best way to create complex jobs [523:25] is to test each step independently to [523:28] confirm that it works before moving to [523:30] the next step. If you try to write the [523:32] full pipeline, it's going to be yeah, [523:34] it's going to be hard to test because uh [523:38] multiple jobs will fail at the same time [523:40] and it's going to be a nightmare. I [523:42] highly suggest to work on complex [523:45] pipelines like this. Create a step, make [523:48] sure that it works and then move to the [523:50] next one. Same as before, I'll wait for [523:52] the CI pipeline to finish before hitting [523:54] the deploy button. I am back. I can [523:57] merge the pull request. Let's do that. [524:00] And let's click on uh the yeah the [524:03] GitHub action to uh trigger the R sync [524:07] job to verify that it works uh [524:09] correctly. Okay. Now again click on run [524:12] workflow. Make sure to choose master. I [524:15] forgot to mention this before but that's [524:17] very important because master is [524:19] protected and whenever we merge a new [524:21] thing it is ready to be deployed. What [524:23] we want to confirm in this step is that [524:26] the backups folder shouldn't be deleted. [524:28] This is uh very important. We can see [524:30] that here we have a few backups and if I [524:33] go back uh the backups folder shouldn't [524:36] be deleted. Let's verify that this is [524:38] the case. Uh okay let's confirm the [524:41] deployment. Click on production and [524:44] approve and deploy. Let's click on the [524:46] deploy job. We now should have a new job [524:49] which is uh going to run the rs sync [524:52] command. And as you yeah as you can see [524:54] here it is. Okay. So we can see that the [524:57] rsync command is going through [524:58] everything uh in the GitHub repository [525:01] and it says done. Okay. Uh we we [525:05] deployed successfully. But the question [525:06] is do we still have the backups folder? [525:09] Okay. We still have it. Let's see what [525:11] we have inside it. Amazing. So, we have [525:14] everything there. But if we have removed [525:16] this rync exclude file, things will have [525:19] changed. Uh but basically, you get the [525:21] idea. If you want to keep uh files that [525:24] are inside your virtual machine but [525:26] don't exist on GitHub, you should try to [525:29] add a file like this and specify it in [525:32] the arguments. Now, the next step is to [525:36] populate the environment variables. [525:38] Okay. So first of all let me try to add [525:41] the backups folder again to r sync [525:44] exclude it was so it was backups uh save [525:47] the file and exit. Now if we go to back [525:50] end uh sorry let's go to back end let's [525:53] run ls-a you can see that we have thev [525:57] file and I have created this uh manually [526:00] when I when we were testing the the back [526:03] end but we can also create this uh [526:06] directly from the deploy job because we [526:08] have all the secrets we want and if we [526:11] uh look at env.example example. You can [526:15] see that this is the only thing we need. [526:17] We just need to change the mini search [526:19] master key because the other ones are [526:22] not yeah they are not secrets. This is [526:25] just the endpoint and uh this is the [526:27] index name and the environments. It can [526:29] be either development or production but [526:31] the secrets uh is the master key. This [526:34] is what we will try to add. Let's try to [526:37] do that first. Let's go back here. Let's [526:41] click on the repository. Uh okay. So we [526:45] have merged. That means we need to go [526:46] back again to master and let's try to [526:50] pull the latest changes. Okay. As you [526:53] can see we have added the new step here. [526:55] Amazing. Now I will create another [526:57] branch and here let's call it feature [527:00] and let's say create environment file. [527:06] This step is also small. Uh let's add [527:09] this below uh the rs sync job. You can [527:14] see that the name is create a the [527:16] backend env file. Here we are again [527:19] using the SSH action command because it [527:21] allows us to run scripts. Here we are [527:24] using cat and we are redirecting all of [527:26] this content to this file we are [527:29] creating. So this is the the endpoint we [527:32] are this it's local that's why we are [527:34] using local host. The index name this [527:37] you can change it based on the name of [527:39] your index. And here we are using that [527:42] environment variable that we get from [527:43] secrets to populate this environment [527:45] variable inside this file. And finally [527:48] environment is set to production. And we [527:50] end the file. That's all we need to do [527:52] to create a to create thev file. Uh [527:55] let's uh commit this civil [528:00] file. Commit this. Publish the branch to [528:04] origin. Let's go to GitHub and let's [528:08] create the pull request. Okay, we have a [528:11] problem. I need to change this to ghost [528:13] like this. Okay, great. Now, let's [528:15] create the pull request. I'll wait for [528:18] the pipeline to finish and then I will [528:20] come back. Let's merge the pull request [528:23] because the CI pipeline succeeded and [528:27] let's test the deploy job. Click on [528:31] actions. Let's find manual deploy. Let's [528:35] run the workflow on the on the master [528:37] branch and let's see if the environment [528:39] file will be created. This is a simple [528:42] job so it shouldn't fail but yeah it's [528:45] always uh a great idea to test. Let's [528:48] click on review deployments, choose [528:51] production and deploy. Click on the [528:53] deploy job and let's wait for the final [528:57] step to run. As you can see it is there. [528:59] So that means that we uh yeah we are [529:02] using the latest version of that deploy [529:05] file. The RC command has finished and [529:08] now we and we we were also able to [529:10] create the backendv file. That's [529:13] amazing. Yeah, we can verify that we [529:16] have the file there. Um let's see. So [529:19] let's run ls-la. [529:22] And as you can see, today is it's June [529:25] 28th, um 11:24 a.m. So that means that [529:29] it's uh correct. Great. Let's go back to [529:33] VS Code again. Let's switch to the [529:35] master branch. Let's pull the latest [529:38] changes. [529:40] Now, we need to install the Python [529:42] dependencies. Let's say that we added a [529:44] new library that the back end depends [529:47] on. If we don't install it, the back end [529:50] will fail. So we need to add a step to [529:53] install the dependencies. But the [529:54] problem that we need to account for is [529:56] that let's say that we have 20 [529:58] dependencies and uh we install all of [530:01] those inside the virtual environment [530:03] folder. Uh let me show you. Let's go to [530:06] um v to the virtual machine. You can see [530:08] that we have the virtual environment [530:10] folder. Let's say that something bad [530:12] happened and we were able to install 10 [530:15] libraries but the rest failed. If we [530:18] change this virtual environment folder [530:21] and um yeah this not all not all [530:24] dependencies were installed this will [530:25] cause a problem. So what we need to do [530:27] is this step should be atomic meaning if [530:30] it fails we need to roll back and reload [530:34] uh the virtual environment. This means [530:36] that we are going to take a copy of this [530:38] virtual environment folder. We will [530:40] rename it something else. Uh for example [530:42] VNV uh backup. Then we will use pip to [530:46] install the dependencies. That will [530:48] create a new virtual environment folder [530:50] for us. If uh everything went well, [530:53] great. We will delete the backup folder [530:54] that we created and we will keep the new [530:57] one. But if something failed, we will [530:59] roll back and reuse the the backup [531:02] folder so that um we don't accidentally [531:06] break the back end. First of all, we [531:08] need to update the rsync exclude file. [531:12] we need uh to uh update the backend [531:15] section. Uh here are the things that we [531:18] need to add. Remember that I showed you [531:20] that we need to create the logs folder, [531:23] the J unicorn start etc. And here we [531:26] also have the socket. All these things [531:28] do not exist on GitHub. And if we deploy [531:32] that was the problem that I had is that [531:34] these files and folders were deleted. [531:37] But now I have added them here and uh [531:40] yeah they shouldn't be deleted anymore. [531:42] Um so I have gone through the steps that [531:44] we um followed in the previous sub uh [531:47] chapters and you can see that I have the [531:50] socket file I have the scripts and I [531:53] also have the logs folder and I have [531:55] added them to rsync so that they won't [531:58] be deleted anymore in the future and to [532:01] uh also verify that the back end is [532:03] working fine you can see that we can use [532:05] supervisor ctl and here this is the [532:08] application and it is saying running [532:10] which is uh Great. This is what we want [532:12] to see. And if I also open the browser [532:16] and go to this website, you can see that [532:18] it is working. If I click on blogs, [532:21] yeah, they are coming in. And if we [532:23] click on uh let's go to the conclusion [532:27] and let's start clapping normally it [532:29] should work. Yes, you can see that if I [532:31] refresh the page, the the count uh was [532:34] updated because the data is stored. Uh [532:37] and yeah, it means that the back end is [532:39] running. Maybe search is running. [532:41] Everything is working perfectly. Okay. [532:44] Now let's go back. We let's commit this. [532:47] Uh let's say sure update r sync exclude. [532:52] Uh let's commit this. Now we need uh we [532:54] have created the branch. Great. Now we [532:56] need to add the new job inside this [533:00] file. Now let me add let me paste the [533:03] code here. It's long but I'll try to [533:05] explain it. uh to you. The new job name [533:08] is called install Python dependencies. [533:11] And here I have tried to emphasize that [533:14] we are going to uh atomically swap the [533:17] virtual environment. We use the SSH [533:20] action GitHub action to run a script. So [533:24] let's look at what we have here. First, [533:26] we define this virtual environment uh so [533:29] that we can go inside back end. And here [533:31] is the first step. We want to create a [533:34] backup of the virtual environment [533:36] folder. But first, we need to check if [533:38] that directory exists. If it does, we we [533:41] use the move command to create a backup. [533:44] And the new folder name will be virtual [533:46] environment backup. After that, we will [533:49] create a brand new virtual environment. [533:51] We activate it and we use pip install to [533:55] install the dependencies. And here so [533:58] yeah if uh this command failed we are [534:01] going to show the else case which is [534:03] going to remove the virtual environment. [534:07] This is the roll back strategy that I [534:09] was talking about. If it if it if things [534:12] doesn't work as expected we will fall [534:15] through the else block. We will show the [534:18] error to understand what happened and [534:20] then we roll back. But first we need to [534:23] deactivate the virtual environment. We [534:25] remove it and then just to double check [534:28] we check if the virtual environment [534:30] backup exists. If it does, we move it [534:32] back to virtual environment and we exist [534:35] because the installation failed. But if [534:39] everything went well, we will do the [534:41] cleanup. So we have created a new [534:43] virtual environment. It has all the [534:45] dependencies. So now we no longer need [534:47] the backup. We can just delete it. And [534:50] at the end we echo that the virtual [534:52] environment setup has completed. And now [534:55] I can we can commit this um let's say [534:58] pet and here let's say add uh virtual [535:02] environment or let's change this to CI [535:05] because this is what we are working [535:06] with. We can commit and remember that [535:10] the rsync exclude file is now looking [535:13] like this. We can publish the branch. We [535:16] go to origin and let's now click on pull [535:21] requests to create another one that we [535:24] can deploy to pass this new step in the [535:27] job. Okay. Now let's change this to uh [535:31] origin. Let's remove all of this. So [535:34] let's see. So this is the branch we want [535:36] to merge it with master. Okay. And here [535:39] let's change the title to CI add virtual [535:42] environment atomic setup. Click on [535:45] create pull request. Now we just need to [535:48] wait again because uh yeah we need to [535:51] ensure that the CI passes before testing [535:54] the new deploy job. I am back. As you [535:57] can see we can merge the pull requests. [535:59] Let's do that quickly. Now let's click [536:01] on actions. Let's scroll down to manual [536:05] deploy. Click on run workflow and hit [536:08] run workflow. And at the same time I'll [536:11] try to uh looking at the virtual machine [536:15] to see if um when the backup folder will [536:19] be uh created. But first let's wait for [536:22] the build the job to finish and then we [536:25] can run the deployment job in order to [536:29] test the atomic virtual environment uh [536:33] creation. Let's hit review deployment. [536:37] Click on production and approve. Okay, [536:40] let's click on the deploy job and here. [536:44] Okay, let's do clear and ls-la. [536:49] We are syncing the files. Let's hope [536:52] that the logs folder and the other ones [536:53] do not get deleted now. Okay. And now we [536:56] are installing the dependencies. And as [536:58] you can see, we have the virtual [537:00] environment backup. And now we created a [537:03] new virtual environment where we are [537:05] going to install the dependencies. And [537:07] we can confirm this by the date. You can [537:09] see that this virtual environment was [537:11] created uh a week ago. But now this one [537:14] was created today. And we can also see [537:16] that the logs file that the logs folder [537:19] these files were not deleted because we [537:21] have updated the content of the rync uh [537:25] exclude file. Amazing. This is working [537:28] really well. And you can see that the [537:30] deploy job um finished successfully. [537:33] Which means that now if I run ls-la, [537:35] this folder will be removed. Let's [537:38] confirm that. And yep, that's it. That's [537:40] what we what happened. And you can see [537:43] that the date on this folder is um June [537:47] 28th. This is working really well. Now, [537:49] let's go back to VS Code here. I need to [537:53] go back to the master branch cuz we need [537:56] to pull the new changes. Let's hit okay. [538:00] Um let's wait for a while. Okay, we have [538:03] a new job here. Great. Now the next step [538:06] is to uh reload the back end as well as [538:11] the engineext configuration because if [538:13] you have made changes to the front end [538:15] or back end, we need to restart the [538:18] application so that the changes take [538:20] effect and for that we will let's go [538:23] back to the website. You can see that [538:24] here in the navbar I have so this is my [538:28] name. I will try to change that and we [538:31] will hit deploy and then come back here [538:33] to the website to see if um yeah the new [538:36] modifications uh are going to be live. [538:40] Let's try to search for the navigation [538:42] bar component and here. Okay. So, let me [538:46] search for my name. Here it is. And I [538:49] will add uh let's say 2026. Okay. So, I [538:53] have changed the uh changed that file. [538:56] Let me create a new branch. Commit that. [539:00] Close the file. We will close the rsync [539:02] excl exclude because we no longer need [539:04] it. And now I will try to add this job. [539:09] Uh let me indent this correctly. So it [539:12] says restart the backend service. Here [539:15] again we are using the SSH action GitHub [539:19] action and inside this job. Here is what [539:22] we are doing inside script. We check we [539:25] try we run the restart all command using [539:29] supervisor ctl. If this succeeds it [539:33] means the back end restarted [539:36] successfully. Otherwise we fail the job. [539:39] Uh after running this we will wait for [539:41] let's say 5 seconds so that uh we can [539:44] run again the status command to check [539:46] that the output will say running. This [539:49] is another layer of confirmation that [539:52] will tell us if the back end is is [539:54] working. Okay. So we will try to push [539:57] this and then I will add another job [539:59] after this that will reload X. Let's [540:02] commit ci restart back end and below [540:08] this I will add uh let me go back here [540:12] let's go down below and let's paste the [540:15] new job uh I will collapse this because [540:19] uh we have uh seen it and uh now we just [540:22] want to focus on the reload engine x job [540:25] here inside the script first we need to [540:27] check that the configuration is correct [540:29] if the configuration is not correct we [540:31] are not going to reload EngineX because [540:33] if we do then we are going to break the [540:35] yeah we are going to break the website [540:37] and we don't want to do that. If the [540:39] syntax is okay we move to the next if [540:42] block we run again here we run the [540:45] reload engineext command. If this [540:48] succeeds great otherwise we are not [540:50] going to run it because the yeah the [540:53] configuration is not correct. And at the [540:55] end we show deployment complete. This is [540:58] the last job in this deploy if workflow [541:02] and if this succeeds we have um created [541:06] uh a good CD workflow that when I hit [541:10] deploy after changing my applications [541:12] code it's going to be live in a few in a [541:15] minute or two. Uh so here let's add [541:18] reload enginex. Let's commit this. I'll [541:21] publish the branch and create a pull [541:24] request. Then I will wait for a few [541:26] minutes for the pipeline to finish and I [541:29] will come back. I am back. Let's merge [541:32] the merge requests or the pull request. [541:34] Let's now go to actions. Click on manual [541:39] deploy. Let's uh where is it? Yeah. And [541:42] here let's click on run workflow on the [541:44] master branch. Let me split the page on [541:48] two so that we can see the changes uh [541:51] quickly. Okay. Now let's click on review [541:55] deployments and let's approve and [541:57] deploy. Let's click on deploy. We'll [542:00] follow the steps. We are checking the [542:02] repository. We download the front end [542:05] artifacts. [542:06] Here are the backups. Okay, we are are [542:10] syncing the files. [542:12] We have installed the dependencies [542:14] without an issue. And now we are trying [542:17] to restart the backend service. You can [542:20] see that the message is saying [542:21] restarting the fast fast API back end. [542:24] Let's hope that this works. We have seen [542:27] that we were able to stop and start the [542:30] the backend service but we have a [542:33] problem. So here it says sudo a terminal [542:36] is required to read the password meaning [542:38] that the v sudo uh change that we did [542:42] didn't work. Let's try to investigate [542:45] why. So here what was what was the last [542:49] message we have seen pass API backend [542:52] that restart is started successfully. [542:53] Okay now let's go back here and let's [542:56] search for that message. Okay so [542:58] supervisor ctl start all worked well but [543:01] I think what happens is that maybe the [543:04] status uh is not allowed because here we [543:07] need to run sudo. So let's verify that. [543:10] Let's open the terminal again. Let's run [543:13] sudo v sudo and let's see what we have [543:16] inside this file. Let's scroll down. [543:19] Okay, you can see that here we have uh [543:22] yeah that's the problem. So here we are [543:23] allowing just restart all. We don't have [543:26] status. That's the problem. So what we [543:30] can do is copy the command. I'll try to [543:33] copy this. Let's go back to the [543:36] terminal. And here let's try to add it [543:39] after this one. Let's type user bin and [543:43] I will paste this command. Supervisor [543:45] ctl status all. Save the file. Exit. Now [543:51] go back to uh let's go back to GitHub [543:54] and let's restart the failed jobs. So [543:58] here uh let me try to make the screen [544:00] bigger to so that I can see what's [544:01] happening here. I will click on rerun [544:04] failed jobs. Let's uh rerun that. Let's [544:08] now click on review deployments and [544:11] let's hope that this time the we won't [544:14] have an error when trying to reload the [544:16] back end. Okay, now we have reached the [544:19] other the step that failed before. But [544:21] now as you can see it uh it didn't fail. [544:24] That's great. And um okay, so now we [544:27] have the same problem while trying to [544:30] reload EngineX. And I think I know the [544:33] um uh how to solve it. I think yeah we [544:36] have the same issue if we look at here [544:40] let's see so we have reload engine x but [544:43] we forgot to add uh engineext-t because [544:47] we are checking the syntax if we go back [544:51] to vs code let's collapse this because [544:54] it succeeded it didn't fail now we need [544:57] to basically uh try to work on this one [545:00] you can see that systemct ctl reload [545:02] engine x is already here it systemctl [545:06] reload engineext. But when we try to uh [545:10] check the configuration, we run [545:12] engineext-t. [545:14] We need to also add this one. To fix [545:16] this, we need to find where the binary [545:19] of is located. But I think normally it [545:23] is located at user sbin [545:26] /enx. [545:28] But we can verify that. I have saved the [545:31] file and now I can run which engine x. [545:34] And as you can see this is the path that [545:35] I have added in b sudo and um okay let [545:39] me just verify it one more time user/ [545:42] spbin/engineext and then I added the [545:45] comma. Okay great. So now with this we [545:48] won't um yeah we won't have an error [545:51] when trying to run that specific step. [545:54] Okay let's go back again to GitHub and [545:56] let's try to rerun this failed step. Uh [546:00] let's use rerun all failed jobs and uh [546:03] again let's uh rerun this. I will [546:06] refresh the page to get the confirmation [546:09] is production and approve. Let's click [546:11] on the deploy job. Hopefully this time [546:14] it should work and then I will go back [546:16] to this tab refresh the page and [546:18] hopefully we should see that 2026 [546:22] uh will appear here. Let's collapse this [546:25] rync step the create uh backups etc. Now [546:30] we are installing the Python the Python [546:32] dependencies. Uh we arrived at um the [546:36] last steps. This one will not fail [546:39] because we have confirmed that. Uh let's [546:41] just hope that reloading engine X will [546:43] also work and yes it did. Amazing. Uh [546:47] let's uh look at this. So as you can see [546:49] everything is green now. And as you can [546:51] see here, GitHub will also give you the [546:53] link so that you can click on it to see [546:55] if everything uh is looking great. Okay, [546:58] this is the moment of truth. Let's [547:00] refresh the page. And as you can see, we [547:02] have 2026. And this will work when um it [547:07] doesn't matter if you change your front [547:09] end or back end. Both will both uh will [547:11] work. If you change your back end, we [547:13] restart supervisor. We restart yes [547:16] supervisor using the supervisor ctl [547:18] command. And if your front end changes, [547:21] we also reload the engine X. Awesome. [547:24] So, as you can see, we have built the [547:26] full pipeline. And this is how uh how I [547:28] want you to build it. Don't try to write [547:31] it fully. No, go step by step. Try to [547:34] add one step at a time. Verify it [547:36] because as you can see right now, we try [547:38] to add these two steps. They fail. It's [547:40] not a problem. We can go ahead and fix [547:43] them before moving to the next steps. [547:46] And once you do that you you learn how [547:48] to handle errors and it's uh also [547:52] amazing when you try when you fix the [547:54] problem and see uh the green check mark [547:57] uh it makes you happy and developing it [547:59] like this is also fun. Uh so yeah we [548:02] have finished the deploy job and this is [548:04] how it works. So now in the future you [548:07] have seen now that I was creating put [548:09] requests and then I merged them. That's [548:11] the first step. create your uh create a [548:14] new feature branch. Develop your uh the [548:17] yeah your feature or bug fix or [548:19] whatever. Mer make sure that the CI [548:22] pipeline passes. Then merge that request [548:26] with your master branch. Then after that [548:29] move to the second uh step which is to [548:32] deploy. So go to actions and then hit [548:35] the button on the master branch. Don't [548:37] forget that. And uh after that your [548:40] steps are going to go one by one and [548:42] then your application will uh update [548:44] accordingly with the new code that you [548:46] pushed to master. And just like that we [548:49] have finished chapter 5. It was a long [548:52] one and uh this this was the I I would [548:55] say the most interesting chapter because [548:57] we have merged everything we have [548:59] learned before and we try to automate as [549:01] many things as possible. We have one [549:03] more chapter left. In that one we are [549:04] going I will it's a it's not a long one. [549:07] uh I will just show you tools that you [549:09] can install uh on your server so that [549:12] you can monitor let's say uh HTTP [549:14] requests that are hitting your server [549:17] etc. So these are just tools that are [549:19] nice to have and that are easy to [549:21] install and configure. [549:23] We have arrived at module 6. We are [549:26] going to focus on observability. You [549:30] will learn how to parse native engineext [549:33] traffic using go access. I will show you [549:37] how to deploy a dashboard where you can [549:40] see this engineext data using HTTP basic [549:44] authentication. We are going to learn [549:46] about rotating logs and we are going to [549:50] use a chrome job for that. We are going [549:53] to visualize the CPU and memory usage as [549:57] well as process trees using BTOP. And [550:01] finally, I will show you how to find [550:04] people that are or bots that are trying [550:07] to get data from your virtual machine [550:11] and we are going to block those IPs. Now [550:14] let's go to the terminal and let's SSH [550:17] to the virtual machine. Here is my [550:19] terminal and as you can see I am inside [550:21] the machine. I am going to start by [550:24] installing go access. Here we are using [550:27] ibuntu. We can install it very easily [550:30] using this command sudo apt install go [550:34] access. I already have it on my system. [550:36] So that's why it didn't take too much [550:38] time to install it. And how do we use [550:40] this? Okay, let me show you. First, we [550:42] need to try to look at the this path [550:46] that stores the logs. You can see that [550:49] here we have a lot of logs because we [550:51] have just one website on this server. It [550:54] is called my website_com. [550:57] We are seeing this. But if you are if [550:59] you have a virtual machine that is that [551:01] is hosting multiple websites then pick [551:03] one or multiples. So here we are going [551:06] to pick my website. We want to look at [551:09] the access logs and you can see that [551:11] some of them are regular files but some [551:13] of them were compressed and this is what [551:16] I was talking about in the beginning. [551:18] So, Linux is trying to rotate the the [551:20] logs, meaning it it tries to compress [551:23] them to save disk space. But first, we [551:26] are going to look at just this one [551:28] because it's a regular file that we can [551:30] read. And later, I will show you to in [551:33] how to include the zipped files also. [551:36] Now, we know the logs are stored here [551:38] and we know that we need to pick this [551:40] specific uh website. Here is the command [551:43] that we need to use. I will paste it and [551:46] then I will explain what it does. Here [551:48] we are using pseudo because we need to [551:49] read files from this specific directory. [551:53] And here I need to copy this and uh [551:56] replace this placeholder with this [551:58] specific file name. And here the log [552:01] format is set to combined. This is the [552:04] for the default format that enginex [552:06] uses. So if you have if you are using a [552:09] different format, make sure to change [552:11] this option. That's all we need to do. [552:14] Now if I hit enter, let's see what [552:16] happens. You can see that we get this [552:18] amazing in real time dashboard. This web [552:23] server received 122 requests today. And [552:28] here we have 54 unique visitors. Go [552:31] access is showing you the data that is [552:34] stored in the engineext logs. And here [552:37] it is also shows you where it where it [552:40] found the data. And this is the normal [552:43] file that I was talking about. And here [552:45] it shows you so the uh the unique [552:48] visitors we I we have just one person [552:50] that was that was hitting the the web [552:53] server. We can keep scrolling or you can [552:55] hit tab to move to the second uh tab or [553:00] to the second um section. Here we have [553:03] requested files. Some were requesting [553:06] the site map. Some were just visiting [553:08] the root directory. Here we have the [553:12] static requests. So you can see that [553:14] some were checking the robot cxt file. [553:17] We don't have 404s. Okay. Here we get [553:20] the IP addresses of people or bots that [553:24] are trying to reach the web server. Here [553:28] it shows you the operating systems. You [553:29] can see that we have a lot of useful [553:31] data that we can read and try to [553:34] understand what is happening. And here [553:36] it also shows you where the traffic is [553:38] coming from. You can see that some were [553:41] able to find the website from Google. [553:43] Some were uh some traffic was coming [553:45] from our website itself. And here we got [553:48] the HTTP status codes per request. So [553:52] most requests here succeed but some of [553:54] them have client errors. Great. And as [553:57] you can see all of this data is coming [553:59] from one single file. And later I will [554:03] show you how to increase this range by [554:07] incorporating the files that were [554:09] compressed. But before we do that, I [554:11] will show you if you don't if you prefer [554:13] to see a dashboard instead of sshing to [554:19] the machine having to run this command [554:21] to get this terminal dashboard. We can [554:24] tell go access to generate a an HTML [554:27] page that we can access from our [554:30] browser. And this is what we are going [554:32] to do now. But since this data contains [554:35] sensitive information like IP addresses [554:38] and other stuff we don't want everyone [554:40] to see that page. So we are going to use [554:43] HTTP basic authentication to block [554:46] access to that specific page. So you [554:49] need in order to see the data you need [554:51] to provide a username and a password. [554:54] And in order to do that here I will hit [554:57] Ctrl C to exit. We need to install a [555:00] tool called Apache2 utils because it has [555:05] that functionality that allows us to [555:07] create to generate an encrypted password [555:11] file. Great. We have installed Apache to [555:15] utils. Now let me clear the screen and [555:18] let's use it. Here is how we use that [555:21] utility. So here we tell it that we want [555:23] to generate a as I said an encrypted [555:26] password file and we will put that here [555:28] in this location and here the user is [555:31] going to be uh admin. I hit enter and [555:35] now we need to provide a password for [555:37] that specific admin user. Now I need to [555:40] retype the new password and it says add [555:43] in password for user admin. Now we can [555:46] use this username and password to lock [555:50] that specific page. But before we do [555:52] that, we need to generate the go access [555:57] html page. And in order to do that, let [556:01] me go back to this specific command [556:04] where we generate where we get the [556:07] terminal dashboard. And we need to [556:09] adjust a few things. The first part is [556:12] going to stay the same. So we are [556:14] pointing at a specific log file but we [556:16] need to add the following option before [556:19] log format. So here I will add this and [556:23] as you can see here we add dash O which [556:26] basically will try to generate this [556:30] specific HTML page from the data. Once I [556:33] hit enter, you can see that it is going [556:36] to put it inside front end this. And [556:38] this allows me to be able to use this [556:42] file and to check it from my browser. [556:44] And if I want, I can check the engineext [556:48] configuration. It says okay. And now I [556:50] can try to reload it so that engineext [556:54] can serve that specific file. So let's [556:57] go to Firefox. And here let's go to uh [557:01] the website. And now because that [557:04] analytics html file is in the this [557:08] folder, I can access it. But you can see [557:10] it didn't work here because I need to [557:12] create I need to update the engineext [557:14] configuration in order to serve this [557:17] file. First let's u make sure that when [557:20] we go to web app front end and this we [557:25] need to ensure that we have the file [557:27] there. As you as you can see we we do. [557:29] So we have analytics.html HTML. Let me [557:33] make sure that I have typed it [557:34] correctly. Yes, I did. Now that's great. [557:37] We need to open the enginex [557:39] configuration [557:41] and let me try to do that. We use the [557:45] nano. So here I am using nano but you [557:47] can use other text editors. And here we [557:49] are pointing at my my website com inside [557:54] the sites available folder. And here you [557:57] can see that we have we need to come [557:59] here. So here we are serving uh index [558:02] html, we are serving assets and we are [558:06] serving other stuff. So what are we [558:08] going to do is we are going to add [558:11] another location block but we are going [558:14] to do that before the spa fallback [558:18] routes. So we can do it maybe before [558:20] assets or before index html. It it [558:24] doesn't matter. Okay. Uh I will just [558:26] copy the following code and I will paste [558:29] it here. And now I need to try to indent [558:33] this. And here you might see that I have [558:35] changed something here. Root instead of [558:38] specifying this path I have this v this [558:42] variable that I'm using here. I did this [558:45] because uh before I didn't have this [558:48] coming soon page. So when I was uh [558:51] because here I had I was serving other [558:53] stuff. So I just uh changed this [558:56] variable to point to this coming soon [558:59] but basically it's just a variable that [559:01] is holding a path and here so here you [559:04] can see that root is pointing at web app [559:06] front end this and this is why this was [559:09] not working now I can save the file exit [559:14] let's check engineext-t [559:17] it says that the configuration is okay [559:19] and now we can reload the page sorry I [559:22] made a mistake Let's try to run this [559:25] again and let's hit enter. Okay, now [559:27] let's go back to the browser and let's [559:29] refresh the page. And as you can see, I [559:32] can't access that page because I need to [559:35] provide my password. And this is what we [559:37] are using in the configuration. I forgot [559:40] to explain it. So let's go back to this [559:43] file and let's see. You can see that [559:45] root is pointing at the folder because [559:48] there we have the analytics.html [559:51] file. And here we add this o basic to [559:56] tell engineext that this endpoint or [559:59] this file is restricted. We cannot [560:01] access it. And here is what we are [560:04] using. We are using also and here is the [560:07] encrypted file that we have created. So [560:09] if the user or if the person that is [560:11] trying to get this resource if they [560:13] provide the correct user and password [560:16] then serve this file for them otherwise [560:19] don't serve it and show them this [560:22] challenge which is what we are seeing [560:24] here. Now I type admin because that's [560:26] the user that I have created and here I [560:29] need to provide the password. If I make [560:32] a mistake let's say I remove the last [560:35] character it's not going to let me in. [560:37] So I need to provide the correct [560:39] credentials in order to see that [560:41] dashboard. And as you can see now I am [560:44] in. So we need to wait uh for a few [560:46] seconds. Um I'm not sure what is [560:49] happening. Do we have an error here? [560:51] Okay. Okay. Okay. So I see the problem. [560:53] Remember that we are using content [560:55] security policy which is restricting [560:59] this project from loading some some [561:02] data. That's why it is not working. If [561:05] we go, so we need to fix this and this [561:07] is really cool because as I told you [561:09] when you apply the CSP policy, you have [561:13] to keep attention to it and if you [561:16] change your front end, you need to [561:18] always test it to see if there is some [561:20] new dependency that you added or [561:22] something that is not loaded correctly. [561:25] And this is what what happened here. But [561:27] we can solve this easily by going back [561:30] to the engineext configuration. And uh [561:33] let's go up to see what we have. You can [561:35] see that here we have the content [561:37] security policy. And this is what we [561:40] have done. And what we could do here is [561:42] we can uh [clears throat] h have the [561:44] header here just for analytics. HTML so [561:47] that we can override this. Uh okay. So [561:50] we can come here and I will I will try [561:54] to add that. I will paste that here. Let [561:57] me save the file. Exit. Let me check [562:00] that the configuration is okay. And now [562:04] I can reload. I will I will just make [562:07] sure that uh the website is going to [562:09] load correctly and then I'll try to [562:11] explain to you what is happening there. [562:13] Okay. So, we still have a problem here. [562:15] It says that the page settings blocked [562:18] an inline script uh because it violates [562:21] something. Okay. Let's go back and let's [562:23] see what we have here in the [562:24] configuration. I think what I will do [562:26] here since this is a page that only [562:30] admins can access. I will add unsafe [562:34] inline just so that we can see how the [562:36] dashboard looks like. So here we have [562:38] unsafe eval and after that I will add [562:41] unsafe inline. But please never do this [562:44] on your main production site because [562:46] this is a page that is protected. it's [562:50] not it's not going to be an issue. [562:52] That's why I'm adding this and to just [562:55] make sure that it is working. So here [562:57] now I refresh the page and as you can [562:59] see we get the dashboard. It's the same [563:01] information that we will see in the [563:03] terminal. But if you prefer to say that [563:06] you want to see this from your phone or [563:08] you don't have access to the virtual [563:10] machine, you can use the dashboard to [563:13] look at the data. And here you have you [563:16] can for example open the settings. You [563:18] can change the theme if you like. If you [563:20] want a bright theme, you can do that [563:22] here. And here you can see all the [563:25] panels that we talked about. Here it [563:27] shows you the unique visitors per day. [563:30] It shows you the files that were [563:31] requested, [563:33] etc. And if you want to learn more about [563:35] Go Access, you can go to their GitHub [563:37] repository. They have a ton of resources [563:40] and it's an amazing project. You can [563:42] also if you want you can uh as I said go [563:45] to GitHub and here they have their own [563:49] example the so on their website they [563:51] show how this might look like. So if you [563:54] want to to come here to read the [563:57] documentation uh that that would be [563:59] great because they have a lot of [564:01] features and if you click on this image [564:03] on GitHub you see that they also have [564:06] their own dashboard and you can see that [564:08] they have a ton of information and you [564:11] can also add maps. It's amazing. It's an [564:14] amazing project. Great. Now let's go [564:16] back to this to my website. You can see [564:20] that we were able to add this and uh as [564:24] I mentioned here we have we just did a [564:27] shortcut just to be able to solve the [564:30] problem with the CSP but as I mentioned [564:34] and never ever use unsafe inline because [564:37] that goes against security. Now let me [564:41] show you how to do this correctly. First [564:43] we need to add nons. Uh so yeah we need [564:47] to add this. Let me show you. We need to [564:50] add nons dash requests ID to the CSP [564:55] only in the script source. Okay, because [564:58] this is related to loading inline [565:00] scripts. And after that, this was the [565:02] missing piece. I forgot to add it. This [565:06] is what we need to add to uh this [565:08] location block. Here we are passing sub [565:11] filter once off because we want [565:14] engineext to go through the content of [565:16] this file and we don't want it to stop [565:20] uh when it finds the first occurrence of [565:22] the things that we want it to search. [565:24] And here here are the filters we are [565:25] telling it to search for script tags [565:28] that don't have attributes. We want to [565:30] have to search for script tags that have [565:33] attributes. So for example, this one is [565:35] not closed but the first one is closed. [565:38] And for these two we want engineext to [565:42] include the nons value. And this is a [565:46] dynamic ID or yeah request ID that is [565:49] generated whenever we request this file. [565:52] And because of security we want yeah we [565:56] want to make sure to uh include this [566:00] stuff. We are telling EngineX to replace [566:03] these strings with these values and this [566:06] should resolve that problem. So now I [566:09] can save the file. Exit. Let's first [566:12] make sure that the configuration is [566:15] correct. Good. And now we can reload the [566:18] page. Awesome. Now let's go back to the [566:20] page and I will do I will do a hard [566:23] refresh. We can do that by using control [566:26] shiftr. And as you can see it is working [566:29] now. and here. So yeah, we have just [566:31] warnings. We can remove them. So we can [566:33] see that we don't have any errors and we [566:35] can confirm. This is great to play to [566:37] try to play with this because once you [566:39] add a CSP, you will have problems like [566:42] these. And in order to uh understand [566:44] them, you need to read the error that [566:46] you are getting to understand what needs [566:48] to be changed. And here for example, I [566:51] can remove this and see if it is not [566:53] needed. And now I can reload the reload [566:55] the configuration, come back, refresh [566:58] the page. As you can see, we were able [567:00] to load the dashboard but not fully. And [567:02] this is how you do it. Now we see that [567:05] okay, we have a problem. So now we can [567:07] zoom in and read a little bit. It says [567:09] that here uh okay so the pages settings [567:12] blocks an inline script and here from [567:15] being executed we can also click on this [567:18] specific file and as you can see here it [567:20] shows us that we have this specific [567:23] script and what do you see? We have we [567:26] have the type attribute. So it's not a a [567:28] closed script like the one we have here. [567:31] Let's go back to the configuration. You [567:34] can see that here the scripts that we [567:36] captured is closed. Here it is closed [567:40] but this one was not we weren't able to [567:42] load it because now it has an attribute [567:45] and that was the second thing that I've [567:47] added in order to capture that. So let [567:50] me go back and paste the file and here [567:52] let's indent it. And uh we have seen how [567:55] to work with CSP. But I just wanted to [567:57] remind you how we do it. Uh you can use [568:00] LLM to help you with this. Just copy the [568:03] console error, give it your current [568:05] configuration and tell it to give you a [568:07] secure CSP. Sometimes it might tell you [568:10] to uh do the what we did before to get [568:13] rid of the errors quickly, but that's [568:15] not secure. So try to tell it to give [568:18] you a secure CSP and that's also going [568:21] to be very helpful. Okay. Now we can [568:24] save this file after adding this sub [568:26] filter. Let's reload it. I know that the [568:29] configuration is correct. And let's [568:31] refresh. And as you can see everything [568:34] is working perfectly. Amazing. Now let's [568:36] go back to the terminal because now I [568:38] want to talk about log rotation. We have [568:41] seen that when we did ls and looked at [568:45] the var directory, we found that we have [568:49] a lot of zipped files here. And if we [568:52] want to use them, it's not possible. So [568:56] that's why here in the dashboard, we see [568:58] just what is happening today. But let's [569:00] say that I want to see what happened [569:02] during the this month or uh during the [569:05] past year. I need to change the command [569:07] a little bit. So let's see how we can do [569:10] this. For that, we have a utility called [569:13] zeb uh grap. You can search for it. [569:17] Normally it is installed on Linux. So uh [569:20] you can use it and this is how it works [569:22] and it allows you to search inside the [569:25] zipped file. So you don't need to unzip [569:27] them and then search within them. This [569:30] utility allows you to uh search for [569:32] patterns within the file directly and [569:35] it's super useful. We can try we can try [569:38] to do that uh in order to show you how [569:40] that works. I will copy the full command [569:42] and then I will show you what it what it [569:45] does. Let me go back to the terminal. I [569:48] will paste the command here. I need to [569:50] change that placeholder using this [569:52] website. Let's do that quickly. Okay. [569:56] Let's see what we have here. Here you [569:58] can see that we are using a wild card. [570:00] This means we are going to capture every [570:03] file that starts with my [570:05] website_-access.log. [570:09] And this will capture the normal file [570:11] and the other ones that are compressed. [570:14] Now let's look at this Z grab-h [570:17] dot. If we go back to Google, here we [570:20] see the this uh AI generated content. It [570:23] says that here we pass the options and [570:25] the term that we want to search for. And [570:27] here we put dot because we want to [570:29] capture the whole file because that's [570:31] what uh go access is going to use in the [570:34] dashboard. And here we pass the dash h [570:37] option in order to basically tell it to [570:41] hide the file names. And here if you [570:44] search online you'll see that's the this [570:47] confirms it. So dash h means d- no file [570:51] name. And here yeah it's it's not going [570:53] to show the uh the file names in the [570:55] output. But that's how it works. So you [570:58] you give it the the files. Here we use [571:01] the wild card to capture all the the [571:04] files and here we pass that to go access [571:08] and here the dash is telling go access [571:10] to read the unccom log data from the [571:14] standard input because here we are using [571:16] pipe it's going to take the output from [571:18] this and give it uh to go access uh from [571:22] the standard input again so we here we [571:25] have - o web app and we put that inside [571:27] the and we use the combined log format [571:30] that and here we get a problem. So it [571:32] tells us permission denied. That's uh [571:35] understandable. So we need to add sudo [571:37] and uh here is what it is saying we get [571:41] permission denied because when we do [571:43] sudo like this it is going to be applied [571:45] to just z grab and uh after piping that [571:49] we are going to use that permission. So [571:51] we need to change that command to uh to [571:55] look something like this. We need to run [571:56] it inside this shell execution script. [572:00] We are using p sudo and we use bash and [572:03] we pass it the full command that we want [572:04] to that we want to run. And this will uh [572:07] use the pseudo privileges. And this [572:09] normally should work. And as you can see [572:11] we don't get the permission denied error [572:13] like we did before. And now since this [572:16] is inside the this folder I don't need [572:18] to re to reload the engineext [572:20] configuration. I can just update yeah [572:22] refresh the page. And as you can see the [572:25] the duration changed. So now it is [572:28] saying 28th of June to 12th of July [572:32] which is amazing. Uh normally it will be [572:35] 2 weeks max. But we will solve this [572:37] issue because Linux is going to first [572:40] compress the files but after two weeks [572:43] it is going to download uh to remove or [572:45] to delete old files. And uh yeah now you [572:49] can see we get more data. Now you can [572:51] see the number of visitors per day and [572:54] the number of hits. A visitor can hit [572:57] the server multiple times. That's why [572:58] the number of hits is usually higher [573:00] than the number of visitors. Amazing. [573:04] But what do we want is that we want to [573:06] automate this generation process. We [573:08] don't want to log to the virtual machine [573:10] and every single day generates this uh [573:14] updated uh static HTML file. We want to [573:18] automate this and we know how to do how [573:20] to do that. So we are going to use chron [573:22] tab and we are going to add uh this uh [573:25] file or this command to the end of that [573:27] file. Okay, let me copy the full command [573:30] and I'll paste it here. Here we are [573:33] going to run it uh hourly. So that's why [573:37] we have zero here. It's going to make [573:39] sure to run zap every hour for us. And [573:43] now the only thing I need to change is [573:45] this. Let me save the file and I'll just [573:47] copy that from here. Let's go back to [573:50] Chrome tab and let me change this and [573:54] paste it here. Great. Now we can save [573:56] the file. Exit. And now this is going to [574:00] automatically generate the dashboard. [574:03] And let's say tomorrow if I want to see [574:06] um the stats. I can just go back and uh [574:10] refresh the page. But we still have a [574:12] problem where Linux is deleting old [574:15] files. But we are going to come back to [574:17] that issue and resolve it. For now, we [574:19] will uh stop a little bit and we will go [574:22] to GitHub. I want to show you an amazing [574:25] project that you will find very useful. [574:27] It's called Btop. So, if you go to Btop [574:31] GitHub, it's a way to monitor resources [574:34] and it's super super amazing. The [574:37] application that they have developed [574:39] looks very nice. What I want you to do [574:41] is open the releases page and here the [574:44] last version is 1.4.7 4.7 and here they [574:47] have the pre-ompile binaries uh that we [574:50] can download but since we are here in [574:53] the terminal we need to use the wget get [574:56] command to download that so I'll just [574:58] download this one and I will use um [575:01] first let's use ls okay we have the file [575:04] there I will use tar to unzip that okay [575:07] now we have btop so we need to go inside [575:10] there sorry let's go inside btop and [575:14] this is the source code we can just use [575:16] pseudo make install to compile this [575:19] project. Okay, you can see that this is [575:21] the installer and it installed the [575:24] binaries uh in their respective folder [575:27] so that we can just type btop and uh the [575:29] program uh will run without a problem. [575:33] So now if we type btop it says that okay [575:35] we need to resize the window and uh [575:38] voila as you can see we have everything [575:40] here but sometimes [575:43] Linux will not allow btop to get uh [575:46] sensor data. So for that we can you can [575:50] run this command. So if you get that [575:51] problem you run pseudo set cup and that [575:55] will allow btop to read the sensor data [575:59] without an issue. After doing this, we [576:02] can go back one directory and remove the [576:07] the the files that we have inst uh [576:09] downloaded because now btop is running [576:12] without an issue. And as you can see, [576:14] btop is a replacement for htop or top or [576:19] any uh one of these commands because [576:22] here you can see yes, we get the same [576:24] information but it's hard to read [576:26] visually. Uh it's hard on the eye. But [576:29] Btop is uh is an is an amazing uh [576:32] terminal user interface. As you can see, [576:35] it looks really clean. Let me zoom in a [576:37] little bit just to see what is happening [576:39] here. You can see on top we have the [576:41] CPU. Since this is a lowend virtual [576:46] machine, we have just one core. Uh here [576:48] is that CPU. We have uh 512 MGB of RAM. [576:54] You can see is here. We see that [576:55] currently we are using 48% of memory and [576:59] uh 50% 50 52% is available. We also have [577:03] swap. This is what we have created in [577:05] the beginning. So we are allocating 2 GB [577:08] from disk as a swap memory to help us [577:11] because we don't have a lot of uh RAM [577:13] and uh here so we have disk we have 8 GB [577:18] and here we have network. So it show it [577:20] shows us download and upload speeds and [577:23] here we have the processes and here you [577:26] see you can you can filter you can uh [577:30] switch between tree view so if you hit E [577:33] it's going to show you individual [577:35] processes uh but if you want to see the [577:38] tree you can hit E to yeah yeah group [577:41] the processes uh like so if you want to [577:45] sort you can use the arrow keys [577:48] currently we are sorting by memory but [577:50] you can sort with other stuff. I usually [577:52] keep it to memory because here as you [577:54] can see I don't have a lot of it. So if [577:56] I see that my server is struggling I [577:58] would like to see what is the process [578:01] that is using a ton of resources so that [578:03] I can go here maybe hit K to kill that [578:08] specific process if it is using too much [578:10] memory and uh is making my application [578:13] work u poorly. Okay so that's how we do [578:17] it. You can also filter. So for example [578:19] here I you know that we are using J [578:21] unicorn. So we can look at that. And as [578:24] you can see here we have all of this and [578:26] we can hit E. And here you can see that [578:29] our application is using these three J [578:33] unicorn workers because that's what we [578:36] have specified. But here you can see [578:38] that we have some orphaned uh processes [578:41] that are left here maybe when we were [578:44] reloading the the back end but we forgot [578:46] to delete them and these are consuming [578:49] memory. So what we can do here we can [578:51] just hit K and kill them one uh one by [578:55] one because you can see that some of [578:56] them are using 5 MB of memory and they [578:59] are doing nothing and we know that here [579:03] when we set up the J unicorn file we [579:06] have specified that we want to have [579:08] three Unicorn workers that's why we have [579:11] three of them there and you can see that [579:13] they are grouped by the master J unicorn [579:16] process that's how you know if those are [579:20] orphaned or not. And we can also verify. [579:22] So if I go back to my website, it should [579:26] I yeah it should work without any [579:28] problem. So if I go back here, you can [579:31] see that here I have the coming soon [579:33] page. But that's just something that I [579:35] have added recently. So if I open [579:37] another terminal window and update. So [579:41] let me show you what I'm talking about. [579:43] Here I have this maintenance file just [579:46] to allow me to change between this page [579:49] that you are seeing right now and uh [579:51] this page that contains my uh website. [579:56] So let me reload EngineX. Let's go back [579:59] refresh the page and you can see I get [580:00] the website. And remember last time we [580:03] did the change where we added 2026 here. [580:05] And if I go back to blogs you can see [580:07] that the back end is still running. And [580:10] uh we can also verify something because [580:14] now if I try to kill one of these [580:16] workers what will happen? Let's see. It [580:18] is going to be killed. But as soon as [580:20] that happens J the manager so this [580:24] process manager is going to see that hey [580:27] we have two workers now but uh the the [580:31] configuration says that I need to have [580:32] three workers. So it is going to spin a [580:34] new one to uh fulfill that. If you don't [580:38] remember where we have that, you can [580:39] always go back to web app. Let's go [580:43] there. We open backend and let's go to [580:47] back end. Let's use cd instead of ls. [580:51] And let's see here we have you can see [580:54] that we have j unicorn start. Let's look [580:56] at that file. And as you as you can see [580:59] here we have specified that the number [581:02] of workers should be always three. And [581:06] here what when we are on J unicorn it is [581:09] going to uh ensure that we have three [581:13] UVORN workers. That's why when I tried [581:16] to kill uh one of these workers I I [581:21] don't see two. So one gets deleted and a [581:24] new one comes in. And how do we know [581:27] that? Just look at the process ID. So [581:29] here is the current process ID. Let me [581:31] take a screenshot quickly just to verify [581:34] that this is the case. Um here is the [581:37] screenshot. I will try to kill it again. [581:40] And here is the number. So you see [581:42] before it was uh 827 but now it changed [581:45] to 990. [581:47] And that's amazing. This is the what we [581:50] have in the configuration. And now if I [581:52] go back and refresh the page, you can [581:53] see this is still working because and uh [581:57] these processes are what's handling the [582:00] the requests that are coming from the [582:03] browser. And you can see this is why [582:05] Btop is amazing because it allows you to [582:07] interact with it uh using your mouse and [582:10] keyboard simply instead of top and other [582:14] programs that are hard to use. Okay, I [582:17] have now showed you how to use Btop. [582:20] Now, let's talk about identifying [582:24] bots and uh people that are trying to do [582:28] malicious stuff with your server. Okay? [582:31] And let me also show you how to [582:35] understand what I'm talking about. So, [582:37] if I go back to analytics.html, [582:40] you can see it is still working even [582:42] though we changed to my website. You can [582:45] see that if we focus on requested files. [582:49] Let me see if I can zoom in. Yes, we [582:52] can. But I think what we can do, we can [582:54] also hide some panels. So we can see [582:58] that here we have unique visitors. Can [583:01] we hide that? Yes, we can. Okay. And now [583:03] we have requested URLs. Here you can see [583:06] that this is what we currently have. [583:08] Some people are trying to get the site [583:10] map. Okay, that's fine. But look at [583:11] this. Someone is trying to do some fast [583:14] traversal inside the virtual machine. [583:17] This is malicious. This is not something [583:18] that a normal user will do. We can keep [583:21] looking. This one is this person is [583:23] trying to login somehow. Uh let's see. [583:26] This person is trying to get Yeah, you [583:29] can see there are a lot of malicious [583:31] things. This guy is trying to get [583:33] password and u some configuration [583:36] environment files etc. These are not [583:40] normal users. Basically, there is a [583:42] person or a group of people that are [583:45] building programs that are going to the [583:49] through the internet and trying to find [583:51] web servers. Once they they find your [583:53] web server, they try to find if there [583:55] are vulnerabilities. Basically, if you [583:57] are not good at security, you might leak [584:00] some files. And if they find that you [584:03] are leaking some passwords or some [584:05] common files that we see here in the [584:07] requested files, they will try to uh get [584:11] as many things from you as possible. You [584:13] can see that here they are also trying [584:15] to get some AWS credentials or stuff [584:17] like that. And these bots they might hit [584:20] your server a lot. So if you if you see [584:23] here here so for example this is a new [584:25] web server. I am not getting a lot of [584:27] hits but this will increase from time to [584:30] time but we want to solve this. We want [584:32] to block these people because they are [584:35] just wasting bandwidth. Now let's see [584:37] how to do this. We are not going to do [584:38] it manually. Meaning I am not going to [584:40] come here. Okay look at this request [584:44] come down see which IP uh because here [584:47] we have IP addresses. You can see that [584:50] here we have the IP addresses. So, we [584:52] need to correlate them together and see, [584:54] okay, this IP address is hitting my [584:56] websites a lot. Like, look at this one [584:58] for example. It says hitting the website [585:01] mo almost 2,000 times. That's inhuman. A [585:04] human person will not go through your [585:06] websites and try to refresh the page [585:08] 2,000 times in uh the span of two weeks. [585:11] So, that's how you know that you have [585:13] some outliers, some uh automated [585:16] programs that are trying to get [585:18] something from your website if it if [585:20] they found it. So here this our security [585:23] is great. So and we don't here we are [585:25] not using PHP. So the resources that [585:29] they are trying to get are not in on the [585:31] website. So that's fine but just so that [585:34] uh we make their life hard. We can't [585:37] block them. So let's see how to do this. [585:39] Now we understood the problem. Let's see [585:40] how to solve it. Let's exit now from [585:43] Btop. We no longer need it. You can do [585:46] two things. You can either look at the [585:48] dashboard here and see the the IP [585:51] addresses that are that have the high [585:54] amount of hits or we can ex uh tell go [585:58] access to give us a JSON file. So if we [586:02] do sudo go access let's look at this [586:05] command we can change it a little bit [586:08] here we are looking just uh at the the [586:11] today's log and here we do dash o to [586:15] specify the output but we can here in [586:18] this case it is an html file but we can [586:21] also generate a JSON file instead of [586:25] html let me do that I'll go here and I [586:29] will remove this path and replace it [586:31] with this one. You can see that I have [586:33] parsed the log file and I have generated [586:36] a JSON inside uh the web app folder. [586:40] Let's ensure that we have the file [586:42] there. And yes, we do. So now we can [586:45] write a simple Python file to analyze [586:49] this data. Here is the content of that [586:51] script. And let's see what we are trying [586:53] to do. First we we load the file. So if [586:56] we find this great we will have this [586:58] data dictionary and otherwise we exit if [587:02] we don't find it we we load the hosts [587:04] and from that we get data and we will [587:07] print the IP addresses. I will save the [587:10] file and I will run it just to see what [587:13] we will get. Okay so this is what we are [587:16] getting. This is the traffic that we get [587:18] today. Okay. So these are all the IP [587:21] addresses that uh reached the web [587:25] server. And here you can see for example [587:27] this IP address sent 76 uh hits today [587:32] and these these uh these ones send just [587:36] three four five. Okay. So these are low. [587:38] That's not a problem. Even 76 it's not [587:40] that big of a problem. But if I go to my [587:44] own personal website which which is um [587:48] an old [clears throat] one uh it has [587:51] been deployed for more than a year. In [587:53] that case I have some cases where one IP [587:57] addresses one IP address will send in [588:00] 6,000 requests 10,000 requests. Those [588:03] are clearly automated bots that I [588:06] usually block without thinking about [588:08] that. But let's say that this IP address [588:11] uh is a bot. It's not a normal user. [588:14] What we will do? We analyze this output [588:17] and we see okay so these are not doing [588:19] bad things or they are not wasting [588:22] bandwidth. We can keep them now but if [588:24] in the future they change their behavior [588:26] we can spot that. So now we copy the IP [588:29] address and we use the uncomplicated [588:32] firewall to plug them. So we use sudo [588:35] UFW. [588:37] We type deny from this IP address to any [588:41] and this will block that specific IP [588:43] address. And you can see UFW says that [588:46] we have a new rule. Let's confirm that. [588:49] I will clear the screen and I will run [588:51] sudo UFW status. And as you can see, we [588:56] denied this IP address from accessing [588:59] anything from the web server. So now [589:02] when they try to make a request is going [589:04] to be dropped. it's not going to reach [589:06] Ajax at all and that's what we want to [589:08] see and this is how we block nefarious [589:10] actors that are trying to do us harm. [589:14] This was the first sub chapter we have [589:16] talked about observability. Uh but in [589:19] the next sub chapter we are going to [589:21] focus on reliability. That's one is also [589:24] important. We are going to try to put [589:27] your server under stress. Meaning that [589:30] you need to know this virtual machine [589:33] how how much traffic can it support. So [589:36] if I have 100 people who are trying to [589:39] access the website at the same time, [589:41] will my application be able to handle [589:43] that? I cannot just say it like this. I [589:45] need to test that. I need to put my [589:47] server under stress to confirm that yes, [589:50] we can support that. Otherwise, no, we [589:53] need more resources. So we need to use a [589:55] tool that will help us answer this [589:57] question. We will also see solve the [590:00] problem with engineext logs. So we we [590:02] are going to tell index hey please don't [590:03] delete the files that are older than 14 [590:06] days. Instead delete files that are [590:08] older than a year or something like [590:09] that. And I'll also show you a tool that [590:12] will that will allow you to analyze the [590:14] disc the disk space to see the uh what [590:19] is what are the things that are [590:20] consuming a lot of storage so that you [590:22] can delete them if you want to. and uh [590:25] more things. So that's what we are going [590:27] to talk about in the next subchapter. In [590:30] this sub chapter, we are going to talk [590:32] about reliability. Let me show you what [590:35] I mean by that. I've mentioned that we [590:37] need to simulate concurrent user traffic [590:40] locally using a any tool. And in the [590:44] Python world, we have locust, which is [590:46] an amazing tool that will allow us to [590:49] simulate a lot of people trying to [590:51] access our website. to answer the [590:53] question, can my website handle x amount [590:56] of traffic? So with locos, we have a [590:59] class called fast http user that will [591:02] allow us to boost execution capacity by [591:05] five times. So that's something that I [591:06] want you to use to benefit from it. And [591:09] we are going to use this tool called [591:11] NCDU to analyze the disk space to see [591:16] what are the things on our virtual [591:19] machine that are consuming space to [591:20] determine if we can delete it or not. If [591:22] we are running out of storage of course [591:24] we are going to use systemd journal D. [591:28] So we are going to enforce memory [591:30] constraints and build a monthly [591:33] automated cache and package cleanup [591:35] system. So we want the system to remove [591:39] things that are normally not used by our [591:42] application monthly. So this is just [591:44] really uh something that we can add and [591:47] it's going to run automatically uh for [591:49] us to save us some space. Amazing. [591:52] That's what this these are all the [591:53] things that we are going to look at. Now [591:56] let's go to the terminal to apply this. [591:58] So last time we saw that uh the last [592:01] thing we did is to block an IP address [592:04] that is trying to do bad things to our [592:07] virtual machine. Okay. Now we can [592:10] continue from this. We need to install a [592:12] tool called locust. But we need to be [592:16] careful. We should not install that here [592:18] on the virtual machine. Instead we need [592:21] to install it on our laptop. Because if [592:24] you install it here on the virtual [592:25] machine and you point it at your server, [592:28] the data that you are going to get is [592:30] going to be incorrect because Locos is [592:33] going to consume some memory and your [592:35] web server is trying to fulfill all [592:38] those requests that are sent by locos [592:40] and it's going to fail or it's not going [592:42] to fail but it's not going to handle all [592:44] of them because locos is there is [592:46] consumer uh CPU and memory. So we want [592:49] to put that on our laptop and then [592:52] perform the the test. And in order to [592:55] install it, you use pip install locust. [592:58] That's how we do it. And um after that, [593:02] you create a Python file uh locally to [593:05] perform the test. Okay. I will open the [593:07] terminal, a new terminal locally, go to [593:10] the correct directory and I will come [593:13] back. I am back and as you can see, I [593:16] have opened a new terminal. I am now [593:18] inside the project. I can go to back end [593:21] and here I have created that file inside [593:25] tests load because this is a uh load [593:29] test and this is the file that I was [593:31] talking about. It is called locust file. [593:33] You can put it wherever you want but in [593:35] my case I decided to put it here because [593:38] it is some type of testing. Now let me [593:41] open VS code to show you uh the contents [593:44] of that file. So let me zoom in and [593:48] let's see what we have here. Obviously [593:50] you need to change this based on your [593:52] application and you want to test a few [593:55] endpoints. So here I have a few [593:57] variables and but these are uh the [594:00] things that you can that you can tell [594:02] locus to test. So you create specific [594:05] tasks. Here I have a few of them and [594:08] each task receives a weight. So a task [594:12] with higher weight is going to be [594:14] preferred over a simple task. So here I [594:17] am giving a higher weight to the uh to [594:20] the latest and uh recommendation [594:23] endpoints as well as the search [594:25] endpoints because these are tasks that [594:28] are heavy that are going to do a lot of [594:30] work and I want locus to uh focus on [594:34] them so that uh because if we have a lot [594:37] of requests that are doing search that [594:39] will use more resources and here in your [594:42] test let's just look at one uh just as [594:44] an example so we tell block us to uh go [594:47] to this endpoint here. We uh we take [594:50] random inputs. So we take a random [594:53] query. Here are the search terms that I [594:55] have here. So I can search for Inkscape, [594:57] Python, M search and um I want to uh [595:02] randomly take the article type, sort [595:05] field and sort order. And because this [595:07] is a post endpoint, I need to have a [595:10] payload. And after doing that I can uh [595:14] send uh a post request using the client. [595:18] And here the so you can see that this is [595:20] the the client that we are that we have [595:22] here. And here is that fast HTTP user [595:26] class that I was talking about that will [595:28] boost the execution capacity by around [595:30] five to six times. And uh you can see [595:33] that it's simple and uh you can do the [595:35] same thing for your other endpoint. But [595:38] again this is specific to my [595:39] application. These are the endpoints [595:41] that I have there. But if you have other [595:43] things, you can change um the endpoints [595:46] to suit your application or your back [595:48] end. Okay. So this is how we write that [595:51] locus file. But now what do we need to [595:52] do? Let me show you. So after coming [595:55] here first I need to activate the cond [595:57] environment because uh locally I I like [596:00] to use Anaconda and here inside that [596:03] virtual environment I already have [596:04] locus. So what I need to do just type [596:07] locus and hit enter. Okay. So there is a [596:09] problem with loading imports. That's not [596:11] an issue. I think I need to go back to [596:14] back end and then run it like this. We [596:17] use locos and we tell it here is the [596:19] file. Okay, now it is working. And as [596:22] you can see by default locos will pick [596:26] the this port 8,89. [596:29] But if you want to specify a port, you [596:33] can do it like so. You can come at the [596:36] end and add the host option uh to add [596:39] the endpoint. But let's just keep things [596:42] simple and let's uh go to this website. [596:46] Okay. So this is how locus looks like. [596:48] Let me use light mode so that you can [596:50] see things clearly. Now what do we need [596:54] to do? Okay. So let's change the number [596:57] of concurrent users. At peak we will [597:00] have 50 people trying to access the [597:03] website. Here the the spawn rate or the [597:06] ramp up rate is going to be five. This [597:08] means uh locos will try to add five new [597:13] users every second. So that's uh what it [597:16] means and host here we need to put the [597:18] production website and make sure to do [597:22] this testing only when uh your website [597:25] is not used by people. So for example [597:28] before launch you will come and do the [597:31] testing or if you have a staging area [597:33] before production you can test that but [597:35] never test production because if you [597:38] increase the number of users that might [597:40] create problems and if you want you can [597:43] also open the advanced options but let's [597:45] keep things simple. Once we do this, we [597:48] will have two things here in this [597:49] terminal. We will see some uh some data [597:52] after stopping and because here we also [597:55] are using the dashboard, we can see it [597:57] here. So that's fine. But we can also I [597:59] can also use go access to see that [598:03] interactive terminal dashboard uh to get [598:07] the data. So let's try to do that. So we [598:11] we have sudo go access. So this is what [598:14] we have used right. Let's run that again [598:17] and let's see. So currently we have 214 [598:20] requests. Okay, that's fine. Uh I'll try [598:23] to put this uh below like this. Let me [598:27] also put uh this on top like that. Uh [598:32] but let me decrease uh the font size a [598:34] little bit so that we can see this [598:36] better. Okay, are you ready? So this [598:39] page is fine but since it is going to be [598:42] generated every hour we are not going to [598:45] see realtime data that's why I am going [598:48] to close it and close these tabs because [598:50] we don't need them. Okay let's hit start [598:52] and let's see what will happen. So okay [598:54] so now we are get we are creating the we [598:58] are generating the requests and look [598:59] here you can see that we are getting [599:01] some traffic and here we are also uh so [599:04] this is the amount of data that we are [599:06] consuming right now and uh okay so it is [599:10] running and make sure to run this uh for [599:13] 5 to 10 minutes don't let it running for [599:15] an hour or more because that might [599:18] create a problem maybe your ISP or if [599:21] you are using cloudflare or uh other [599:24] stuff they might detect that you are [599:25] under a DOS attack and they might block [599:28] the IP. So just run this for 5 minutes [599:30] and then get the data and see uh if your [599:33] server was able to handle all that [599:35] traffic or not. So I will wait for 5 [599:37] minutes and then I will come back and [599:39] let me try to make this bigger so that [599:41] we can see things better. Okay. So here [599:43] you see the number of requests. You can [599:45] also stop for example if you reach 8,000 [599:48] or something like that. So currently we [599:50] have we we served 1,000 requests. Here [599:53] you get the the data that we are going [599:56] to look at after that. So you here we [599:58] have the average response time, minimum, [600:01] maximum and the current requests per [600:04] second. So for example, this endpoint is [600:07] receiving three requests. Again, as I [600:09] said, I will wait for 5 minutes and then [600:11] I will come back. I am back. We have [600:14] reached 5,000 requests. I will stop now [600:17] because I have the data. Okay. So we can [600:20] read this table or we can also check the [600:23] other tabs. If we click on charts that [600:26] gives us uh that will plot the data. Uh [600:28] so we can see that we started at 123 and [600:32] we finished at 1211 and uh so here what [600:35] do we have? So we have the total [600:36] requests per second. Uh on average we [600:40] were getting 16 requests per second and [600:42] it also shows us the requests that [600:45] failed. So that's interesting. We also [600:48] has 16 requests that were failing per [600:51] second. We have here the response time. [600:54] So normally it's small um but sometimes [600:57] we get these spikes. That's interesting. [600:59] This is the ramp up. So this is the [601:02] number of users. So here we started [601:04] small but then we reached the maximum [601:06] value set to 50. Okay that's good. But [601:09] if you are curious why sometimes we get [601:12] spikes in the response time that's [601:14] normal. This is what we call tail [601:17] latency. And if you are curious, there [601:19] is this amazing article that you can [601:21] read and uh so by Andrew you can see [601:24] that here on this image it it shows you [601:27] that most requests are going to have low [601:30] response time but some of them are going [601:32] to have [601:34] long response times and this is what we [601:36] call tail latency. It has nothing to do [601:39] with your application so don't worry [601:41] about it. Uh but if you want to uh read [601:44] more about it to see why that happens [601:46] even though the request is going to take [601:49] the same amount of time to process, you [601:52] can read this article. I'll make sure to [601:55] uh it is already on the documentation. [601:58] So you can find it there uh if you go to [602:00] the repository. And again just a [602:04] reminder if you don't if you don't know [602:06] what I am talking about, this is the [602:08] repo that uh that I'm that you can go [602:11] to. So it's called fullstack deployment [602:13] handbook and inside the book folder you [602:16] can find the content for this specific [602:20] module and here if you search for tail [602:24] latency you'll find that article here we [602:27] can also go to uh the terminal let's see [602:31] what we have here and we can stop this [602:32] and this will also show us some data. So [602:35] if you it's the same data that we have [602:37] in the dashboard. So you can see that [602:38] here we have 5,000 requests in total and [602:42] these are the error. So you can see that [602:44] we have 429. [602:46] Ah so now I understand why we have this [602:49] error because in my API endpoint I have [602:52] rate limiting. So that's why we get 429 [602:56] because once we reach I think 20 [602:58] requests per uh per minute uh I block [603:01] the request. So that's why we were [603:03] getting this. So I was confused why we [603:06] have a lot of errors, but that's fine. [603:08] So again, that's just another layer of [603:10] security or it's a layer to block people [603:14] that are that want to scrape your data [603:16] or something like that. So that's why we [603:18] have this. So that's completely [603:20] understandable. [603:22] That's amazing. I told you that the rate [603:23] limiting is done by minutes. So this is [603:26] why we have this specific recurring [603:28] shape because here if you look at 12 six [603:31] okay so the this specific user was below [603:34] the limit but once they reach that the [603:36] number of errors starts to increase [603:38] because they made a lot of requests but [603:41] after the end of that minute we reset so [603:43] that's why they are able to make [603:46] successful requests but then it [603:47] increases. So now let's go back to the [603:50] statistic to the statistics page to see [603:52] what we have here. So on aggregate the [603:55] average response time is 100 [603:57] milliseconds. That's amazing. So if we [603:59] look at the search uh endpoint it's done [604:03] is in less than 100 milliseconds. That's [604:06] that's really great. But if we go to max [604:08] which is the SP the tail latency we uh [604:11] we have to wait for 2 seconds. But that [604:14] happens very rarely. So what you want to [604:16] look at is basically the 50th [604:19] percentile. So that's what we have here [604:21] in this chart. this green. Yeah. So, if [604:24] I remove this, if we look at this green [604:27] graph, you can see that this is what we [604:29] want to care about because most requests [604:31] are going to be here. And you can see [604:33] that the response time is slow is small. [604:35] It's less than 100 milliseconds. So, [604:38] that's that's really great. Uh that's [604:40] all I wanted to show you here in locus. [604:43] You also can look at failures, [604:45] exceptions. We don't have those. And if [604:47] you want, you can download the data. So [604:49] now we can close this because we have [604:51] stopped the server here and I can also [604:54] close that. And if you look at Btop you [604:56] can see that the server was doing a lot [604:58] of work uh here. But after that yeah now [605:01] uh it is sitting idle because there are [605:03] no requests. I can also close that. And [605:06] here if you look at this dashboard you [605:08] can see that here are the yeah that's [605:11] 5,000 requests that we received today. [605:14] And here are the end points that were uh [605:17] that that received a lot of work. And [605:19] you can see this is based on that those [605:22] weights that we specified in the tasks. [605:25] So track search were the ones that had [605:28] the highest weight. That's why the we [605:30] have a lot of uh hits. Amazing. Now we [605:34] can close this and we can look at the [605:36] NCDU utility. You know if you want to [605:40] look at this disk this space you can use [605:42] du-h [605:43] and that gives you this information you [605:45] can aggregate it stuff like that but [605:47] again it's not uh a tool like btop where [605:51] you can interact with it so that's why [605:53] there is a tool called ncdu that allows [605:56] us to inspect the disk util us usage uh [606:00] with a terminal user interface and that [606:03] tool is called ncdu and on ibuntu you [606:06] can use a to install it and once that's [606:09] done you can use sudo incdu and here we [606:13] are going to do a full scan starting [606:16] right at the root directory so that will [606:19] take some time but as you can see [606:21] already we have a nice in application [606:24] running in the terminal yeah we can use [606:26] to analyze the disk usage and as you can [606:30] see I can use the arrows to go through [606:32] all of these and here let me show you [606:34] what we have here so currently the total [606:37] disk usage is 6.2 GB. Here you can see [606:40] that under the user folder we are use we [606:43] have we are using 2 GB and here my web [606:47] application is using 300 mgabytes. If I [606:49] want to understand what is inside user I [606:51] can hit enter. Now I am inside can go to [606:54] library. Okay. So here we have things [606:57] related to Linux. I should not touch [606:58] that. Here we have Python. I should not [607:00] touch that because I am using it. Um so [607:03] that's how you use it. If I go want to [607:05] go back, I use I go up and uh uh so that [607:09] will take me uh to the user folder and [607:12] now I can go to maybe home. Let's see [607:14] what we have there. Okay, so we have my [607:16] user there and this these are all the [607:18] things. So here we have NVM because we [607:20] are using node, we have cache. So maybe [607:24] if you want to remove the pip cache, we [607:26] can come here and hit D. That will [607:29] delete it. Okay. And if you want to [607:31] learn more about the commands, you can [607:32] hit you can press the question mark and [607:35] that will give you this help panel that [607:38] shows you how to move. So if you want to [607:40] move your cursor, you can use the up and [607:42] down keys. If you want to open a [607:44] directory, you hit enter. If you want to [607:47] delete, you use the D key, etc. And you [607:50] can uh use the up and down arrow to see [607:53] the other commands. So let's hit Q to [607:56] close. And let's close out of incu. And [608:00] to do that we use Q in order to close. [608:02] And that's how we use incu to analyze [608:05] disk space and to delete uh any folder [608:08] that is no longer needed. Okay. So now [608:11] let's try to fix the issue with uh [608:14] rotating logs. In order to do that we [608:17] will open the journal D configuration. [608:20] And here I want you to search for this. [608:22] So in order to search, you type, you hit [608:25] Ctrl W and you paste the term that you [608:28] want to search for and just hit enter [608:31] and there it is. So what we need to do, [608:33] we need to uncomment this in order to [608:35] tell Linux to change the amount of of [608:39] logs that we want to use because by [608:41] default on Ubuntu, we allow uh systemd [608:46] journal D. So this damon is going to [608:48] consume up to 10% of your total disk [608:51] volume just to store logs which is a [608:55] lot. So if you want to change that [608:57] default you come here you unccommend [608:59] this option and you set it to 100 [609:02] megabyte or any other value. So now [609:04] after we save this we need to restart [609:07] the service. So how do we do that? We [609:10] use systemctl and we restart it and we [609:13] now we can run status to see if it is [609:16] running and yes it is. That will uh [609:19] reduce the amount of space that we [609:21] allocate to logs. But in order to fix [609:24] the engineext issue we go here uh etc [609:29] log rotate and we go to engineext. And [609:32] as you can see here under rotate we have [609:35] 14 days. So after for everything older [609:38] than 14 days will be deleted. I will [609:40] change that to 365. [609:43] And this value depends on the disk space [609:46] that you have left. So if you don't have [609:48] a lot, this is going to cause you a [609:50] problem because storing 365 files, it's [609:54] a lot. Verify that you have enough space [609:56] before doing this. Okay. Yeah, I'll just [609:58] do it because in my case, it's uh it's [610:00] not an issue. And I will exit. And it [610:03] also depends if your website receives a [610:05] lot of traffic because let's say that I [610:07] only received let's say a thousand [610:09] requests a day that's not a lot. So the [610:11] log file is going to be small but if I [610:13] receive 100,000 requests that's a lot. [610:15] So one file is going to be very huge. [610:18] Multiply that by 365 [610:20] and uh imagine the amount of disk space [610:23] that you will be consuming. So yeah now [610:26] we won't we won't delete older files and [610:29] in the dashboard we will be able to see [610:31] a large uh duration so that we can see [610:34] what how what is happening across months [610:37] or weeks. Finally we are going to create [610:40] an automated script. I will put it [610:43] inside user local bin because if you [610:46] remember when when you put a a bash [610:48] script there you can run it in any [610:50] location. So that's the benefit of [610:52] putting it right there. And this [610:55] specific file it's small it's not that [610:57] big and this is what it is going to do. [611:00] So here set e is awesome. So if we have [611:04] an error is any step we are going to [611:06] terminate and we are going to exit [611:07] because we want to run all of these [611:09] steps without an issue. First we clean [611:13] um anything from apt and we use auto [611:17] remove with dash y to accept the the [611:21] things that we are going to delete. The [611:23] journal ctl vacuum time is a command [611:27] that will delete logs that are older [611:29] than 2 days. Here we are removing pip [611:33] cache and pmppm cache. And we print echo [611:36] system cleanup completed. And I want to [611:39] do this let's say once a month. Okay, so [611:41] I will save the file, exit. I will use C [611:44] cho to make it executable. Let's go back [611:48] to chron tab. Let's go to the end of the [611:50] file and let's add the following line. [611:54] And this is going to run this um script [611:58] at 3:00 a.m. in the morning on the first [612:02] day of every month. This is what we have [612:05] why we have this. And the logs are going [612:07] to be stored inside this file. But we [612:10] can see if we are getting the correct [612:13] echo message. Okay. Now I can save and [612:16] exit and that will be run that that [612:19] specific script will run uh once a [612:21] month. But you can also test it because [612:23] as I mentioned once we put that thing [612:25] there you can see that once I hit tab it [612:28] detects it. And now I can hit enter and [612:30] it says that I need to use pseudo. Okay [612:33] that's understandable. And because we [612:35] have put it inside p sudo that's fine. [612:37] So it's it is going to run without an [612:38] issue. So now it is uh using apt get to [612:42] clean the cache. This is the vacuum. So [612:45] you can see that we deleted logs that [612:47] are older than 2 days and you can see [612:48] they are using a lot of space. So here [612:51] we have maybe seven uh so 82 megabytes [612:55] and we have the correct echo message and [612:58] this is what we should see in that log [612:59] file. Amazing. We have finished this [613:02] reliability subchapter. Now we have one [613:06] more subchapter that I want to add [613:08] before ending this book or this course [613:10] and that one will be will be about [613:13] continuous security. If you remember [613:15] before uh we have created a pipeline [613:20] that runs uh let's see so we have SUS [613:23] and we also have a light security scan [613:26] using OASP zap but that one runs in 2 [613:29] minutes so that's why we have included [613:31] it in the pipeline because it is not [613:34] going to block us but there is a a heavy [613:37] where it is so this one is lightweight [613:39] but there is a heavy version of this [613:42] that takes a lot of time that's why we [613:44] should not put it in the pipeline [613:45] because otherwise instead of waiting for [613:47] 8 minutes we might wait for 30 minutes [613:49] or more that's why I didn't include it [613:51] in the pipeline but in the next sub [613:53] chapter I will show you how to create a [613:56] GitHub action that will run daily and [613:59] that action is going to run that heavy [614:02] security dynamic security scan. So [614:05] that's what we are going to see in the [614:06] next sub chapter. It's uh it's a bonus [614:09] that I wanted to add to show you um how [614:12] that happens. And the beauty about that [614:15] one is that it's going to run [614:16] automatically. We don't need to trigger [614:18] it with a pull request or something like [614:20] that. It is going to run let's say every [614:22] day at 3:00 a.m. or uh something like [614:25] that. We can specify maybe even hourly [614:27] if you want. And uh once when we see an [614:29] error there, it means hey we have a [614:31] problem that we need to solve. But never [614:33] include that in your pipeline because [614:35] that will make it very very slow. We [614:37] arrived at the last sub chapter in [614:40] module 6. It is titled continuous [614:43] security. Let's see what we are going to [614:45] talk about. We have mentioned in the end [614:48] of the previous sub chapter that in the [614:50] pipeline we are performing a light [614:52] dynamic testing. So we want to move from [614:55] that state into performing a complete [614:58] synchronized stack. And that's what I [615:01] mean by a heavy security scanning [615:04] because we need to run the front end, we [615:06] need to run the back end and other [615:08] services. In my case, that other service [615:10] is M search. So we want to have all of [615:12] these things interacting together while [615:14] running the dynamic testing. We are [615:17] going to also use OAS, but we are going [615:19] to configure it a little bit to perform [615:21] that heavy scan. And as I said, we are [615:25] going to use a an automated GitHub [615:28] action that will use the chrome syntax [615:30] like we did in the chrome tab uh [615:32] recently to run this scan following a [615:35] specific uh frequency or interval. And [615:39] like before we are going to use that uh [615:42] zap rule set that specific TSV file to [615:46] uh avoid uh false alarms. So things [615:49] related to engineext security headers [615:51] stuff like that. We are here in my [615:54] second account. This is my fork. I need [615:57] to uh click on the repository. Uh let me [616:00] remove these uh tabs. We don't need [616:03] them. And here on GitHub, you can hit [616:06] the dot key to open uh VS Code directly. [616:10] So if you want to do things quickly, uh [616:13] you can do them that way. So just hit uh [616:16] the dot key and that that will open VS [616:19] Code. Uh so here it says the extension [616:22] yes allow login. Let's continue. The [616:26] only thing we need to add is a new uh [616:28] workflow and here we are in the master [616:31] branch. So we cannot push directly to it [616:34] because it is protected. So let's create [616:36] a new branch. Let's call this feed daily [616:40] bas scan and let's create it from the [616:44] head. Let's switch to that branch. Okay. [616:47] Now open the GitHub folder and add this [616:51] specific file inside the workflows [616:53] folder daily deep security scan. I will [616:58] copy the full content and I will paste [617:00] it here. You will see that this is [617:02] similar to the other file where it is [617:04] this dust scan. Uh I'll put them next to [617:07] each other so that we can see we can see [617:09] the difference. The only thing that we [617:11] have here, you can see that here we have [617:13] on workflow call. But here we also add [617:17] this option which is to schedule this. [617:20] So this means that this GitHub action [617:24] will run daily at 6 UTC. We don't need [617:29] to. Yes, we have the option to run it [617:32] manually, but on top of that, GitHub [617:35] will read this chrome syntax and it is [617:38] going to understand that we wanted to [617:41] run this for us every day at 6 uh UTC. [617:45] And that's the the only the the first [617:48] difference, but we have another one. So [617:50] here uh so because this is a deep scan, [617:52] we want to time it out. So if something [617:55] bad happened maybe uh we didn't install [617:58] dependencies correctly or something was [617:59] not working uh well we don't want this [618:03] job to get stuck. So we are going to [618:05] fail if it takes more than 30 minutes [618:08] and this value can be adapted based on [618:10] the scale of your application. So if you [618:13] have a lot of endpoints and a lot of [618:15] services the scan will take a lot of [618:18] time. So you want to run it maybe two, [618:20] three or five times to get the average [618:22] time and then put it here. But if we see [618:25] if we look at the other stuff, let me uh [618:28] decrease the font size. First of all, we [618:30] define the environment variables for me [618:33] search. But then the next steps are [618:35] similar. We check the repository. We [618:38] install Python. We install the backend [618:41] dependencies. Uh we are here. We after [618:45] that we install me search. Uh okay then [618:49] we start it we use that script to see [618:53] the so let's see so here we are starting [618:55] mill search and uh here you can see that [618:57] we have that seed script here it is here [618:59] is that same step and we also start the [619:02] back end so here maybe I could uh put [619:06] this here to have the same order let's [619:09] put it there you can see that here we [619:10] have se test data we start the backend [619:14] server and after that so here we stop [619:17] because we were testing just the back [619:19] end. We weren't testing the other things [619:22] in the stack. So we can see that here [619:24] after arriving at this we just pointed [619:26] at the open API JSON because we have a [619:28] fast API back end that will tell Zap all [619:32] the end points. But here we continue. We [619:35] install we set up PNPM. We install [619:38] NodeJS. We install the front end [619:40] dependencies. We build the front end. [619:43] And then we start it. Here we are using [619:45] pmppm preview because we want to [619:47] simulate a production environment. We [619:49] are not using pmppm dev because that is [619:51] a dev server. We want to use pmppm [619:54] preview. And here we uh like before we [619:57] are using a for loop just to ensure that [619:58] the front end started and and it is [620:01] ready to uh be used. And finally we [620:04] perform the full scan this time. And if [620:07] you look at the image that we are using [620:09] it's different. So here we have action [620:11] API scan because this is uh used for API [620:15] endpoints but here we are using the full [620:18] scan and here you can verify the version [620:22] to see if there are new ones. If there [620:24] are make sure to update and here the [620:27] target also changes. So we are pointing [620:28] at the front end. It's like a user that [620:30] will go there and try to do malicious [620:33] stuff. uh we are we don't we don't just [620:35] point at the back end and we are reusing [620:38] that TSV file and uh yeah the this the [620:42] other options are are the same but after [620:44] doing this you can see that we are going [620:46] to test everything in our stack mainly [620:48] search the front end and the back end as [620:51] you can see both of them are similar we [620:52] just uh added running the front end and [620:56] uh we changed the image that we are [620:58] using here to action full scan amazing [621:02] now let's commit this file. So let's [621:05] come here and let's say that field add a [621:09] new workflow for full dust scan commit [621:13] this and push it to this new branch that [621:15] we have created. And now we can create a [621:18] new merge request and point it at my [621:21] master branch. And I hope that we don't [621:25] have a problem with the dependencies. I [621:28] think we will because it's been a few [621:30] weeks since I've worked on this. But if [621:33] we yeah if I get some problem while [621:36] auditing the dependencies I'll try to [621:38] resolve them and I will come back and uh [621:40] Okay. So let's wait for this to finish [621:43] and then we can go to the actions tab [621:46] and look for that new file and then we [621:48] can run it manually from here. Let's [621:51] wait for this to yes thankfully the [621:54] auditing step passed. That's amazing. Uh [621:58] so that means that this will will not [622:00] fail hopefully because we haven't [622:01] changed anything in the source code. I [622:04] am back and as you can see we can merge [622:06] the pull requests because we don't have [622:09] any error. Let's confirm the merge and I [622:12] can also close this tab because I no [622:15] longer need it. Let's go back to master [622:18] and yeah as you can see it merged now. [622:21] Now we can click on actions and we [622:24] should see this new scan. Yes, here it [622:27] is. Daily deep security scan. Let's [622:30] click on it. And as you can see, [622:32] normally if we wait, it should run at 6 [622:34] in the morning every day. But now it is [622:38] 12:00 a.m. or 12:00 p.m. So I cannot [622:41] wait for that. So let's just hit run [622:43] workflow. Make sure that it is on master [622:45] and hit run. And uh yeah, it's just one [622:49] simple job. If we click on it, we will [622:52] see the the steps. We installed Python [622:55] and now we install all the dependencies [622:58] that we need for the back end. We [623:00] downloaded M search. Now we start it and [623:04] we need to seed it with data so that the [623:06] end points will work and won't return [623:10] errors. We have restarted the started [623:12] the back end and as you can see we sent [623:14] a request to help to ensure that server [623:16] has started. Now we are installing [623:19] NodeJS [623:21] the front end dependencies. We are [623:23] building the front end. That shouldn't [623:25] take a lot of time. Okay, that's done. [623:27] Now we start the front end using PMP [623:29] view. And we see the front end is ready. [623:32] Great. And now we are starting the full [623:35] scan. Hopefully that I haven't forgot [623:39] anything to add to that TSV file. [623:43] Normally this should pass because uh I [623:45] haven't changed anything in the back end [623:47] and I have tested this before on my [623:49] personal project or on my websites that [623:52] I have deployed to production. So it [623:54] shouldn't fail. But since this will take [623:57] some time, I will wait for it and then I [623:59] will come back. I am back and as you can [624:01] see the scan didn't fail. That's a good [624:04] sign. And in my case it took 6 minutes. [624:07] You can see that this is already uh [624:09] taking three times more time than the [624:12] light one. The light one took 2 minutes [624:15] and this one took six. So, and it runs [624:17] successfully and this is going to run uh [624:19] every single day. But as I said, if you [624:22] have a a large application, my personal [624:24] work is not that big. So, I have a few [624:26] endpoints that's why it took 6 minutes. [624:28] But if you have a lot of applications, a [624:31] lot of services in your app, this time [624:33] will increase drastically. And that's [624:36] all I wanted to show you in this [624:37] subchapter. Thank you for watching. The [624:40] next video will be about we will look at [624:43] all the things that we have learned [624:45] together across this long journey that [624:48] we took from the first sub chapter where [624:50] we were just talking about what we are [624:53] going to do. And now as you can see we [624:55] are able to perform scans, block IP [624:59] addresses and do a lot a lot of things. [625:03] And I hope that you have learned a ton [625:04] from this course. The the next video [625:06] will be simple. It's not going to be [625:08] very long, but we will try to at least [625:10] give you something to that that you can [625:13] use to basically tell you um maybe [625:16] remind you of all the things that you [625:18] have learned. Congratulations on [625:21] reaching the end of the course. You have [625:24] successfully journeyied from a bare [625:27] Linux server to a fully hardened [625:32] self-sustaining production application. [625:34] These are all the things you have built. [625:37] In the first two chapters, I have showed [625:41] you how to harden SSH access by removing [625:46] the ability to authenticate with [625:48] password, removing the ability to [625:51] connect as root user. You also learned [625:55] how to play with the uncomplicated [625:58] firewall. We have learned how to use [626:01] engineext socket proxying to talk to our [626:04] J unicorn/ Uvic application. On chapter [626:09] three and four we have focused on [626:12] performance and data. Here you have [626:15] learned how to isolate the the me search [626:20] engine by creating a new user that we [626:23] called me search and that user was [626:26] managing the this service and uh we also [626:31] made sure that the environment variables [626:34] that med search requires are stored in a [626:37] way that cannot be read by other users. [626:40] We have played with go access which [626:43] allows us to read the enginex logs to [626:47] create really beautiful dashboards that [626:49] shows us what is happening on our web [626:52] server. We have also learned how to use [626:54] cloudflare CDN to serve the static files [626:58] quickly across the world across the [627:00] globe and we have we have used SSL [627:03] certificates to make your website [627:06] secure. And finally in the last two [627:08] chapters we talked about automation and [627:11] security. Here you have learned about [627:14] pre-commit hooks so that you don't push [627:16] code that is broken. So you need to pass [627:20] a few tests in order to be able to [627:22] commit and push. You have learned about [627:25] CI/CD testing pipelines, atomic [627:28] rollbacks, automated [627:31] dynamic application security testing and [627:34] also uh testing the libraries that we [627:38] are using in our application. After [627:40] doing everything, now our application is [627:43] ready for production. Now your system is [627:46] observable. You have go access which [627:49] allows you to read the logs and get [627:52] realtime metrics. It's automated meaning [627:56] we will have zero downtime. We have seen [627:59] that even if we kill a UVON worker the J [628:04] Unicorn process manager will create one [628:06] for us. So even if you have a lot of [628:09] traffic and maybe one of the workers [628:11] dies everything is automated. So you [628:14] will have zero downtime when we deploy [628:17] our our application. If something [628:20] happened while installing the [628:21] dependencies, the application is not [628:24] going to crash. We are going to roll [628:26] back and use the all dependencies that [628:29] we had before the deployment. And [628:32] finally which is this is the most [628:33] important point security. We have made [628:36] sure that people cannot log in using SSH [628:40] as root or by using a password. We you [628:43] have SSL certificates that renew [628:45] automatically so that people who connect [628:47] to your website are connected securely [628:50] and we have done a lot of testing jobs [628:53] in our CI/CD pipeline. I really hope [628:56] that you enjoyed going through this [628:58] course with me and now we are completely [629:01] ready to build, launch and scale your [629:03] features with absolute confidence. Thank [629:06] you for following along and happy [629:08] building.