CI/CD in Production with Jenkins – Complete DevOps Course

[00:00] Welcome to this comprehensive CI/CD [00:02] course designed to take you from [00:04] foundational concepts to [00:05] production-grade implementations. Over [00:08] the next 17 and a half hours, you will [00:10] gain a solid real world understanding of [00:12] modern DevOps workflows using Jenkins as [00:15] the automation server. You will work [00:18] through nine hands-on projects [00:20] culminating in a complex dev sec ops [00:23] mega project. By the end of this course, [00:25] you will have the skills and context [00:28] required to build, secure and operate [00:31] reliable software delivery systems in [00:34] any professional environment. Hello, I [00:36] am Varun Jooshi and I welcome you to [00:38] cloud with VJOS where we talk about [00:41] cloud DevOps and everything in between. [00:45] What you are about to watch is a fully [00:48] consolidated CI/CD master class bringing [00:51] together 11 lectures and over 17 and 1/2 [00:55] hours of structured in-depth learning [00:58] into a single video. This video is for [01:02] anyone who wants one complete place to [01:06] understand how CI/CD actually works in [01:09] production without jumping between [01:12] disconnected tutorials. This is not just [01:15] a Jenkins course. This is a CI/CD first [01:19] masterass with Jenkins used as the [01:22] primary orchestrator. We start from [01:25] absolute basics and move step by step [01:28] towards advanced production grade CI/CD [01:31] implementations. [01:33] Every concept is introduced with [01:36] context. So you understand not just what [01:39] you are doing but why it is done that [01:42] way in real engineering teams. While [01:45] Jenkins is the main tool used most of [01:49] what you will learn here is orchestrator [01:52] agnostic. The concepts apply equally to [01:55] GitHub actions, GitLab CI and other [01:59] modern CI/CD platforms. Once these [02:02] fundamentals are clear, switching tools [02:05] becomes super easy. This master class is [02:09] designed as a complete end toend [02:12] learning journey. You will walk through [02:15] nine hands-on CI/CD projects and the [02:18] course concludes with a dev sec ops mega [02:22] project that runs for nearly 3 hours and [02:25] 40 minutes. In this final project, we [02:28] bring everything together in a [02:30] production style setup. So you can see [02:33] how real CI/CD systems are actually [02:37] built and operated. All demos in this [02:40] course are intentionally chosen to [02:42] reflect real world scenarios. There are [02:46] no toy examples and no artificial [02:49] workflows. What you see here closely [02:52] mirrors the patterns, decisions and [02:55] tradeoffs used in real production [02:57] environments. Some ideas are repeated [03:00] intentionally. CI/CD foundations are [03:03] critical and repetition helps these [03:06] concepts stay with you when you start [03:08] applying them in real projects. Along [03:11] with the video, you will have access to [03:14] a well-maintained GitHub repository that [03:17] goes hand inhand with the lecture. These [03:20] notes contain the exact YAML files, [03:24] Jenkins files, configurations, and [03:26] commands used during the demos. More [03:29] than just notes, think of this [03:32] repository as a written companion or [03:35] transcript of everything we are [03:37] learning. So you always have one place [03:40] to revisit and revise concepts. I have [03:43] intentionally kept the notes structured [03:46] day-wise instead of merging everything [03:49] into a single large readme. This makes [03:52] it easier to navigate, practice and [03:55] revise without confusion. The repository [03:58] link is available in the description. [04:00] For completeness, I have also retained [04:03] the original intro and outro of each [04:06] lecture. So you can easily map sections [04:09] of this long video back to the [04:12] corresponding notes. This course is not [04:16] meant to be watched passively. I [04:18] strongly encourage you to practice every [04:21] lecture along with me. Try to understand [04:24] the why behind what you are doing. Not [04:26] just commands or configuration. break [04:29] things, fix again, experiment with [04:31] variations. [04:33] This process is what makes the learning [04:36] stick for the long term, especially in [04:39] CI/CD and DevOps where real [04:42] understanding comes from hands on work. [04:46] The more you engage with the demos, the [04:48] more confident you will feel when [04:50] applying these concepts in real [04:53] production environments. [04:55] Before starting this course, you should [04:57] have basic understanding of Docker and [05:00] Kubernetes. Everything else is covered [05:04] from the ground up. In this video, we [05:08] begin by building strong CI/CD [05:10] fundamentals. You will first understand [05:13] how modern software delivery works, how [05:17] SDLC has evolved and why build [05:20] automation and CI/CD exist in real world [05:24] teams. From there we break down CI/CD [05:27] end to end covering how pipelines work, [05:30] how branching strategies are applied in [05:33] production and how code moves from [05:36] commit to deployment. Once the [05:39] foundation is clear, we move into [05:41] Jenkins. You will learn what Jenkins is, [05:44] how it fits into the CI/CD ecosystem and [05:47] how to install and run it using Docker. [05:51] We start with freestyle jobs and [05:53] gradually go deeper working with [05:55] parameters, git branches and tags, [05:58] triggers, and chronbased automation so [06:01] you understand how Jenkins is actually [06:05] used dayto-day. With that base in place, [06:08] we move into hands-on projects. You will [06:10] build, push and deploy applications [06:13] using Jenkins. First with freestyle jobs [06:16] and then with Jenkins pipelines. From [06:19] there we move into production style [06:22] CI/CD by working with multibranch [06:25] pipelines, GitHub pull requests and real [06:28] branching workflows used by engineering [06:31] teams. The course then expands beyond [06:35] Jenkins itself. You will learn where [06:37] build tools like Maven fit into the [06:40] DevOps workflow and how they are used [06:43] inside real CI pipelines. We then [06:47] introduce Sonar Cube to bring in dev sec [06:49] ops practices covering code quality [06:52] security scanning and enforced quality [06:55] gates inside CI/CD pipelines. All of [06:59] this come together in dev sec ops mega [07:02] project where we build a production [07:04] style setup using a multi-AZ Amazon EKS [07:09] cluster integrated with Jenkins Maven TV [07:13] sonar cube Amazon ECR and GitHub. We [07:17] close the course by covering Jenkins [07:20] shared libraries showing how teams move [07:24] from copy paste pipelines to reusable [07:27] maintainable production grade CI/CD [07:30] design. [07:31] Your feedback genuinely matters to me. [07:35] If this course helps you in production, [07:37] clears your interviews or strengthens [07:40] your CI/CD fundamentals, I would love to [07:44] hear about it. And if you feel something [07:47] is missing or unclear, tell me that too. [07:50] I read the comments and reply. Your [07:54] feedback directly influences how future [07:57] content is shaped. By the end of this [08:00] master class, you will not just know [08:03] Jenkins, you will understand how CI/CD [08:06] works in real world and how to apply it [08:09] confidently in production. If this [08:13] course helps you, liking the video, [08:16] subscribing to the channel, or leaving a [08:18] comment goes a long way in supporting [08:21] the work and helping it reach more [08:24] learners. If you would like to support [08:26] the channel further, consider joining as [08:30] a channel member. I don't lock content [08:32] behind pay walls, so membership is [08:35] simply a way to back the free learning I [08:38] share with everyone. So with all of that [08:42] said, let's get started and I wish you [08:45] all the very best. Hi, I am Vun Jooshi [08:48] and welcome to day one of our Jenkins [08:52] basics to production course. In this [08:55] very first lecture, we are setting the [08:58] stage by covering the fundamentals every [09:02] DevOps and S sur engineer must know [09:05] before touching Jenkins. And here is the [09:09] agenda for today's lecture. Firstly, we [09:12] will discuss the phases of software [09:15] development life cycle SDLC. How modern [09:19] software moves from idea to production. [09:23] Next, we will discuss waterfall and the [09:27] agile DevOps model. Why we shifted from [09:31] big bang releases to continuous [09:34] delivery. Then we'll move on to compiled [09:37] versus interpreted languages. What [09:40] happens to your code before it runs. [09:44] Next we will discuss the key building [09:47] blocks which are artifacts, runtime and [09:51] dependencies. And finally the build [09:54] process which is of extreme importance [09:57] for us DevOps engineers both for [10:01] containerized and non-containerized [10:04] workloads. So you understand how [10:07] applications actually get prepared for [10:11] production. By the end of this session, [10:14] you will have a production level [10:17] understanding how software is built and [10:21] deployed which will make everything we [10:24] do with Jenkins later far more clearer. [10:28] So let's get started. Phases of SDLC. [10:32] Firstly, SDLC is software development [10:36] life cycle. And this is what we will [10:40] use. What you see on screen is what we [10:43] use in agile and devops model which is [10:47] the latest and the greatest one. So now [10:51] I want you to understand that there are [10:54] millions of applications that are around [10:57] the globe. Are all the applications [11:00] adhering to this SDLC framework? No, it [11:04] is not that. But whether an application [11:08] adheres to all these phases depends on [11:12] the maturity of the application. So if I [11:15] talk about an enterprisegrade [11:18] application like Amazon.com or [11:20] Netflix.com, [11:22] they would definitely be adhering to the [11:26] SDLC process. Now SDLC at large is not a [11:32] tool. It is not something that you can [11:34] purchase. It's an ideology. An ideology [11:38] that you must have in your application [11:42] if you want your application to have the [11:46] latest and greatest modern design. [11:50] Hence, SDLC is a logical framework which [11:55] defines how you should be developing [11:58] your application. And in this lecture we [12:02] will start by discussing the different [12:05] phases in SDLC. Now like I said SDLC is [12:10] not a tool. It is a framework. But [12:13] within the phases of SDLC, you will of [12:17] course be using different tooling. For [12:20] example, if I talk about the first [12:22] phase, this is requirement gathering. [12:24] What would requirement gathering? What [12:26] are the tools that I would use in [12:28] requirement gathering? I would be using [12:31] Microsoft Teams, Zoom or WebEx for [12:34] meetings. I could be using Excel or [12:37] Microsoft Office in general to prepare [12:40] my documentation. I could also be using [12:43] tools that I would use to create my [12:46] diagrams for example diagrams.net [12:49] or eraser.io etc etc. So what I'm trying [12:54] to tell you is in each of these phases [12:57] you will be leveraging different tools [13:01] but SDLC at a very high level is just a [13:06] framework. Now let's discuss the first [13:09] point requirement gathering. What [13:12] happens here? As the name implies this [13:14] is the place where you gather everything [13:18] you can about the application. Now when [13:22] I talk about an application there are [13:25] two things that we talk about. First is [13:28] functional and second is non-functional. [13:32] Now what are functional requirements? [13:35] Functional requirements are what the [13:38] system should do. For example, if we are [13:41] developing a banking application, then [13:44] it should be able to show users their [13:47] account balance, their passbook. They [13:50] should be able to check if they can [13:52] apply for a loan, if they are eligible [13:54] for a loan. They should be able to [13:56] calculate interest rates and so on. That [14:00] is the functional requirement from a [14:02] banking application that they should be [14:04] able to do these things. Then what are [14:08] nonfunctional [14:10] requirements? Now nonfunctional [14:12] requirements are anything that are not [14:15] directly application. For example, what [14:18] should be the infrastructure on which [14:20] this application must be deployed? [14:23] Whether we have to adhere to any [14:26] compliance standards or not. What will [14:29] be the RTO and RPO for my application? [14:33] What will be the DR strategy for my [14:35] application? What will be the [14:38] availability requirements for my [14:40] application? These are nonfunctional. So [14:43] functional are essentially anything that [14:46] your developers will be coding for the [14:49] application to work properly and [14:52] non-functional are anything that will be [14:54] supporting your application. For [14:56] example, the underlying hardware will be [14:59] helping my application to run. So [15:02] underlying hardware is a non-functional [15:05] requirement. I hope functional and [15:08] non-functional is clear to you. Then [15:11] third point is compliance and security [15:14] standards. Now at this point of time I [15:17] want to differentiate between security [15:20] and compliance. Security is a subjective [15:23] term for organization A. Security [15:27] implies not allowing port 22 to the [15:31] outside world. Not allowed at all. For [15:34] organization B. Security could mean [15:37] allowing port 22 but only from a [15:41] specific public IP. Both of these are [15:45] security requirements and these are [15:47] subjective. So security in general is [15:51] subjective. For one organization or for [15:54] one application something could be [15:56] secure and for another something could [15:59] not be secure. But when I talk about [16:01] compliance, compliance is dealt with [16:05] different compliance standards. So if I [16:09] say my application is compliant to GDPR, [16:13] then my application must meet the [16:17] requirements that are laid out in GDPR. [16:20] Now GDPR is a popular compliance [16:23] standard when I talk about Europe. So in [16:26] the requirement gathering section itself [16:30] you decide what all are the compliance [16:34] standards that your application must [16:37] adhere to. Now these compliance [16:39] standards could be PCI, DSS, GDPR, ISO [16:43] 27,01 [16:45] even HIPPA. If you are doing insurance [16:48] within your banking application then you [16:50] must also adhere to HIPPA. So in the [16:53] requirement gathering phase I am just [16:56] saying that these are the compliance [16:59] standards I must adhere to. I am not [17:02] defining how we will adhere to them. I [17:05] am just saying we must be adhering to [17:08] compliance standards. Similarly we must [17:11] have a DR. Now how that DR is achieved [17:15] we don't know yet. We want high [17:17] availability but how that high [17:19] availability will be achieved we don't [17:22] know that all those things will be done [17:24] in the design phase which we'll discuss [17:27] next. Now before I move to design let's [17:30] also see plan for reliability and [17:33] environments. Now in the requirement [17:35] gathering phase itself we decide how [17:38] many environments do we need. Do we need [17:41] just dev staging and prod or do we also [17:45] need a QA or a testing environment as [17:48] well? So all these things are decided in [17:52] the requirement gathering phase. Now [17:55] once all these things are collected your [17:58] program manager would be the person who [18:00] would be overseeing everything and will [18:03] be working with different project [18:05] managers that are part of that project [18:08] and then you folks your devops [18:11] architects your application architects [18:13] your developers will be participating in [18:16] these meetings and you will have [18:18] documents that will have all the [18:21] requirements. Then there is the design [18:24] phase. Now design phase is architecture [18:27] and API contracts. Here is where you [18:32] define how you will achieve whatever was [18:36] defined in the requirement gathering [18:38] phase. Now I want you to clearly know [18:41] that these phases are for both [18:44] developers and devops. It is not just [18:47] the DevOps team who will be [18:49] participating in all these phases [18:51] because we are talking about taking an [18:53] idea to running software. So when I talk [18:58] about creating an application, it is not [19:00] just DevOps. You will also have [19:03] developers here. So design is [19:05] essentially how the system should [19:08] function and interact. Second point is [19:11] testability and observability [19:14] built in. Now in this phase itself you [19:16] will design for everything. So be it [19:18] your observability posture, security [19:21] posture, testing posture, high [19:23] availability, DR here you will be [19:26] defining how you will achieve whatever [19:29] was gathered in the requirement [19:32] gathering phase. So security and [19:34] branching policies. Now you know that [19:37] whenever we talk about developing a [19:40] software you know that the entire code [19:43] is contained in some sort of source [19:46] control source control like GitHub or [19:49] GitLab and there are native products [19:51] available from cloud providers as well. [19:54] And just a small note here AWS has [19:57] discontinued its source control service [20:00] which was AWS code commit. So you will [20:03] have to go elsewhere if you are on AWS. [20:06] You can be using something like GitLab, [20:09] Bitbucket or GitHub. And what will be [20:12] the branching policy who all will be the [20:15] collaborator. So all these things are [20:18] done in the design phase itself. And [20:20] then the next phase is development. Now [20:24] development is the phase where the [20:27] actual work starts. First is branching [20:31] strategy and code review. So whenever [20:33] developers are writing the code, they [20:35] will be writing in their local machines [20:38] and then they will be pushing the [20:40] changes to your central repository. [20:43] Second is local builds and unit tests. [20:46] So developers will be performing local [20:49] builds and unit tests. What are unit [20:51] tests? We will see them in a little [20:53] while. Shift left security. Now firstly [20:58] let's understand what is shift left. Now [21:01] previously what used to happen was the [21:03] security was only dealt with when we [21:06] were deploying the application or when [21:08] the application was completely deployed. [21:11] But now we are adhering to the shift [21:14] left approach. Why we are saying it [21:16] shift left? Because if you go from 1 to [21:19] 8 you are going left to right. [21:23] Previously it was security was applied [21:26] here or here but now we are saying shift [21:30] left from the requirement gathering [21:32] phase itself start thinking about [21:36] testing security and automation. Hence [21:40] even when the developers are developing [21:44] SAS and SCA are still performed. What is [21:48] SAS? SAS ST is static application [21:52] security testing and SCA is software [21:56] composition analysis. Now when the code [22:00] is being written the code is static and [22:03] that is why static testing is performed. [22:06] When the application is in running state [22:09] then we perform something called DAT. [22:12] DAT is dynamic application security [22:17] testing. So static scanning when the [22:20] code is being written. So developers are [22:23] in their idees and these static testing [22:26] tools are integrated within the IDE and [22:30] then when the application is running you [22:33] have something called DA ST and we'll be [22:36] seeing that shortly. Now there are tools [22:40] that are popular. So if you want to [22:42] perform static testing then you have [22:44] sonar cube which can test your code. you [22:47] have check marks for DAT, you have OASP, [22:51] ZAP and there are plenty of tools that [22:53] are available at your disposal. Then the [22:56] last point is pre-commit hooks and [23:00] feature flags. Now what are pre-commit [23:03] hooks? Pre-commit hooks are local [23:05] guardrails that enforces code quality [23:09] security even before the code is pushed [23:13] to your GitHub repository or to your [23:16] choice of source control. Now let's talk [23:20] about the fourth phase that is the build [23:23] phase. Now I want you to understand that [23:26] when I talk about build and deployment [23:29] phase, these are of paramount importance [23:32] when you are releasing a new version of [23:35] an application or when you are deploying [23:37] the application for the very first time. [23:40] So in this section we will briefly talk [23:43] about the build stage and then we will [23:46] zoom in into this as we progress with [23:49] this lecture. Now first point is compile [23:52] and package code. So whenever you have a [23:55] source code it is in raw format. You [23:58] must compile it and then package it. And [24:01] once you do this packaging you receive [24:03] something called an artifact. So [24:06] whenever you are packaging a software [24:09] then the output that you receive is [24:11] called an artifact. See do not limit [24:15] your imagination here. So whenever you [24:17] are building something whatever the [24:19] result of that is is known as an [24:22] artifact. So even if you are building a [24:26] container image the container image that [24:28] you receive is also an artifact. So [24:32] whatever is the result of that build [24:34] stage produces some output and that [24:38] output is called artifact. Second is [24:42] reproducible build pipelines. Now if you [24:46] have a stringent process for build then [24:50] whatever your artifact is in your local [24:53] machine the artifact will work fine in [24:56] your production pipelines as well in [24:59] your development pipelines as well. For [25:01] example, if I'm creating a container [25:02] image and I run that container image [25:05] onto my laptop that has Docker engine, [25:07] it will work and any machine that has [25:11] Docker engine installed all over the [25:13] internet, that container image will [25:16] behave the same way in that machine [25:20] because the underlying Docker engine was [25:23] the only requirement for that container [25:26] image to run as a container anywhere. [25:29] Similarly, if your artifact is received [25:33] in one machine, that artifact should be [25:36] reproducible. So, whatever results that [25:39] you see in your local machine, you [25:42] should also see those results in your [25:44] production and dev pipelines. Here that [25:48] is what we mean by reproducible build [25:51] pipelines. Next point is artifact [25:54] registry storage. So whatever your [25:57] output you receive from the build phase [26:00] that output should be stored in some [26:04] sort of registry. So when I talk about [26:06] container images you have different [26:08] image registries that are available that [26:11] you can use to store your images. You [26:14] can use docker hub you can use AWS's [26:17] elastic container registry and so on. [26:20] when I talk about application artifacts. [26:22] So application artifacts are [26:24] predominantly the artifacts that you [26:27] receive post building applications [26:30] source code and that output can be [26:33] stored also in a registry. Now there are [26:36] different sort of registries for them. [26:39] For example, you can use Nexus or JROG [26:42] registries to save your artifacts or you [26:47] can also use native tooling that is made [26:50] available by the cloud provider. Next [26:54] point is supply chain security. In [26:57] parenthesis you see Sbombs and Sbombs [27:01] means software bill of materials. Now [27:04] software bill of materials they comprise [27:07] of everything that is used to build a [27:10] software. It could be your image, it [27:13] could be your application, it could be [27:16] your artifact etc etc. So essentially [27:20] this line implies all the supply chain [27:24] must be secure. So if there is code you [27:28] should be using SAS and dashed. If there [27:30] is container you should be using static [27:33] and dynamic scanning. So what is static [27:36] and dynamic scanning when I talk about [27:38] containers? So if your container image [27:40] is there in a registry then your [27:43] container image must be scanned. If you [27:46] have a running instance of that [27:48] container image that is if you have [27:50] running containers then you should be [27:53] scanning your containers as well and [27:55] this is also called runtime scanning. So [27:58] if I am talking about your containerized [28:00] workloads, there should be a two-step [28:03] scanning mechanism. First, scanning [28:05] should happen for your container images. [28:08] Second, the scanning should also happen [28:10] for your running containers. And there [28:13] are multiple tools available for this. I [28:15] will let you explore Trivy, Anor, Sneak [28:18] and so on. Then the next phase is [28:22] testing phase and in parentheses you see [28:25] continuous. Now testing is also adhering [28:29] to shift left approach when we are [28:31] talking about modern SDLC. So right from [28:34] the development phase the testing is [28:37] ongoing which is why you see in the [28:40] second point we have unit tests [28:42] mentioned here. Now let's understand [28:45] what testing means and what are [28:48] different types of testing that you will [28:51] see in your day-to-day work. Now testing [28:54] in general implies testing something [28:58] whether it is working as expected or [29:00] not. That is testing. So for example, if [29:02] I want to test whether my connection to [29:04] the internet is working or not, I'll try [29:06] to ping google.com and if it works, I'll [29:09] know that my connection to the internet [29:11] works. Now let's see what testing means [29:14] when we talk about a software. First is [29:18] unit and integration test. And I want [29:21] you to take example of the same banking [29:23] application that we have been talking [29:25] about. So unit testing is my banking [29:28] application could have several [29:30] functionalities. It could be used to [29:33] check your account balance. It could be [29:34] used to check your passbook. It could be [29:37] used to apply for loan and so on. So [29:40] this application can be divided into [29:43] different units. And that is what unit [29:46] testing means. Unit testing means [29:48] testing different units of your [29:50] application. So if I want to test let's [29:53] say account statement, I will see [29:55] whether the user will be able to log in [29:57] and check its account statement. So that [29:59] is called unit testing. Now what is [30:02] integration testing? Integration testing [30:04] is how your application behaves when two [30:07] or three components come together. So if [30:10] I want to check my account balance along [30:12] with my passbook then [snorts] these two [30:16] components of the application must work [30:18] together. Hence when you are testing [30:21] multiple components of an application it [30:23] is called integration testing. [30:25] Essentially how the application behaves [30:27] when multiple components come together. [30:30] Second is performance and load test. [30:33] Performance implies how the application [30:35] is behaving. What are the metrics? what [30:38] is the speed at which the requests are [30:40] catered and load tests are how my [30:43] application is behaving when there is [30:45] let's say 90% load on my application [30:48] then in testing itself you have security [30:51] scans you have SAS and DAST you see [30:55] there is no stage or phase named [30:58] security because security must be in [31:02] mind from point 1 through8 so you should [31:06] be taking care of security testing and [31:09] automation in all the stages of your [31:13] SDLC. [31:15] So security scans we have already [31:17] discussed SAS and dashed. Then there is [31:20] pen testing. Pen testing is penetration [31:22] testing and these this testing is [31:25] typically performed once in 3 months or [31:28] once in 6 months or once in a year. [31:31] Pentesting involves knowing how the [31:35] application will behave when there is an [31:37] attack for example. Then the last one is [31:40] smoke and regression suites. Now smoke [31:44] testing is a quick testing. So if let's [31:46] say you released an application, you [31:48] quickly see whether the application is [31:50] accessible or not. This is a simple [31:52] smoke testing. Regression testing is you [31:56] test the application thoroughly. whether [31:59] all the functions that my application [32:02] was supposed to perform are being [32:04] performed by the application or not. So [32:06] regression is a thorough testing. Now [32:11] let's move to deployment. Now in [32:13] deployment we know how the deployment [32:17] should be. You first will be deploying [32:20] into dev stage and then that will be [32:22] graduated to staging and then that will [32:25] be graduated to prodage and we'll be [32:28] discussing this in greater detail as we [32:31] progress with this course. Second is [32:34] progressive delivery strategies. There [32:36] are multiple delivery strategies like [32:38] blue green canary. What you use depends [32:42] on what strategy was finalized in your [32:46] design stage. Now next is infrastructure [32:49] as code. So you should be using tools [32:52] like terraform, helm, anible to do the [32:55] deployment and configuration. The next [32:58] is roll back and roll forward playbooks. [33:02] Roll back is how you will roll back the [33:04] application. You should have strategies [33:06] in place. Roll forward is if let's say [33:10] you have deployed the application and [33:11] you can correct the error quickly. How [33:14] should you be doing that? that is called [33:16] roll forward. Then next is operate and [33:20] maintain and this is the operations [33:23] phase of the application. So here you [33:26] will be dealing with monitoring and [33:28] alerting. So you should have an [33:30] observability posture that will ensure [33:33] you can operate and maintain the [33:35] application properly and you will be [33:37] patching the resources, upgrading the [33:39] resources. You will be taking care of [33:41] the DR in case something fails you will [33:43] be restoring it from the backup incident [33:46] response and scaling. All these things [33:48] will happen in the operations phase. And [33:52] the last one is feedback and improve. I [33:56] want you to know that this is a critical [33:59] phase. This is where you collect the [34:02] mistakes that you have made over the [34:04] past. You make a collection of that and [34:07] then you also take feedback from the [34:09] users. Users will be telling you what [34:12] features do they want, what else would [34:14] they like to see in the application, [34:16] what are the problems that you are [34:18] facing. And then you as well as devops [34:21] folks would have proper observability [34:24] posture to know what all are the metrics [34:28] that were not behaving properly. So you [34:31] will take that into account. You will be [34:33] discussing those things with your [34:34] development teams and all these things [34:37] will be jotted down in feedback and [34:41] improvement section. And once you have [34:43] done that then you will again move to [34:46] the requirement gathering phase. You [34:47] will collect whatever you have gathered. [34:49] You will again design and this process [34:52] will go on. No application is perfect. [34:57] This process continues till the [35:00] application is alive. That is what [35:04] modern application development looks [35:07] like. Now before we move ahead, I [35:10] quickly want to discuss waterfall model [35:13] and the agile model. So when it came to [35:16] waterfall it was sequential phases [35:19] feedbacks came late risky big bang [35:22] releases. So what used to happen was a [35:24] work was given to a software development [35:27] team. They would continue working for 4 [35:29] months, 6 months over a year and then [35:31] they will go to their client to show [35:34] them what they have built. And the [35:35] client most times would have said I [35:38] didn't ask for this sort of application. [35:40] You have not done it right. Then the [35:42] entire process had to be repeated. So [35:45] this resulted in risky releases and this [35:50] was not acceptable. So that is where [35:53] agile and devops come into picture. In [35:56] DevOps you make small frequent updates [36:00] with rapid feedback. Hence the [36:02] deployments are safer. You discuss with [36:05] your clients that this is what we are [36:06] going to do and then you deploy it. If [36:09] you take example of big e-commerce [36:12] website like amazon.com, [36:14] they would be pushing hundreds of [36:17] changes per day. [36:19] >> [snorts] [36:20] >> And why they can do that? They can do [36:22] that because first they are using the [36:25] agile and devops model for SDLC. So what [36:30] I'll do is I'll take you to the GitHub [36:32] repository of this course and here I am. [36:35] I'll go to Jenkins basics to production. [36:39] I will go to day one here. And in day [36:43] one I want to discuss waterfall versus [36:46] agile. And here you can see the [36:48] differences. Clearly delivery big bang [36:52] release at the end small frequent [36:54] incremental releases. When we talk about [36:57] agile DevOps feedback comes late [37:00] feedback is continuous here. Flexibility [37:03] is rigid hard to adapt once requirements [37:07] set. This one is adaptive. Risk is [37:10] higher. Issues found late are costly. In [37:14] the agile and devops methodology we say [37:17] fail fast. fail fast in development so [37:20] that we know this code or this [37:23] application is not working as expected. [37:26] You fix those changes, you again deploy, [37:29] you again deploy until you fix the [37:31] problem permanently. So that is how [37:34] agile and devops work. Team [37:36] collaboration siloed. This is crossf [37:39] functional. So dev, QA and operations [37:43] all work together. examples year-long [37:46] banking system projects agile devops [37:49] Netflix Amazon deploying updates daily I [37:53] hope this section is clear to you now [37:56] let's move on to next section that is [37:59] programming languages now firstly I want [38:02] to take a moment to explain you why I [38:05] want you to know these things see you as [38:09] devops and s engineers will be working [38:11] with applications all the time you will [38:14] be deploying them you will be rolling [38:16] back and so on. Hence you must [38:19] understand what programming languages [38:22] are used when we talk about applications [38:25] and how these programming languages are [38:28] segregated. And I will tell you if you [38:31] do not understand this you will not be [38:33] able to understand automation tools that [38:36] are used in the build phase of SDLC and [38:40] the ownership of build phase is with [38:43] DevOps folks. So let's understand [38:46] programming languages. When I talk about [38:49] programming languages, they can be [38:51] categorized into two. Compiled and [38:54] interpreted. Now let's discuss compiled [38:57] languages first. Translated before [39:00] execution to machine code or to byte [39:04] code. So if I talk about compiled [39:06] languages, let's understand what these [39:09] are. Compiled languages requires some [39:12] sort of compilation. Now what that [39:15] compilation will do is it will take the [39:17] source code and it will convert that [39:20] source code into a nonhuman readable [39:24] format. Now that nonhuman readable [39:27] format could be machine code or it could [39:31] be byte code. So if I talk about C, C++ [39:34] and Go, they use machine code. If I talk [39:38] about Java and C they use byte code. So [39:42] when I talk about machine code, they do [39:45] not require any runtime. What is [39:48] runtime? I'll come to it. But when I [39:50] talk about byte code, a runtime is [39:54] required. For example, if you have a [39:56] Java application, then you would need [39:59] JVM for that Java application to work [40:02] properly. Which is why you would have [40:04] seen that if you install anything that [40:06] requires Java underneath, you first must [40:10] install the Java runtime which is JVM. [40:14] Now before I start explaining this, [40:17] let's understand these terms. What are [40:20] these? First is artifact. Like I told [40:22] you before, artifact can be thought of [40:25] as a build output. Now that build output [40:28] can be dot jar. When I talk about Java, [40:32] it can be war. War is also associated [40:35] with Java. It can be.exe or it could [40:38] also be a Go binary. Now what are [40:42] runtimes and interpreters? These are [40:45] execution engines. Example JVM, [40:49] Net CLR, Python, NodeJS. No, Python and [40:53] NodeJS are interpreted languages. But I [40:57] will explain a small thing right here. [41:00] So when I deploy a Java application then [41:03] I require a JVM runtime and if I am [41:08] running a Python application then I will [41:10] need a Python interpreter. Similarly if [41:13] I'm running a NodeJS application I would [41:16] require a NodeJS interpreter. Now we [41:20] will discuss this deeply in a later [41:22] section but for now I just want you to [41:24] have a basic understanding. Then there [41:27] is middleware for extra app services. [41:30] Now there are specific formats that are [41:33] related to Java that are war and ear you [41:38] don't see here but we'll discuss that in [41:40] a little while. Now they require special [41:44] app services. Now these apps could be [41:46] Tomcat, JBoss or Web Logic. Now what [41:51] exactly is middleware? Middleware is a [41:54] layer of software that sits between the [41:56] application code and the runtime and we [42:00] will see this in a little while. Then at [42:02] the very last it is dependencies. Now [42:05] dependency could be any dependency. If [42:08] I'm having an application, my [42:10] application could be dependent on a [42:12] configuration file. It could be [42:14] dependent on a specific version of [42:16] something. Hence, anything on which my [42:18] application depends on can be thought of [42:21] as dependencies. What I want to do is I [42:25] will take some time to specially discuss [42:27] Java. And why I am doing that? Because [42:30] Java and Python are the most popular [42:34] programming languages that you as DevOps [42:37] folks will be dealing with. So if I move [42:40] to the right of the screen then the [42:43] artifact format for Java could be jarw [42:48] and year. Now firstly [42:52] needs runtime all of these require JVM [42:56] runtime but only war and ear require a [43:02] middleware. Mostly you will see jar. So [43:07] you will not be deploying a middleware. [43:10] But in case your application requires [43:13] Tomcat, Jboss or Web Logic then you [43:16] would be generating artifacts in war and [43:20] ear format. JAR is simple Java archive. [43:23] WAR is web application archive. E is [43:27] enterprise archive. And as the name [43:29] implies this is used in enterprisegrade [43:33] environments. And you can read the [43:35] GitHub notes if you want to know more [43:38] about this. But for now I will leave [43:40] this here because this is the highle [43:42] information that you need. The key point [43:44] here is when you are talking about jar [43:48] you don't need a middleware but with war [43:51] and ear you need a windowware but JVM is [43:55] required in all these three formats. now [44:00] requires build stage and in parenthesis [44:04] you see compiler plus tools. Now when I [44:07] talk about compiled languages as the [44:10] name implies compiled languages they [44:13] require a compiler that is what we [44:15] understand. So compiler will convert the [44:18] source code into nonhuman readable code [44:21] which is machine code or byte code and [44:25] machine code is for these languages. [44:26] Bite code is for Java and C. Now I will [44:30] move to the left of my screen. So there [44:34] are different compilers for different [44:36] compiled languages. For Java the [44:39] compiler is called Java C. For C it is [44:42] GCC C lang C++ this. Go this. There are [44:47] build tools at your disposal. Now we [44:51] will see why these build tools are [44:54] needed later but for now I want you to [44:57] understand that build tools do not just [45:01] invoke compilers they handle [45:03] dependencies packaging tests and [45:06] artifact publishing. See if you have [45:10] written your source code in Java then [45:12] there could be multiple dependencies [45:15] that must be handled for your source [45:17] code. If you are on Mac, you have [45:19] homebrew which is a package manager for [45:21] your Mac. Similarly, these build tools [45:25] act as package managers and they resolve [45:29] the dependencies that are there in your [45:32] code. For example, Maven is a popular [45:36] build tool for Java. [clears throat] And [45:38] this build tool will not just invoke the [45:43] compiler, it will handle the [45:44] dependencies, packaging, testing and [45:47] even artifact publishing. And we [45:51] understand that artifact is the result [45:54] of a build output. So Maven and Gradel [45:59] are popular build tools and we [46:01] understand this artifact format could be [46:03] jarw [46:05] and e dot class is not widely used. It [46:10] is a very niche use case. Then there is [46:14] C and you can read about what are the [46:16] compilers and what are the common build [46:19] tools. But here I am telling you that [46:22] when I talk about build tools, build [46:25] tools are not just for compilation. They [46:29] can do a lot more than compiling a [46:33] compiled language. Now let's move to the [46:37] right of the screen to see other points. [46:40] Faster execution CPU direct. Now why [46:43] there is faster execution? This machine [46:46] code that is non-human readable or byte [46:49] code can be easily understood by the [46:53] CPU. But just a caveat here, bite code [46:56] runs on a runtime like JVM. So there is [47:00] some sort of translation that this JVM [47:04] does before sending the inputs to the [47:07] CPU. But at large compiled languages are [47:11] faster than interpreted languages. Now [47:14] portability. If I talk about [47:16] containerized, we know that [47:18] containerization [47:20] in general is best for portability [47:23] because you package everything that the [47:26] application needs in the container image [47:29] and wherever that container image runs, [47:32] your software will run there. But when [47:35] it comes to non-containerized these [47:38] artifacts are specific to OS and CPU [47:41] plus they need correct runtime and if [47:45] there is a mismatch between these then [47:48] the software will not run properly and [47:51] this is understood. So if for example a [47:54] Java application is designed for Mac and [47:57] I try to run it on Windows, it will not [47:59] work. So the it is OS and CPU dependent [48:03] plus needs correct runtime. What if I am [48:07] trying to run an application in a [48:09] machine that does not have JVM? It will [48:13] not work there or maybe it needs a [48:17] specific version of JVM then also it [48:20] will not work. Now the key point is code [48:24] is hidden. So as we understand that [48:28] these tools the build automation tools [48:31] like Maven for Java will convert the [48:35] source code into byte code and if it [48:39] does that then that code cannot be read [48:42] by humans because it is binary only. Now [48:46] examples of compiled languages are Java, [48:49] C, C++, Go, Rust and there are plenty [48:54] more. Now let's discuss interpreted [48:58] languages executed by interpreter at [49:02] runtime. So understand previously when [49:05] we were discussing compiled the source [49:08] code was compiled into a nonhuman [49:12] readable format but here these are not [49:16] compiled prior. What happens is you [49:20] write the code and then there is a [49:23] interpreter that does the execution line [49:28] by line. So there is no explicit build [49:32] step here. Previously there was a build [49:35] step which is why we had automation tool [49:38] come into picture and do the build for [49:40] us. In interpreted there is no build [49:44] step and the execution is line by line [49:48] by the interpreter. Portability, [49:51] containerized workload, of course they [49:53] are portable, but non-containerized [49:55] workloads, for example, if you are [49:57] trying to run a Python application in [49:59] your laptop, then your laptop must first [50:03] have a Python interpreter and all the [50:06] dependencies must also be met. So for [50:08] example, if your application requires [50:11] Flask framework of Python, then that [50:15] must be installed. version mismatch and [50:18] if you already have flask maybe you do [50:21] not have the version that is needed by [50:23] my application. So the version mismatch [50:26] risk is always there. Now this is an [50:30] important point code visible. So if you [50:34] have a Python application when you share [50:37] that application with your user then the [50:41] code is visible to them. Examples of [50:45] interpreted languages are Python, [50:47] NodeJS, Ruby. Now let's move to the next [50:51] section. See, I want you to know that [50:55] you as DevOps engineers will not just be [50:58] deploying resources onto containerized [51:02] workloads. You will also be deploying [51:04] resources or your applications onto [51:08] non-containerized workloads. Now in this [51:11] section I want to give you a flavor of [51:16] how compiled and interpreted languages [51:19] will behave in both of these [51:22] environments. And I am telling this [51:25] specifically from the point of view of [51:28] the build step. Let's scroll down. First [51:32] let's see what happens to the [51:34] containerized world and we are talking [51:36] about compiled languages first and we [51:40] know compiled languages there is a build [51:43] step where you do the compilation and [51:46] what is compilation changing the source [51:48] code into a nonhuman readable format. So [51:53] first you see source code then here you [51:56] have tools like Maven, Gradel, make for [51:59] different programming languages. Maven [52:02] and Gradel are predominantly used for [52:04] Java. So let's say our source code is in [52:08] Java. Here we have Maven. Maven did all [52:12] the things for us. So Maven handled the [52:14] dependencies, packaging, testing and [52:17] then finally artifact publishing. That [52:20] is what you see here. Then the artifact [52:23] was generated and let's say the artifact [52:26] was generated in dot jar format. What is [52:30] the next step? We are talking about [52:32] containerized workload. So when when we [52:34] talk about containerized workloads in [52:37] the next step we'll be creating a docker [52:39] file. Now here is the docker file. What [52:42] I'm saying in this docker file is from [52:45] open jdk. Why I am mentioning open jdk? [52:49] Because understand for this jar to run I [52:54] need the JVM runtime and open JDK is the [52:58] base image that has that JVM runtime [53:02] available which is required by my jar [53:07] file and in this step I am copying the [53:11] app dot jar file and here in step two I [53:16] am building the container image. [53:19] And once the container image is built [53:22] using this docker file, we can finally [53:25] deploy it as container. Now I want you [53:28] to clearly know that when I'm talking [53:31] about containerized workloads, it is a [53:34] twostep build. First I am building the [53:38] code and then I am building an image. So [53:43] here when Maven was used to publish this [53:46] artifact, I copied this artifact in my [53:49] docker file and then an image was [53:52] created which had this artifact. Now [53:56] let's see how non-containerized [53:59] workloads behaves when we talk about [54:02] compiled languages. Here everything [54:05] remains the same. It is just we do not [54:09] have a docker file. We are deploying [54:11] onto a VM. But this VM must have [54:16] everything that this needs. Now what do [54:20] we need? Firstly, we need a runtime. If [54:22] we are deploying a Java application, we [54:25] need a JVM runtime. Then second is [54:29] middleware. Now if we were receiving the [54:32] output in war format then we would also [54:35] require a middleware and that middleware [54:38] could be your Tomcat or Job Boss. Now [54:42] that we have this understanding let's [54:45] see how interpreted languages will [54:49] behave like Python. So if I move here I [54:53] have containerized workloads and here I [54:56] have interpreted languages. See the [54:59] source code is this and I'm directly [55:03] moving to the creation of docker file. [55:06] There is no build step that involves [55:10] compilation or packaging. What I am [55:14] doing is I am saying this docker file [55:17] from Python 3.11. [55:20] Here why I am using the base image of [55:23] Python because I know for my Python [55:27] application to run it requires an [55:30] interpreter. And why is interpreter [55:32] required? So if I move here, we learned [55:36] here that when I talk about interpreted [55:38] languages, they require an interpreter [55:42] at runtime. That is why if you want to [55:46] run any Python-based application in your [55:48] laptop, your laptop must have a Python [55:52] interpreter. So I'm saying from this and [55:55] this image will have the Python [55:57] interpreter. I'm saying copy source. So [56:00] I'm copying the source code into my [56:02] docker file and the resultant container [56:05] image will have this source code and [56:07] here I'm installing the dependencies. [56:11] I'm saying pip install-r [56:15] requirements.ext text and this file will [56:18] essentially have all the packages that I [56:22] use and pip is the package manager for [56:25] Python which will install all the [56:28] dependencies and I'll have my image [56:30] build here and then I can deploy. Now [56:34] let's contrast this with how [56:36] non-containerized [56:38] workloads will behave. So here I have [56:42] the source code. Now the source code [56:44] could be in any interpreted language. [56:46] Here it is let's say py for python. Now [56:51] we deploy it onto the vm and here are [56:54] the must have interpreter must be there. [56:58] So this vm must have python [57:01] dependencies. Dependencies should have [57:03] been installed using pip npm or gem. pip [57:08] is for python. npm is for NodeJS and gem [57:12] is for Ruby. Last one is isolation. Now [57:17] this is an important piece. Okay. Venv [57:20] is virtual environment for Python. NVM [57:24] is node version manager. RBNV is Ruby [57:30] environment. I want you to know that [57:32] there could be multiple applications [57:35] deployed onto your VM. One application [57:37] might require a specific version of [57:39] Python. Another application might [57:41] require a different version of Python. [57:44] In that scenario for Python, you can use [57:46] virtual environment. Now, what this will [57:49] do is this will allow you to have [57:51] different versions of Python for [57:54] different applications and similarly you [57:57] can do this for NodeJS and Ruby as well. [58:02] I hope you have understood this. Now [58:05] before I let you go, I want you to know [58:07] one more thing. You understand that the [58:10] build process is automated and build [58:13] process is automated using automation [58:17] tools like Maven, Gradel or for C you [58:21] have make, CMake and so on. So what if [58:25] these build automation tools were not [58:28] there. So for compilation it is fine. [58:30] You could have done the compilation but [58:32] what about dependencies? What about [58:34] packaging? What about tests? What about [58:37] artifact publishing? This would have [58:39] been a chore for you. So if I move to my [58:42] GitHub repository, what I have done is I [58:46] have one section where I am defining why [58:49] build automation matters and it is there [58:52] because you don't have to have that [58:54] classic problem works on my machine. So [58:59] whatever artifact you generate are [59:02] reproducible, traceable and environment [59:05] agnostic whenever you are using build [59:08] automation. So you want whatever [59:11] artifacts are generated should work the [59:14] same way in dev staging and prod. And [59:18] this has version logged as well. So [59:22] results are predictable and safe to [59:26] promote. Now before I let you go, I want [59:29] you to know one thing. We have discussed [59:33] SDLC and these are the phases of SDLC [59:38] that we discussed. I want you to [59:40] understand that there must be some tool [59:44] which could help us with these all [59:48] things. What is that tool? And the tool [59:51] that will help us with all these things [59:54] are CI/CD tools. And in the next [59:58] lecture, we will explore this in a [60:01] detailed way. So for now, that's a wrap [60:05] for day one of our Jenkins basics to [60:08] production series. Today we laid the [60:11] foundation by breaking down SDLC phases, [60:14] contrasting waterfall with agile, [60:17] exploring compiled versus interpreted [60:20] languages and understanding artifacts, [60:23] runtime dependencies and build [60:25] processes. In the next lecture, we will [60:28] zoom into continuous integration, [60:31] testing, delivery, and deployment and [60:34] then start connecting the dots on why [60:37] Jenkins is such a popular choice for [60:40] automation. If you have any questions, [60:43] feel free to ask in the comment section [60:46] and I will reply. If this helped, do [60:49] consider liking, sharing, and [60:51] subscribing. It will mean a lot to me. [60:55] Thank you very much. Hi, I am Varun [60:57] Jooshi and welcome to day two of our [61:00] Jenkins basics to production course. In [61:04] this lecture, we will first discuss Git [61:07] and how teams collaborate with commits, [61:10] branches and pull requests. Next, the [61:13] continuous practices, continuous [61:16] integration, testing, delivery, [61:19] deployment and monitoring. Then the two [61:22] production styles you will often see git [61:26] flow-based branching and trunkbased [61:29] branching and how promotions flow from [61:33] feature to dev to stage to production. [61:36] Finally we place Jenkins in that path as [61:40] the orchestrator and reinforce the [61:43] principle build once promote by digest. [61:48] This lesson uses simple visuals and [61:52] concise theory that sets upcoming [61:56] Jenkins demos. Please give it your full [62:00] attention. Together with day one, it [62:04] forms the base for the rest of the [62:06] course. By the end, you will understand [62:10] the path from a push to a production [62:13] release and exactly what Jenkins [62:17] automate. Let us begin. So let's start [62:21] with Git branching the essentials for [62:25] CI/CD. What exactly is Git? Git is a [62:30] version control for code. Now I want you [62:34] to imagine that you are a developer and [62:36] you are writing code and also testing it [62:39] in your own laptop. You wrote a code, [62:42] you tested it, then you performed few [62:45] more changes and you tested it. Then you [62:47] performed few more changes but this time [62:50] the testing failed. Now you want to [62:54] revert to the previous stage. How do you [62:57] reach that code? Now if you are [63:00] maintaining four versions of the code, I [63:03] do not want you to manage four files, [63:06] one file for each version. We want some [63:10] way via which I just have to manage one [63:13] file and there is some system that is [63:17] ensuring that the history of important [63:22] junctures in the development phase of my [63:25] application are captured properly. And [63:28] this is where git comes into picture. [63:31] And whatever I'm saying the junctures of [63:34] your application that you want to be [63:36] saved, these are called comets. So [63:39] essentially a comet is a local snapshot [63:43] of changes. For example, if you have 10 [63:46] commits, you essentially have 10 local [63:50] snapshots of those changes and you can [63:54] go back to the snapshot that you want to [63:57] work with. That is how git simplifies [64:01] this. And you see here DVCS. This stands [64:05] for distributed version control system. [64:09] That is what git is. Why are we calling [64:12] it distributed? Firstly because you have [64:16] the entire code base in your laptop and [64:20] then you also have a code base in [64:23] somewhere remote. Vun why do I need the [64:26] code base to be in remote? Because what [64:30] you want is collaboration reviewing and [64:34] CI. We will come to CI later which is [64:37] continuous integration. But if you want [64:40] your fellow developers to collaborate [64:43] with you, then you must have a [64:46] centralized location from where the code [64:50] can be accessed by all your team [64:52] members. That is why we are saying this [64:56] system is distributed. You have your [64:59] changes locally and then you also have a [65:03] remote where your code base is also [65:06] being managed and anyone who wants to [65:08] collaborate or review your code can do [65:11] it from the remote. But Warun, what if I [65:15] am doing some changes and those changes [65:18] are not yet reflected in the remote? [65:20] Then wouldn't the other developers who [65:23] are accessing that code will be [65:26] accessing a previous version? Yes, you [65:29] are right. That is why you have [65:32] something called push. So if you have [65:35] written a code and you want your [65:37] teammates to see what is the latest [65:41] change that you have done in the code [65:44] then you will do a push send local [65:47] commits to the remote and what if a new [65:51] developer has joined your team. So if [65:53] you see in this diagram here, you have [65:56] developer one and developer two is the [65:59] person who has recently joined your team [66:02] and your team is working on application [66:06] one which I have spelled wrong here. [66:09] This is a repository for application one [66:12] and we are calling it remote. We call [66:14] this local. Whatever you are doing in [66:16] your laptop is local and whatever is [66:18] happening in that remote is called [66:21] remote and that remote location what are [66:24] the possibilities for that remote [66:26] location we'll see in a little while. [66:28] But if developer 2 joins and developer [66:31] two wants to see what is happening in [66:34] the codebase of application one, dev two [66:38] can perform a get clone first and then [66:41] if developer two wants to see the recent [66:44] changes whatever dev one is pushing or [66:47] any other developer is pushing then dev2 [66:51] can just perform a get pull and he will [66:55] have the latest changes available. in [66:58] the local copy that is being maintained [67:02] in his laptop. But Warun what if [67:04] developer 2 has some changes to propose? [67:08] The same what dev one is doing. Dev2 [67:11] will perform the changes and push it so [67:14] that all the other developers can see [67:17] the changes that are being sent to the [67:20] remote by dev 2. As simple as that. This [67:25] is what git is doing for you. Now in [67:28] this lecture I won't be covering the [67:30] commands because we'll be doing thorough [67:32] demos as we move ahead with this course. [67:36] This lecture is purely conceptual [67:39] because I want your foundations to be so [67:42] strong that rest of the lectures are a [67:46] cakewalk for you. Now the next is where [67:49] to host your git repos. Understand that [67:52] when you are sending changes to your [67:55] remote, you want that remote to be [67:58] highly available because you cannot [67:59] trust your laptops because your laptop [68:02] is not highly available. If your laptop [68:04] crashes, God forbid, your data is gone. [68:08] Hence, you cannot just rely on your [68:11] laptop when you are talking about a [68:13] critical application. You have a remote [68:16] and that remote must have provisions [68:20] that allows high availability and they [68:23] must have an SLA so that you as [68:27] developers and DevOps engineers know [68:31] what should you expect from the platform [68:35] that is hosting your git repository. Now [68:39] application one repo that you see here [68:42] this is known as a repository when we [68:45] are talking about git. Now there are [68:48] various options you have github you have [68:51] gitlab and azure repos. So you can [68:55] always use the functionalities provided [68:58] by your cloud provider as well. But I [69:00] want you to know that AWS code commit [69:04] and GCP's CSR are legacy for new teams. [69:08] If you are existing users, you may use [69:11] it for some time I believe, but [69:13] eventually you will have to migrate if [69:15] you want to leverage any new features [69:18] that other competitors are providing. So [69:22] you have two options here. cloud hosts [69:25] which are GitHub, GitLab, Bitbucket and [69:28] there are more of course then there are [69:30] provider suites which only Azure is [69:33] there which is now providing Azure repos [69:36] then there is selfhosted if you do not [69:39] want a public service like GitHub GitLab [69:43] you can have it privately you can create [69:46] a GitLab CE or ei Garrett all these [69:50] allow you to use selfhosted [69:54] repository hosting platforms. I want you [69:57] to understand one thing. If you are [69:59] self-hosting, you must ensure that your [70:03] solution is highly available. Now let's [70:08] talk about what are branches in Git. A [70:11] named line of work. If you see here, [70:14] there is a main line and main line is [70:18] the one that has all your code right [70:21] now. Let's assume that. And then if you [70:24] want to introduce a new feature in your [70:26] application, for example, if I'm [70:28] managing a banking application and I [70:30] want to add a new feature which will [70:33] allow users to top up their loan, then [70:36] what I will do is I will create a [70:39] feature branch and that feature branch I [70:43] will name loan. Now I want you to know [70:45] from this stage itself that you can have [70:49] multiple feature branches. If you are [70:52] introducing multiple features, you can [70:54] have multiple feature branches and in [70:56] each feature branch there could be [70:59] multiple developers working. So broaden [71:02] your imagination from this stage itself. [71:05] So that whatever we learn fits in and [71:09] main is equal to the source of truth. So [71:12] if I go to my GitHub repository, I'm [71:15] going to my browser here and you see we [71:18] have this Jenkins basics to production [71:20] GitHub notes where I am capturing all [71:23] the nodes for this lecture and you see I [71:26] have a branch here that is named main. [71:30] Now some call it main, some call it [71:32] master but typically you will see it [71:34] main and often this name persists. You [71:38] do not rename this to something else [71:41] just for consistencies sake and then you [71:45] can create multiple branches if you have [71:48] a requirement for that. When you are [71:51] working with a production grid [71:53] application you will often see there are [71:56] multiple feature branches created based [71:59] on your requirement. And again do not [72:02] limit your imagination. I only have main [72:05] here. There are certain implementations [72:09] where you will see there is a dev [72:11] branch, there's a stage branch, there is [72:14] a release branch, there is a prod branch [72:16] and so on. It is what your application [72:20] architect decides and then based on that [72:25] you work on that project and again you [72:29] could see multiple branches based on [72:33] specific applications. Then what is a [72:37] PR? What is an MR? Firstly, they both [72:41] mean the same thing. There is a lot of [72:43] confusion. PR is pull request. MR is [72:46] merge request. And I prefer the term [72:50] merge request. But GitHub is more [72:52] popular. Hence PR term is more popular. [72:57] But I like MR more because it conveys [73:01] the message within its full form itself [73:04] that what it is doing. What is a merge [73:07] request doing? For example, you were [73:09] working on your feature branch here. You [73:12] created your feature branch from this [73:15] stage. You performed your changes. You [73:17] performed your commits. This was your [73:19] first commit. Then this was your second [73:21] comet. And then you performed your [73:24] respective commits. And then you decided [73:28] I want this feature to get merged with [73:32] the main branch because now my [73:36] application is ready to test whatever [73:39] feature I was working on with the main [73:43] application. So you perform a merge [73:45] because I want you to understand at this [73:47] stage itself that feature branch does [73:50] not hold the full code. Your full code [73:54] right now is in the main branch. You [73:57] cannot perform end to end testing of [74:00] this until and unless you merge it with [74:04] your main branch. And once that merge is [74:07] performed, then you can test the entire [74:10] application along with the feature that [74:14] you have recently introduced. Now here [74:17] is a catch. Should I be allowed to [74:21] perform a merge in the main without any [74:25] tests? No, you should not be allowed. [74:28] Hence, you have the concept of protected [74:32] branches and status checks. So, what [74:35] happens is whenever I say a branch is [74:38] protected, there are certain rules in [74:41] that branch. And a rule that you would [74:44] often see is a push is not allowed to [74:49] these sort of branches. I cannot [74:52] directly perform a get push in the main [74:56] branch because main branch has a rule [75:01] that prohibits anyone to perform a push [75:05] or a force push. only PRs are allowed or [75:10] only MRS are allowed and even before the [75:16] merge is allowed, there are certain [75:19] checks that are run to ensure whatever I [75:23] am doing, whatever I am introducing in [75:26] the main branch will not break the [75:29] existing functionality of the main [75:31] branch. If my changes were to disrupt [75:36] the existing application, then what good [75:39] are my changes? And this is the main [75:41] branch that caters my production. I do [75:45] not want anyone to walk in and merge to [75:48] my main until and unless I perform [75:52] certain checks. That's why you see here [75:55] PRs only, no direct pushes, requires [75:59] reviews. Now in case you are deploying [76:02] to your production then you might want [76:05] that someone manually reviews it and I [76:10] use the word might and you will learn as [76:13] you progress with this course why I am [76:16] saying you might need a manual review [76:19] here. Last is CI must pass and be up to [76:24] date. Now imagine a scenario with me. So [76:27] here is my feature loan. Let's say [76:30] developer one was working here. [76:32] Developer one opened a PR and then there [76:35] is a developer two who was working in [76:37] let's say some another feature and [76:39] developer two also raised a PR because [76:42] she was also done with her work. But [76:45] what happens is let's say her PR got [76:49] executed first because she was making a [76:52] small change and then this branch had a [76:56] new commit and that commit is here. [76:59] Let's say I'll copy this. I'll paste [77:02] this. There is a new commit. So from my [77:05] PR's perspective I'm not dealing with [77:08] the latest commit of the main. Hence [77:12] this is not a valid design which is why [77:15] I'm saying here CI must pass and be up [77:19] to date branch must be up to date. So [77:22] whatever I am doing should be done with [77:25] the latest commit in main and latest [77:28] commit that is successful because this [77:31] merge will only happen when all the CI [77:36] checks will be performed. And if you are [77:38] hazy about these things, do not worry. I [77:41] have a diagram later in this section [77:44] which will help you solidify these [77:47] concepts. Just create a mental picture [77:49] right now. Whatever we are discussing, [77:52] we will be discussing it [77:53] diagrammatically as I progress with this [77:56] lecture. Now last is why are we studying [78:00] it? If the topic of the lecture is CIC [78:05] CD, firstly whenever there is a push or [78:09] a PR, there is a CI build that starts [78:13] and we will dissect this why this [78:16] matters as we look into the nitties and [78:20] gritties of the CI process. CI is [78:23] continuous integration status gates the [78:28] merge and what is this status? This [78:30] status is status checks. I will only [78:33] allow this PR to be approved when my PR [78:39] status is green. And what on what all [78:44] things does the PR status depend? It [78:47] could depend on whether the CI run was [78:50] successful or if there is a specific [78:52] file in a specific location. There could [78:55] be hundreds of permutations and [78:58] combinations based on which I will [79:01] approve this PR. Lastly, CD promotes the [79:06] build artifacts to dev stage broad as [79:10] you might understand. So, there are two [79:13] terms CI and CD and in the subsequent [79:17] section we will be zooming into these [79:21] terms. I hope you have a good [79:23] understanding of how git fits into the [79:27] larger picture. If not, do not worry. We [79:31] will learn more as we progress with this [79:33] lecture. So, I'll scroll down and this [79:37] is what I want to discuss now. CI and [79:40] CD, continuous integration and CD is [79:44] continuous delivery or deployment. But I [79:48] also want to discuss two more terms that [79:51] are continuous testing and continuous [79:54] monitoring. These two also work from [79:58] start to end. There is also continuous [80:01] security because we want everything that [80:04] we are doing is secure. So whenever we [80:07] are doing certain steps in CT we also [80:11] integrate certain security level checks [80:14] so that the application that we are [80:17] deploying using the pipeline is secure [80:21] to run in production. So with that [80:23] information let's start this discussion [80:27] of continuous practices in DevOps 10,000 [80:31] ft overview. I want you to understand [80:34] these things at a very high level. so [80:36] that rest of the lecture is easy to [80:39] understand. First is continuous [80:42] integration. The first is build and test [80:46] every push PR automatically. Now, right [80:51] away I want you to think of two things. [80:54] When I'm saying push and PR, that [80:57] implies I'm talking about a git [81:00] repository, right? And when I'm talking [81:03] about build and test, I am talking about [81:07] the build stage. And if you remember [81:10] while we were discussing the build stage [81:12] in our previous lecture, we know what [81:15] happens in the build stage. If I talk [81:17] about interpreted languages and you want [81:19] to deploy your code onto let's say a [81:22] containerized environment, then your [81:24] build will be a two-step process. First, [81:27] you will be compiling the language. [81:28] Let's say if it is Java, you will be [81:30] let's say creating an artifact in dot [81:33] jar format and then you will be creating [81:36] a container image out of that. So the [81:38] build is essentially a two-step process. [81:40] In interpreted it is just a one-step [81:43] process. You take the code and you [81:45] create your container image and then [81:49] there are plenty of other things that [81:50] are performed that are related to [81:52] security. So when I talk about [81:55] continuous integration, I want you to [81:58] understand that whatever happens with [82:01] your git repository and whatever happens [82:05] in the building stage is called [82:07] continuous integration. The question [82:10] then becomes why we are calling it [82:13] continuous integration because every [82:16] change that you are performing in your [82:19] git repository is automatically built, [82:22] tested and integrated the moment it is [82:27] shared. So understand these points. [82:30] Second is fast checks enforce merge [82:34] protection on the branch. Now it is not [82:36] that everything that is pushed will be [82:38] built. There will be certain parameters, [82:42] certain rules that will be in place. [82:45] Only if those rules are met only then [82:49] the build will start. So we will see [82:52] what those rules are later in this [82:54] section. And as I just told you [82:58] compiling, unit testing, quick security, [83:00] dependency scans, all of these things [83:03] will happen in the continuous [83:05] integration stage itself. What are the [83:08] tools that are used in this stage? [83:11] Jenkins, GitHub actions, GitLab CI. Do [83:15] not worry. Once we finish this section, [83:17] I'll tell you how the story is staged [83:21] from CI to CM. Then we have continuous [83:26] testing. Now I want you to clearly know [83:29] that testing happens across the pipeline [83:32] and we know these are just logical [83:36] constructs. These could be happening at [83:38] all stages. If I talk about monitoring, [83:41] monitoring happens from start to finish. [83:45] Can you say that monitoring only happens [83:47] in this part of the pipeline? No. And [83:50] same applies for testing and security. [83:53] And we saw that in day one as well. [83:56] Whenever we are talking about a [83:57] continuous design, testing, security, [84:00] automation, monitoring, all these [84:03] things, they are happening on every [84:06] stage of the pipeline and hence at every [84:10] phase of SDLC. Of course, you cannot [84:13] perform monitoring when it comes to [84:15] requirement gathering. You cannot [84:17] perform security at this stage. But [84:20] again all these things all these logical [84:23] constructs are kept in mind whenever we [84:26] are discussing any phase of the SDLC. [84:31] Then let's come back here and the next [84:34] point is ads integration API UI [84:38] performance security suites. Now at this [84:40] stage itself I want you to know that [84:43] testing can be thought of as two-phased [84:46] process. Okay. First phase is pre-eploy [84:50] when you have your artifact with you. [84:52] Your artifact could be the container [84:54] image or your jar file or any other [84:58] artifact that results from your build. [85:02] Firstly, you want means to test that [85:06] artifact. Then when your application [85:09] runs using that artifact post you have [85:11] done the deployment, you want that also [85:14] be tested. So first you are checking the [85:17] static things and then you are checking [85:20] the things that are running. So when I [85:22] talk about testing you have both of [85:24] these you have s when I talk about [85:28] anything pre-eploy and post deploy you [85:32] have dashed and other related tests for [85:36] example you cannot perform integration [85:39] API UI performance tests until and [85:43] unless the application is running but [85:46] security suites the vulnerabilities you [85:49] can start doing these sort of tests from [85:52] the very beginning itself from the point [85:55] where developer is writing the code [85:57] because you know like we discussed in [86:00] our day one's lecture that there are [86:02] plugins that are in your idees that are [86:06] ensuring the code that you are writing [86:09] is not vulnerable or it is not using any [86:13] dependencies that possess [86:15] vulnerabilities. [86:17] So runs premerge, pre-eploy and post [86:21] deploy. Last is there are different [86:23] tools that could be used to do these [86:26] things. JUnit, TestNG, Cyprus, JMeter [86:30] and so on. There are plenty more. You [86:32] will learn about all these as we [86:34] progress with this course. Then I talk [86:36] about continuous delivery. And before I [86:40] start this discussion, I want to [86:42] differentiate between continuous [86:44] deployment and continuous delivery. Now [86:47] continuous delivery is anything that [86:51] requires manual approval before [86:53] deploying. So when you think about [86:55] continuous delivery, think of a person [86:58] that is there to approve human approvals [87:02] and gate. So human intervention is there [87:05] before deployment. Whenever we are [87:08] talking about continuous delivery then [87:11] what is continuous deployment? When [87:13] there is no human intervention that is [87:17] no human is approving anything before [87:21] deployment. Now you must be thinking but [87:24] Vun that's a bad thing. No that's not a [87:27] bad thing. If there are no humans [87:29] involved, then you must ensure that your [87:34] CI, your testing and your deployments in [87:38] staging and development are so well [87:41] rehearsed that you do not require anyone [87:45] to approve before your code is deployed [87:49] onto production. So again when I talk [87:52] about continuous deployment auto deploys [87:56] to prod again prod is the key word. If [88:00] you are auto deploying to staging or you [88:02] are auto deploying to development it is [88:05] not continuous deployment continuous [88:08] deployment implies when you are [88:10] deploying the code without manual [88:13] intervention in production environment. [88:18] I hope that clarifies any doubts that [88:21] you had between delivery and deployment. [88:25] So let's read the pointers now. Keep [88:28] main always deployable with versioned [88:31] artifact. That makes sense. And I told [88:33] you this briefly in the previous stage. [88:36] Your main should only hold the code that [88:40] can be easily deployed. Then promote the [88:43] same artifact across environments. [88:46] Important point. So for example in [88:49] development stage if I have a build that [88:53] resulted in the container image then you [88:56] should be using the same container image [88:59] in the staging and in the prod [89:01] environment. So essentially you did the [89:04] build of the container image only once [89:08] and then the same build was promoted [89:11] onto staging then promoted onto [89:14] production. So I want you to keep these [89:17] definitions in mind and again if [89:20] anything is not making sense right now [89:22] do not worry we have a diagrammatic [89:25] section where we'll understand all these [89:27] things step by step. Next is human [89:31] approval gates before production that we [89:34] have already discussed. There are tools [89:36] that will help you with this. Jenkins [89:38] pipelines, GitHub actions, GitLab, CICD [89:42] and then if I talk about continuous [89:45] deployment, there is auto deploy to prod [89:47] when all checks pass. This is we have [89:50] discussed progressive delivery, you have [89:53] to choose what is your deployment [89:56] strategy, whether you want to use [89:58] Canary, whether it is blue green feature [90:01] flags and so on. And as we progress, [90:04] I'll discuss all these three briefly. [90:07] Then there is rapid roll back on failed [90:11] health signals. You would want to roll [90:13] back if something goes bad and there [90:16] must be processes in place that will [90:19] help you do the same. There are [90:22] different tools that can be used with [90:25] continuous deployment and those tools [90:28] are Argo CD, Flux, Spinacer and there [90:31] could be more. Argo CD is a popular tool [90:34] which I will also cover in my channel. [90:37] Maybe the next course will be on Argo CD [90:40] itself. Then lastly I want to discuss [90:43] continuous monitoring. Now we know [90:46] observability has three pillars. There [90:49] is metrics, logs and traces. And [90:53] observability can be thought of from two [90:55] perspectives. There is an infrastructure [90:58] perspective where your Kubernetes [91:00] clusters, your servers, all these things [91:02] come into picture and then there is [91:04] application perspective. Now application [91:07] perspective will cover anything that is [91:10] application specific that is application [91:14] performance monitoring is there from [91:17] applications perspective. So you would [91:20] want to have all these three things so [91:24] that you can properly judge the health [91:28] of your application and your [91:30] infrastructure. Now we discussed this in [91:33] detail in our Kubernetes observability [91:36] section. I will link that lecture as [91:38] well. If you want to know deeper about [91:40] observability then you can definitely [91:42] watch that lecture. So you should be [91:44] able to visualize health of your [91:47] application and your infrastructure. [91:50] Detect regressions trigger incident [91:53] response. So this is part of your ITSM [91:56] process. You should have these things in [91:58] place to ensure that your application [92:01] and infrastructure is in good health. [92:05] Tools there is Prometheus, Graphana, [92:07] Elastic Stack, Data Dog, New Relic. [92:10] There are many tools and then there are [92:13] offerings from cloud providers as well. [92:16] Azure has Azure monitor logs. AWS has [92:20] the cloudatch suite, X-ray and so on. [92:24] All these things can be used for APM and [92:28] also for infrastructure observability. [92:31] Now if I have to stitch this entire [92:34] story, CI proves a change is safe to [92:39] merge. We know that and we will dissect [92:42] this more as we progress. TT widens the [92:46] safety net because that is where the [92:48] testing is being performed. TD keeps [92:51] releases always ready. CDP that is [92:55] continuous deployment ships [92:58] automatically continuous monitoring [93:00] watches everything and informs the next [93:03] change. Now what why and what of [93:08] Jenkins? What is Jenkins doing here? [93:10] Jenkins is your orchestrator. It is an [93:14] automation server which will do whatever [93:18] you ask Jenkins to do. If you ask [93:21] Jenkins to pick code from location A, [93:24] paste it to location B, it will do that [93:26] for you as well. But we leverage Jenkins [93:31] with CICD and this entire lecture will [93:35] tell you how. So as we progress we will [93:38] see where Jenkins fit and why in my [93:43] opinion this tool must be learned by any [93:47] aspiring CI/CD engineer. So let's come [93:52] below before I start discussing the [93:55] pipeline I want to discuss branching [93:58] strategies and I want you to know that [94:00] we are not talking about environments [94:03] here. environments as in we are not [94:05] talking about your development [94:07] environment or your prod environment [94:09] here we are talking about branches. So [94:12] when I say gitflow based CI/CD it works [94:16] on the premise of environment branches. [94:19] Now environment branches as in there [94:21] could be multiple feature branches and [94:24] there will be a dev branch, a stage [94:26] branch and a prod branch. And as you [94:29] understand the code will be promoted. [94:32] First the developer will do the changes [94:34] in the feature branch. Then a PR will be [94:37] created to the dev branch and from dev [94:39] branch the code will be promoted to the [94:41] stage branch. And from stage branch the [94:44] code will be promoted to the prod [94:46] branch. And similarly codes from these [94:49] respective branches will be deployed to [94:52] the respective environments. For [94:53] example, code from the prod branch will [94:57] be deployed onto the prod environment. [94:59] Code from the dev branch will be [95:01] deployed onto the dev environment. So [95:04] keep this in mind. The second one is [95:07] trunkbased. Now trunk based is you have [95:10] multiple feature branches but there is [95:12] only one main branch and that main [95:15] branch is the single source of truth. So [95:18] what happens is there are multiple [95:20] checks which allows you to deploy code [95:24] from that one single main branch to [95:27] different environment. So you can use [95:29] that single branch to deploy the code [95:32] onto the dev environment to the stage [95:34] environment and to the prod environment. [95:36] But Wun how does it help? Isn't there a [95:41] rule of thumb that I can use? The rule [95:44] of thumb is if your changes are very [95:47] frequent. So if I talk about a busy [95:50] application that sees 100 changes per [95:53] day, 50 changes per day, then they would [95:56] probably be using this approach because [95:59] this is used when you want to fasttrack [96:02] everything. And Git flow is typically [96:05] used h when you have a stable [96:08] application. Maybe you are performing [96:10] changes once a week or twice a week and [96:12] so on and again these things are not [96:15] written in stone. You could be starting [96:18] with this strategy even when you are [96:21] performing change once a week or once a [96:24] month. It is dependent on application [96:27] and these are not the only two [96:31] strategies branching strategies that you [96:33] would be using. you would see others as [96:35] well but 90 95% of the projects would be [96:39] using these two branching strategies [96:41] which is why I want to discuss these and [96:44] again once you understand these two rest [96:47] of the branching strategies will become [96:50] a cakewalk for you so let's quickly read [96:53] these points multiple longived [96:56] environment branches we have just [96:58] learned that dev stage pro lines and [97:00] then you have feature branches CI [97:02] triggers PR are between environment [97:04] branches. I just told you that in the [97:06] introduction itself and let's contrast [97:08] whatever we have read with trunkbased [97:12] single trunk main as is the source of [97:14] truth short-lived feature branches only [97:17] and then you have the main branch where [97:20] a PR is created CI triggers PR into main [97:24] merge result that's fine build only [97:27] happens once this is interesting okay so [97:31] when you are using multiple branches [97:34] then also So the artifact that is [97:36] generated artifact as in the container [97:38] image or the jar file or both they are [97:41] only generated once in both of these [97:44] scenarios and I do not want to explain [97:47] this to you right now we'll explain this [97:49] in the diagram section governance strong [97:53] reviews approvals and traceability now [97:55] since we are using different branches [97:58] for different purposes for different [97:59] environments it is self-explanatory that [98:03] this approach approach has stringent [98:06] controls and is good when it comes to [98:10] traceability. But having said that, do [98:12] not think of trunkbased as any less. The [98:16] checks are lean. But again, it depends [98:19] on what you configure it to have. So for [98:22] example, maybe when you are deploying [98:25] changes to dev, you do not have those [98:27] many checks. But when it comes to prod, [98:29] you have a checklist of 150 checks that [98:31] must be performed. So that makes [98:33] trunkbased really secure. So it depends [98:37] on how you have configured your CI [98:40] server to interact with these branches [98:43] or how you have configured your CI [98:46] server to perform the checks. But all [98:48] tools have you integrated with your [98:51] automation server which will ensure how [98:56] and when the changes are deployed onto [98:59] your production staging and dev [99:02] workload. So the control is in your [99:04] hands. How you configure will determine [99:07] what level of governance each of these [99:10] approach will have. If you do not have [99:12] any governance in gitflow based then [99:14] again this will create chaos. It's not [99:18] that if you use this there will be no [99:20] chaos because unmanaged pipelines are a [99:24] nightmare for a devops engineer. The [99:27] next point is tradeoff more merges [99:30] branch drift risk tradeoff is needs [99:33] small PRs solid tests and we'll see this [99:36] in the diagram. use cases, regulated [99:38] apps, release trains, use cases, [99:41] microservices, SAS, rapid delivery. Now, [99:44] here is the mental note. This is [99:47] important. The first line is important. [99:49] In both styles, you build once and [99:53] promote the same artifact. So, the [99:55] artifact is not generated three times [99:58] for different environments. It is only [100:00] generated once. Then there is gitflow [100:04] uses environment branches. Promotions [100:07] are PRs between branches. Dev stage prod [100:10] that deploys that artifact. Trunkbased [100:14] keeps all the code in the main branch. [100:17] Promotions update environment [100:19] configurations. For example, there will [100:21] be a different configuration for dev and [100:24] there will be different configuration [100:25] for stage and pro so on to deploy that [100:28] artifact. No environment branch merges. [100:33] So there is only one branch main that is [100:35] the source of truth. Then there are [100:37] multiple feature branches. Now let's [100:40] discuss gitflow based CI/CD environment [100:43] branches. So we will first be discussing [100:45] git flow where we have three branches [100:48] like I talked dev stage and prod and we [100:52] have multiple feature branches. Here [100:54] first I'm going to show you the [100:56] transition from feature to dev and then [100:59] in the next diagram we'll see how the [101:02] artifacts or how the code gets promoted [101:05] from dev to stage. Firstly flow will be [101:09] feature first then to dev then to stage [101:12] then to prod. This is I'm calling a [101:14] beginner's flow because I have not added [101:16] all the bells and whistles here. There [101:19] are many more functionality that could [101:21] be leveraged but I promise you if you [101:25] understand this flow whatever flows that [101:28] we are discussing in today's lecture any [101:30] CI/CD pipeline and from after this [101:33] lecture itself you will be able to make [101:36] sense of those because these concepts [101:39] are the foundations. So let's see what [101:43] we have here. We have a developer here. [101:47] developer has written the code. [101:48] Developer commits. Now commit happens in [101:51] the local branch and then when you do [101:55] the push it is only then the code goes [101:58] to the remote. So developer performs a [102:01] push to the feature topup loan. And [102:04] again like I discussed before I am [102:07] calling this top up loan because let's [102:10] say this developer is working on a [102:13] feature that will allow users to top up [102:16] on their loan. For example, if they have [102:18] a two lakh loan right now they can [102:20] increase that to 2.5 lakhs. Second step [102:25] is opens PR to dev. Again I told you [102:28] that this developer cannot directly push [102:32] to dev because dev branch is a protected [102:35] branch and you see a lock symbol here. [102:38] Again I want you to clearly know that [102:42] all the feature branches that gets [102:44] created these are the branches where [102:46] your developers would work. So they will [102:48] be able to push directly onto only these [102:52] branches and rest of the branches they [102:55] will not be able to directly push [102:57] because they are protected branches. Now [102:59] as soon as a PR is open to dev a CI [103:04] runs. Now understand we are not saying [103:09] that the PR gets gets approved or [103:11] whatever was here is merged onto the [103:15] dev. We are saying a CI runs and this is [103:20] your CI pipeline. And here are the rules [103:25] that decide when can this change be [103:30] merged with dev. Firstly, this is a [103:32] protected branch. It says that PRs only [103:35] no direct commits. This part we [103:37] understand. And then there's a second [103:39] point that says require status checks to [103:42] pass. And here our status check is only [103:46] one that is CI must be green. So what I [103:49] am saying is when I run this CI pipeline [103:53] only when this CI pipeline gets [103:56] successfully executed and I have a green [103:59] here only then merge PR into dev. If [104:04] this fails then tell the developer that [104:07] fix the errors that you see in the logs [104:10] and then repush and the CI will rerun. [104:15] So we are clearly saying do not approve [104:19] the PR until and unless the CI is green [104:22] and if the CI is green then merge the [104:25] changes onto dev. So let's see what is [104:29] happening now. Here what happens is CI [104:33] tests dev plus feature dev is unchanged. [104:37] So what this will do is it will take the [104:40] changes that the feature branch has and [104:43] it will take the latest commit from the [104:46] dev and run this CI pipeline. If I want [104:50] to know how this feature will impact my [104:53] application or how this feature will [104:55] integrate with my application or [104:58] whatever I have coded will it at first [105:00] integrate with the app code or not I [105:03] need to test this with the latest commit [105:07] in the dev because that is the main [105:10] application. So when I do that what this [105:13] CI will do is it will only perform the [105:17] changes on the feature plus it will take [105:22] the the code that is in the dev branch [105:25] and dev remains unchanged. Now we know [105:28] what CI pipeline will do. It will check [105:31] out first from the GitHub repository. It [105:34] will check out the code and then it will [105:37] build and test it. I want you to know [105:40] that Jenkins is orchestrating all of [105:44] these. You will be configuring how to [105:47] reach your GitHub repository in Jenkins. [105:50] You will be configuring what tests to [105:52] perform, what builds to perform in your [105:55] Jenkins. You will be defining where to [105:58] deploy in your Jenkins. So Jenkins is [106:01] the orchestrator that will make [106:04] everything happen for you. So in the [106:06] third step we perform this then the [106:09] status of the PR it will either be green [106:13] or it will be red. If it is red let's [106:15] say if the build fails because of XYZ [106:18] error then the merge will be blocked and [106:21] the developer will have to fix the code [106:24] repush and rerun the CI. If it succeeds, [106:29] we will do the merge and we are also [106:33] deploying it onto our compute. Now I [106:37] want you to know this compute could be [106:40] your Kubernetes cluster. This compute [106:43] could be your virtual machine or this [106:46] compute could be serverless. I am saying [106:49] all these three here because at this [106:52] stage of your learning I do not want to [106:55] bombard you. But in next section in the [106:58] next diagram we will make this specific [107:01] to Kubernetes because it will involve [107:05] more nuts and bolts again. So let's read [107:10] what is happening. CI runs on the PR [107:13] tests dev plus feature. Green enables [107:16] the merge. Red blocks it until you push [107:20] a fix. So until and unless this is green [107:23] the PR will not be merged into dev. [107:27] After merge you may autodeploy to the [107:31] dev environment. Now I want to introduce [107:34] something that will help you in [107:37] designing and thinking in a better way. [107:39] When I talk about dev environment you [107:42] can use a blue color there. If I talk [107:45] about stage, you can use yellow there [107:47] because staging environment denotes [107:50] pre-pro and it is like handle with care [107:53] and then prod is typically denoted by [107:57] green color which implies success, [108:00] stable and live. So if there are any [108:02] errors or something that is denoted by [108:04] red like you see here. So keep these [108:07] colors in mind. So if you are creating [108:10] any diagrams use blue for dev, amber or [108:13] yellow for stage and green for broad. [108:17] Now that we have understood this step, [108:21] let's see what happens when we try to go [108:23] from dev to stage with the help of next [108:27] diagram. Okay. Now I am calling this a [108:31] promotion PR. Here we are moving from [108:35] dev to stage deploy before merge flow [108:40] and I will explain what this means in a [108:43] little while and like I told you in the [108:45] previous section that we'll now be using [108:47] a Kubernetes cluster hence you only see [108:50] a Kubernetes cluster here and now let's [108:53] see what is happening in this diagram. [108:56] So first is what happened in the dev [108:59] build. So I oversimplified this by [109:02] saying that you can deploy into any [109:03] environment and that is how you would be [109:07] handling projects. There could be [109:09] applications which will be deployed onto [109:10] virtual machines. There will be [109:12] applications that will be deployed onto [109:14] serverless and so on to Kubernetes as [109:17] well. But now we will focus on a [109:20] Kubernetes cluster. If this were done [109:23] for a for an application that is [109:26] supposed to be containerized then let's [109:29] see what all steps would be performed in [109:31] the dev state. So here I'm saying what [109:34] happened in dev build. So the first is [109:38] build once and push. So what would have [109:42] happened in the build phase is once the [109:45] application artifact was received let's [109:47] say it was a Java application. And if [109:49] you are not understanding these terms, [109:51] go and watch day one. Okay. So you have [109:53] dot jar and based on dot jar, you [109:57] created a container image and you push [110:00] that onto an image registry. Now that [110:03] image registry could be GitHub container [110:06] registry which I am using or it could be [110:09] your cloud native solution. So what we [110:13] have done is understand that we are [110:16] calling this repository banking loans [110:19] and then we have something like this [110:22] here and what are these now typically [110:26] when you see the tagging convention here [110:29] this is the commit ID so commit ID is [110:33] whatever you would have received here [110:34] that commit ID along with the version of [110:38] the application so let's say this was [110:41] version 1.4.3 of the application. Hence, [110:45] I have written both of these the commit [110:48] ID and the version of the application. [110:53] Another point is capture digest [110:55] immutable. Now I want you to clearly [110:58] know that if you have not enabled tag [111:00] immutability in your image registry then [111:04] anyone can push another image with the [111:07] same tag and it will overwrite your [111:10] existing image and that is not something [111:13] you want. And even if you have tag [111:16] immutability, it is still recommended [111:20] that you use image digests when you want [111:24] to pull a certain image. Warun, what do [111:28] you mean by that? If you see here, I [111:32] just want you to concentrate on this [111:33] point right now. I'm saying the image is [111:36] this at the rate sha and there is an [111:40] image digest here. So in production [111:43] implementations, in enterprise [111:45] implementations, you will always be [111:48] using digests when it comes to container [111:52] images. You will not be using tags in [111:55] enterprise production environments. Now [111:58] why is that? The reason number one is [112:01] these image digests or simply called [112:04] digests are immutable. You cannot change [112:08] them. And I want to demonstrate this to [112:11] you. I'll go to my iterm. And if I press [112:16] ls, you see I have a docker file here in [112:19] day2. And if I cat on this and it's a [112:21] simple docker file. We are saying just [112:22] from this and echo hello from image. Now [112:26] let's do one thing. Let's make an image [112:29] out of it. So I will write docker build- [112:34] t. I will call this cloud with vos. so [112:38] that I can quickly push it forward slash [112:41] let's call it loans and here let's say [112:44] the commit ID was 777 [112:48] yt [112:50] let's say this is the commit ID let's [112:52] add two more digits here so this is the [112:54] commit ID now I'll do one more thing [112:56] here I'll add hyphen t and then I will [113:00] write here version 144.3 [113:05] let's say and then I will again add [113:08] hyphen t. I will paste that again and [113:11] this time I will write v 1.443 [113:17] and then I will add a dot because I want [113:19] to tell that my docker file is present [113:22] in the current working directory and [113:24] this is my build context. I want you to [113:26] clearly know that I am creating a single [113:29] image here and I am adding three tags to [113:33] that image. Tag number one is of my [113:36] commit hash. Tag number two is of my [113:41] commit hash along with the version of [113:42] the application. And then the third tag [113:46] should just be of the version of the [113:50] application. Let me just quickly correct [113:52] this. So I will remove this. And the [113:54] third tag is of just the version of the [113:57] application. But Vun, why are you [114:00] creating three tags? So let's understand [114:03] this. When I talk about version tags, [114:05] these are used by a product QA and [114:08] support so that they can map incidents [114:11] to a release quickly. So these are great [114:14] for change logs and dashboards. If I [114:17] talk about let's say commit tag which [114:20] you see here, this one is a commit tag [114:22] because here is the commit ID. Then [114:25] these are used by developers and sRes so [114:29] that they can map an image to the exact [114:32] commit. So if you see this you exactly [114:36] know which commit was done to create [114:40] this version of application because you [114:44] would see what changes were made to the [114:47] code and the entire codebase when you [114:49] see that comet. Now what is this [114:52] combined tag used for? Now combined tag [114:56] is best for both in one glance. You get [114:59] to know the version as well and you get [115:01] to know the commit ID as well. But in my [115:04] opinion creating this tag is of [115:07] paramount importance because you see [115:09] both the things at one place version at [115:12] one place and also the commit ID so that [115:14] you can directly check what was the code [115:17] like when you were in this version of [115:20] application because you have the commit [115:22] ID with you. Now let's get to the crux [115:25] of it and build this image. I'll press [115:27] enter and this has started building the [115:31] image and it will tag this as well. So [115:33] if I run docker images, I see that there [115:37] are three tags associated with my image. [115:40] The point I want you to concentrate on [115:43] is the image ID. So understand the [115:48] contents of all these tags is the same. [115:52] Hence the image ID that you see is also [115:56] the same. Now image ID is not same as [115:59] digest. You will only get the image [116:02] digest when you push this to your image [116:06] registry. So my image registry here is [116:08] docker hub. So let me push this to [116:10] docker hub so that I can show you the [116:14] image digest as well. So I'll write [116:16] docker push. I will write hyphen a. I'll [116:20] tell you what hyphen a will do. And here [116:22] I will write cloud with varosh and it is [116:26] loans. Now what this will do is this [116:30] will push all the tags to dockerhub. [116:33] I'll press enter and this has started [116:36] doing its job and along with this this [116:40] command will also show me what is the [116:44] image digest. So let's wait for that. [116:46] And you see all these digests are same. [116:50] And if you want to look at it docker [116:53] images um I'm forgetting it is hold on [116:58] digest docker image hyphen h I want to [117:03] grab for digest sorry docker images [117:08] h yeah it is digests hold on docker [117:13] images digests and here I can name my [117:16] image cloud with varosh loans I want to [117:19] check it for this. So if you see the [117:22] digest is exactly the same. So even if [117:26] someone changes these tags, if I do not [117:29] have let's say tag immutability enabled [117:32] then this digest will always remain the [117:35] same and this digest is same because all [117:39] these tags they are holding the same [117:42] content. Even if there is a slight [117:44] change in the image this digest will [117:48] change. So understand this point in [117:50] production scenarios you will always be [117:53] using digest to pull the images. So [117:57] let's go back to our browser and I want [118:00] to show this to you in the dockerhub [118:03] console as well. So I'll refresh this. [118:06] I'm in the repositories. I'm in my cloud [118:08] with VJOS account and you see the loans [118:11] is here. I'll click on this and if I go [118:14] to tags you see I have three tags. [118:18] version tag. There is the commit tag and [118:21] then I have a combine tag here. But all [118:25] of these they have the same digest. And [118:28] if I click on this I will get to see the [118:31] entire digest. And this is the format of [118:34] the digest. If I write this and then at [118:37] the rate this digest then it will pull [118:41] this image itself. So I hope this point [118:44] is understood. Now capture the digest [118:48] publish manifest.json [118:50] Vun why are we doing this? So in the [118:53] build stage you are building an image. I [118:57] want you to understand and this happened [118:59] in the dev build. Okay in the dev build [119:02] now we are just promoting dev to stage. [119:04] Now in the dev build understand that we [119:08] built an image and our registry could be [119:12] holding multiple images. How do we tell [119:15] the deployment part of our CI/CD that [119:19] you have to use this image for [119:21] deployment? There must be a file which [119:24] will have all the information that my [119:27] Jenkins can use to deploy my application [119:31] and that information can be provided [119:34] using a file. And that is why in the dev [119:38] stage itself when the build happened and [119:41] my our Jenkins which is our CI/CD pushed [119:45] the container image to our image [119:47] registry which is GHCR in our example it [119:51] created this file for us. Now this file [119:54] can be created anywhere. It could be in [119:57] your S3 bucket. It could be in a GitHub [120:00] repository or it could be part of the PR [120:05] request. But I want you to know that [120:07] your Jenkins server should have access [120:10] to that file because once Jenkins server [120:13] have access to this file, Jenkins would [120:16] know what to deploy because like I said [120:20] there will be plethora of images in your [120:22] image registry which could have multiple [120:25] digests but we want to aim for the [120:28] digest that we are looking for. And here [120:32] you see I have added all this [120:34] information the time tags the all the [120:36] tags commit sha version digest and the [120:41] repository. So I hope this makes sense [120:44] to you. All this happened in the dev [120:48] stage itself in this stage itself but I [120:52] did not want to complicate things that [120:54] is why I didn't show these things here. [120:57] So now let's see what actually happens. [121:00] So first is opens promotion PR to stage. [121:05] We are calling this a promotion PR. [121:09] There we were just calling it a PR. Mind [121:13] the nuances. Look for the nuances. Opens [121:17] promotional PR to stage. Now as soon as [121:21] the PR is opened CI runs it is like [121:25] before itself. Now when will the [121:29] deployment happen? So let's see what are [121:32] the branch protection rules and what [121:35] should happen for the code to merge [121:39] successfully. When will the PR be [121:41] approved? So firstly PR only no direct [121:45] pushes. Second is dev run green for the [121:48] promoted commit. So this pipeline should [121:52] have been successful only then you can [121:55] move that artifact or that code to the [122:00] staging stage. So that's what we are [122:02] saying dev run green for the promoted [122:04] commit up to date with base include [122:08] latest stage and I told you this the [122:11] stage should be up to date like we [122:13] discussed before there should be no [122:16] commits that were performed after this [122:20] PR request was created in the stage [122:23] branch. Fourth is require these statuses [122:27] to pass. What is pre-flight? Pre-flights [122:31] are part of the CI itself. Then there is [122:34] pre-flight config deploy to stage. [122:37] Pre-flight manifest as in this manifest [122:41] must exist. So CI is ensuring we will [122:44] only merge this when there is a [122:48] pre-flight manifest and this is the [122:50] manifest. Pre-flight config. What is [122:53] pre-flight config? It is these things. [122:57] For example, Helm lint should be [122:59] successful. Helm template should be [123:02] successful. All these are performed in [123:05] the CI stage itself. And only when all [123:09] these are successful, then the PR is [123:13] merged. Then we are saying deploy to [123:15] stage, stage smoke, stage integration. [123:19] All these tests we have discussed [123:21] before. Lastly, once all these are [123:24] successful, then I want at least one [123:29] person to review this. It could be your [123:32] TL, it could be the QA team or it could [123:35] be the owner. Understand this manual [123:38] review process is optional. Some might [123:42] not have this. Now I am not sure whether [123:46] you have noticed it or not but here we [123:50] have the deployment step before the PR [123:55] merges and when we were discussing [123:58] feature to dev flow we had this at the [124:02] very last. Now first I want to explain [124:05] you why this is the case when I'm [124:09] talking about staging. Staging is like [124:12] prepro. Whatever is there in prepro, you [124:16] want those things to be extremely [124:20] perfect. You want the deployment to be [124:23] successful. What if the CI pipeline [124:26] succeeds but the deployment failed and [124:29] the merge was performed? We don't want [124:31] that. We want to only perform the merge [124:35] when the deployment is also successful. [124:39] So it makes sense because staging is [124:41] pre-pro and production of course will [124:44] adhere to the same design. So only [124:47] perform the merge once continuous [124:51] integration is green and the CD is also [124:55] green that is continuous deployment or [124:57] delivery. Both of these things must be [125:01] green for my merge PR into stage to be [125:06] successful. But Vun why is it like this [125:11] when it comes to feature branch? Firstly [125:14] I want you to know there is something [125:16] called fast feedback. The primary goal [125:20] of the feature to dev flow is to get new [125:23] code from multiple developers integrated [125:27] into a single shared environment as [125:30] quickly as possible. Now waiting for a [125:33] full deployment to succeed before [125:36] merging here would create a merge queue. [125:40] If deployment fails, the PR stays open [125:42] because understand there are multiple [125:44] developers who are working in their [125:46] respective feature branches blocking the [125:49] next developer who wants to merge their [125:52] code. Hence in this scenario you would [125:55] often see that the merge is performed [126:00] before the deployment stage because [126:03] again development is there to perform [126:06] testing left right and center. There [126:08] will be multiple PRs that will be raised [126:12] for dev branch compared to devto stage [126:16] or stage to prod. Another thing is [126:18] catching failures early. It's meant to [126:22] be a place where things might break. So [126:24] it's okay if my deployment fails after [126:27] the merge as long as the developer who [126:31] caused that havoc is ready to fix those [126:35] things. So here the agenda is fail fast [126:41] and recover faster. That's the reason [126:45] why you do deployment at the very last. [126:49] Okay. Now let's continue with this [126:52] discussion. Now if I talk about CI [126:55] pipeline here we are doing three things [126:58] pre-flight, deploy and post deploy. So [127:02] whenever CI pipeline is created the [127:04] first step is pre-flight and we know [127:08] what are the pre-flight steps. [127:10] Manifest.json [127:12] should be reachable that is here. Now it [127:14] could be hosted in different platforms [127:16] like I told you S3, GitHub or it could [127:18] be also part of your PR's description [127:22] and I want you to know that whenever you [127:24] create a PR provide proper description [127:26] in your PRs okay what this will do what [127:30] is the expectation and all of these [127:32] things second is image must be pullable [127:36] by digest so all these things CI will [127:39] verify config renders cleanly now you [127:43] could be using Helm, you could be using [127:46] customize. So you would apply those [127:49] testings in the CI phase itself because [127:52] you do not want those things to fail [127:55] when you are in deployment stage. You [127:57] want to fail fast if at all you are [128:01] supposed to fail. Here we are saying [128:04] Helm upgrade install loans. We are [128:07] upgrading this and you see what are the [128:11] variables that we have defined. [128:13] Variables are defined from the [128:16] manifest.json. [128:17] Here we are using jQuery. If you have [128:20] seen the JSON path lecture in our [128:22] Kubernetes series then this will make [128:24] sense to you. We are quering this [128:27] manifest and finding what is the value [128:30] of image. The value of image is this. [128:34] What is the value of digest? The value [128:36] of digest is this and based on this the [128:39] whole image along with the digest is [128:42] formulated and you have a deployment. So [128:47] these things are checked pre-flight. Now [128:50] the deploy phase refer details from [128:53] manifest.json [128:55] all the details are here. Deploy to [128:58] stage by digest with stage config only. [129:02] And what is the stage config that will [129:04] be saved in your values hyphen stagey? [129:09] So all things are connected. Now I want [129:12] you to concentrate on the parenthesis as [129:14] well. No rebuild. So it is not that [129:18] staging will have its own rebuild and [129:20] then prod will have its own rebuild. [129:23] Nothing like that. Whatever was built [129:26] was built in this stage itself. And that [129:29] same artifact is being promoted to stage [129:32] and being promoted to prod. So image is [129:36] this one by digest not tag. I'm [129:38] stressing on this point. Use digests in [129:42] enterprise production grade [129:43] implementations. Then next is post [129:46] deploy only things that need a live [129:49] stage and I discussed this briefly. [129:52] There are certain tests that needs to be [129:55] performed in the live running [129:57] environment and that can only be [129:59] performed once the deployment is done. [130:02] So here the smoke test smoke test is [130:04] quickly checking whether the website is [130:06] accessible. You are able to login. You [130:07] are able to click on the tabs or not. [130:10] All these things are part of post deploy [130:13] checks and configuring whether your [130:15] readiness probes your livveness probes [130:17] and all these things are successful and [130:19] they check your application application [130:22] paths application ports all of these [130:25] things can be checked in this stage and [130:27] then small integration text one two [130:29] happy path so if you are applying for a [130:32] loan are you able to proceed that's one [130:34] happy path second happy path could be if [130:37] you want to download your statement are [130:39] you able to do that that second happy [130:41] path and there could be several happy [130:43] parts that you want to verify when you [130:46] are testing this small integration as in [130:48] how are two parts of your application [130:51] behaving. All these things will be [130:53] performed post deploy. What if there is [130:56] a failure? In that case you should be [130:59] able to roll back helm roll back or [131:02] whatever you are using you should be [131:04] able to roll back and once the roll back [131:07] is successful you again fix it on dev [131:10] build new digest and repromote and then [131:14] finally let's say all of these things [131:17] succeeded the CI succeeded properly then [131:20] the deployment will be done and only [131:22] when the deployment is successful the PR [131:25] status will be screen and the merge will [131:28] be performed. So whatever was in dev [131:31] stage is now part of your staging code [131:36] and then if the something is wrong the [131:38] merge is blocked you fix indev new [131:42] digest repromote and that's the entire [131:45] process you follow now let's see what I [131:48] want to discuss next so next is stage to [131:53] prod versus dev to stage I want you to [131:56] know that when you are dealing with [131:59] stage stage and prod environments. Then [132:01] when you talk about stage, stage part is [132:04] heavier post deploy. Okay. But when you [132:07] have a production application running, [132:10] it is more heavier on the pre-eploy [132:13] stages. So pre-eploy is you will have [132:16] more checks in your CI when it comes to [132:19] prod when compared with stage because in [132:23] production you do not want to perform so [132:26] many tests in your running application [132:28] because staging is prepro these [132:32] environments in enterprise applications [132:35] are similar if not same. So whatever [132:38] configuration is there in your staging [132:42] environment is also there in the [132:44] production environment. So when you have [132:47] a running application it is suggested in [132:50] prod you mellow down your testing [132:52] because you have already performed all [132:53] that testing in your staging stage but [132:57] have pre-eploy tests they are more when [133:00] I talk about the prod stage so I only [133:04] wanted to talk about this when it comes [133:07] to the promotion from stage to prod rest [133:11] I will quickly read it for you but they [133:13] are pretty much similar if not name of [133:16] what we discussed just now. Promotion PR [133:20] references stage proven digest we know [133:22] plus commit and version. We have seen [133:25] this no rebuilds promote the same [133:29] artifact by digest. So whatever [133:31] container image was created in the dev [133:34] is used in stage and also in prod. Third [133:38] is broad branch stricter PRs only [133:42] approvals config validated. For example, [133:46] here we only had one manual approver. In [133:49] the stage to prod part, you could have [133:52] two or three manual approvers, one your [133:56] team lead, another your QA lead and then [133:59] there could be one that is your change [134:01] manager. So all these things must be in [134:04] place. CI pre-flights manifest reachable [134:09] digest pullable both of these we have [134:11] already discussed then here is flags off [134:15] so whenever you talk about deploying an [134:18] application or upgrading an application [134:21] there are two perspectives there is an [134:23] infrastructure perspective and then [134:25] there is an application perspective now [134:29] we must have means in both of these to [134:33] control how the upgrade is performed. So [134:37] when I talk about a production [134:39] application, you would let's say you are [134:41] using canary based approach for your [134:44] upgrades. Now Canary is you gradually [134:48] roll out the changes for example first [134:50] only for 20% of the users then 60% of [134:54] the users and then let's say 100% of the [134:57] users. So this is how canary is [134:58] performed and at infrastructure layer [135:01] you could be performing these at your [135:04] load balancer controller level or you [135:07] could be having your service mesh which [135:10] is helping you with canaries and [135:12] eventually your deployments will be [135:15] handling the traffic. So that is how you [135:18] will cater it from an infrastructure [135:20] perspective. But how about application? [135:24] Now when I talk about infrastructure, [135:26] you cannot control how the new feature [135:30] is behaving. From infrastructure, you [135:32] can control the entire application that [135:35] that entire application will only [135:37] receive 20% of the traffic first. The [135:40] upgraded version will receive only 20% [135:42] of the traffic and then 60 and then 100 [135:45] and so on. Whatever your permutation and [135:47] combination is. But when it comes to [135:50] that specific new feature, you do not [135:52] have any control over it from an [135:54] infrastructure perspective. But when I [135:56] talk about applications, modern [135:58] applications, what they will do is they [136:01] will typically release the new version [136:03] of the application with the flags off. [136:06] Now flags off, what will this do is this [136:10] will disable the new feature when you [136:12] first deploy the application. So first [136:15] it they will test how the application is [136:18] behaving with that feature disabled. If [136:21] the application works fine only then the [136:25] flags are enabled and these flags are [136:28] still enabled in a staggered way. So [136:31] first only 30% of the users will be use [136:33] able to use that feature then another [136:36] 50% then 100% and so on. This can differ [136:39] application to application organization [136:41] to organization. But what I want you to [136:44] understand is these things can be [136:47] controlled from both infrastructure and [136:49] application level. But from [136:51] infrastructure you cater to the entire [136:55] application but application teams can do [136:58] this per feature level as well. So that [137:01] is what flags off means. So when you are [137:03] deploying application onto production [137:06] you will typically have flags off but in [137:09] staging these flags are often on at the [137:12] beginning itself and again you can test [137:15] various permutations and combinations in [137:17] staging as well because these things are [137:20] specific to project application [137:22] organization. The next is cautious [137:25] rollout. In cautious rollout you have [137:27] canaries. Canaries we have already [137:29] discussed. Blue green is you have two [137:32] environments and you deploy to the new [137:35] environment and the previous environment [137:37] can then be used to deploy the next [137:40] version of the application. So you [137:42] maintain two full versions of your [137:45] infrastructure. So for example, if you [137:47] are using Kubernetes, then you will have [137:49] two Kubernetes clusters. When you [137:52] upgrade your application, you will [137:53] upgrade that onto the second Kubernetes [137:56] cluster and your traffic, your users [137:59] will be served from the second [138:01] Kubernetes cluster and once there is [138:04] another release of your application then [138:06] you would again be using cluster one to [138:09] run your upgraded version. So this [138:11] continues this approach is costly but if [138:14] your availability requirements are like [138:16] that then you can also use blue green. [138:19] Canaries is the one which is most [138:21] popular. In Canaries the shift is [138:24] gradual. So first 10% of the users will [138:27] be served with the new version then 30 [138:30] then 50 then 100. Now what I'm saying is [138:33] guarded by SLOs's. The strategy in both [138:37] stage and prod will be same. For [138:39] example, the strategy in stage will [138:42] still be Canary, but in staging, you [138:47] would be deploying it for 40% of the [138:49] users, then 60%, and then 100%. But in [138:52] production, you could first be only [138:54] deploying for 10% of the users, 20%, [138:57] then 50, then 60, then 80, then 100. So [139:00] you are more conservative with your [139:03] percentages when it comes to production [139:06] post deploy checks health synthetics [139:08] brief dwell period this we have already [139:10] discussed responsibility split and we [139:14] saw this at the very first stage is [139:16] heavier post deploy prod is heavier [139:18] pre-eploy if signal fails auto roll back [139:22] fix and repromote so this is all there [139:27] was to get flowbased CI ICD. Now let's [139:31] see what happens when you are dealing [139:34] with trunkbased CI/CD feature. As we [139:38] understand from our previous discussions [139:40] that in this you have main branch that [139:43] is your source of truth and then you [139:46] have multiple feature branches where [139:49] your developers will be working. I want [139:52] you to know here the source of truth is [139:55] only main. That is why you see the main [139:57] branch is being used to deploy onto [140:00] different environments. Dev in blue, [140:03] yellow in stage and green in prod. So [140:07] let's see how this works. So developer [140:10] commits and pushes. As soon as the push [140:13] happens, PR is opened and then CI runs. [140:18] As we know here is the CI. CI will be [140:21] performing the checkout, build and test. [140:24] And based on the results of this PR a [140:29] merge will happen. And you can see here [140:31] what all things are performed by our CI. [140:34] It's build succeeds, unit tests are [140:37] performed, lint format clean, quick [140:39] security scans, up to date with mains. [140:42] Now I have deliberately chosen different [140:45] level of things when it comes to CI but [140:48] you could be doing all these in others [140:51] diagrams as well. But I wanted to give [140:54] you what all are your options so that [140:57] you understand in a better way. Then we [141:01] have a PR stage. Here is the result. So [141:05] if it is green, if all these succeed, [141:08] build, unit, lint, security scans, up to [141:10] date, the build succeeds only then the [141:15] merge will happen. And once the merge [141:18] happens on the main then you can publish [141:22] it. Understand that the publishing part [141:25] is called as continuous delivery, [141:28] continuous deployment. If there are no [141:30] approvals that are taken in this stage. [141:34] Now I want to explain postmerge actions [141:38] on the main build package once publish [141:41] image plus digest example this also [141:44] publish manifest. Now I want you to know [141:46] that you would be using tools like Helm [141:49] and customize to do your deployment. So [141:53] those things are carried forward from [141:56] the previous discussions that we have [141:59] done thus far. So here what we are [142:02] saying is auto deploy no approval in dev [142:05] stage there is no approval. So reads [142:09] manifest.json JSON. This is what will [142:11] happen when the deployment will be done [142:14] in the dev environment. The manifest [142:16] will be read and who is reading the [142:18] manifest? It is your Jenkins server, [142:21] your automation server that is doing [142:24] this. It will read the manifest.json [142:27] deploy by digest and perform smoke and [142:30] health check whether the application is [142:32] working fine or not. Health is your [142:34] readiness and livveness probes. Smoke [142:36] tests are whether you are able to access [142:38] the application or perform basic [142:40] functionality in the application or not. [142:42] Second is once this is successful we are [142:45] saying that we can deploy the same [142:47] digest in the QA environment. But I want [142:50] you to know that here when will this be [142:53] run? When there will be a promotion job [142:56] plus approval. Now approval is as I told [143:00] before it is optional but typically you [143:02] would see in staging stages there is [143:05] approval your TL or your QA will be [143:08] approving your request. So stage here is [143:12] deploy same digest perform integration [143:14] contract API UI plus performance [143:17] security and we have already discussed [143:20] all these tests in our day one's lecture [143:24] and finally so when you are deploying to [143:26] the production cluster there will be [143:28] first be a promotion and these things [143:30] will be taken care by your Jenkins [143:32] server itself there will be two [143:34] approvals your approvers could be your [143:36] change manager your TL your application [143:40] owner and so on. So these are the steps [143:43] that will be performed for deploying it [143:46] to the prod environment. We'll be [143:48] deploying the same digest again that is [143:50] the important piece. We are using the [143:53] same artifact for deployment in multiple [143:55] stages cautious rollout plus SLO [143:58] guardrails ready roll back. Now if I [144:02] want to summarize this PRCI proves the [144:06] merge then the merge happened build once [144:09] on main promote that exact digest [144:12] through dev stage prod with increasing [144:15] confidence. I want you to know that if [144:18] it was deployed in dev and that failed. [144:21] I want you to know that merge has [144:23] already happened then you must fix the [144:27] problem in your main. Your developers [144:29] must be equipped to fix this [144:31] immediately. Which is why when you are [144:34] using this sort of design, your testing, [144:36] your CI is very stringent. It takes care [144:41] that everything is in place before the [144:44] actual deployment takes place. But again [144:47] let's say if your application requires [144:49] GPU then you must ensure that in CI [144:53] stage itself you are pulling your [144:56] deployment environment to see whether [144:58] those nodes are available or not. So you [145:00] have to think like that if you are [145:04] adhering to this design then testing [145:06] must be extremely stringent. I hope you [145:10] have understood this and then finally I [145:12] want to discuss why and what of Jenkins [145:16] and I'll just read through these points [145:18] because we have already understood these [145:20] points throughout this lecture. Jenkins [145:23] Y and what it is open-source automation [145:26] server per CI and CD runs pipeline as [145:30] code from your repo and we have seen all [145:34] these things in action in our diagrams [145:37] again automation server not just for [145:40] CI/CD it can also do CT it can do any [145:44] automation that you want Jenkins to do [145:47] there are plethora of plugins available [145:50] for Jenkins which can help you integrate [145:53] various AWS, Azure or GCP services and [145:58] you can leverage those why teams use it [146:01] portable and extensible with with rich [146:04] plugins. Gates merges with green red [146:07] status checks fits git flow and trunk [146:10] based workflows. We have just seen these [146:12] enables build once promote the same [146:16] digest and we have also seen this how it [146:18] usually runs. We will be discussing [146:21] these things in detail that what are the [146:24] patterns that you can use for Jenkins [146:27] how Jenkins will scale when there are [146:29] plethora of jobs running in parallel so [146:31] we'll see all these things what it isn't [146:34] Jenkins is not a gith host a registry or [146:37] a monitoring tool it is an orchestration [146:40] engine it is an automation server [146:42] alternatives and compliments now if you [146:45] have a githops based workflow then you [146:47] will often see that Jenkins works in [146:50] conjunction with your Argo CD, Flux or [146:53] your any other tooling similar to that. [146:55] But there are alternatives to Jenkins [146:58] like GitHub actions GitLab CI. That's [147:02] all I wanted to cover. That wraps up day [147:04] two of Jenkins basics to production [147:07] course. Today we reviewed Git and [147:10] branching essentials. Decoded the [147:12] continuous practices. Compared Git flow [147:16] and trunkbased strategies, walked the [147:19] promotion flow from feature to dev to [147:22] prod and reinforced build once promote [147:25] by digest. You also saw how Jenkins [147:29] orchestrates builds and promotions. In [147:33] the next lecture, we'll install Jenkins [147:35] and run our first pipeline. If you have [147:38] any questions, feel free to ask them in [147:41] the comment section and I will reply. If [147:44] you have any questions, feel free to ask [147:46] them in the comment section and I will [147:49] reply. If this helped, do consider [147:52] liking, sharing, and subscribing. It [147:55] will mean a lot to me. Thank you very [147:57] much. Hi, I am Vun Jooshi and welcome to [148:01] day three of our Jenkins basics to [148:04] production course. In today's lecture, [148:07] we will clarify what Jenkins is and why [148:11] it matters. Then we will compare [148:14] installation options across environments [148:17] so you know what fits the lab versus [148:20] enterprise. Next, we will go hands-on [148:24] with Jenkins installation with Docker. [148:27] Finally, we will create a freestyle job [148:30] to run commands, archive artifacts and [148:33] see built-in environment variables in [148:37] action. So, let's get started. [148:41] What is Jenkins? Jenkins is an [148:44] open-source automation server and [148:46] open-source is the keyword here. So when [148:51] I talk about Jenkins, Jenkins is [148:53] communitydriven [148:55] and this falls under the umbrella of [148:58] continuous delivery foundation also [149:01] called CDF. So like when we were [149:03] discussing Kubernetes, Kubernetes is [149:06] open-source communitydriven and it sits [149:09] under the umbrella of CNCF. Jenkins is [149:13] also communitydriven but it sits under [149:16] the umbrella of continuous delivery [149:19] foundation CDF. Now Jenkins is CI/CD [149:23] orchestrator that will help you in all [149:27] the stages of your application. It will [149:30] help you in the coding part. It will [149:32] help you in the build phase, package [149:34] phase, test phase and deploy phase. And [149:37] again I want you to understand that [149:39] orchestrator is someone that is [149:43] connecting the dots. For example, when [149:45] you are deploying an application, there [149:48] is an infrastructure architect, there is [149:50] a DevOps architect, then there is an [149:53] application architect and so on. And [149:55] then there is a chief engineer. Chief [149:58] engineer is the one who connects all the [150:01] dots together to give a proper end toend [150:05] solution. Similarly, Jenkins is an [150:08] orchestrator which is connecting all the [150:11] dots for you. For example, at all of [150:14] these phases, you will have different [150:17] tooling that will be helping your [150:19] application. If I talk about code, you [150:21] will be using git and you will also be [150:23] using a git host, git host like GitHub, [150:26] Bitbucket, GitLab, etc. When I talk [150:29] about build, you will be using different [150:31] tools here. You would be using docker if [150:33] you are generating container image as an [150:36] artifact. You could be using maven [150:38] gradal ant which helps you in the build [150:41] phase of java application. Similarly [150:45] when it comes to package you can use [150:47] maven to package a java application. [150:49] When it comes to test you will be using [150:52] different tooling for test as well. You [150:54] will be using tools that can perform s [150:58] that is static application security [151:00] testing. Then you will have tools that [151:03] will help you with DAST that is dynamic [151:06] application security testing. Similarly, [151:09] when it comes to deploy, you could be [151:12] using Helm, you could be using customize [151:15] or you could be just be deploying to [151:17] plain EC2 instances. Each of these [151:20] stages have their own tools. So what I [151:23] want you to understand is Jenkins is not [151:27] replacing these tools. What Jenkins is [151:30] doing is it is acting as an orchestrator [151:34] which will connect all these different [151:36] tools to deliver your application. So [151:41] that is why we are calling Jenkins a [151:44] CI/CD orchestrator. Third is pipelines [151:48] as code. And whenever you see something [151:51] as code, I want you to imagine version [151:54] control. So with Jenkins whatever [151:57] pipelines that we write as in your [152:00] pipeline can have one stage two stage 10 [152:03] stage it depends on your application. So [152:06] what you can do is you can code those [152:09] stages into something called a Jenkins [152:12] file. And once you have that Jenkins [152:15] file, you can version control that [152:17] Jenkins file. So that as you progress [152:20] with your infrastructure, as you [152:22] progress with your pipeline, you can see [152:26] the evolution of your pipeline. So for [152:29] example, how was my pipeline when [152:32] application one was introduced? Now 5 [152:35] years down the line my application has [152:38] so many bells and whistles. How does my [152:41] pipeline file looks like? So that helps [152:45] you understand the pipeline evolution as [152:48] well. Next is rich plug-in ecosystem. [152:52] Now see, I just said that Jenkins is an [152:58] orchestrator. But I want you to [153:00] understand if Jenkins is creating a [153:04] container image for you, [153:06] then Jenkins must have a tool that helps [153:11] in creating a container image. And we [153:14] know Docker is one tool that can help [153:17] you with that. So if you are expecting [153:20] Jenkins to create container images for [153:23] you then [153:25] Jenkins must have docker installed. [153:29] Similarly if you want Jenkins to check [153:33] out code from your git host then Jenkins [153:37] must have g installed so that it can [153:40] interact with your remote. So in that [153:45] scenario, git must be installed in your [153:48] Jenkins. So understand if you want [153:52] Jenkins to do something, if you want [153:54] Jenkins to deploy using Helm, then [153:57] Jenkins must have Helm configured. [154:00] Similarly, [154:02] we talk about rich plug-in ecosystem. So [154:05] plugins are something that helps Jenkins [154:09] integrate with other tooling. For [154:12] example, if you want to push an image [154:15] into AWS's ECR, then Jenkins has a [154:19] plug-in that will help you do the same. [154:23] If you want to use AWS's secrets [154:26] manager, then you have a plug-in that [154:29] will help Jenkins interact with the [154:32] secrets manager. So, I want you to [154:35] create that picture in your head right [154:38] now. So if you want your Jenkins to [154:42] integrate with some notification engine [154:44] then there must be a plug-in that you [154:47] can install in Jenkins and that plug-in [154:51] can help you interact with your [154:53] notification service. So Jenkins has a [154:58] rich plug-in ecosystem get workflow [155:02] friendly. What is this? So let's go to [155:04] day two. In day two, what we said? We [155:07] said that developer will perform a [155:09] commit. Then developer will raise a PR [155:12] to dev and as soon as this PR is raised, [155:16] the continuous integration will run. [155:18] Essentially, Jenkins will trigger. How [155:21] is this possible that when you are [155:24] raising a PR in your GitHost like [155:26] GitHub, it will trigger Jenkins that is [155:29] running in your on premises? How is that [155:32] possible? It is possible because [155:35] whenever there is a PR created, you can [155:40] configure your GitHost to send a web [155:44] hook and based on that web hook, Jenkins [155:48] will trigger the pipeline. That is why [155:51] we are saying get workflow friendly. [155:54] Whatever happens in your GitHost, for [155:57] example, GitHub, if there is a [156:00] collaboration request or there is a [156:02] review or there is a PR, then you can [156:05] configure your GitHost to send a web [156:09] hook to Jenkins and Jenkins based on [156:12] that can trigger the pipeline. extremely [156:16] powerful and you will see this in [156:18] production used a lot. Whenever there is [156:21] a PR created a Jenkins pipeline is [156:24] automatically triggered. Next is free [156:28] core enterprise option. So at the very [156:31] core, Jenkins is free as we know it is [156:34] open-source, but there are enterprise [156:37] options and cloud bees is an [156:40] organization that is a major backer of [156:44] Jenkins. You can have support taken from [156:48] cloudbes. If you want Jenkins support, [156:50] now I want you to know at this stage [156:52] itself whenever you are using an OSS [156:55] application or software in your [156:57] environment to run your production [156:59] application, then you should always take [157:03] enterprise support for that and not just [157:05] OSS. So if you are on AWS, you are [157:09] running an enterprise application on [157:10] AWS, then also you would pretty much [157:14] always have business support from AWS [157:17] because there are services whose control [157:21] plane is sort of hidden from you. If I [157:24] talk about transit gateway, you just see [157:26] the software level things. What if you [157:29] want to know deeper insights? Then you [157:32] will have to involve AWS support for [157:34] that. Similarly, if you are using [157:37] Jenkins, which is open-source, and you [157:39] are stuck somewhere, your only [157:41] troubleshooting option would be to surf [157:44] the web. But what if you do not get an [157:47] answer? You want an enterprise support [157:50] of Jenkins? If you are running an [157:52] enterprise application with Jenkins, now [157:57] how it usually runs, it runs anywhere [158:00] and we'll explore the installation [158:02] options soon. Docker, VM, bare metal, [158:05] Kubernetes clusters and so on, [158:08] controllers and agent. So this is an [158:11] important point. The installation that [158:13] we will do right now at this stage of [158:16] this course will be using only one [158:19] machine and that one machine or one [158:21] container will be acting as both [158:25] controller and agent. So what are these [158:28] controllers and agents? Think of them as [158:31] control plane and data plane. Control [158:33] plane is the part where you configure [158:37] what the system must do and those tasks [158:42] are achieved by the data plane. So data [158:44] plane has agents and the control plane [158:48] instructs the agents on what to do. And [158:51] what is that these agents will be doing? [158:53] These agents will be running pipeline. [158:55] For example, if you have 10 agents, [158:58] maybe five agents are supposed to work [159:00] when you are dealing with the pipelines [159:03] associated with application one and the [159:05] next five agents are supposed to work [159:08] when there are pipelines run associated [159:11] with application two. Then the next one [159:15] is credentials and policies. I want you [159:18] to know that there is a store that you [159:21] can use within Jenkins to store your [159:23] credentials. You can do that. But in [159:27] enterprises, you would often see that [159:29] your Jenkins pipelines are integrated [159:32] with third-party tools. Now, these third [159:34] party tools could be your Hashi Corp [159:37] vault or it could be the native secrets [159:41] management service that is provided by [159:43] major cloud provider. For example, you [159:46] can integrate AWS's secrets manager. [159:49] Now, why would you want to do that? I [159:51] want you to know that if Jenkins is [159:54] deploying something onto Kubernetes, [159:56] then Jenkins must have credentials to [159:59] authenticate with your Kubernetes [160:01] cluster. Where do you save that data? [160:04] You must have secure means to access [160:07] your Kubernetes cluster. If you are [160:09] deploying something onto traditional [160:12] VMs, then you would want to have SSH [160:17] keys to access that VM over SSH. Where [160:20] do you save your SSH keys? Or if you are [160:24] accessing anything using tokens, where [160:28] do you save those tokens? Hence you must [160:32] have means in place via which you can [160:35] save these secrets information securely. [160:40] Often times you would see third party [160:42] plugins are integrated so that you can [160:45] leverage such tools. Now that we have [160:48] this understanding let's move to the [160:51] installation section. Jenkins [160:53] installation options. There are plenty [160:56] of these. Let me tell you when it comes [160:58] to docker containers this is the option [161:02] that you would use when you are doing [161:04] labs in development environments and in [161:07] small teams. So it is the quickest way [161:10] for labs and PC's portable consistent [161:13] across all the operating systems and [161:15] that is the flexibility that containers [161:18] will give you. Let's say you have [161:20] everything configured. You can create an [161:22] image out of that running container and [161:24] then distribute that image if anyone [161:26] wants to replicate your Jenkins [161:28] architecture. So containers give you [161:31] that flexibility. Lightweight we know [161:34] that cons persistent scaling needs extra [161:38] design and this is true. So if you are [161:41] using Docker Swarm, there is not an [161:43] intelligent plug-in yet that can help [161:47] you scale. And what do I mean by scale? [161:50] I told you that there is a controller [161:54] and then there are agents. So when there [161:57] are jobs that do not yet have agents [162:01] associated with them, then what the [162:04] controller will do? Controller will spin [162:06] up more agents. And where are these [162:09] agents living? So if I talk about the [162:12] Kubernetes style Jenkins, then these [162:15] agents will be running in the pods. If I [162:18] talk about VM and bare metal, these [162:20] agents will be running in the VMs. If I [162:24] talk about Docker container environment [162:26] and if I'm using Docker Swan, they would [162:29] be running as Docker containers. So what [162:32] I'm saying is there is not a proper [162:35] fullfledged plug-in that can help you [162:39] scale those agents in docker swarm. [162:42] Hence you would not see this used in [162:45] enterprises but we will be using it for [162:48] testing in the early stages of our [162:51] course. Then let's talk about VM bare [162:54] metal. Let's say you are running on [162:56] premises and you want to run your [163:00] Jenkins but Vun how will I ask my [163:05] controller to spin up agents in my on [163:08] premises environment so what you have is [163:11] you have plugins available from VMware [163:14] open stack and so on that you can [163:17] leverage so that when controller demands [163:21] agents it can spin virtual machines that [163:24] will run those agents. Now, ideally, one [163:27] agent is equal to one virtual machine. [163:30] So, for example, let's say you are [163:33] running 10 pipelines. All those 10 [163:35] pipelines can be run by a single agent [163:38] or you can distribute the work by [163:41] spinning multiple agents. Now, I don't [163:43] want to explain this in much detail [163:46] right now because in the demo section, [163:48] this will make much more sense. So once [163:50] we move to demos for this which will [163:53] come later in this course then we'll [163:55] understand this deeply but for now I [163:57] want you to keep this in mind that [164:00] whenever there are more and more and [164:02] more pipelines in your Jenkins [164:04] implementation you can have scaling at [164:08] Jenkins as well. It is not that once you [164:10] install Jenkins onto a single VM that [164:12] single VM is all you have. your [164:15] controllers can spin up more agents [164:18] based on the load on Jenkins. So when [164:22] you use a VM, it's a good enterprise fit [164:25] best for compliancedriven stable scale [164:28] environments. Now another question that [164:30] comes to mind is support. If you want to [164:33] take support, there is cloudbes and [164:35] there are other organizations that can [164:37] give you enterprise level support for [164:39] your Jenkins server. So you can always [164:42] take that and I would recommend to take [164:44] them if you are running an [164:46] enterprisegrade application in Jenkins. [164:49] Next is Kubernetes. Jenkins on [164:53] Kubernetes. Firstly, I want to tell you [164:55] right away that do not think that this [164:57] is the best solution. No, it is not. The [165:01] best solution depends on what the [165:03] requirement is. If for example whatever [165:07] you are deploying you are deploying onto [165:09] traditional VMs that implies that client [165:13] or that application doesn't even have a [165:15] Kubernetes cluster. Will it make sense [165:18] to propose Jenkins on Kubernetes to that [165:21] client? Not at all. So in that scenario [165:25] VM could be a good choice. But if the [165:28] client already has a Kubernetes cluster, [165:31] then deploying Jenkins onto Kubernetes [165:34] is a good option. So your solution to a [165:39] problem should not be based on the most [165:42] fancy and latest tech available at your [165:45] disposal. It should be based on the [165:48] requirement of the client. And this is [165:50] not just for Jenkins. This is applicable [165:54] for the entire IT universe. Do not [165:58] design solutions based on what is the [166:00] latest and greatest available. Always [166:03] design based on what is the need of the [166:07] client. You have to be costconscious. [166:10] But if the client requires high [166:12] availability, you need to tell them that [166:15] dear sir, high availability is directly [166:19] proportional to the dollar sign. So if [166:22] you want me to design an environment [166:24] that will never ever break then you have [166:27] to shell some dollars. So let's get back [166:30] to this topic again. So Kubernetes [166:32] Jenkins on Kubernetes. So cloud native [166:35] scalable enterprise ready consider [166:39] operational complexity. Now if you are [166:40] not running Kubernetes and you just spin [166:43] up a cluster to run Jenkins then of [166:45] course it's an operational complexity [166:48] best for cloudnative organizations [166:51] needing elastic scale again here as well [166:54] the fun remains the same when controller [166:56] wants to spin up more agents you will [166:58] see more Jenkins pod running in your [167:01] cluster now these two options are kind [167:05] of niche so if let's say you just [167:08] migrated to the cloud and you spawned a [167:10] Jenkins server there. Now you have two [167:13] options either you can spin up a virtual [167:15] machine and then configure Jenkins [167:16] yourself and then maybe later take [167:18] Jenkins support but you also have an [167:23] option where you can go to the [167:24] marketplace of your respective cloud [167:26] provider and search for managed Jenkins [167:30] that is also available here. So this [167:32] talks about that you can use that. Now [167:36] this is cloudb's enterprise jenkins. Now [167:40] cloudb's calls it cloudb's cia. So what [167:44] they do is there are two things. First [167:47] is support. Now if you are already [167:50] running Jenkins you can go to cloudbes [167:53] and tell them that I want support from [167:55] you and they will support you whenever [167:57] there is a break fix in your Jenkins [167:59] environment. That's one use case. [168:01] Another use case is there is cloudb's CI [168:05] which is based on Jenkins but they have [168:07] add extra bells and whistles that can [168:10] help you make your Jenkins environment [168:13] have different policy controls and [168:16] compliance toolings. So this can help [168:18] you with multi-team multicontroller [168:21] agent management and I will let you [168:23] explore this further but what I mean is [168:26] instead of Jenkins you have cloudbi [168:28] cloudbi also uses Jenkins under the hood [168:32] but they have added few more features [168:35] that are specially useful for [168:37] enterprises so if you want you can check [168:39] that out. So the crux of this is if you [168:42] are running enterprise application on [168:44] Jenkins then you should ideally have [168:47] enterprise support for your Jenkins and [168:50] if you already have a Kubernetes cluster [168:52] you could deploy your Jenkins onto [168:54] Kubernetes. If you do not have a [168:56] Kubernetes cluster or if for XYZ reason [168:59] you want your Jenkins environment to be [169:02] separate from the Kubernetes cluster [169:05] then you can also choose this option. [169:07] Now there are scenarios when you might [169:10] not want to deploy Jenkins in the same [169:12] cluster where your application runs and [169:14] we will see those scenarios as we [169:17] progress with this course. Now let's go [169:20] to the CLI and install Jenkins. So first [169:24] I'll go to my GitHub notes here. I will [169:27] go to Jenkins basics to production. And [169:30] if you are finding this series helpful, [169:33] I would highly encourage you to, you [169:35] know, if you are liking my content, like [169:37] the video, subscribe to the channel and [169:40] you can always start the repository if [169:42] this is helping you. Now I'll go to day [169:44] three. Now here I will click on lab [169:48] setup and then I'll first copy this [169:50] command and then I'll tell you what is [169:52] going on. I'll go to the terminal. I [169:54] will paste it here. What I'm saying is [169:57] docker run-d [169:59] is run in the background. Name of the [170:02] container will be Jenkins. Restart is [170:05] unless stopped. So what I'm saying is if [170:07] I manually stop this container then keep [170:11] it stopped. But if I let's say restart [170:14] the docker engine in my map then you [170:16] should automatically restart Jenkins [170:20] container as well. hyphen p is the port [170:23] mapping 8080 to 8080. So I'm saying the [170:26] host port on my docker host when someone [170:30] hits that you take them to the container [170:34] port 8080. And what is listening on port [170:37] 8080? Jenkins is running in my container [170:41] that is listening on port 8080. And then [170:45] again hyphen p 50,000 to 50,000. So the [170:48] host port 50,000 in my docker host is [170:52] listening and it will send the request [170:54] to the 50,000 port in container. So what [170:57] is this used for? At later stage in this [171:00] course we'll be using agents. Agents use [171:03] 50,000. Hence I have kept it here for [171:06] the sake of completeness. [171:08] Hyphen V. What we are saying is create a [171:13] named volume called Jenkins home and map [171:18] that to forward slash whereward/jenkins [171:22] home in the container. Warun what is [171:25] Jenkins home? Jenkins home is something [171:29] that you must preserve at all times [171:32] because this is the directory which is [171:35] holding all data. Jenkins the pipelines [171:39] you are configuring the plugins you are [171:41] adding all of those things are saved [171:43] here. Now if something is that critical [171:46] automatically you should be thinking how [171:49] should I back that up. So you should be [171:51] backing this up. If you are deploying [171:54] your Jenkins in a containerized [171:55] environment and your orchestrator is [171:57] Kubernetes, then you should be storing [172:01] this in a PVC and that PVC should be [172:05] coming from a block storage or from a [172:08] file storage but this must be preserved. [172:12] If you are running Jenkins on a VM [172:14] environment, then you should be storing [172:16] this onto a block storage or onto a file [172:20] storage. Next is hyphen E. E is defining [172:25] the environment variables. We are saying [172:27] the environment variable TZ which is for [172:30] time zone. The value for that is Asia [172:33] Kolkata. And I am doing this so that the [172:36] logs I see makes sense to me because I [172:40] am in India. Then Jenkins [172:42] forward/jenkins [172:44] LTS. What is this saying? This is saying [172:46] that I want you to use a repository that [172:50] has the name Jenkins and the tag is LTS. [172:54] I will press enter now and this will [172:57] create the Jenkins container for me. So [173:00] if I run docker ps I see my machine has [173:04] a single container running and that [173:06] container is named Jenkins. Now I said [173:09] that anything that hits on port 8080 on [173:13] my docker host should be mapped to port [173:16] 8080 in the container. And I told this [173:20] in my docker course that whenever [173:22] someone is accessing your docker host [173:25] first there is docker host then there is [173:28] the container. And you mention these [173:30] ports in the same way. First there is [173:32] docker host and then there is the [173:34] container port. they are not always the [173:37] same. Hence, this distinction is [173:40] important to know which is the docker [173:42] host port and which is the container [173:45] port. So, my docker host is listening on [173:47] port 8080. So, I'll go to my browser and [173:51] I will access local host on port 8080. [173:57] I'll press enter. Voila. I see Jenkins [174:01] is here. And it says to unlock Jenkins [174:04] take the password from this path. So let [174:07] me copy this path. I will go to my [174:11] terminal here. I will run docker exec. [174:16] Name of the container is Jenkins. And I [174:19] want born again shell which is bash. [174:21] I'll press enter. And here I can see my [174:25] credentials. Before I do that, I do not [174:27] like this bland terminal prompt. So let [174:31] me make this colorful. I'll press enter. [174:33] This is now colorful. Let's copy that [174:37] path again. I'll copy this and then I [174:40] will cat on this path. And it shows me [174:43] the password. I will copy this. I'll go [174:46] to my browser. I'll paste this. I'll [174:49] press enter. And this is asking me to [174:52] customize Jenkins. Now either I can [174:56] install suggested plugins or I can [174:58] select the plugins and install them. I [175:01] will for this demo click on install [175:03] suggested plugins. Now while this is [175:07] getting started, let's wait. Okay, let's [175:09] do one thing. Let's understand what this [175:11] is doing. So these all are the plugins [175:14] that are being installed. So what is [175:16] folders? Folders is there to you know to [175:19] put your jobs into respective folders. [175:23] OASP markup formatter. This is for HTML [175:27] pages. Then there is build timeout. If [175:29] there is a faulty build, you do not want [175:32] that build to run till infinity. There [175:35] must be a timeout. For example, if you [175:37] know your pipeline will run in 30 [175:39] minutes. You will set a build time out [175:42] of 45 minutes, 15 minutes buffer. Hence, [175:45] build timeout allows you to configure [175:48] that value. These all are plugins. Okay. [175:51] Timestamper whatever we are doing in [175:53] console there will be a timestamp [175:55] associated with our actions and this [175:56] timestamper plug-in is helping with that [175:59] workspace cleanup and this is a system [176:02] utility task and gradal these are [176:06] specific to Java pipeline is the plug-in [176:09] that allows you to create pipelines in [176:11] Jenkins GitHub branch source this is [176:13] specific to GitHub pipeline GitHub [176:16] Groovy libraries again associated with [176:18] GitHub pipeline graph view. This is [176:21] cosmetics. This will help you see your [176:24] pipelines in a graphical view. Git, as [176:27] we know, this is our source control [176:29] system. SSH build agents. These are used [176:33] when you want to SSH into virtual [176:35] machines. Matrix authorization strategy. [176:39] This pertains to authorization. As the [176:41] name implies, LDAP is lightweight [176:44] directory access protocol. This is used [176:48] for SSO single sign on email extension [176:51] as the name implies. If you want to send [176:54] emails to someone, this extension can [176:55] help you with that. Mailer again [176:58] associated with mails. Dark theme is [177:02] what we all love. Let this complete. [177:05] Credential bindings injects credentials [177:08] into builds. Okay. Now it is saying [177:11] create first admin user. So let's call [177:14] this user cloud with varosh and then [177:18] I'll set up a password and full name is [177:23] varun jooshi and email address is abc [177:27] the ratexyz.com [177:29] because I don't want to use this feature [177:31] right now. I will click on save and [177:34] continue. Now instance configuration. [177:36] Now this is the Jenkins URL. Now this [177:38] Jenkins URL is important because when [177:42] you are implementing an enterprisegrade [177:44] Jenkins then this Jenin URLs can be [177:47] carried forward to your emails. So for [177:50] example if some job has failed and your [177:54] fellow members want to see the error [177:56] logs then they can just click on Jenkins [177:59] URL along with you know the full path [178:01] and they will be right away taken to the [178:03] Jenkins screen. So this is important to [178:05] configure but right now since we have [178:07] deployed it onto the local host that is [178:09] our machine we will leave it at this [178:12] save and finish start using Jenkins and [178:16] let's the first thing that I want to do [178:18] is enable dark mode so that we do not go [178:21] blind. So I'll hover over my user icon [178:26] click on appearance and then I can [178:29] choose dark and apply. I'll save this [178:32] and this is the welcome screen of [178:35] Jenkins. You will see this and first is [178:38] create a job. We'll be using this but I [178:41] want to give you a small tour. I'll [178:44] click on settings here and here is where [178:47] you will be spending a lot of time as a [178:50] Jenkins administrator. You see here you [178:53] can provide credentials. Then there is [178:55] credentials. Then there is security. [178:57] Here is users. There are many options. I [179:00] will let you explore that because each [179:03] line under the main heading tells you [179:05] what this thing is doing. But in today's [179:09] lecture, we'll be concentrating on tools [179:11] and maybe plugins as well. This is nodes [179:14] add remove your agents and so on. So [179:16] let's do one thing. Let's directly [179:19] create a job so that we know the basics [179:23] right. So I'll click on create a job. [179:25] Here I have multiple options. I can [179:27] create a multibranch pipeline folder [179:30] pipeline and freestyle project with this [179:33] lecture. I want to start with a [179:35] freestyle project. I'll name this intro [179:39] freestyle. [179:40] I'll press okay. And this has created a [179:44] job. Now it is asking me to configure [179:47] that job. Intro freestyle configuration. [179:50] Here description you always should be [179:53] giving for production jobs. For now, I [179:55] will skip this. For now, I will skip all [179:58] of these. Source code management. I do [180:00] not want that. I'll let none be [180:02] selected. Then these are the triggers. [180:05] Like I told you, how will a Jenkins [180:08] pipeline get triggered? There is poll [180:11] SCM as well. SCM as in source code [180:14] management. Here your Jenkins can pull [180:18] your git host like GitHub after let's [180:21] say an hour or so to check if there is [180:24] some work for that pipeline that is also [180:28] possible. Then there is GitHub hook [180:30] trigger. This is the one that we [180:32] discussed the web hook build [180:34] periodically. It's a chron job build [180:37] after other projects are built. This is [180:39] like chaining. If that project is [180:41] completed then you start this job. Then [180:44] you have trigger builds remotely example [180:47] from scripts. We will be using none of [180:49] them here because we will be running our [180:53] job. We'll be triggering our job using [180:56] manual method that is by clicking the [180:59] mouse button on build. Now then here are [181:04] certain environments that you can set. [181:05] We'll also leave this for now. Here is [181:09] what I want to do in today's lecture. I [181:12] want to click on add a build step. You [181:15] see there are multiple options to choose [181:18] from. What I will do is I'll click on [181:22] execute shell. So if I go to my Jenkins [181:25] container first. Okay, let's go here. I [181:28] will clear my screen. I will click on [181:30] cat etsy and then I will check OS [181:35] release. And this will show me the OS [181:40] statistics for my container. Now I want [181:43] to do another thing. I will go here and [181:46] here I have execute shell. So what I [181:49] will write here is I will write cat [181:51] oscos [181:54] release and then I will click on apply [181:57] and then I will save this. My job is [182:00] saved. A simple intro freestyle job. [182:03] Here I have multiple options but what I [182:07] want to do is I want to click on build [182:09] now. I'll click on build now. This will [182:12] start the build process for this job. [182:15] And here are the builds history. And you [182:19] see number one build of this job is [182:23] successful. So what we are calling these [182:26] multiple runs are we are calling them [182:29] builds. So if I click on build now again [182:32] I see a build two is created and build [182:36] two I have multiple options with build [182:39] two either I can click on it to see [182:41] those options or I can click on this [182:44] tiny drop-down and here again I have [182:48] multiple options. I can delete this [182:50] build edit build information but what I [182:53] am concentrated on right now is console [182:56] output. I will click here. Voila. And [182:59] this is giving me the same thing that I [183:03] saw here. So then the question becomes [183:06] are both of these same. I want you to [183:10] understand that execute shell in Jenkins [183:14] whatever we configured here. So this was [183:17] the job we were on. And if I click on [183:19] configure I will go back to the same [183:21] screen where we configured the job. So [183:24] we added execute shell as a build step. [183:28] And then we wrote cat this here. Now [183:32] this is a non-interactive [183:35] shell and this shell is on the assigned [183:38] node. Additionally, it is in this [183:41] workspace. What is the workspace? We [183:45] will see what that workspace is, how to [183:48] explore that. But for now understand [183:51] this is a non-interactive shell in a [183:55] specific workspace. And that workspace [183:58] is of this job that we have created. And [184:03] here is your actual shell. This is an [184:07] interactive shell in the containers [184:10] default working directory or the users. [184:14] So I hope that thing is clear. Now at [184:18] this stage itself I want to show you one [184:21] more thing that will clarify how this is [184:26] different than this. So if I write print [184:31] env here and I press enter you see all [184:35] the environment variables that are [184:38] defined in this container. So if you see [184:42] this is the one that I just defined you [184:45] know to make the prompt colorful and [184:48] this is the environment variable that we [184:50] set while we were creating this [184:52] container. Now when I ran catoss release [184:57] this also gave me the same output as [185:01] execute shell gave. But now what I want [185:05] to do is I want to add here print env. [185:10] And I want to save this and then I want [185:13] to click on build now. A build now will [185:17] create a third build. And what I'll do [185:20] is I'll go to the console output. And [185:23] you see here I have so many more [185:26] environment variables. You see build ID [185:30] is three because I am in the third [185:33] build. Job base name is intro freestyle. [185:38] This is the name of our job. So I want [185:42] you to clearly understand the [185:44] distinction between execute shell that [185:48] we used here and what is the shell that [185:52] is of the container. Now, Vun, what are [185:56] these environment variables that are [185:58] specific to this? So, if you click on [186:00] this link, the list of available [186:02] environment variables and you see here, [186:05] these are all the environment variables [186:08] that are available at your disposal. [186:10] Now, why are these environment variables [186:13] important? These are important for [186:15] traceability. Now imagine if your job [186:20] has failed and you saw the logs you [186:23] would want to know to which build these [186:26] logs belong to which job this logs [186:30] belong. Hence build number build ID job [186:34] name all these are important and you [186:38] should be knowing these things. So [186:40] whenever you are let's say creating a [186:43] Jenkins job you should always think to [186:48] implement these environment variables [186:51] somewhere so that you can correlate to [186:54] which job this artifact belongs to. So [186:58] if I'm creating a container image, I may [187:00] want to tag that container image with [187:03] the job name or the build number so that [187:07] I know exactly which build within my job [187:12] was used to create that container image. [187:16] So I always want to know which artifact [187:20] resulted from which build. Hence using [187:24] these environment variables will help [187:26] you. For example, node name. Maybe I [187:29] want to know on which Jenkins agent this [187:33] job ran. Hence, I would want to have [187:35] node name. So, based on requirement, we [187:39] can always use this. Now, again like the [187:42] example I gave you, maybe when a job [187:44] fails, I want to send an email and that [187:46] email can have the build URL so that [187:50] whenever someone clicks on that URL, [187:52] they directly land on the build page. [187:56] And as we progress with this, we will [187:59] explore more of these environment [188:01] variables. But this is an important [188:04] concept. That's why I introduced these [188:07] so early. So let's go back to our GitHub [188:11] notes and I will scroll down and let's [188:15] use these environment variables. So I [188:18] have these here. So I'll quickly copy [188:20] this. I will go to our job, the job that [188:24] we created. And here I will remove all [188:26] of these and I'll paste this here. And [188:29] then I will save this. And I want to [188:33] show this to you what we are doing. So [188:35] we are saying the workspace name here. [188:38] We are echoing the workspace. The build [188:40] we are echoing the job name along with [188:42] the build number. Then we are echoing [188:45] the node on which this job ran. So I [188:48] will click on save and then I will click [188:50] on build now. And we see the build [188:53] number four has completed. I'll click on [188:55] console output. And you see here [188:58] workspace is where Jenkins home. And [189:02] where is Jenkins home? Jenkins home is [189:06] in the volume that resides in our docker [189:09] host and we have mapped that onto our [189:12] container. That's why it is said you [189:14] externalize this. So if for example this [189:17] container fails then I will map the same [189:20] volume and I will get my all jobs back [189:24] and here is the name of the job and then [189:27] you see here is the node on which this [189:29] job was run. It's a built-in node and [189:32] then the workspace is this we have [189:34] already seen. Now in the build part we [189:37] said the job name and the build number. [189:39] So the job name is intro freestyle and [189:42] the build number is fourth. You see here [189:44] it is the fourth build. Now let's see [189:47] what we want to do next. So if I scroll [189:50] down. So print env I have already shown [189:52] this to you. This is again I will let [189:55] you do this. It is echo report for this [189:57] build number into build report. So this [190:00] is like generating an artifact. We are [190:03] generating an artifact for this build. [190:06] So let's do this because I want to show [190:07] you one more thing related to this. So I [190:10] will go to the same job. You can create [190:12] a new job as well. I'll click on [190:14] configure and here I will come down. I [190:17] will paste this here. I will apply this [190:21] and then I will save this. So see what [190:24] is it is doing. It will be echoing this [190:27] a build number and then that output will [190:30] be directed to build report.ext. And [190:35] where is this? We have not specified any [190:39] absolute path. So let's see where this [190:41] file gets created. And if you are [190:43] guessing it is the workspace then you [190:45] are absolutely right. So let's click on [190:48] apply save and then I will click on [190:50] build now. And there is a fifth build. [190:53] Now if I go to the console output you [190:56] see build this this total four files are [190:59] there. Build report exists there. Now [191:02] what is there? It is here. So let's go [191:05] to our terminal. I will cd into [191:09] workspace. Then intro freestyle. I'll [191:11] press here ls and you see build-en [191:15] report is here. So if I c this file you [191:18] will see report for build 5 because this [191:22] is our build 5 and we invoked the build [191:26] 5 variable here. And if I go here you [191:28] can see that we defined here that we [191:32] want the build number for whatever build [191:36] this is and this was build five. So I [191:40] hope that is clear and there is one more [191:43] thing I want to show before I let you [191:45] go. I will click here and then let's [191:47] click on configure again but this time [191:51] what I want is I don't want anything [191:54] like this. Let's close this. Let's [191:56] delete this step. Let's go to add build [192:00] step and I will click on invoke tople [192:05] maven target. So here what we'll be [192:07] doing is Maven as we learned in our day [192:10] two is a build automation tool that is [192:13] used for Java. Here our goal is to just [192:17] know the version and version is simple [192:19] like we do here get - version it's [192:23] simple. So we get the version of git [192:25] when we do this. So here when you do MVN [192:29] that is for Maven - version then you [192:32] should see the version of Maven but my [192:34] machine does not have Maven installed [192:37] which is why I see command not found. So [192:40] if I go back here I have written version [192:43] here. Now let's do one thing. I will [192:46] click here. I will open this in new tab. [192:48] So I have the settings open in new tab. [192:50] I will go to the tools section and here. [192:54] So I'll scroll down here. I will add [192:57] Maven and it is asking what do you want [193:00] to name this. So I will call this Maven [193:03] cloud with V Josh so that it is [193:05] identifiable for me that I created this [193:08] then install automatically that is fine [193:10] and here you can choose the version of [193:13] Maven that you want to install. I will [193:15] click on apply and then I will save [193:17] this. Now this essentially will tell [193:20] Jenkins to install Maven. Again there [193:25] are two ways to do this. I could have [193:28] installed it directly on my container [193:31] because I want Maven to be there if I [193:34] want to run Maven specific commands. So [193:37] there are two ways. One is I use apt to [193:40] install it. Other way is I use tools [193:44] section here to let Jenkins do that [193:48] internally. So if you go to the GitHub [193:51] repository, I have explained both of [193:55] these methods and I would encourage you [193:57] to give them a read. So I'll come here [194:01] and then I'll click on apply save. I [194:04] want to show you one thing. I'll click [194:06] on build now and this should fail and [194:08] this has failed. Now why has this [194:10] failed? I'll show you the reason. It's [194:12] saying same Maven is not found. It's [194:14] something related to that. So let's do [194:17] one thing. I will go here and I will [194:19] click on configure again. I'll scroll [194:22] down. So I added this Maven but I did [194:26] not define which Maven's installation I [194:30] want to use. So if I click on the [194:32] drop-down, I have to choose Maven that I [194:34] just configured. So I selected that. Now [194:37] I'll click on apply and save. And if I [194:40] run this again, this should now give me [194:43] the version of Maven. So it has [194:45] succeeded. Now if I click here, console [194:47] output and you see Apache Maven 3.91 [194:53] is installed in this machine. Now you [194:56] would be thinking Wun which one should I [194:59] use? I would encourage you to use this [195:03] tools part because this allows you to [195:06] configure multiple versions of Maven and [195:09] then in your job you can call the [195:12] version that you want to use. Now you [195:16] now War Vun which one to use? So firstly [195:19] I want you to know that when you use [195:22] this method you have the flexibility to [195:25] have multiple Maven versions. So if you [195:29] go here you can click on add Maven and [195:32] add a different version of Maven and you [195:35] can call that within your job like we [195:38] did. So we named it Maven cloud with [195:40] VJOS and we called it you can name it [195:42] something else and then call it but when [195:45] it comes to your apt if you install it [195:48] on the container then you are tied to a [195:51] specific version. Now of course there [195:53] are ways around it but it increases the [195:57] complexity. So if you wish to use [195:59] multiple versions of Maven then this is [196:02] the best place to configure Maven or any [196:05] other tool that you see in the tools [196:08] list. Before I let you go I want to [196:11] cover one more quick thing. So if I go [196:13] to my diagrams Jenkins roles and [196:16] responsibilities there is Jenkins [196:19] administrator. This is the person who [196:21] will be configuring Jenkins who will be [196:23] ensuring that Jenkins is backed up. all [196:26] the plugins are up to date. The patching [196:29] of the Jenkins node and so on will be [196:31] responsibility of Jenkins administrator [196:35] and Jenkins administrator is typically [196:38] part of the DevOps teams. But who are [196:42] the owners for Jenkins jobs? Who will be [196:45] creating the pipelines in Jenkins? Now [196:48] these could also be your developers and [196:50] of course DevOps engineers are also here [196:54] and QA teams. The crux of this is your [196:59] developers may also need access to [197:01] Jenkins because they want to test and [197:04] build their pipeline. So keep these [197:07] things in mind. That is all I wanted to [197:10] cover in this lecture. That's wrap for [197:13] day three. We installed Jenkins in [197:16] Docker, reviewed practical install [197:18] choices for lab and enterprise and built [197:22] a freestyle job that printed environment [197:25] details, archived artifacts, and [197:28] validated different installation [197:30] options. You now know how Jenkins runs [197:34] commands, where state lives, and how to [197:38] make tools available the right way. Next [197:41] up in day four, we will move to [197:44] pipelines Jenkins files in Jenkins. If [197:47] you have any questions, feel free to ask [197:50] them in the comment section and I will [197:52] reply. If this helped, do consider [197:55] liking, sharing, and subscribing. It [197:58] will mean a lot to me. Thank you very [198:00] much. Hi, I am Vun Jooshi and welcome to [198:04] day four of our Jenkins basics to [198:07] production course. Today we will look [198:10] inside Jenkins home to understand what's [198:14] stored and what you should back up. We [198:18] will then improve our freestyle jobs. We [198:21] will add parameters, pick a branch from [198:24] Git, set up triggers, pull SCM and web [198:27] hooks and create a nightly job that [198:32] makes small report and cleans the [198:35] workspace. We will keep it practical, [198:38] easy to audit, easy to repeat and safe [198:43] on disk space. This lecture will build [198:47] the foundations of whatever complex [198:51] pipelines we will be creating as we [198:54] progress with this course. So let's get [198:57] started. So let me go to the terminal. [199:02] from the past lecture. We have a Jenkins [199:05] container created and we will be using [199:08] the same container for all the demos [199:11] that we'll be doing as part of this [199:14] lecture. And as you can see our Jenkins [199:17] is exposed on port 8080. So what I'll do [199:21] is I'll go to my browser and I'll just [199:24] type localhost [199:27] 8080. [199:29] And here you see our Jenkins and it did [199:32] not ask me for the credentials because I [199:35] clicked on keep me signed in. Here is [199:38] the job that we created the last time. [199:40] So what today I want to do is I first [199:43] want to explore [199:45] Jenkins home. So let's exec into our [199:48] Jenkins container. Docker exec it. [199:53] Jenkins is the name of my container and [199:56] I want to run bash. And this is not [200:00] Kubernetes. So this will not come here. [200:03] I'll press enter. And here I am in the [200:05] Jenkins container. So before we discuss [200:09] anything, let's see what exactly is the [200:14] variable Jenkins home. So let's echo [200:18] Jenkins [200:20] home. And what you see here is Jenkins [200:23] home resides in this directory. Now, [200:28] right off the bat, I want you to know [200:31] that anything that is inside this should [200:35] ideally be backed up somewhere because [200:40] even if your Jenkins instance, your [200:42] Jenkins controller crashes for XYZ [200:45] reason, it can read the data once you [200:49] reattach the previous volume. So, keep [200:53] this backed up. Now let's cd into this [200:58] directory and see what all files and [201:01] folders exist in this directory. So I'll [201:04] go here. I will press ls-lr. [201:09] We will not be discussing all of these [201:12] but we will be discussing the critical [201:15] ones. For example, let's first take the [201:19] files that I want to discuss. So [201:21] config.xml, XML. This is your global [201:25] Jenkins configuration. Then you see [201:29] Jenkins location configuration. So this [201:32] will have your Jenkins base URL. I want [201:37] you to concentrate on a few things. You [201:40] see Hudson in a lot of places. What [201:43] exactly is Hudson? Now previously [201:46] Jenkins was known as Hudson when it was [201:50] developed by Sun Micro Systemystems but [201:53] in 2011 Hudson as a project was handed [201:58] over to the community to the Jenkins [202:00] community let's say and then they [202:03] renamed it to Jenkins and since then we [202:07] have known this as Jenkins now even [202:11] today few of the code still has Hudson [202:15] in it and they have not changed it which [202:18] is why you will see Hudson mentioned in [202:22] some of the places. Now let's discuss [202:25] rest of the things you have war here. [202:29] War is a directory as we understand from [202:33] our past lectures that when we talk [202:36] about Java the packaging formats could [202:39] be earwar [202:42] and so on. So war is typically used when [202:46] you are talking about a web application [202:49] and Jenkins in itself is written in [202:53] Java. And you must be thinking but Vun [202:56] we are accessing this over HTTP. Where [203:00] is the web server? Now I want you to [203:03] know that this war has a built-in web [203:07] server and this web server typically is [203:11] Jetty. Now there is secret dot key [203:14] secret key not so secret all these [203:17] things are used for encryption and [203:21] decryption plugins will hold your [203:24] plugins users will hold the users that [203:28] are being created in your Jenkins tools [203:31] will hold tools like git maven that we [203:34] install using the manage Jenkins section [203:38] secrets as the name implies will hold [203:41] all your credentials. Then you have logs [203:44] which will hold the logs, updates, all [203:47] the updates. Most of this is [203:49] selfexplanatory. [203:51] Now there are two directories that I [203:55] want you to concentrate on. First is [203:58] jobs. Second is workspaces. Now I want [204:03] you to think of these as control plane [204:06] and data plane. And in our past lecture [204:08] we briefly discussed about it. When we [204:11] are dealing with production [204:12] implementations of Jenkins, you will [204:15] have a control plane and that control [204:17] plane will not be running any jobs that [204:20] will just be saving the critical [204:22] information. So if I talk about a [204:24] typical control pin which is also called [204:27] controller you will see that it will [204:30] have all these directories and the files [204:33] that you see here but it will not have [204:37] workspace because workspace is a scratch [204:41] space that is created to execute your [204:45] pipeline. So in production [204:47] implementations all of these you will [204:49] see are part of the control plane node [204:52] because these are holding the [204:54] information about Jenkins and is also [204:58] storing the jobs. I am saying storing [205:01] the jobs. I'm not saying executing the [205:04] jobs. The executing part is offloaded to [205:08] the data plane. And what is the data [205:11] plane? The data plane has agents. Now [205:13] these agents could be virtual machines [205:16] or containers that are running your [205:19] pipelines or that are running your jobs. [205:23] So understand jobs are created as part [205:27] of the control plane and you will see [205:29] all the jobs created as part of the [205:31] control plane and whatever execution is [205:34] performed that execution is performed in [205:38] the workspace directory. So in our [205:41] current implementation of Jenkins, we [205:44] have a merged control plane and data [205:47] plane which is why you see a workspace [205:51] directory and workspace directory is the [205:53] directory where a scratch space is [205:56] created so that our pipeline can run. So [206:00] if my pipeline or my job is creating a [206:04] container image then that container [206:06] image will be created in the workspace [206:10] directory. And since we are using a [206:13] collapsed design if we create a [206:15] container image right now as part of our [206:18] pipeline you will see that the workspace [206:21] directory will have the job name and [206:24] inside that job name you will see the [206:27] container image. But if this job were to [206:30] be executed by an agent by a data plane [206:34] that is a separate component in that [206:37] scenario you will see that that [206:40] container image is generated and created [206:43] in that agent node itself not in the [206:47] control plane. Now I want you to know [206:50] that it is imperative that you have [206:53] backed up your jobs. So all of jobs that [206:57] you create in Jenkins should be backed [206:59] up. Hence I recommended Jenkins home [207:03] should be backed up. Now I want to [207:06] discuss the control plane and data plane [207:09] in a little detail and as we progress [207:12] with this course we will see demos in [207:16] which we will be leveraging agents to [207:19] run our job. So let's go to the diagrams [207:22] and I'll scroll down and here there is a [207:25] control plane and then there is a data [207:29] plane. Control plane is also known as [207:32] controller. If I talk about data plane [207:34] there are agents. Now these agents could [207:37] be like I mentioned virtual machines or [207:40] they could be containers. Each agent can [207:44] have a single executor or they can have [207:47] multiple executors. Now understand it [207:50] this way. If my agent has a single [207:53] executor and this is something that you [207:55] define and we'll see this in detail [207:57] later. So if I have one exeutor that [208:00] implies this agent can run one job at a [208:04] time. If I have two executors that [208:06] implies this agent can be running two [208:10] jobs at the same time. So it is defining [208:13] the parallelism. Then let's talk about [208:16] the controller that is the control pane [208:19] stores. So jobs, builds, artifacts, [208:23] credentials and plugins. I should not [208:26] mention artifacts here and I'll let you [208:28] know why I'm saying this. So all these [208:31] are part of the control plane. So [208:33] whatever directories and files that you [208:35] saw in Jenkins home excluding the [208:38] workspace will typically be seen in the [208:41] controller. Orchestrates [208:44] Q should scheduleuling labels security [208:48] UI API. Now what are labels? We'll see [208:51] in a while. But understand that the [208:54] control plane is the orchestrator. Now I [208:56] want you to contrast this with [208:58] Kubernetes. In Kubernetes you had the [209:01] control plane. Control plane was the [209:03] brain of the cluster and then there is [209:05] data plane. Data plane is the one that [209:07] is running your jobs. Similarly in [209:10] Jenkins control plane is the brain and [209:13] then there is a data plane that is used [209:15] to execute your jobs. Understand in [209:19] Kubernetes as well your control plane [209:22] could also be running your app workloads [209:26] but that is not a recommended design. [209:28] Control plane should just be running [209:31] control plane component. Similarly in [209:34] Jenkins we are right now using a [209:36] collapsed design where our control plane [209:39] and data plane is handled by the [209:41] controller itself. But as we progress [209:44] we'll improve this architecture. So the [209:47] key takeaway is the control plane is the [209:51] orchestrator. So executors is equal to [209:54] zero in production. No builds here. So [209:57] nothing builds in the control plane. [209:59] Backup Jenkins home upgrades and configs [210:03] live here. Dispatches work to agents [210:07] receives results and logs. As simple as [210:11] that. Now let's understand what are [210:14] labels. I want you to know that for [210:17] certain jobs I might require an agent [210:20] that has Java installed and for certain [210:24] jobs I might require an agent that has [210:27] Python installed. And for some I might [210:30] need docker installed because I'm [210:32] creating a container image. So I can [210:34] have a vast variety of job types that [210:38] have certain dependencies on which those [210:42] jobs can run on which agents those jobs [210:46] can run and labels help you achieve [210:49] that. For example, if I have a label [210:51] associated with an agent that this agent [210:55] is supposed to run this sort of jobs, [210:57] then that label can be used when the job [211:02] demands that specific requirement. So [211:05] then let's talk about data plane run [211:08] build steps. It is understandable that [211:11] this executes your job. Provides [211:14] capacity via executors as we discussed. [211:18] Advertise capabilities with labels. Now [211:21] what capabilities? Capabilities as in do [211:23] I have Python installed? Do I have [211:25] Docker? What kind of resources I have? [211:27] Maybe you have certain labels for highMP [211:30] compute agents. Those agents are running [211:33] your strenuous jobs which run for let's [211:36] say 15 20 minutes. In that scenario, you [211:39] will label them accordingly. So, broaden [211:41] your imagination and do not just limit [211:44] yourself to think that labels are just [211:47] dealing with Python or Docker or Java is [211:51] installed or not. These labels could be [211:54] based on maybe SSD disk. This agent has [211:58] SSD installed. So, if you want to [212:00] process something extremely fast, then [212:03] use the agents with these labels. [212:07] Anyway, ephemeral or static, these [212:11] agents can be created based on demand. [212:15] For example, I need to execute three [212:17] jobs and that those three jobs require [212:19] three containers that is ephemeral based [212:22] on need or I can have few agents always [212:25] running. So that as soon as my job kicks [212:29] in, as soon as my pipeline starts, I [212:32] have agents to process that job or [212:36] pipeline. and they can scale [212:38] horizontally based on the number of jobs [212:41] that are pending. Next is keep clean. I [212:46] told you that workspace is a scratch [212:49] space created to execute the job. So [212:53] whatever artifact your job is generating [212:56] and what are these artifacts? These [212:58] artifacts could be the container image [213:00] that was generated or this could be your [213:03] packaged code. Maybe you have your Maven [213:08] acting in your build step and that Maven [213:11] is compiling first the source code into [213:14] byte code for Java and then you are [213:18] packaging that into jar format and in [213:21] that scenario jar becomes your artifact. [213:25] Once your build is done, maybe you are [213:28] creating a file and that file upon [213:31] creation is used to store the build [213:35] information. For example, what's the tag [213:37] of the container image that was created, [213:39] what's the digest, what is the timestamp [213:42] and so on. Your agent's storage could be [213:46] filling very fast if your jobs are [213:49] continuously generating artifacts and [213:51] you are not cleaning them. In that [213:54] scenario, you should have jobs that are [213:57] cleaning your workspaces. Now, I want [214:00] you to clearly know that once the [214:03] artifact is generated, you won't keep [214:06] that artifact in your agents. You will [214:10] be storing them somewhere. Say for [214:12] example, if I'm generating a container [214:15] image in my build step, then I would [214:17] want that container image to be stored [214:21] in an image repository like ECR. If I'm [214:26] generating certain packages, I would [214:29] want those packages to be stored in [214:33] let's say Nexus or JROG Artifactory. If [214:37] I'm generating certain files, maybe I [214:41] want those files to be stored in an [214:46] object storage like S3 or blob or maybe [214:50] file storage like EFS. So before [214:54] cleaning up workspaces, keep these [214:57] things in mind that certain artifacts [215:00] that are generated require permanent [215:03] storage somewhere. So you should be [215:05] pushing those artifacts before [215:08] performing a cleanup. And in today's [215:12] lecture in the last demo we will be [215:14] seeing how to do that when you are [215:16] dealing with a freestyle job. Now I hope [215:21] you have a good understanding of Jenkins [215:24] home. With this understanding let's [215:28] directly go to the demos. So I will go [215:32] to my browser. I will create a new item [215:36] and let's call this mini demos and it's [215:41] a freestyle project. I will click on [215:44] okay. Previously we did some basic stuff [215:47] but today I want this to be a little [215:51] closer to production. I'm not saying [215:53] exactly like production because of [215:55] certain reasons and as we progress with [215:57] this course we will learn those things. [215:59] But for now, I want these five demos [216:03] that we'll be doing today, mini demos to [216:06] create the base for you so that you have [216:09] an understanding that okay, Jenkins can [216:11] do this, Jenkins can do that as well. Oh [216:14] yeah, can it connect with my SCM? Yes, [216:16] it can. What will happen when it [216:18] connects and so on. So this will give [216:21] you a taste of Jenkins as a whole and [216:25] how you would be using this in [216:27] production and as we progress we'll be [216:30] creating realistic pipelines that are [216:32] used in production and what we have [216:35] learned thus far will form the base of [216:38] it. So now without further ado let's get [216:41] started. This project is parameterized. [216:45] What does parameterized mean? parameters [216:49] allow you to prompt users for one or [216:52] more inputs that will be passed into the [216:55] build is an important point here. Now [216:59] how this is important for example let's [217:01] say if I am running this pipeline for [217:06] dev then do not perform any testing and [217:10] this can be done using parameters. What [217:14] if you have three environments and you [217:18] want to deploy your artifact onto only [217:22] stage that can be achieved using [217:26] parameters. So let's see an example and [217:29] see how these work. So if I click on add [217:33] parameter there are a lot of options [217:35] available. Few of them we'll be [217:37] exploring in this lecture and few of [217:39] them as we progress with this course. So [217:43] first let's choose boolean. And what is [217:46] boolean? It can either be true or it can [217:50] be false. So what I will parameterize is [217:53] I will run tests. And here I will check [217:58] set by default. And when I check set by [218:00] default, run tests will always be true. [218:04] Again, right now we will not be [218:07] performing any actual tests. I just want [218:10] you to picture. Imagine what you could [218:14] be doing in production implementations. [218:16] When we are dealing with production, we [218:18] will do run tests and if it is yes, [218:21] there will be actual tests performed. [218:23] Maybe you are scanning your image for [218:26] vulnerabilities. Maybe you are [218:29] performing some sort of unit testing and [218:31] so on. Right now it is just flavor. As [218:36] we progress, we'll see more. So I will [218:38] create a parameter called run tests and [218:41] I will check it set by default so that [218:44] this is true by default and then I will [218:49] click on add parameter. Here I will add [218:52] choice parameter as the name implies. [218:55] This will give you choice whether it is [218:56] dev, whether it is stage, whether it is [218:58] prod or whether it is the main branch, [219:01] whether it is the dev branch and so on. [219:04] Another choice could be do you want to [219:06] deploy this now or do you want to deploy [219:09] it 10 minutes later and so on. Broaden [219:13] your imagination. So I will click on [219:15] choice parameter and my choice parameter [219:19] would be env so that my users have a [219:22] choice whether you want to run this [219:25] build for dev or you want to run this [219:27] for stage or you want to run this for [219:30] prod. That's it. Once I have done this, [219:34] I will scroll down and here we'll do the [219:38] same thing. I'll click on add build step [219:40] and I will click on execute shell. Now [219:44] again I wanted to execute something [219:48] substantial and I also did not want to [219:51] waste time typing everything for you. So [219:55] what I'm doing is I will just paste what [219:58] I have and then we will walk step by [220:02] step on what is happening here. I'm [220:05] saying echo run test is equal to run [220:08] test and based on what you have defined [220:11] true or false this will fill that [220:14] environment is equal to env. So what we [220:17] are saying here is environment is [220:20] whatever you supply while running this [220:23] pipeline or your job this will come here [220:28] then I am echoing something I'm saying [220:32] report for build number this should be [220:36] sent to a file called report env.ext [220:41] text. I want you to clearly know that we [220:44] are generating an artifact. Anything [220:47] that results from a build can be thought [220:50] of as an artifact. So here the artifact [220:53] is a file. We are saying if run tests is [220:57] equal to true then echo running tests [221:01] then you sleep for 1 second and echo. [221:03] Okay. Then we are saying if you are not [221:06] running the test for example if run [221:09] tests value is false then you echo [221:12] skipping tests as simple as that then we [221:16] are closing the if else now I will save [221:20] this okay let's see one thing before we [221:24] progress I will go to my Jenkins I will [221:27] press ls here I will cd into workspace I [221:31] will press ls now see that you do not [221:35] see anything in the workspace because I [221:38] have not yet executed it. So this job [221:42] does not need a workspace right now. [221:45] Workspace as I told you is a scratch [221:48] space which will only be there once you [221:50] run something. We have not run the job [221:52] yet. But if I go one step back and then [221:56] I cd into jobs and I press ls, you see [221:59] mini demos here because I saved the job [222:03] that I created. And once I run this, I [222:06] should see the workspace directory also [222:10] has something. So let's build with [222:13] parameters. Previously, you only saw [222:16] build here. As soon as we parameterized [222:19] this job, you see build with parameters. [222:21] Let's click here and you see there is a [222:25] default tick in run test because we set [222:29] to the default and default is true when [222:32] it comes to running tests in our [222:35] scenario. Environment we said [222:38] environment is a choice parameter which [222:40] is why if I click this I get the choice. [222:44] Now I will choose stage here. You can [222:46] choose anything that you want. And then [222:49] I can click on [clears throat] [222:50] build. And this has started building. [222:54] Let's see what happens. The build has [222:57] succeeded. That's why you see a green [222:59] tick here. I can click here and then I [223:02] can check the console output or I can [223:05] just click here and then click on [223:07] console output. But before I show you [223:09] this, I want to run ls here. You see now [223:14] workspace has something for us. This is [223:17] mini demos. And if I cd into mini demos, [223:20] you see the artifact that was generated [223:24] is here reporty- stage. Why stage? [223:27] Because we said environment. And while [223:30] running this job, we said the [223:32] environment is stage. If I cat this [223:34] file, I will see the contents that we [223:38] asked this to have. And again I want you [223:41] to clearly know if I go here if I click [223:44] on configure we are using some builtin [223:49] variables as well and you can access [223:52] those built-in variables from here. I [223:56] have just added the build number. So if [223:58] I go here and I search for build you [224:02] will see that there is a variable build [224:06] number which I have leveraged. And since [224:09] this is the first build you see here [224:12] report for build number one here another [224:17] thing I want you to understand these are [224:19] typically what you would add in your [224:22] script so that you as administrator and [224:26] rest of your folks clearly know under [224:30] which build number under which g commit [224:32] and we'll come into git in a little [224:35] while under which job this something was [224:38] executed. So if I search for job, there [224:41] are variables that you can use that are [224:44] specific to job. For example, job name, [224:47] job base name and so on. As we progress, [224:50] we'll using most of these variables in [224:54] this course. So now let's go to mini [224:57] demos and see the output of the job that [225:01] we just ran. And this is a simple [225:04] output. It is echoing whatever we said [225:08] and the job has successfully completed. [225:11] Now let's move to demo two. And what is [225:15] our demo two? In demo two we will be [225:17] using git SCM plus branch as a [225:21] parameter. So let's get started with [225:24] this. So I will use the same project [225:27] that we created. I'll use the same job [225:30] that we created and that job is mini [225:32] demos. I will just configure it as per [225:36] our requirement. So what I will do is I [225:40] will first close this boolean parameter. [225:42] I will also delete this choice [225:44] parameter. Then I will click on add [225:47] parameter. And this time I will add a [225:51] string parameter. And string will allow [225:54] you to pass a string while you are [225:57] running this. Previously we used [225:59] boolean. So for that the options are [226:01] just true or false. But for string [226:04] parameter you can supply the value [226:07] that's in your mind. Maybe you want to [226:10] supply the name of the branch. The name [226:12] of the branch is production. So you will [226:13] type production while running this [226:16] build. If you are running that build for [226:18] development, you will type development [226:21] as part of that. So here what I am [226:24] defining is a branch. Let's say I'll [226:27] write branch here. The default value for [226:31] branch that I want to keep in my build [226:34] is main. And I'll show you why we are [226:38] using this. And then I want to [226:41] concentrate on source code management. [226:46] Previously we have been using none but [226:49] this time I want this to connect to my [226:52] git system to my source control. and [226:56] which source control you should be using [226:58] in production implementations. You will [227:01] always be using private repositories. [227:04] That goes without saying your [227:06] application code will be in a private [227:10] repository. And if you choose private [227:13] repository, you will have to supply [227:15] credentials. But I do not want to [227:17] discuss anything credentials in this [227:19] lecture. Hence we'll be using public [227:22] repository to see how you can connect to [227:25] it. So I will just go to cloud with [227:28] varosh and then I will choose the [227:30] repository Jenkins basics to production. [227:34] I can simply copy the URL from here or I [227:38] can copy the URL from here. I will copy [227:41] it. I'll come here and it is asking what [227:44] is the repository URL. I will just [227:46] quickly paste it here. Since this is a [227:49] public repository, I do not have to [227:51] provide any credentials. I also want you [227:54] to know that whatever steps we are doing [227:57] in any of these demos, they are well [227:59] documented in the GitHub repository so [228:02] that you can do it along with me as I [228:06] progress with this lecture. Now here [228:08] what I will add is I will add branch [228:13] here. Now what this will do is it will [228:17] fill this branch based on whatever [228:21] branch I specify while running this job. [228:26] Important point to understand. So we [228:29] have parameterized this so that while [228:32] running the job user has the option to [228:35] supply the branch name. Now if I do not [228:37] want that I can hardcode the branch name [228:41] here itself. So I will scroll down and [228:45] then again in this execute shell step we [228:48] will type something. I'll just quickly [228:50] copy what I want the message to be here. [228:53] I will select it all and I will paste [228:56] it. Simple. I'm echoing built from [228:59] branch this. Then I'm running a command [229:02] get version. Then I'm running echo [229:04] latest commit. And then I'm printing the [229:06] git log. I only want the latest commit [229:08] here which is why I'm running get log [229:12] one line and I have added hyphen n1 [229:14] which will give me the first result [229:17] which is the latest commit and [229:19] doublepipe is true. So what is this [229:23] doublepipe? So let's spend some time [229:26] here. Okay. So if let's say you have [229:29] command a and then you have command b. [229:33] What this says is run B only if A [229:38] failed. So if A failed then B will run. [229:44] Then Vun why this is useful here. So [229:47] whenever you are running scripts what [229:49] happens by default is if you are running [229:51] any commands what happens by default is [229:54] if this command fails for example then [229:57] the upcoming command will not be run. So [230:00] let's say if this weren't here and if [230:03] this failed for some reason then this [230:06] command would not have run but we want [230:10] all the commands that are there in our [230:13] execute shell step to run which is why [230:17] we have added this true here. So even if [230:21] this fails the next command will run and [230:24] the next command is true and whenever [230:26] you are running true that will result an [230:29] exit code of zero which is what we want [230:32] that is successful. The command ran [230:34] successfully and the next command will [230:38] execute. Another thing is if we are here [230:41] I want to also discuss this. If you see [230:43] A and and B, this implies run B only if [230:50] A succeeded. So this is also you could [230:55] see in certain scripts. Now this is [230:58] important to understand which is why I [231:00] have used it deliberately because I know [231:04] for a fact that if we run this this will [231:06] be successful but you will often see [231:08] this sort of design in production [231:10] implementations as well. Now that we [231:12] have understood this, let's see what is [231:13] happening. Then I have I'm echoing [231:16] Jenkins variables are git branch that I [231:18] am using. I'm using a Jenkins variable. [231:20] This is not something that I have [231:22] defined. This is available at my [231:24] disposal. So if I go here and I search [231:28] for get branch then you see this is [231:33] defined by Jenkins. This is built-in [231:36] variable. Then get commit. This is also [231:38] a built-in variable. I'll click on save. [231:41] Then I will click on build with [231:43] parameters. As expected, it is asking me [231:46] to supply a branch name. And if I supply [231:50] a wrong branch name, let's say main with [231:52] a spell error. If I click on build, this [231:55] will fail because understand my [231:59] repository does not have any branch with [232:03] the name m a i n. It does have a branch [232:08] that has the name main but it does not [232:11] have any other branch. So ideally this [232:14] should fail and it did and it is saying [232:17] your branch does not exist something [232:19] along the same line. So let's fix this. [232:23] I will go to this and I will click on [232:26] this job and then let's say build with [232:29] parameters and this time let's keep it [232:32] main. And if I click on build this time [232:35] the job will succeed. So let's wait for [232:38] this. And if I click here, if I go to [232:41] console output, what is this? Get [232:45] version, get config, get ref barse, get [232:49] config, get checkout. [232:52] And I want you to notice one thing. [232:54] Okay, let's go to configure again. The [232:58] first command that we were running is [233:02] echo built from branch. Correct? And if [233:05] I scroll down, echo built from branch [233:09] starts from here. Whatever happened [233:12] here, [233:14] we didn't explicitly [233:16] ask for this to happen. Then why was a [233:21] checkout performed? I want you to [233:24] clearly understand that Jenkins always [233:28] performs this get checkout phase [233:31] automatically when SCM is configured. So [233:35] if I go here to my iterm and then I am [233:41] in the workspace mini demos. If I press [233:43] ls what is this? It cloned everything in [233:48] my public repository. But I did not have [233:51] a get clone step explicitly mentioned. [233:55] So understand this behavior. When you [233:58] are using SCM, there are certain steps [234:01] that are automatically performed by [234:05] default and cloning is one of them. And [234:08] if I go back here, then it is running [234:11] the commands that I specified. get [234:14] version get log one line and so on. So [234:17] if I run git log just plain git log it [234:20] will tell me everything about few of the [234:23] last commits. If you see here today I [234:27] performed one commit where I updated day [234:30] four and if I add one line this will [234:33] give me a succinct output. So if I do [234:36] this and add one line this will give me [234:39] a shorter output. it will not show me [234:42] the author date and so on. And if I want [234:46] only for a specific commit, not the [234:49] specific comet, if I want it for the [234:51] latest commit, I can do hyphen n1. It [234:54] will just give me the hash for the [234:57] latest commit. So I hope this is [234:58] understandable. This is part of get. But [235:00] I just wanted to show this to you. I [235:03] hope this is understood. Let's now do [235:05] one thing. Let's go to the next demo. [235:08] And I go to my diagrams. And the next [235:10] demo is pole SCM trigger no web hooks. [235:15] Now you must be thinking what is pole [235:18] SCM. So let's go to our browser. I will [235:22] go to configure. We'll edit the same [235:24] job. We do not need this. We the project [235:28] is not parameterized. We do not. Yes, we [235:31] do need this. But here what I will do is [235:34] I'll change this to blank so that it can [235:37] perform the build on any branch that is [235:40] available and that is typically main in [235:43] our example because if I go to my [235:45] repository the main is the only one that [235:48] exists. I will scroll down and this time [235:50] I will choose poll SCM. Now pole SCM [235:53] what this will do is it will pull the [235:57] SCM that is your git host every interval [236:01] that you specify here. Let's say if we [236:03] specify every 2 minutes then it will [236:06] pull the SCM every 2 minutes. [236:09] Understand? It will do the polling. It [236:12] will not run the job every 2 minutes. So [236:15] for example, if it pulled at 10:00 a.m., [236:18] it will pull again at 10:2. It will [236:20] again pull at 10:4 and so on. But it [236:24] will not run the pipeline every time. [236:26] But if it pulled and there was a push or [236:30] there was a commit, it will trigger. But [236:34] there must be something for it to [236:37] trigger. If there have been no commits, [236:39] then this will not trigger. It will only [236:42] trigger when it has something to do. So [236:45] in the poll SCM section, what I will do [236:48] is let me just quickly copy that. I am [236:52] copying this here. Now this is saying [236:57] run every 2 minutes. Now if you are [237:00] aware with chron expressions I have [237:03] chron tab guru opened here. If you [237:06] supply this this implies this chrome [237:10] expression will run whatever is [237:12] specified every 2 minutes. So if I go [237:16] here I have specified h as well. Now [237:19] what is h? Now this is specific to [237:22] jenkins. What this is doing is H spreads [237:26] schedules by hashing the job name. So [237:30] each job gets its own predictable minute [237:35] slot instead of everyone firing at the [237:39] same time. Now imagine if you have [237:41] multiple jobs and all those jobs are [237:44] triggered frequently. In that scenario, [237:47] you will load the Jenkins and the [237:49] GitHost at exactly the same time. You [237:53] may not want that which is why H is [237:56] used. So if I remove this H, you would [237:59] see that Jenkins will also give me a [238:02] suggestion that use this if you want [238:06] proper balancing. So I'll add age and I [238:09] will then add forward slash. Then I will [238:13] come below and here I will add a simple [238:17] logic. I'll copy and I will paste it [238:21] here. And here I'm just typing echo [238:24] pollingdriven build get log on one line. [238:26] I want to get the latest three commits. [238:31] And then again double pipe true. We [238:33] understand why we are using this. [238:35] Although there is no command that is [238:37] running after this. So I want you to [238:40] understand one more thing here. Why [238:43] would you use pole SCM when you can [238:46] simply configure web hooks? Web hooks we [238:49] will see later. What web hooks will do [238:50] is whenever there is let's say a commit [238:54] or a push or whatever you have [238:56] configured in that scenario a web hook [238:58] will be created and delivered to your [239:00] Jenkins so that your Jenkins can run the [239:03] pipeline. Understand whenever you commit [239:06] github or your gith host will deliver [239:10] something to your Jenkins a payload to [239:13] your Jenkins saying that you should [239:15] trigger the pipeline. That is one [239:17] scenario. But what we are discussing [239:19] here is here our Jenkins is pulling the [239:23] gith. Now why would you use this? You [239:25] would use this in highly secured and [239:28] governed environments where you do not [239:31] want any entity any entity that is [239:34] outside to access your Jenkins server. [239:39] So inbound you have not allowed you have [239:42] only allowed outbound to your gith host [239:46] so that your Jenkins server is the only [239:50] one that is initiating the connection. [239:53] So in secure environments you would [239:56] often see this being used or they would [240:00] have the gith host in the same subnet [240:03] itself as in completely private so that [240:07] you can also use web hooks in that [240:10] scenario. Now that we have this [240:12] understanding let's save this and then [240:15] build this now. So you see this has [240:18] succeeded and it will not run again. You [240:22] will not see seven after 2 minutes here. [240:24] Build number seven. That's because we [240:27] have not commit anything into our [240:30] repository. We do not have any changes [240:33] there. Now that we understand this, [240:36] let's move to the next demo. Demo four, [240:40] web hook trigger. Now I want you to [240:42] clearly know that I am using a container [240:45] which has a private IP address. I'm [240:48] accessing the same on localhost which of [240:52] course will not be accessible by GitHub [240:56] or any git host but I wanted to touch on [241:00] this so that you can understand this [241:02] theoretically and once we move to the [241:05] advanced demos we will see web hook [241:09] triggers as well. So I will go to my [241:11] browser and then I will configure this [241:16] job. I will scroll down and here instead [241:20] of poll SCM I will use GitHub hook [241:24] trigger for get SCM polling and here I [241:29] can just write some dummy data and that [241:33] is echo triggered by web hook branch is [241:37] git branch and you know git branch is a [241:41] defined environment variable get log n I [241:44] want to see the latest commit and then [241:47] you have true here. What I will now do [241:50] is I will take you to GitHub. So if I go [241:53] to my this repository and I go to [241:55] settings and then I go to web hooks. So [241:59] here I can configure a web hook. I can [242:01] click on add web hook. It will say me [242:04] give me the payload. So payload will [242:07] typically be https [242:10] slashward slash. Here you will have your [242:13] domain name or your public IP address [242:16] and then I will correct the spelling. It [242:18] is your and then after this I can you [242:22] know just have the GitHub web hook. [242:25] That's it. You will typically have this [242:26] here and here you would choose [242:28] application JSON and then a secret here. [242:31] Remember that use HTTPS in production [242:34] implementations and we'll see that as we [242:36] progress with this course. And here you [242:39] can choose whatever you want for the [242:42] trigger of this web hook. Which events [242:45] would you like to trigger this web hook? [242:48] Now just push events, send me everything [242:52] or let me select individual events. So [242:54] you have so many options at your [242:57] disposal. You can trigger it when a PR [242:59] was created. You can trigger it when [243:02] there are merge groups. Whatever you see [243:04] here, you can use. So here is the pull [243:06] request. This is widely used and then [243:09] you the default is just with pushes. So [243:12] for this demo let's you can choose just [243:15] the push event and I am not creating [243:17] this web hook because that will hold no [243:19] meaning right now because this path I do [243:22] not have right now but you get the idea. [243:26] So if I come here I can just save this [243:29] and then I can trigger it manually right [243:32] now. But in production implementations [243:34] whenever let's say there is a PR this [243:36] pipeline or this job will trigger [243:39] automatically. Now with this [243:42] understanding let's see what is the [243:45] console output. Whatever we asked is [243:48] displayed here and here you see the [243:51] branch name is main. That is what we are [243:54] using when we talk about our repository. [243:58] We only have one branch here. Now let's [244:01] move to demo five. And demo 5 is the one [244:05] where we'll be doing cleanup. So here [244:08] we'll be creating a job that is supposed [244:10] to clean up the workspace. So I'll go to [244:14] the browser. I will go to this job. I [244:17] will configure it. And now what I will [244:19] do is I will have a schedule. So what I [244:23] will do is I will come below and here I [244:26] will use another thing. I will use build [244:30] periodically. I'm saying you must run [244:34] this job at 1:30 a.m. every day. So if I [244:39] come here to the chron guru, I can have [244:43] one here. So this says at 1:30 you run [244:47] this job. And if I come here, I can fill [244:52] this as well. Let's do one thing. Let me [244:56] copy this. what I want to run. I want to [244:58] copy this and then I will paste it here. [245:01] So right now I am not performing any [245:04] cleanup. I just want to show this to you [245:07] for I'm just echoing nightly maintenance [245:10] on this node and the node that we are [245:12] using is called built-in. We are running [245:14] our jobs in the controller itself. Echo [245:17] workspace is this. And then I am sending [245:19] this information to report.ext. Hence [245:22] report.ext becomes our artifact. I will [245:25] save this. I will run this build now. [245:28] And if I go to my terminal now, I press [245:32] I will hit Q. And then if I do ls, I see [245:36] there is a report.ext here. And you also [245:40] see a get clone was performed. And why [245:42] was this performed? If I go to my [245:45] browser and I go to configure here, then [245:50] you will see we have not yet removed [245:53] this part. So get checkout is [245:55] automatically performed for us even [245:58] though we are not using it here but [246:00] since we had it here we selected SCM get [246:05] which is why the clone is being [246:06] performed and you see this here. So if I [246:09] cat report [246:11] you will see that I have whatever I [246:14] echoed in the the workspace for this is [246:17] where Jenkins home workspace mini demos. [246:20] Now what I want is I want to add [246:23] parameters that will clean up this [246:26] workspace. So let me do that. I will add [246:28] it here. And what we are saying is echo [246:31] backing up artifact. So what will happen [246:34] is in actual production implementations [246:37] you will have here docker push commands [246:41] or you will be pushing your artifacts or [246:44] your files into their respective places [246:46] and then you will start the cleanup. So [246:49] understanding this is critical. You [246:50] won't clean up without saving the [246:53] artifacts somewhere external or [246:57] somewhere where they are safe. So I'll [246:58] save it and then I will click on build [247:01] now. Now essentially what will happen is [247:04] it will clear the workspace and why it [247:07] will clear the workspace because we have [247:10] defined at the very last rmf [247:15] workspace. So if I go back to that [247:17] location, I press ls, I do not see [247:20] anything now because we have performed [247:23] the cleanup of the workspace. So that is [247:27] all I wanted to cover in this lecture. [247:29] That wraps up day four. You saw how [247:32] Jenkins store state and you upgraded [247:35] freestyle jobs with typed parameters, [247:38] traceable git checkouts, event chron [247:41] triggers, and safe cleanup after [247:44] archiving. You now have a solid [247:47] foundation to move from ad hoc jobs to [247:50] reliable CI. Next up in day five, we [247:54] will step into pipelines, Jenkins file, [247:57] parallel stages and promotion patterns. [248:00] If you have any questions, feel free to [248:03] ask them in the comment section and I [248:05] will reply. If this helped, do consider [248:08] liking, sharing and subscribing. It will [248:10] mean a lot to me. Thank you very much. [248:13] Hi, I am Varun Johi and welcome to day [248:16] five of our Jenkins basics to production [248:20] course. Today we will make Jenkins [248:23] actually ship an application. We will [248:26] set up Docker outside of Docker which is [248:30] also called D O D. So our Jenkins [248:34] container can use the hosts Docker [248:37] engine. We will add credentials safely [248:41] and then build, push and deploy a tiny [248:45] flask app end to end. So let's get [248:49] started. [248:50] Now on the left you see what we have and [248:55] on the right what we want. Before [248:59] explaining this diagram I want you to [249:02] understand that we have a Jenkins [249:05] container. that Jenkins container is [249:08] running commands for us and we have seen [249:11] that we were able to run commands [249:13] specific to git we were able to run [249:16] commands specific to Maven and so on. So [249:20] any commands that we were able to run [249:23] that binary was available to our Jenkins [249:27] container or we configured that binary [249:32] using tools section in manage Jenkins. [249:37] So essentially what I'm trying to convey [249:39] is whatever tools that we are using with [249:44] Jenkins those tools are somewhere [249:48] available to our Jenkins container. So [249:52] if in this lecture we want to build [249:56] images and we want to push those images [249:59] to a container registry then we must [250:03] have an OCI compliant tool in our [250:08] Jenkins container so that we can build [250:11] container images and then push them onto [250:13] container registries. For us the tool of [250:17] choice here is Docker. You could use any [250:20] other tool that can be used to create [250:23] OCI compliant container images like [250:25] builda, canico and so on. But in this [250:28] lecture, what we will do is we will [250:31] configure docker in our Jenkins so that [250:36] we can use docker to create OCI [250:40] compliant container images. Let's see [250:44] how we are going to achieve that. So on [250:48] the left you see what we have. We have a [250:52] docker host and docker host is [250:54] essentially my laptop and in that laptop [250:58] I have installed docker desktop which is [251:01] why I am able to run containers and you [251:04] see this is a Jenkins container here and [251:07] I am able to access Jenkins container [251:10] from any browser in my laptop. As soon [251:13] as I access local host on port 8080, [251:17] this is understood because I am [251:19] forwarding anything that is coming on my [251:22] local hosts port 8080 to my Jenkins [251:27] container and that's how Jenkins is [251:29] accessible and whenever I am running any [251:33] docker specific commands then docker cli [251:37] in my laptop is interacting with the [251:40] docker demon and That is how I am able [251:44] to run any docker specific command. So [251:46] if I go to my terminal here and if I run [251:50] docker ps then I see that there is a [251:54] single Jenkins container that is running [251:58] in my docker host and I know this [252:01] because I can run docker specific [252:03] commands and why I can run this because [252:05] I have docker cli and docker demon [252:09] configured in my docker host or in my [252:12] laptop because I installed docker [252:15] desktop. top. So the crux is docker CLI [252:19] interacts with docker demon to run [252:22] docker specific commands. Then what is [252:25] where run docker.sockwaron and why have [252:28] you mentioned it here? So if I talk [252:32] about a Linux machine there could be [252:34] multiple processes in that machine and [252:37] if one process wants to speak to another [252:40] process they can use something called a [252:43] socket file. Now socket file when being [252:46] used they bypass the TCP IP part. So [252:51] essentially what you do is you [252:53] communicate between these two processes [252:56] using a socket file and extension to [252:59] these socket file is sock. So [253:02] essentially when docker CLI is [253:05] interacting with docker demon and they [253:07] can be thought of as two processes and [253:09] these two processes are part of docker [253:12] desktop at a very high level and when [253:14] these two processes docker cli wants to [253:17] interact with docker demon it is using [253:21] this specific file. So this is the [253:24] architecture we have. Now what we want [253:27] to do is we want to install Docker in [253:31] our Jenkins container so that we can run [253:35] docker specific commands. Now one way to [253:38] do that is I install the entire thing [253:42] into this Jenkins container and entire [253:44] thing is I installed Docker CLI along [253:47] with the Docker demon and once I do that [253:51] I can run Docker specific commands and I [253:54] I can also run containers but I want to [253:58] take a different approach. Now if I had [254:01] installed both CLI and demon into this [254:04] Jenkins container then that architecture [254:08] is called dcker [254:12] in docker. So essentially if I would [254:16] have done that then I could create [254:18] containers within this container. That's [254:21] pretty cool. But in this lecture I do [254:24] not want to do d i n d. What I want to [254:27] do is d o d and d o d stands for docker [254:33] outside of docker. Now what happens in [254:36] this is you just install docker cli in [254:40] the jenkins container then vun if I am [254:44] creating a container image and I do not [254:46] have docker demon installed then how [254:49] would that work? this will work because [254:52] what you do is you just install this [254:56] docker CLI and you instruct this docker [254:59] CLI to communicate with the docker demon [255:03] that is installed onto the docker host. [255:07] So essentially what we are doing is [255:09] whenever I do this whenever I run this [255:12] sort of architecture then there are two [255:14] docker clis that can interact with my [255:17] docker demon. The first one is the one [255:20] that is installed in Docker host which [255:21] is hello the obvious. And then second we [255:26] have this docker CLI that we will [255:28] install onto the Jenkins container. And [255:31] if this Docker CLI runs Docker PS it [255:35] will see this output right now. And if [255:39] this Docker CLI runs Docker PS, it will [255:43] also see the same output. Why? because [255:46] they both are using the same docker [255:50] demon and docker demon you will also [255:52] hear people call it docker d. So if they [255:56] are using the same docker demon then [255:59] they will see the same output. So this [256:03] is what we are going to do. But there is [256:05] one more point that I want to cover and [256:08] that point is for this docker CLI to [256:12] communicate with this docker demon I [256:15] must be sharing this file with my [256:19] Jenkins container because docker cli by [256:23] default looks for this file when it [256:25] wants to interact with the docker demon. [256:27] If it doesn't find this file this will [256:30] not able to do that interaction. So what [256:34] we want to do in this lecture first is [256:37] we want to mount this file as a volume [256:41] onto our Jenkins container then install [256:44] Docker CLI so that this Jenkins [256:47] container can interact with the docker [256:49] demon that is installed in our docker [256:52] host. So you now clearly understand what [256:56] do is and what d is. So d in summary is [257:03] when this Jenkins container has both [257:05] docker cli and docker demon and d o is [257:09] when you just have docker cli and this [257:12] docker demon is shared between your [257:15] docker host and your Jenkins container. [257:19] As simple as that. So with this [257:23] understanding what we have to now do is [257:26] we have to delete the existing container [257:28] we have and we have to recreate this [257:32] container by adding additional volume. [257:36] So let's do this. So I'll go to my CLI. [257:39] I will first docker stop Jenkins and [257:45] then docker rm jenkins. Now we will not [257:49] have to do the installation the whole [257:52] thing again because like I mentioned [257:55] before we have externalized Jenkins home [257:58] and Jenkins home is where everything [258:01] Jenkins is available. So we will be [258:03] using the same volume that we had [258:06] before. So I'll just paste the command [258:08] here and everything that I'm doing today [258:11] or any lecture I have in this channel. [258:15] Everything is available in the GitHub [258:17] notes so that you can follow step by [258:20] step along with me. So here I'm saying [258:23] docker run-d run in the background. Name [258:25] of the container is Jenkins. Restart is [258:27] unless stopped. So if I restart my [258:30] docker then you restart this container [258:32] as well because I only want you to not [258:36] restart it if I have deliberately [258:38] stopped this container. Then I'm saying [258:40] hyphen p port put forwarding anything [258:42] that hits on port 8080 send it to 80080 [258:45] on my container. This is for my agents. [258:48] We are not using them. Here is the [258:51] important piece. So this Jenkins home [258:54] already has what we have configured thus [258:57] far. We are just mounting this onto this [259:00] path in our new container. And this is [259:03] the interesting piece. This is the [259:05] second named volume that we are [259:06] mounting. Here we are saying v run [259:10] docker.sock. Find this and map this onto [259:14] vr run docker.sock. As simple as that. [259:16] Here I'm defining the time zone and I [259:19] want to use the Jenkins image which is [259:21] the latest. So I'll press enter and this [259:24] will create the Jenkins container for [259:26] me. So if I go to my browser and I [259:31] access local host on port 8080, [259:37] then I will see my Jenkins container up [259:41] and running and the previous jobs that [259:44] were there in my Jenkins installation [259:46] are still there because all of this is [259:49] stored in Jenkins home and we used the [259:52] same volume, the named volume that was [259:55] used before. So I'll go to my terminal [259:59] again and I want to show you something. [260:02] So what we will now do is I will docker [260:05] exec it and Jenkins is the container I [260:09] want to exec into and I want a bash [260:11] here. So I am logged into my Jenkins [260:14] container. So if I run ls-l [260:17] v where run docker doss sock I press [260:22] enter you see the ownership of this file [260:25] is with root but we are logged in with [260:28] our Jenkins user I want to mention one [260:31] thing while practicing for this lecture [260:34] I already added others as readrite so [260:38] you see this is your owner then you have [260:42] group and then you have everyone else [260:44] that is others. So others also have [260:47] readrite. So for me this is okay. But if [260:50] you are running this you will have to [260:52] run chod triple 6 on this file so that [260:59] when Jenkins user tries to access this [261:03] file it can access it. So if I press [261:05] enter operation is not permitted. What I [261:08] will do is I will exit from here and [261:10] then I will do the same but this time I [261:13] will login into my Jenkins container as [261:16] user root. So I'll press enter then. And [261:19] now I want to run this command again. So [261:23] if I run this, it will not make any [261:26] difference for my specific installation. [261:28] But if I do ls-l [261:31] and I type this path here. Let me remove [261:35] this. So what I will see is this has the [261:39] others have readride and you should also [261:42] see this. So run this command in your [261:44] installation as well. So once I have [261:47] done this I will again exit from this [261:50] container. And now what I want to do is [261:55] I want to install [261:58] docker cli in my Jenkins container. So [262:02] again what I will do is I will again [262:04] login as root. I will do this and I will [262:08] clear my screen so that we have more [262:11] real estate and then I will go to the [262:14] GitHub repository of this course and [262:17] I'll scroll down install Docker CLI [262:20] inside the Jenkins container. I will [262:22] click on this and then you see this is [262:26] what you can use to install Docker CLI. [262:30] So I will just use this one here and I [262:33] will go here and I'll paste this here. [262:35] I'll press enter and this will install [262:38] Docker CLI for me. [262:43] Now we can exit this. Remember we have [262:45] given our Jenkins user access to v run [262:50] docker.sock [262:52] file. So I'll exit it and then I will [262:54] log with a normal user. And that is what [262:57] you should be doing when you do not need [262:59] root user when you do not need the [263:01] highest privilege then login using a [263:04] normal user. And that this is not just [263:07] applicable for Jenkins but as a [263:10] principle in the IT industry. Whenever [263:12] you are using Linux boxes login with the [263:15] user that has only the required [263:18] privileges. So I'll go here. I'm logged [263:20] in as Jenkins. Now I'll clear my screen. [263:23] And if I run docker version now, voila, [263:26] I see docker version is 28.4.0. [263:30] And this is the build. And why I can run [263:32] this command? Because I installed docker [263:37] cli. And that docker cla is interacting [263:40] with the docker demon installed in my [263:43] docker host. So if I run this command in [263:45] my terminal dockery- version, then you [263:49] will see the same output. You are seeing [263:52] the same output because both of these [263:54] are interacting with essentially the [263:57] same docker demon docker d. So the build [264:02] is also same 465 and you see 465. So I [264:05] hope this concept is understood. Now [264:08] comes the interesting thing. If I run [264:09] docker ps, I see I see the same Jenkins [264:15] container because like I told you the [264:18] docker demon is common. So now that we [264:23] have this understanding, let's see what [264:26] we have to demo in this lecture. So I [264:29] will go back to my diagrams and I will [264:32] scroll down and this is what we are [264:35] going to do. So let's see what is [264:38] happening in this diagram. We have [264:40] Jenkins at the very top. Jenkins is the [264:43] orchestrator that will ensure all of [264:46] these things are done seamlessly. All [264:49] the steps are done one by one. And [264:52] before I start with this demo, I want to [264:54] tell you one thing. I did this demo six [264:58] seven years back when I was learning [265:00] these tools and with advancements few of [265:03] the things have changed. But when I [265:06] completed this project, when I completed [265:08] this demo, I was so exuberant that wow, [265:14] I created a container automatically. The [265:17] code came from somewhere, the build [265:19] happened somewhere, the deployment [265:21] happened somewhere else. So this was a [265:24] very cool feeling for me and I hope once [265:28] you do this demo, you will have the same [265:31] feeling. So let's get started. So the [265:34] first step is what we are going to do is [265:37] we'll be creating a private repository [265:39] and we'll be using GitHub for that and [265:41] you can use any git host but we'll be [265:45] creating a private repository because [265:48] that is what you will see in production [265:50] and if you have configured private [265:52] repository then you will have to [265:53] configure credentials because that is [265:55] how Jenkins will access your repository [265:59] and what will we have in that [266:00] repository? We will have our source code [266:03] and we'll have our docker file. Once [266:07] that is done, we'll ask Jenkins to build [266:10] container image for us using the command [266:12] we know docker build. And why will [266:15] Jenkins be able to run this command? [266:17] Because we have just configured docker [266:19] in our Jenkins container. Docker cli to [266:23] be specific. Then in the next step, we [266:25] will push this image. Now understand we [266:28] will be pushing this image to docker [266:31] hub. To push this image to docker hub [266:35] jenkins must have the credentials for [266:37] docker hub. Only then it will be able to [266:40] push into the docker hub account named [266:44] cloud with vjos. So we'll be configuring [266:47] the credentials using the available [266:51] Jenkins plug-in. And why I say available [266:53] because while we were configuring [266:55] Jenkins when we were installing Jenkins, [266:57] we selected an option which said install [267:01] the recommended plugins which is why [267:02] most of the plugins that we'll be using [267:05] in this lecture are already there. And [267:09] then we will be finally once the [267:11] container image is pushed we will be [267:14] creating a docker container using that [267:18] image. So this is end to end from [267:22] creation from code to deploy. So we are [267:26] doing everything we are taking out the [267:29] source code the CI part we are building [267:32] it we are pushing it and then finally we [267:35] are deploying it and then at the very [267:38] last we'll perform a quick smoke test. [267:41] What is quick smoke test? Smoke test is [267:44] just checking whether the application is [267:46] able to perform the basic [267:47] functionalities it is supposed to [267:49] perform. For example, we will just check [267:52] whether our application is accessible [267:55] via the browser or not. So I hope you [267:59] have an idea of what we are going to do. [268:02] So let's get started. So I will go to my [268:06] browser. I will go to dashboard of [268:10] Jenkins. I will close this one and then [268:13] I will come at the very last here and [268:17] then I will create a new item and this [268:19] time as well we'll be creating a [268:21] freestyle project but do not worry in [268:23] the next lecture we'll be converting [268:26] whatever we are doing here into a [268:28] pipeline that is Jenkins file so do not [268:31] worry that stage is about to come we are [268:34] moving step by step so that we can cover [268:38] everything so let's come here and let me [268:42] name this job. So let's call this cloud [268:45] with varos and then I will add d o d and [268:50] then I will add flask because it is a [268:53] flask application that we'll be [268:54] building. So I'll be showing you the [268:56] code in a while. So for now just select [268:59] freestyle and we'll click okay. Now here [269:02] I will select git but we do not have a [269:06] repository yet. So let's create a [269:09] repository. I'll go at the very top and [269:12] then from here I will choose cloud with [269:14] varosh and then here I will choose [269:18] create a new repository here and then [269:21] new and what do I want to call my [269:24] repository I will call it cloud with [269:27] vjoy [269:30] private [269:31] repo I will not give any description [269:34] let's keep it like that this is not [269:35] public this is private so ensure you [269:38] change this cloud with VarJosh private [269:40] repo. I will click on create repository. [269:43] This will create the repository. I will [269:46] add a few files into this repository. So [269:49] what I can do is upload an existing file [269:52] here. I will click on choose your files [269:56] and then I will go to Vun Jooshi here [269:58] courses [270:00] Jenkins basics to production day five [270:04] under the Python directory. I have three [270:06] files here. I will just open them. Do [270:08] not worry what is inside them. We'll see [270:10] that in a little while. So all these [270:12] three files are being uploaded. I will [270:15] click on commit changes. And you see my [270:18] private repository has three files. Now [270:22] I have these files but I want some means [270:25] via which Jenkins can access my git [270:29] repository, my private git repository. [270:31] So I will go here. I will click on [270:35] settings here and I'll scroll down. I [270:38] will go to developer settings personal [270:41] access token pat tokens classic. Let's [270:44] use them. I already have one. This is [270:46] the one that I'm using for this courses [270:49] GitHub repository. So I'll create a new [270:51] token generate new token classic. And [270:55] here I will type Jenkins GitHub [270:59] integration so that I know what is this [271:02] used for. I will give the repo level [271:04] privileges. And let's keep the access [271:06] for 7 days. And I will click on generate [271:10] token. So here is the token that is [271:13] generated. Ensure you keep this token [271:16] somewhere safe. After this lecture [271:19] before uploading this video, I will [271:21] delete this token. So it is fine that [271:24] I'm sharing this right now. So I'll copy [271:26] this and then I will paste this token in [271:29] my browser so that I can use it later. [271:32] Then I will go here and then I will type [271:36] the repo URL first. So I'll go here and [271:39] then I will go to the repository that I [271:43] just created CWVJ private repo. I will [271:46] go to code. I will copy this. I will go [271:50] to my Jenkins now and I will paste this [271:54] URL here. And as soon as I do it, this [271:57] will give me an error that you have to [271:59] provide some credentials. And I have not [272:02] created any credentials for it. So if I [272:05] click on the drop-down, I don't see any [272:07] credentials. So I will click on add here [272:10] and go to Jenkins. This is fine global [272:13] credentials. And then it is asking me [272:15] what type of credentials do you want to [272:18] create. Now for this one, I will just [272:21] click on username and password for now. [272:24] Username is not something we have right [272:28] now. We do have but we are using PAT [272:31] personal access tokens to access our [272:34] private repository. So we can provide [272:36] any username. This will be used as a [272:38] dummy. So for this I will just write CW [272:42] VJ GitHub so that I know that I created [272:46] this username for GitHub. Then I will go [272:50] to password and in the password section [272:52] is where I have to provide this token. [272:55] So I will provide this token here and [272:58] then I have to give it some ID. So I [273:01] will call it GitHub [273:04] pat. So keep this name meaningful [273:08] because it is via ID that you will call [273:11] your credential. So give this an [273:14] identifiable name. So I will click on [273:18] add. Once I do that, I will see in the [273:21] drop-down there is cloud with VJOS - [273:24] GitHub. That was the username that I [273:26] specified. Then you see the token here. [273:28] If I click this, this should just work. [273:31] You see that error, that red error is [273:35] now vanished. So I will come down and [273:38] here we have the master branch and in [273:40] GitHub what you see a main branch. So [273:44] I'll just remove this and blank for any [273:47] we have only one branch that is main. So [273:50] this will be taking action onto that [273:52] branch. I will scroll down. I will [273:55] scroll down. Now here what I want to do [273:59] is add build step and add build step I [274:03] will use the same execute shell. But [274:06] before I do anything else let's go to [274:09] Visual Studio Code and see what are we [274:12] going to configure. So I have three [274:15] things in the Python directory. There is [274:18] app. py, there is a docker file and then [274:21] there is a requirements.ext. [274:23] So first let's explore our code. Our [274:27] code is written in Python which is using [274:30] the flask framework and this is the [274:33] framework that you can use for web apps. [274:36] When it comes to Python, it is a very [274:39] simple code which will just print the [274:41] message welcome to cloud with VJosh tip [274:44] is built with flask shipped by Jenkins [274:48] running in docker and then we are [274:51] exposing our application on port 5000. [274:54] So this is where I am defining that my [274:58] application my python application is [275:00] going to listen on port 5,000. And then [275:04] let's see the requirements.ext. [275:06] Now what happens is when you are dealing [275:08] with a production grid implementation [275:11] you mention all the packages that are [275:15] required by your application in a [275:17] different file that is called [275:19] requirements.ext [275:20] so that all the packages can be found in [275:24] one place that okay this application [275:26] requires these all packages. Now our [275:29] Python application only requires flask [275:32] which is why you only see flask here. So [275:35] let's go to the docker file and [275:37] understand what we are doing here. We [275:39] are saying from python this this is the [275:42] base image that we are using. Of course [275:45] it's a python application which is why [275:47] we are using slim python image. Then I'm [275:50] saying work diir is / app. Essentially [275:54] what we are saying is whatever commands [275:56] that I run next you run them in this [276:00] directory. It is defining the current [276:04] working directory for this. In the next [276:06] instruction that is the copy instruction [276:08] we are copying the requirement.ext [276:12] to work diir. So if I write this this is [276:15] the present working directory and which [276:18] is the present working directory it is [276:19] forward/ app. In the next instruction [276:22] that is run. We know that run is used to [276:24] install or run any commands that you [276:27] want to run. So here we are using pip. [276:30] What is pip? pip is a package manager [276:32] for python. We are saying pip install no [276:36] cache directory r and this is the [276:41] requirement.ext [276:42] from where you can fetch the packages [276:45] that I want you to install. Now I want [276:48] you to know that whenever you are [276:49] creating container images, your aim is [276:52] to have everything in that container [276:55] image that is required by your [276:57] application and nothing else. So for [277:00] example, when you use pip for installing [277:03] something, you have the download cache [277:05] and so on. So here what I'm saying is [277:08] discard the cache only install this. So [277:12] this will install flask. Then I am using [277:14] the copy instruction. Copy instruction [277:17] is used to copy the code to the /app [277:21] directory and that is what app. py and [277:23] app. py has my code and then I'm saying [277:28] expose 5,000. This is a documentary [277:31] instruction which is saying the [277:33] application that we are building listens [277:36] on port 5,000. And finally I am running [277:40] python application. I'm saying python [277:43] run app. py run the code that is in app. [277:47] py as simple as that. Then I also have a [277:50] dot docker ignore and you can see what [277:53] all things are here. So this is what my [277:57] git has. Now I want you to clearly know [278:00] that we discussed in the last lecture [278:03] that if you go to where Jenkins home and [278:07] if I do ls then whatever jobs we are [278:10] creating those jobs will be stored here [278:13] and whenever you run that job so the [278:16] build happens in a scratch space and [278:18] that scratch space is work space. So [278:22] whenever I perform this get check out [278:27] here. So for example I have written this [278:30] here. So whenever I'm using this plug-in [278:33] what this plug-in will do is it will [278:35] perform a get checkout. So it will clone [278:38] whatever is there in my repository in [278:41] the workspace. So essentially if I cd [278:44] into workspace you will soon see when I [278:47] run this there will be a new workspace [278:50] with the name whatever is the name of [278:53] this cwvj double o flask will be created [278:57] for us. Now let's come to the meteor [279:01] part that is execute shell. So I will [279:05] copy and paste commands and I will [279:08] explain you step by step. I wanted this [279:10] to be complete which is why I'm not [279:12] typing every command that will make this [279:15] lecture extremely long. So let's not do [279:17] that. So I will copy the first command [279:20] that I want to run and I will paste it [279:22] here. Now read what we are doing. Docker [279:25] login uses injected environment [279:28] variable. Understand like I told you [279:31] before we will be running docker [279:33] specific command. We'll be performing [279:35] docker push. And for that to happen [279:39] clearly what we want to do is we want to [279:42] perform a docker login. But Warun what [279:45] are these? You have not created any [279:47] credentials for docker hub yet. So let's [279:50] do that and then we'll come back to this [279:53] step again. So I will scroll up here. I [279:56] will say yeah environment. In the [279:59] environment section I will choose use [280:02] secret text or files. I'll click on this [280:05] and I'll click on add binding and here I [280:08] will be choosing username and password [280:12] separated. Okay. So the username [280:15] variable is let's type [280:19] docker hub user here and then what is [280:23] the password variable? Docker hub. If [280:25] you're not getting this just hold on [280:27] we'll look into it. And password and [280:30] here it is saying which credentials you [280:32] want to choose. Now this I am creating [280:34] for Dockerhub. But remember in the step [280:37] here we created credentials for GitHub. [280:40] We did not create credentials for Docker [280:44] Hub. So let's do that. I'll click on [280:46] this and open this in new tab. I'll go [280:48] here. I will go to credentials and here [280:52] I will click on this and then I will [280:55] click on this and here I will click on [280:58] add credentials and the username is now [281:02] here you will have to enter the username [281:06] of your specific docker hub account. So [281:09] my username in docker hub is cloud with [281:14] varosh which is why I am typing cloud [281:16] with varosh. You will have to type your [281:18] own. And I think I forgot to explain [281:21] this scope. The scope global implies the [281:24] jobs can also use these credentials. If [281:27] you keep it to system, only your Jenkins [281:31] controller that is your control plane [281:33] and your nodes that is your data plane. [281:35] Your agents can only use these [281:37] credentials. But we want our jobs to use [281:40] these credentials. Hence, I have [281:41] selected global here cloud with VJOS. [281:45] And then here I will add my password as [281:48] well. I will type the password for my [281:51] docker hub. And then ID I will give it a [281:54] ID that I want. Let's call this docker [281:56] hub because this is what I will see in [281:59] the GUI. So now cloud with varos is the [282:03] username. Here I have typed the [282:05] password. I have the ID as dockerhub. I [282:08] will click on create now. So we have the [282:10] credentials for both GitHub and [282:14] DockerHub. So let's go to our pipeline [282:16] again and then I will scroll down. And [282:20] here you see specific credentials. I do [282:23] not see that yet. So let me do one [282:25] thing. I'll just quickly I'll just copy [282:27] this one so that I have to type less and [282:29] then I will close this. I will click on [282:32] add. Add username and password [282:35] separated. So you can also have [282:36] co-joined depending on your requirement [282:39] but I want to specify user and password [282:42] separate which is why I'm choosing [282:44] separated. So here the username variable [282:47] is docker hub user and the password is [282:51] dockerhub pwd. So what we are saying is [282:54] when we are running this pipeline we [282:56] will call these using this variable. Now [282:59] I will tell you why this is helpful in a [283:02] little while. And then here which [283:04] password do I have to use? This is for [283:06] GitHub. I want to choose cloud with VJOS [283:09] here. Cloud with VJOS is my username for [283:12] Docker Hub. So I will come down now. Now [283:15] let's see what we are doing here. [283:18] Understand that when you are using this [283:21] what happens is Jenkins credential [283:24] binding injects this and this at runtime [283:30] and masks them in the console output. So [283:34] when you are using this variables you [283:36] will not be seeing what your passwords [283:40] are in your console output and that is [283:43] what you want. Imagine if you were to [283:46] type your password here. Ctrl Ctrl + V. [283:50] If you were doing this, my username for [283:53] let's say Docker Hub is Vun and then my [283:56] password is let's say VUN. Then you [283:59] would see all these in your console [284:01] output. We do not want that. Hence these [284:04] bindings, whatever binding that we just [284:07] created bindings, this will help us [284:10] redact our passwords. You will just see [284:14] asterisk when you see the console output [284:16] log. Even when I am running echo this [284:19] password, you will not see this. So [284:22] these won't be displayed. The values for [284:25] this won't be displayed. But you might [284:28] be thinking then Wun why are you using [284:32] this format? Those this format is what [284:34] is recommended by Docker. You must be [284:36] thinking why I am not using this because [284:40] I could have used the password using [284:44] this format and Vun you have just said [284:46] that the passwords are redacted. So we [284:49] can use this right and you can of course [284:52] you can use this but the recommendation [284:54] from docker is whenever you are using [284:56] any passwords use this format. So you [285:00] echo this and docker login you ask this [285:03] command to take it from standard input [285:06] and this is the standard input. So I [285:08] hope this is understood. Now let's move [285:11] to the second part. So understand this [285:15] builds first step was docker login. Now [285:19] let's see what is the second step. The [285:22] second step here is image naming. I am [285:26] creating two variables here. The first [285:30] variable is image and image is we know [285:33] this is the path and you could omit [285:35] docker.io because we are using docker to [285:39] run our containers and default image [285:42] path has docker.io already added. I [285:44] could have started this from cloud with [285:46] varosh and let's do that. Let's do that [285:49] and while you are practicing you can add [285:51] this. So I'm saying image the image [285:55] variable has the value cloud with vos [285:58] cwvj flask. So this is what the name of [286:03] the repository is and this is my account [286:06] name tag I am saying concentrate on this [286:10] thing. I am not supplying the tag [286:12] manually. I have variableized it and I [286:15] am saying use the inbuilt variable [286:18] called build hyphen number for my image [286:22] tag. So understand whenever you see your [286:25] image in docker hub let's say with tag [286:27] nine you would know this image resulted [286:31] from build number nine in your Jenkins. [286:34] So try to connect the dot and as you [286:35] progress with this course we will see [286:37] what are other ways to integrate tags [286:40] what all other methods you can use and [286:42] anyone if you are just watching this [286:44] lecture you can get the information [286:47] about built-in variables from here. So [286:49] these are all the built-in variables [286:52] that Jenkins provide you. So if I run [286:54] build hyphen number you see build number [286:57] the current build number such as 153 an [286:59] example. So when we run this for the [287:01] very first time, this pipeline, this [287:03] Jenkins job for the very first time, it [287:06] will have the build number one. So that [287:08] is image naming for you. And then the [287:10] third step is this one. I'll paste it [287:13] here. And what is the third step? We are [287:15] saying build and push exact tag plus [287:18] latest for convenience. Now understand [287:21] we are just echoing building image. Then [287:23] I'm writing docker build t. I am [287:27] creating two tags. The first tag is [287:30] image colon tag and this will take the [287:33] values from here and we have defined [287:35] these values here and then I am adding [287:38] one more tag and which is latest and why [287:41] you are doing this warun understand that [287:44] if someone tries to pull the image [287:46] without supplying any tag then by [287:50] default the latest is added and if you [287:53] do not have latest in your container [287:56] registry then that will error out. So [287:59] for convenience of other developers [288:02] whenever you are creating a container [288:04] image you add the latest tag as well so [288:08] that if someone tries to pull the image [288:11] without supplying any tag then they do [288:14] not face any error. They get whatever [288:17] you have pushed as the latest tag. So [288:20] every time you create a new version you [288:23] also create a latest tag. Now again in [288:26] some of the projects you might see that [288:28] latest tag is not at all available. So [288:32] that developers will have to provide the [288:36] exact tag that they want to fetch. And [288:38] in our previous lectures we have also [288:40] discussed in production you would see [288:43] instead of tags in place of tags right [288:46] now we are using the build number but it [288:49] is the image digest on the basis of [288:52] which the images are pulled and that's a [288:54] discussion for a different lecture. So [288:56] for now understand we are providing two [289:00] tags here. Now let's see what is the [289:03] fourth step and in this step I'm doing [289:07] few more things. Once the image is built [289:10] I'm also pushing it. So here is the [289:13] pushing command. So I'm saying docker [289:14] push this. Docker push this. And why [289:17] will I be able to push this? I will able [289:20] to push this because I've already logged [289:23] into my docker hub. This is extremely [289:25] critical for you to understand. This [289:27] push will fail if your credentials do [289:30] not work. So let's come down and add [289:33] point number four. Four is just [289:37] deploying. So let's come here and I will [289:40] add this and deploy on the docker host. [289:44] D O D echo deploying docker pull docker [289:49] rm this. Now in case you have a [289:51] container that is already running with [289:53] this name then this command will remove [289:55] it. It is for convenience. And then in [289:58] the next step, I'm saying docker run-d [290:03] name. This is the name of the container. [290:05] And then I want to expose my container [290:07] on port 5,000 in the docker host. And [290:10] that will hit on container port 5,000 [290:14] where my application is listening. So I [290:16] hope this is connecting all your dots. [290:19] So if I go back and the last step I want [290:23] to add is the fifth step is this. Let's [290:26] add this. I will save this and then [290:30] let's click on build now. So if I click [290:34] here and I go to console output, this is [290:37] showing me live what is happening. [290:40] Understand we have the git step which is [290:42] why it is performing a g check out and [290:45] see it's saying echoing echo logging [290:48] into docker hub logging into docker hub [290:51] then echo see it did not show us the [290:54] password we wrote echo dockerhub [290:58] password but it did not show the [290:59] password here which is the reason we [291:02] used the binding plug-in then it is [291:04] saying login succeeded image is equal to [291:07] cloud with var this tag is one. Why is [291:10] this one? This is one because this is [291:12] build number one and we asked it to tag [291:16] our images based on the build number. [291:18] And if I scroll down, it is running [291:21] docker build- t it is building that [291:24] image. Then if I scroll down, you see [291:27] all this process when you create docker [291:29] images and then you create a container [291:33] out of that. So all of these things you [291:35] will see here and see docker push. Now [291:38] we are pushing. So that implies our [291:40] login was successful. It is pushing and [291:43] then it says docker push this. If I [291:46] scroll down layer already exists. Scroll [291:49] down. Then it is finally deploying step [291:52] deploying docker pull cloud with vjo [291:54] this pulling from this. And if I scroll [291:57] down docker remove this this it says [292:00] error response from damon. no such [292:02] container because container with this [292:04] name was not there in my docker [292:08] installation right now. So then I'm [292:10] finally running this container and I [292:14] added a sleep 2 seconds. Now it is [292:16] saying hit http localhost 5000 to see [292:19] the application. So I can right click on [292:21] that open link in new tab and here it is [292:25] localhost 5000. Welcome to cloud with [292:28] VJosh built with flask shipped by [292:30] Jenkins running in docker. That is how [292:34] cool Jenkins is. You have successfully [292:38] not just build an application, you have [292:41] deployed it onto docker host and this is [292:45] just the beginning. Imagine if you want [292:48] to deploy it onto Kubernetes, you just [292:50] provide the credentials and the [292:53] information about your Kubernetes [292:55] cluster and it will deploy onto [292:57] Kubernetes. So this lays the foundation [293:01] of whatever we are going to discuss. So [293:03] I hope you are watching these lectures [293:05] in sequence for everything to connect. [293:07] So now what I will do is I want to show [293:09] you a few more things. I will go here [293:13] whatever logs we have seen. So these are [293:15] the logs. I would encourage you to see [293:17] all these logs and compare them with the [293:20] commands that we have run. So I'll go to [293:22] my CLI here. I am in the workspace. I'll [293:25] hit ls. And yeah, now you see there is a [293:27] workspace for this created. So I will cd [293:30] into cloud with this directory. I'll [293:33] press ls. And you see whatever was there [293:37] in my GitHub repository is here. And if [293:42] I run docker images, you will see that [293:45] the image that we just created, we [293:47] tagged it with different tags. Tag one [293:50] and then the latest tag. You see these [293:52] here. If I go to my browser and I go to [293:56] my docker hub, I click on refresh. Here [294:01] we will see cloud with varosh [294:04] flask is here. So if I click here you [294:07] will see there are two tags which are [294:09] available latest and one and you see the [294:13] digest is the same. The digest is same [294:16] because the contents of both these tags [294:20] are the same and we discussed this in I [294:23] think day two of this series. So I hope [294:27] all of these things are making sense. [294:28] Now what I want to do is I want to also [294:31] clean up this directory once this is [294:35] successful. So what I can do is I can [294:38] add one more step. Uh we discussed about [294:40] cleaning up the workspace in the [294:42] previous lecture. Correct? So I want to [294:44] do just that. So I'll click on [294:46] configure. And now understand this will [294:49] also so if I append this I want to [294:52] append this cleanup step. And if I do [294:55] this, this will also create a new build. [294:59] So if I add this and I save this and I [295:01] run this again, what it will do is it [295:04] will create one more image for me. But [295:07] it will also clean up my workspace and [295:09] that is what I want for now. So let's [295:12] click on save and see what it is doing. [295:14] Cleaning workspace. We saw this in the [295:16] past lecture as well. We are using a [295:18] built-in variable that is workspace and [295:20] we are just asking the current [295:22] workspace. just clean this after you are [295:24] done. So I'll save this. I will just [295:27] click on build now and this will again [295:30] build this image and this time you will [295:32] see the tag is two. So let's wait for [295:36] it. And this is the reason if I go to [295:39] configure again now I added this step [295:42] because we know that this container [295:45] would already be there but this command [295:47] will first remove that and then redeploy [295:51] the container with tag 2. So let's go [295:57] here and then let's go to the console [296:02] output here and it is deploying. It's in [296:05] the final steps and it has successfully [296:08] deployed. So interesting thing is now I [296:10] want to go to item hit ls you don't see [296:14] anything because we have cleaned up our [296:17] workspace and a few important things are [296:20] if I go one step back and I hit ls [296:24] whatever credentials I'll go one more [296:27] step back if I press ls whatever [296:30] credentials we have saved they are saved [296:33] in this file they use both encoding and [296:37] encryption so you will not see them in [296:39] plain text but know that they are saved [296:42] here. So even if you delete this Jenkins [296:45] instance you recreated you will see your [296:47] credentials are still safe because they [296:50] are part of Jenkins home. That is all I [296:54] wanted to cover in this lecture. So that [296:57] wraps up day five. You configured DOD, [297:00] used Jenkins credentials, no secrets in [297:03] the script, built a Docker image, pushed [297:06] it to private registry and deployed it [297:08] on the host using pinned tag for [297:11] traceability. You now have a clean [297:14] reproducible CI flow you can extend. [297:18] Next in day six that is project two, we [297:21] will refactor this into a pipeline which [297:24] is also known as Jenkins file. If you [297:27] have any questions, feel free to ask [297:29] them in the comment section and I will [297:31] reply. If this helped, do consider [297:34] liking, sharing, and subscribing. It [297:36] will mean a lot to me. Thank you very [297:39] much. Hi, I am Varun Jooshi and welcome [297:42] to day six of our Jenkins basics to [297:45] production course. Today we will take [297:48] the day five freestyle flow and refactor [297:52] it into a clean Jenkins file. We will [297:57] map stages, check out, build, push, [298:01] deploy, test, use environment variables [298:05] and credentials safely and add practical [298:09] touches like post actions, a deploy [298:12] info.ext artifact for traceability and [298:17] workspace cleanup. you will leave with a [298:20] production shaped pipeline that's [298:24] versioned in git and easy to evolve. So [298:27] let's get started. So before I start [298:30] this lecture, what I will do is I will [298:32] go to my GitHub notes and I want to show [298:36] you the prerequisites for this lecture. [298:39] I will open this one and then I'll go to [298:42] day six. I will use the table of [298:45] contents to go to prerequisites. I will [298:48] zoom in a little. See this is day six [298:52] where we are upgrading the project that [298:55] we did in day five and that project was [298:58] done using a freestyle job and today we [299:03] are upgrading the same job into a [299:06] Jenkins pipeline. Hence I would urge you [299:11] to do watch day five to get that [299:14] understanding. And apart from that we [299:17] also looked into docker outside of [299:20] docker and compared it with dd that is [299:25] docker in docker. So this day day five [299:28] can be seen as a cicd project that will [299:32] take you from basics to production. So I [299:35] would recommend you watch day five and [299:38] once you have done that let's get [299:41] started with day six. Now before I start [299:46] this one, I want to take you to my [299:49] diagrams and show you what we are going [299:52] to build so that it develops your [299:55] interest from the word go. So this is [300:00] what we are going to deploy. You see [300:02] Jenkins at the very top. Jenkins is our [300:05] orchestrator. I wanted this project to [300:09] be production grade. So here we'll be [300:11] leveraging a private git repo. This [300:15] private repo will be hosting our source [300:18] code and the docker file. Then we will [300:21] be building a container image container [300:24] image using docker build. After that we [300:27] will push the build image to our image [300:31] registry or container registry which [300:34] will be docker hub. Next, we'll be [300:37] deploying our container image onto a [300:41] docker host. And finally, we are going [300:44] to test our application. And this is [300:48] called a smoke test. Smoke test [300:50] typically mean that you check whether [300:53] the website is accessible and may check [300:57] the most basic functionality. So in our [301:00] scenario, we will just be trying to [301:03] access our website from the browser. So [301:07] all these steps we'll be doing via [301:10] Jenkins pipeline. I want you to know [301:13] that there are certain steps which will [301:16] warrant us to use credentials because if [301:19] you are trying to access something [301:20] external, something private, you will [301:23] have to supply credentials. In this [301:25] pipeline, we'll have to provide the [301:27] credentials for the git repository and [301:31] our git repository is hosted in the gith [301:35] host named GitHub. And then we'll need [301:38] credentials to push the container image [301:41] that we'll build out of the docker file [301:44] that's in our private repository. We'll [301:46] have to push that. And for that as well [301:49] we'll require credentials for docker hub [301:52] for the account cloud with varos. So we [301:55] will do all these things. So let's get [301:58] started. So I'll take you straight to [302:01] the genkins console that is localhost [302:05] 8080. [302:07] I'll go here. I will click on new item. [302:10] And here I'm going to name my job flask [302:14] pipeline. And once I do that, I'll click [302:17] on pipeline and I will create click on [302:20] okay. Understand that we were working [302:23] with a freestyle project until now. But [302:26] today we are going to discuss the [302:29] pipeline. Hence the item the job that [302:32] I'm creating is of type pipeline. I'll [302:36] click on okay. And you see something [302:38] similar like we saw in freestyle job. We [302:41] have these. Then we have triggers. And [302:44] then we have the interesting part. We [302:46] have the pipeline. Define your pipeline [302:49] using Groovy directly or pull it from [302:53] source control. So what it is trying to [302:56] tell you is when you are writing a [302:58] pipeline, you have two options. You can [303:01] either write your entire PI pipeline in [303:05] here in this screen or you can pull that [303:09] pipeline or you can pull that pipeline [303:12] script from SCM. Now in production [303:16] implementations you would always be [303:19] using pipeline script from SCM. You will [303:22] not be writing your pipelines here. But [303:25] since we are just learning to walk right [303:28] now not run we will start with pipeline [303:31] script and then we will convert the same [303:35] script into SCM. It's just a click [303:38] nothing more I promise. Now that we have [303:41] this basic understanding let's try to [303:45] learn the pipeline syntax. Now for that [303:49] I will take you to Visual Studio Code so [303:51] that we have proper color coding while [303:54] we practice this. I'll right click here [303:57] new file and I'm going to call this file [304:00] Jenkins file. And why I'm naming this [304:03] this way at least as of now is so that I [304:07] get proper color coding. Now you could [304:10] name this anything for practicing but as [304:13] we progress with this lecture you will [304:16] understand what is the significance of [304:19] naming something Jenkins file. With that [304:22] understanding let's write our first [304:25] pipeline. I will start with pipeline. [304:28] This is the top level block. I'll zoom [304:31] in a little. This is a top level block [304:33] that you will always see at the very [304:36] top. I'll hit one space and then I will [304:40] add a curly brace. I want you to [304:42] concentrate on the color coding of the [304:45] curly braces. Right now the curly braces [304:48] are white which implies they are [304:50] associated with the pipeline block. Next [304:54] I will write agent any. Now understand [304:59] when you are running a Jenkins job your [305:03] job must be executed somewhere and in [305:07] Jenkins architecture jobs are executed [305:10] by agents and here I'm saying use any [305:15] available agent more on this in a little [305:19] while now then I have to add what we [305:24] want to do so if I take you back to the [305:26] diagrams [305:27] What you see here is we are performing [305:30] all these steps and all these steps can [305:34] be thought of as stages. There is a [305:37] check out stage, there is a build stage, [305:39] push stage, deploy stage and test stage. [305:42] Similarly, when you are writing [305:45] pipeline, understand all of these things [305:49] will happen under the pipeline. So it [305:52] makes sense to have those things inside [305:56] the pipeline block. But you can have [306:00] that inside the pipeline block. But at a [306:03] very high level, what are these things? [306:06] These things are stages. And this is [306:09] stage one, stage two and so on. So what [306:12] I will write next is I will just write [306:16] stages here. And I will again open a [306:20] curly brace. And as soon as I did that, [306:23] you see there is a color coding. It is [306:27] pink. It tells me anything within the [306:30] stages block must be contained within [306:34] this pink curly braces. I will press [306:38] enter. And then inside stages, I will [306:42] have to mention what all stages are [306:46] there. [306:47] in my stages section. What is my stage [306:50] one? What is my stage two? And so on. So [306:53] what I'll write here is stage. I will [306:57] open a parentheses and then I will write [307:01] checkout because my first stage [307:05] is checkout. If I go here, this is my [307:10] first stage. Now that I have added my [307:13] first stage, there would be certain [307:17] steps that I need to perform within that [307:19] stage. So if something is within this [307:22] stage, then I will have to define that [307:26] something here. So what I will do is I [307:29] will come here and I will add a curly [307:32] brace again and I will press enter. Now [307:36] what is inside this stage? Inside this [307:40] stage there are certain steps that I [307:42] need to perform. So as the name implies [307:47] I will just write steps here and inside [307:51] the steps block there will be steps. So [307:54] I will have to again start a curly brace [307:57] and press enter and then I will write [308:01] what is the step that I need to perform. [308:04] So for now let's just write echo [308:07] checking out from private repo and I'll [308:13] close the single quotes. So the crux is [308:17] this is the pipeline curly brace. It [308:20] ends here. Then there is a stages curly [308:23] brace. It ends here. Then there is a [308:25] stage curly brace that ends here. Then [308:28] there is a steps curly brace that ends [308:31] here. So try to create that mental [308:35] picture. There is pipelines at the very [308:37] top then in the pipeline you will have [308:40] multiple stages and within those stages [308:43] you will have your steps. So as long as [308:47] you understand the foundational concepts [308:50] these things are a cake work for you. [308:53] Now understand that whatever that we are [308:56] doing is being done inside a pipeline. [309:00] Hence you will see anything that we do [309:03] will be written within the pipeline [309:06] block. And I want to tell you one more [309:09] thing at this stage itself because I [309:11] know you will understand that. So I will [309:14] go here. I quickly want to show you. [309:17] Yeah. So here is the pipeline block [309:20] required top level block. And right now [309:23] we have mentioned agent is any. So any [309:26] as in any available node understand in [309:29] our architecture right now there is as [309:32] we have discussed before there is a [309:33] control plane in Jenkins and then there [309:35] is a data plane. Control plane is [309:38] essentially the controller and it is the [309:40] controller where you create jobs and [309:42] then controller will ask different [309:45] agents or different data plane nodes to [309:49] take action and execute those jobs. So [309:52] jobs are created in the controller and [309:55] they are supposed to be executed in the [309:58] agent. Now and we have discussed this [310:01] before. Okay, if you are not [310:02] understanding this, watch the past [310:05] lectures. In our architecture, the [310:08] control plane and data plane is [310:10] consolidated. So we have a single [310:12] Jenkins container that is doing [310:14] everything for us both the controller [310:16] part and also the executor part. Hence I [310:20] have written agent any and this will [310:23] pick up any available agent and that [310:25] available agent is our Jenkins [310:27] controller itself. So if I go here if [310:30] you see this second bullet I have [310:32] mentioned what is the agent label [310:35] docker. Now understand in an [310:38] architecture where you have multiple [310:41] agents how do you call the agent that is [310:45] supposed to perform docker related task? [310:48] You add labels to your agents and we'll [310:52] see this later as we progress with this [310:54] course. But for now understand that you [310:58] can have multiple agents and those [311:00] agents will have labels associated with [311:03] it. And why do we need labels? For [311:05] example, if I'm executing a pipeline [311:07] that requires Docker installed, then I [311:11] will use an agent that has Docker [311:13] installed. If I'm running a Maven [311:16] related build for Java, then I will [311:18] choose an agent that has a label Maven [311:21] so that my job can be executed in that [311:25] agent because that agent has all the [311:28] prerequisites that are required by my [311:31] Maven job. As simple as that. So what I [311:35] will now do is I will get back to our [311:38] pipeline, the code of our pipeline. I [311:41] will copy this thing. I will take you [311:43] back to the browser and then let's go to [311:46] our job and here I will paste whatever [311:50] we have written right and then I will [311:53] save this and you see something [311:56] different here this time you have a [311:59] stages section as well and stages is [312:03] what we call these things all these [312:06] steps that we have performed here these [312:09] can be thought of as stages and whatever [312:11] you define in the stages section within [312:14] your Jenkins pipeline will be displayed [312:18] as stages on this screen. And how cool [312:20] is that? You can see whatever you have [312:24] done in that stages console. So let me [312:28] first build this job so that I see the [312:30] stages being populated. So it is here. [312:33] Now let's go to stages. The first build [312:36] is successful. I will click on stages [312:40] and you see how good this output is. It [312:43] is saying number one build. This only [312:47] had one stage which was check out. It [312:49] started here. It ended here. Only one [312:52] stage was there which was checkout. And [312:54] that stage was successful which is why [312:57] you see a green tick here. And if I [312:59] click on this green tick it will take me [313:02] to the part where we executed something. [313:05] And here we executed check out from [313:08] private repository. We just echoed this. [313:11] Hence you only see this step. So here is [313:14] the stage and here are the steps. Now [313:18] understand as you increase your stages [313:21] all of these will be displayed here. And [313:23] when you click on that stage you can see [313:25] the steps right here. You can see what [313:28] failed. You can see what was done as [313:31] part of that stage. So I really like [313:34] this screen. This gives me a very good [313:36] view of my entire pipeline and rest of [313:40] the things are pretty basic like we have [313:42] seen before. If I click here, I can go [313:45] to console output and see that this job [313:49] was running in a node Jenkins and then [313:52] it was finished successfully. And this [313:54] is what we echoed checking out from [313:56] private repository. Now that we have [313:59] this basic understanding, we understand [314:02] what the syntax looks like. Let's go and [314:06] add a few more stages. Understand that [314:10] if I look at it, this is stage and this [314:14] stage has steps. This stage ended here. [314:18] Understand that here is the stage is the [314:21] cyan curly brace and this cyan curly [314:23] brace ended here. And whatever stage you [314:27] create must be created within the stages [314:30] block. That is understandable. So [314:32] whatever stage you create as long as [314:35] they are inside the stages block you are [314:38] good. So I will copy this stage and then [314:41] I will add this stage here. This is [314:44] stage two, stage three, stage four, [314:48] stage five. As simple as that. But I [314:52] will have to name my stages properly. [314:55] This stage I will call build. This stage [314:59] I will call push. And then next we are [315:03] calling deploy. And after deploy we will [315:07] have test. We have test here. And now [315:10] let's also update these. I will write [315:13] this is the build step. I'll add an [315:16] exclamation mark for effects. I will [315:18] copy this. I will add this here, here [315:23] and also here. And then I will write [315:27] this is the push step. This is the [315:32] deploy step and this is the test step. [315:37] So now we have a pipeline that has five [315:41] stages. And let me zoom out a little so [315:44] that you can see everything at one [315:46] place. So we have this pipeline which [315:49] has five stages and I'm telling you [315:52] ultra honestly if you have understood [315:55] this this structure you have understood [315:58] pipelines there is nothing more I [316:01] promise rest when we add the steps that [316:05] we want to add those are just commands [316:09] and we'll walk them step by step so if [316:11] you understood this you are good with [316:13] Jenkins pipeline I will copy this and [316:16] then I will again go back there. I will [316:20] go to this job. I will click on [316:22] configure and here I will paste whatever [316:26] I have. I will click on save and I will [316:31] build it. Now the job has completed [316:33] successfully. But this time the stages [316:36] would look pretty nice. You see build [316:39] number one is here. Build number two is [316:42] here. And this is our build two. So if I [316:44] click here, it will take me to this [316:47] awesome view. I can see all my stages. [316:50] If I want to see what happened in which [316:52] stage, I just click on that stage and it [316:55] will tell me this is the build step, [316:56] push, this is the push step, deploy, [316:59] this is the deploy step and so on. This [317:02] is super cool. And then I will go here [317:05] and I'll show you the console output. [317:08] And this is also what is expected. Now [317:12] we have this understanding. So let's [317:15] start with making this pipeline really [317:19] awesome at what it does. So I'll go to [317:22] Jenkins basics to production repository. [317:25] I will go to day five because day five [317:28] has the commands that we ran the last [317:31] time. So I want to use the same commands [317:33] because we are doing the same thing. We [317:35] are just doing it with the pipeline. And [317:38] I also want to explain you the [317:41] application that we are deploying once [317:42] more. So that if someone is watching [317:44] only this lecture they have this [317:47] understanding. So first I will copy this [317:50] quickly and then I will go to Visual [317:53] Studio Code. I will just create a new [317:57] file here and I will call it test.yamel. [318:00] I have the YAML extension so that the [318:04] cosmetics are good. Nothing else. I'll [318:06] just paste it here and then I will right [318:09] click on it and I'll write split to [318:11] write. But before that I wanted to [318:14] explain the code that we have with us. [318:16] Just give me a moment. And if I let's [318:20] rename it to something else okay just [318:23] test because I don't like this red. [318:27] Anyway, so now let's understand what is [318:31] happening in our code. So for anyone who [318:33] does not know what we are deploying. So [318:35] this is app. py. In app. py we are just [318:40] creating a simple application python [318:42] application that uses the flask [318:44] framework. And then what we are doing [318:47] here is we are just saying welcome to [318:51] cloud with varosh built with flask [318:53] shipped with Jenkins running in docker [318:56] and our application once deployed will [318:59] be listening on port 5000. understand [319:02] that I have coded my application to [319:05] listen on port 5,000. Then I want to [319:07] show you the docker file. In the docker [319:10] file, I'm saying use the python 3.11 [319:13] slim image. Then I'm saying working [319:16] directory is / app. This implies that [319:20] whatever commands I'm going to run next, [319:23] all those commands must be run in this [319:27] app directory. This becomes the working [319:30] directory. Then I'm copying [319:31] requirements.ext. [319:33] What is inside requirements.ext [319:36] is the information of all the packages [319:38] that are required by my application. So [319:41] if I go to requirements.ext, [319:43] I just have flask and I'm mentioning the [319:46] version for it. Flask because flask is a [319:50] web framework in Python which is used by [319:53] web applications. I'll go to docker file [319:56] again and here I'm copying the [319:59] requirements text file. I'm copying it [320:02] in the working directory which is this [320:04] one. Then I'm saying run pip install-r [320:08] requirements.ext. [320:10] Now understand that I am saying here you [320:14] want to use requirement.ext to install [320:17] the packages and I'm mentioning here no [320:20] cache diir. This signifies that whatever [320:24] download logs or whatever anything else [320:28] apart from the flask framework that is [320:31] generated you just discard that and why [320:33] I'm discarding that because we want our [320:36] container images to be as small as [320:39] possible which is why I'm mentioning no [320:42] cache diir pip as we know is a package [320:46] manager for python then we are copying [320:49] our code app py which is here into our [320:53] /app directory. Then we are just [320:56] mentioning this for documentation [320:58] purposes. This won't actually make our [321:00] application listen on port 5,000. We are [321:03] saying expose 5,000 so that DevOps [321:06] admins would know when they look at the [321:08] docker file that this application yeah [321:11] it listens on port 5,000. Then we are [321:13] finally running our Python application [321:17] by providing a command instruction [321:19] python app.py py app. py is the name of [321:24] our file that has the code. This is a [321:28] simple code that we are deploying. So [321:29] I'll just close this and let's start [321:32] building our docker file here. I will go [321:36] to view and I will just quickly word [321:39] wrap this so that we can see everything. [321:41] Now first step is check out. understand [321:45] that we will have to provide some [321:48] credentials for this pipeline to check [321:51] out our code. So how do we provide [321:54] credentials? So let's go to our browser [321:57] and then go to our Jenkins console. I'll [322:00] go to this job and see what I am doing. [322:04] There is a pipeline syntax here. I'm [322:07] right clicking on it and I'll click on [322:09] open link in new tab. I'll go here and [322:13] this is a very cool place to be. This [322:16] will help you generate certain things [322:20] that you want in your pipeline and we [322:22] will explore this as we progress. But [322:25] you see you can generate so many things [322:27] as we progress with this lecture. We'll [322:29] be using a lot of these and let's start [322:34] this by using git git. We are providing [322:37] the credentials for git which is why I'm [322:40] using this. Now why is this useful? So [322:44] anything that you want your pipeline to [322:47] do you can have certain automated ways [322:51] via which you can embed those things [322:53] into your Jenkins pipeline. And this is [322:56] one way the snippet generator will help [322:58] you learn the pipeline script code which [323:00] can be used to define various steps. And [323:03] in this step what we would want to do is [323:06] we would want to perform a get checkout [323:08] which is why we are here to generate the [323:12] pipeline script for that. So here it is [323:15] saying what is your repository URL. So I [323:18] will go here. I will go at the very top [323:21] and here I want to go to cloud with vj. [323:25] I already have a private repository [323:28] created from the past lecture. If you do [323:30] not know how to do that, watch the [323:32] previous lecture. Otherwise, just open [323:35] your private repo. And this is my [323:37] private repository. It already has the [323:39] docker file app. pyre requirements.ext. [323:42] Okay. I will click here. I will get the [323:46] URL. I'll copy this. And then I will [323:49] come here. I will paste this URL. The [323:52] branch I want is main. Okay. And since [323:56] it is a private repository, it is giving [323:59] this because I'm not yet provided the [324:01] credentials. So I will come here. I [324:04] already have added the credentials for [324:06] GitHub. If you do not know how to do [324:09] that, go to the previous lecture and [324:11] watch that. We created a personal access [324:15] token known as PAT and then added those [324:17] credentials in our Jenkins GUI. So watch [324:20] that lecture and do that. So here I will [324:22] choose this cloud with VJOS - GitHub and [324:26] then include polling. Understand that we [324:30] are not polling any repository right [324:33] now. We'll be manually running this job. [324:35] So for now I will just unselect this and [324:38] then you see include in change log. So [324:42] when change log is true, Jenkins records [324:46] the SCM changes, any changes that you do [324:49] between this build's checkout and the [324:53] previous build checkouts for that job. [324:56] So I would recommend you to check this [324:59] so that you have all the changes with [325:02] you. And then I will simply click on [325:05] generate pipeline script. It has [325:07] generated the script for me. I just have [325:11] to copy it from here and I will go to my [325:15] visual studio code and within the step I [325:19] will remove this echo and I will add [325:22] that step and then for just cosmetics [325:25] what I will do is I will bring this URL [325:28] down so that we can see everything at [325:32] one place. I will just make it a little [325:35] bigger this one. And you see get branch [325:38] main credentials ID is this and then [325:42] poll is false like we supplied and the [325:44] URL of the repository is this. So [325:48] understand in the previous lecture if we [325:51] were writing any shell scripts to do [325:53] that with pipeline we are just using [325:56] this to provide the details about our [325:59] private repository and I'll also go to [326:02] view and word this so that we can see [326:06] everything at one place. Now that we [326:08] have done this let's add the build step. [326:11] Now what were we doing in the build [326:14] step? we were building our container [326:17] image. So what I will do is here is the [326:20] command that we use to build our [326:23] container image this one. So what I will [326:26] do is I will copy this command and I [326:29] will simply paste it here. What I will [326:32] also do is I will add sh here and then I [326:36] will close this single quote. But warun [326:40] what is this sh? Now sh tells Jenkins to [326:45] run a shell command on the agent. So it [326:47] is asking the agent that whatever I'm [326:51] giving in this step you have to execute [326:54] that using a shell. Understand when we [326:56] were doing the freestyle job the [326:58] commands that we were writing were [327:00] already in the block that was named [327:03] execute shell. So the agent was [327:07] automatically executing those commands [327:10] in the shell. But here [327:13] this is the pipeline syntax. We have to [327:15] explicitly tell the pipeline syntax that [327:18] whatever I'm typing now must be executed [327:22] in shell. [327:24] That is why we have added sh. Now [327:28] pipeline also has many built-in steps. [327:31] What are built-in steps? These are [327:33] built-in steps. Hence here we did not [327:36] have to invoke any shell. understand [327:39] that you could perform these things [327:41] using shell as well. You could provide [327:43] the credentials using shell but we do [327:46] not want to do that for things that are [327:49] built into Jenkins and hence we have [327:53] used the thing that is built into [327:56] Jenkins in this step. So I will [327:59] summarize this. SH tells Jenkins to run [328:03] a shell on the agent. Pipeline has many [328:07] built-in steps like get archive arc [328:10] artifacts. We'll see all these as we [328:12] progress. There is with credentials. But [328:15] when you need to run a CLI like docker, [328:18] you use SSH. Now understand if this were [328:22] Windows, you would be using BAT or [328:24] PowerShell, not SH. I hope this is [328:27] understood. But another thing that is [328:30] worrying me right now is image tag. All [328:35] these are variables. Where are the [328:38] values for these variables? We will [328:42] provide them. But for now, let's see [328:45] what we want to do in the push step. Now [328:47] in the push step, we have docker push [328:50] this and this. I will copy this. I will [328:53] come here. In the push step, I will [328:56] paste these here and I will make this [329:00] cosmetically fine and I will add sh here [329:04] single quote single quote. Then here I [329:07] will write sh single quote single quote. [329:11] So now we understand what we are doing [329:14] here. We are pushing the image and we [329:16] have to pick these values from somewhere [329:21] and that somewhere we'll define shortly. [329:24] Let's go to the deploy step. So what are [329:26] we doing in the deploy step? These are [329:28] the things. I will copy all of these and [329:32] then in the deploy stage I will just [329:35] paste those right and then I will move [329:39] this here. I will also move this here. [329:43] And then here I will write sh single [329:46] quote single quote sh single quote [329:50] single quote sh single quote single [329:53] quote and we discussed in the last [329:56] lecture what these double pipes mean. [329:58] These mean that only run this command if [330:02] the first command fails. So we are doing [330:05] this so that our pipeline does not fail [330:07] if this does not succeed. Let's do one [330:11] more thing. We have to define the [330:13] environment variables. And where do we [330:16] define that? There are two ways to do [330:19] it. Here we are using our variables in [330:24] three stages. We are using it in the [330:26] build stage. You see image and tag here. [330:29] We are using it in the push stage. You [330:31] see image and tag here as well. And we [330:34] are also using this in the deploy stage. [330:38] So we have two options. either I can [330:41] define environment variables per stage [330:45] or I can define them for the entire [330:49] pipeline. If I define it for the entire [330:52] pipeline then all the stages can use [330:56] them. If I define it only within a stage [331:01] then only that stage will have access to [331:04] those environment variables. But here [331:07] there are three stages that will require [331:10] that environment variable. So let's [331:12] define the environment variables at the [331:15] pipeline level itself. So that all the [331:18] stages can consume them. So I will come [331:21] here. I will add one more block and this [331:25] block is called environment. [331:28] And this is the way you define [331:30] environment variables. I'll start a [331:32] curly brace as with all the steps we [331:35] have seen in the past. I'll press enter [331:38] and within this environment block I will [331:41] be defining my environment variables. So [331:45] what are my environment variables? These [331:48] are my environment variables. I will [331:51] paste them here and then I will explain [331:54] you what these are doing and we [331:57] discussed them in the previous lecture [331:58] as well. So image is my dockerhub [332:02] account is cloud with varosh and there [332:05] the repo name will be cloud with varosh [332:10] flask and what will be the tag tag will [332:14] be the build number. So we are tagging [332:17] our image with the build number. Now [332:22] let's come to the last step. Last step [332:24] is stage is test. what we want to do in [332:27] the test stage. I just want to do this. [332:31] So I will copy this and I will paste [332:34] this here and then let's come here and I [332:41] will just move it here and let's add a [332:45] semicolon and then here as well. This is [332:49] shell right? So I will write this and [332:52] then I will close this here. Now there [332:56] is one more thing that we have to do [332:58] here. I have just written push but where [333:02] are the credentials? How will this push [333:05] to docker hub if I do not provide the [333:08] credentials in this stage? So let's [333:10] provide the credentials now. So I will [333:12] go to my browser and previously we used [333:17] the pipeline syntax for git. This time I [333:21] want to use it to generate credentials [333:24] for my docker hub. So the sample step I [333:29] wish to use is called with credentials. [333:32] Here is with credentials. Bind [333:35] credentials to variables. Understand [333:38] when we were working with the GUI in the [333:40] freestyle job, we chose under the [333:44] bindings section which credential we [333:47] wish to use. So similarly here as well [333:50] you have with credentials that are used [333:52] to bind credentials to variables. What [333:54] are the variables? We will see it in a [333:57] little while. Here I'm saying with [333:59] credentials bind credentials to variable [334:02] bindings. I will add the bound binding [334:05] username and password separated and we [334:08] have seen this in the past. Here I will [334:10] write docker hub [334:14] user is the variable that I want you to [334:16] use. and then dockerhub_pwd [334:20] is the password credentials. Both of [334:23] these credentials belong to my dockerhub [334:27] account. So I can choose any. I'll [334:29] choose this one. And that's it. I have [334:32] entered the username variable is this. [334:35] The password variable is this. The [334:37] credentials that I want you to associate [334:39] are these. I'll just click on generate [334:42] pipeline. So it is saying with [334:44] credentials username password docker hub [334:48] password variable dockerhub password [334:50] username variable dockerhub user [334:54] understand that this is the credential [334:57] ID. So I will copy this and then I will [335:00] go to my pipeline code here. I will go [335:04] under the steps under the push step [335:07] because that is where we want our [335:09] credentials to be. I will just paste [335:12] that block here and understand that [335:16] whatever we are now performing those [335:20] things are to be performed within the [335:24] with credentials block because both of [335:27] these commands will be using the [335:31] credentials that are here. Understand? [335:34] Now there is one more step that I have [335:36] to do that is we'll see in a little [335:39] while but for now let's cut this from [335:43] here and add it here and it will make [335:47] much more sense to you once I add that [335:50] command as well. So I will come here and [335:53] if I scroll up here is the login [335:57] command. So I will copy this to login. I [336:00] will add this here. SH and then let's [336:05] bring them in one line. So now within [336:08] the with credentials block, I'm starting [336:11] the with credentials block with this [336:14] dark pink. And then I'm closing it. This [336:17] is purple I believe. Anyway, so I'm [336:19] starting this here and closing this [336:21] here. And all the commands that are to [336:23] be run within this with credentials [336:26] block are here. Now what we are doing we [336:29] have discussed this in the past lecture [336:31] as well. We are echoing our docker hub [336:34] password and then we are using docker [336:36] login - u. We are defining the variable. [336:39] So this is the variable docker hub user [336:42] and then we have docker hub password [336:45] which is being picked from here and then [336:48] we are performing a docker push. So what [336:50] we are doing is we have tagged this [336:53] image with two different tags. You can [336:56] see here this there is a latest tag and [336:59] then there is a tag that we have defined [337:02] as environment variable. So we are doing [337:04] these things now. That's it. We are done [337:08] with this. We don't have to do anything [337:10] else. So if I copy this entire thing, [337:14] okay, and I go here to my Jenkins [337:19] console, I will go to my pipeline. I [337:22] will go to configure and then I'll paste [337:26] that code here and let's quickly see [337:30] that we have everything. This is a tag. [337:32] Checkout stage. Yes, build is this. Push [337:36] is this. We have the credentials here [337:39] for git as well. We have everything that [337:43] we want here. We have also supplied the [337:45] credentials. Credentials ID is this. [337:48] This looks good. Push also looks good. [337:50] Deploy is also fine and in the deploy [337:55] stage we are just first we are pulling [337:57] the image then we are removing if there [338:00] is any existing container even if this [338:02] command fails we have true here so that [338:06] the next command will run and next [338:08] command is docker run this one we are [338:11] running a container in the background [338:13] and we know on which port our [338:15] application is listening and finally we [338:17] are doing the testing as well so I'll [338:19] save this I will click on build now and [338:23] our job is running and you can go to the [338:27] console output and see where it failed. [338:31] It has failed. So let's come down and [338:33] see where did it fail. Push skipped due [338:36] to earlier failure. Docker build- t this [338:40] this latest docker not found. Why is [338:44] this docker not found? Hold on. Let's go [338:49] to our terminal. Docker execit. [338:54] The container that I want to exec is [338:57] Jenkins. And I want a bash there. And [339:01] here if I write docker docker not found. [339:05] But we have oh I recreated this [339:09] container. So I did not install docker [339:12] cli yet. So let me install that. So if [339:15] you have performed the steps that were [339:17] part of day five then you should be good [339:20] otherwise just follow along with me [339:22] whatever I'm doing. So I'll go here. I [339:24] will go to cloud with var Josh here. [339:28] Then I will go to Jenkins basics to [339:31] production day five. It's a simple step [339:34] when you have documented everything. I [339:36] will scroll down and here is where I [339:40] want to install Docker CLI. To install [339:44] Docker CLI, I will have to run these [339:48] commands. I will just copy them from [339:50] here. I will go to my terminal and I'll [339:54] paste that. Some troubleshooting, right? [339:57] Failure writing output to destination. [340:00] What destination? I'll have to do this [340:02] as root user. Hold on. I'll exit. And [340:06] then here in this exact command I will [340:08] just add hyphen u root user is root I [340:12] want to login with and then I can just [340:14] paste the command. I'll press enter. And [340:17] if you have any confusions about this [340:19] what we are doing just watch the [340:21] previous lecture. Okay. Okay. Docker CLI [340:24] is installed. What I will now do is I [340:26] will exit from here and then I will [340:29] again exec and then this time I will run [340:32] docker. Okay. So now docker is [340:34] installed. So let's go to our job again [340:39] and I will come here and then let's come [340:43] here and I'll click on build now. It [340:46] will start the build again. Let's click [340:48] here and see the console output now. [340:52] Okay, this looks good now. Things are [340:55] working. It is pushing the docker image. [340:58] Let me also open hub.doccker.com. [341:03] sign in and here less than a minute ago [341:07] you see a push was just performed. So [341:10] let me zoom in a little by our Jenkins. [341:13] So I will go here and then the job has [341:17] finished successfully. I don't want to [341:19] show you the logs from here. I want to [341:22] go to the stages in pipeline. You see [341:25] how good this is. It is saying that your [341:29] third build failed. And if I click here, [341:32] you know, it will tell me what was the [341:35] problem. It is giving the error right [341:37] here. Docker not found. And we [341:39] troubleshooted that. We understood that [341:42] Docker CLI was not installed. Which is [341:45] why we are receiving this error. So if I [341:48] go to push deploy test, these stages [341:51] were not invoked from build. The [341:54] pipeline was directly ended. So this is [341:58] a good thing to do. So here itself I can [342:01] go to build two, build three, build four [342:04] is the one which was successful for us. [342:07] Now each step whatever we asked Jenkins [342:13] to do all those steps were performed in [342:16] each stage. So here you see the cloning [342:18] was done. Then I move to build. In build [342:21] stage our container image was built. If [342:25] I go to the push stage, this is where [342:27] our container image was pushed to our [342:30] image registry which was docker hub. And [342:34] in this deploy step, we deployed a [342:37] container. And you see this is the [342:39] container ID. If I go to test here, you [342:43] can see echo this. Hit this to see the [342:46] app. So if I right click on it, open [342:48] link in new tab. Voila. This is our [342:51] application. and welcome to cloud with [342:53] varos built with flask shipped with [342:55] Jenkins and running in docker. So we [342:59] have now gotten a very good [343:01] understanding of the pipeline syntax and [343:04] how do you do these kind of things using [343:08] pipeline. Now I do not want to finish [343:12] this lecture here. I want to improve [343:16] this pipeline further. So let's do that. [343:20] Firstly, [343:22] let's use I'll go to the browser and [343:26] then I'll go to this flask pipeline. [343:29] I'll click on configure. And this time [343:31] what I want to do is I want to use [343:36] pipeline script form from from SCM. And [343:40] this time I want to use pipeline script [343:43] from SCM. So I'll click here. And you [343:47] see the box where we were typing the [343:49] pipeline just vanished because we have [343:52] now said that the file that you use for [343:58] the pipeline is stored in SCM. And here [344:02] it is saying what is your source [344:04] control? I'm saying my source control is [344:07] get. Now it is asking me the URL. [344:11] understand [344:13] that by default what you will name your [344:18] pipeline file is Jenkins file. Okay. So [344:21] that is why in the script path you see [344:23] Jenkins file is populated by default and [344:29] let's keep it Jenkins file and you will [344:31] see most of the projects will do this. [344:34] Now if they have three different [344:37] pipelines for dev, prod stage then they [344:41] might name their files like Jenkins dev, [344:44] Jenkins prod and so on but Jenkins file [344:48] as a word is pretty much always there [344:52] and I'm not saying you cannot name it [344:54] anything else. I'm just saying it's a [344:57] good practice to keep Jenkins file. Now [345:00] if I move up understand that there must [345:03] be some repository where I'm storing [345:06] this Jenkins file right and I again have [345:10] two options here either I can store my [345:13] Jenkins file in the same repository [345:16] where my code and docker file is or I [345:20] can host my Jenkins file in a different [345:23] repository. Typically you will host your [345:26] Jenkins file with the application code [345:30] itself with your docker file and the [345:33] application code itself. So in this [345:34] lecture we are going to adhere to that [345:38] strategy. So here it is asking me the [345:40] repository URL. I will come here. I will [345:44] copy this URL. I will first let me close [345:48] this and then I will paste this URL [345:51] here. The credentials that I wish to use [345:53] for my GitHub are cloud with var GitHub [345:58] and then this is fine. This is fine. [346:01] This is fine. That's it. But Vonun, [346:04] where is the Jenkins file? Jenkins file [346:07] we'll have to upload in the private [346:09] repository. Right? We are saying you [346:13] Jenkins pick the Jenkins file from the [346:17] GitHub repository. So now we'll have to [346:20] create a Jenkins file and put that file [346:22] in the repository so that when we [346:24] initiate this build, Jenkins just picks [346:27] that file from our source control and [346:32] builds our project. Right? So let's [346:35] click on save here. This is now saved. [346:38] Now let's start building our Jenkins [346:42] file. Now conveniently [346:45] I have initially named this Jenkins file [346:48] so that our job gets easier. Understand [346:53] we do not need a checkout stage now [346:56] because hello we have defined whatever [346:59] we had to define for checkout in the [347:02] GUI. Remember we defined all the things [347:05] here. So let's just entirely remove this [347:09] stage and understand this stage started [347:12] with a cyan curly brace and finished [347:15] with a cyan curly brace. So just look at [347:18] the curly braces when you want to remove [347:20] something so that you do not mess up [347:24] your groovy syntax. So I will just [347:27] remove this from here and then the [347:30] checkout stage is removed. We are [347:32] starting from the build stage. Now there [347:36] are certain improvements that we want to [347:38] do in this pipeline and I want to show [347:40] you what those improvements are. The [347:42] first is use pipeline script from SCM. [347:45] We have already seen that. Now [347:47] understand you can always add post [347:50] actions in your pipeline. What happens [347:53] on success? What happens on a failure? [347:57] And what happens if there is success or [348:00] failure? You can define these things in [348:03] post actions. So what I will now do is I [348:06] will define certain post actions for my [348:10] job as well. So what I will do this time [348:13] is I will copy a certain section and I [348:16] will paste that section. So I'll come [348:18] down. So understand that we have to add [348:22] the post actions. These are the stages. [348:27] stages started here and stages ended [348:31] here. The pink curly brace follow that [348:34] we have to define post actions. These [348:37] post actions are for the pipeline [348:40] itself. We are not defining these per [348:44] stage. We are not defining this for any [348:46] stage. We are defining it for the [348:49] pipeline. Understand the distinction. [348:52] Post actions in this lecture we are [348:55] defining for the pipeline. Hence I will [348:59] have to define at the pipeline level. [349:01] And what is the pipeline level? Pipeline [349:04] level starts at this curly brace. The [349:07] white curly brace. So whatever I am [349:09] defining now should be within this white [349:12] curly brace. Not within the curly brace [349:15] of stages. It must be in the curly brace [349:19] of white of the pipeline. And hence I [349:22] will come here and I will paste that. [349:26] And now what we are saying is and this I [349:30] have just changed the way it looks. [349:32] Okay, it's the same thing. You can also [349:35] do it like this. It's just I want you to [349:38] be familiar of different ways you will [349:40] see this pipeline. So what I'm saying is [349:43] post post actions success echo build [349:46] this is succeeded failure echo build [349:50] this failed always echo build finished [349:54] this is as simple as that now you can [349:57] have various post actions you might want [349:59] to send emails and so on and as we [350:02] progress we will do those things but for [350:04] now I just want you to know that you can [350:07] have these post actions as well. So we [350:10] have added this post action. Let's go to [350:13] the diagrams and see what we want to do [350:16] next. Generate a deployin info.ext file [350:21] and archive it. What is the significance [350:24] of deployin info.ext? [350:27] I want you to know that whenever you are [350:29] building something, you want certain [350:32] information to be captured in a file and [350:35] maybe that file is used by a different [350:38] tool or by Jenkins to deploy your [350:42] artifacts. So for now I will leave it [350:46] here itself. In day two's lecture, we [350:49] understood that this file is generated [350:52] and this is kept somewhere in the object [350:55] storage or in the file storage somewhere [350:57] that Jenkins has access to it. And once [351:01] Jenkins has access to it, Jenkins can [351:04] pick the details from this file and [351:07] deploy your infrastructure. We'll see [351:10] how that works later. But for now, I [351:13] just want you to understand the [351:15] generation part and how this archiving [351:19] works. So for now, what I will do is I [351:22] will just copy this part and paste it. [351:25] Just give me a moment. I'll go to my [351:27] Jenkins file. And I want this file to be [351:31] part of the deploy stage. So what I will [351:34] do is I'll copy it and then I will come [351:38] here and then I will paste it. I've [351:41] pasted a bunch of information. So this [351:44] remains the same what we were doing. [351:47] Okay, we have not changed anything here. [351:50] What we have added is we have added this [351:54] section this. So what we are doing here [351:57] is we are creating a file that is named [351:59] deploy info and this build number and we [352:04] are saying that this file will have the [352:06] build number. It will have the image [352:08] name. It will have the commit ID, the [352:11] branch, the time and the URL. And then [352:15] in the last step we are saying that once [352:18] this file is created you archive this [352:21] file. And I'll tell you what archiving [352:23] means in a little while. But for now, [352:25] see this archived artifact. This this [352:28] this I'll tell you how to generate this. [352:31] For this as well, you just copy this [352:33] name. Whatever name you want the file to [352:36] have. And then you go to pipeline syntax [352:39] generator. Here I hold on here. I will [352:42] go to the pipeline syntax generator. I [352:46] will and this is the sample step. We [352:49] wish to use archive artifacts files to [352:51] archive. I will just put the file name [352:54] here. I will go to advanced and here I [352:57] can fill in the details. For example, I [353:00] want fingerprint all archived artifact. [353:03] So what this means is I will tell you [353:06] later but when you want to deploy the [353:08] same artifact across multiple stages and [353:10] we'll do that later in this course then [353:13] you can fingerprint all your archived [353:16] artifact and then I will simply click on [353:18] generate pipeline syntax and you see [353:21] this has generated this for us. You just [353:24] have to copy this and you paste it in [353:28] your pipeline script nothing more it's [353:31] as simple as that. Now Jenkins true [353:34] tells Jenkins to compute and store a [353:36] unique hash of fingerprint for the [353:39] archived file and that is what we have [353:41] said using fingerprint true and follow [353:44] sim links false if you go here you can [353:47] read about it by disabling this option [353:49] all symbolic links found in the [353:51] workspace will be ignored and it's all [353:54] right for us to ignore that at this [353:56] stage now let's see what is the final [353:59] improvement that we want to do clean the [354:02] workspace at the very end. And if I go [354:05] to the pipeline syntax generator, I will [354:08] just go here and here I will search for [354:12] clean workspace, delete workspace when [354:15] build is done. I will click here and [354:19] here I can go to advanced and it will [354:21] show you what all you want to do. So [354:24] clean this workspace when success [354:26] unstable failure not built aborted. In [354:29] any circumstance we are saying just [354:31] clean the workspace. I will click on [354:34] generate pipeline script and this is the [354:38] only step that you have to do for this. [354:40] So let's do one thing. If you remember [354:42] in the past lecture in the freestyle job [354:45] we cleaned the workspace as well. So [354:47] here too we will do that. So I will go [354:50] to my visual studio code and I will come [354:53] down and I want to add one more stage at [354:57] the very last and that stage will be [355:00] doing the cleanup for me. So I will come [355:03] here and then I'll come here. I will go [355:09] here and then I will add a stage. And in [355:12] this stage what I want to do is clean up [355:17] and I will close this. And then what are [355:19] the steps for this stage? I will open a [355:22] curly brace. I will write steps here and [355:25] then I will again open a curly brace. [355:28] And this is where I want that step to be [355:30] added. And I told you whatever is [355:33] builtin can be added directly. And that [355:36] is what we have done. If I scroll up [355:39] with credentials which was builtin, I [355:41] added that directly. I did not have to [355:43] write any SSH for that. So for this two, [355:46] I have just added cleanup workspace. And [355:49] once the cleanup is done, then we will [355:52] run the post action. So I'll save this [355:55] file. Now I will have to add this [355:57] Jenkins file to my SCM. So I will go to [356:00] my private repository. Here it is day [356:04] six. Oh, not day six. It is my private [356:08] repository. I will go to add file, [356:11] upload file and then I will click on [356:14] choose your file and then here I can [356:17] choose Jenkins file. I will choose this [356:20] file. It is uploading that file here and [356:22] then commit changes. It will commit the [356:25] file here. Now let's come to Jenkins and [356:30] run our pipeline. So I will go to this [356:33] flask pipeline. I will click on build [356:36] now. Oh, it has failed right away. Let's [356:38] see the console output. Oh, pardon me. [356:42] Let's go back. Let's see what the [356:44] problem is. If I scroll down, I have [356:47] this master selected and we don't have a [356:50] master branch. The branch that we have [356:53] is main. So, I will come here. I will [356:56] just remove this. I'll I can leave it [356:59] blank. It will pick any or I can just [357:01] write main here. I will save this and [357:04] let's run it again. Oh, it has failed [357:07] again. Column 9 stage. Oh, I have to add [357:11] a space. I believe space I have not [357:15] added. That's the magic of doing things [357:17] live, right? I will go here. I'll have [357:20] to first save it. Understand that I have [357:22] to upload the file again with the [357:24] changes. I could have done the changes [357:25] here itself, but anyway. So, I will [357:28] upload the file again. I'll choose the [357:31] file. And this one is the file. And then [357:34] I will click on commit changes. I'll [357:37] come here. Pipeline build. Now it has [357:40] failed again. Stage cleanup. Hold on. I [357:44] think there is something wrong with [357:45] this. Oh, I think this does not matter [357:49] having you know space or no space. [357:52] Stage. Oh. Oh. Oh. [357:55] Oh. [357:57] Do you get what the problem is? This [358:00] part the stage part should have come [358:04] after the cyan curly brace. Why? See [358:10] interesting. So you have stage test [358:12] here. Okay. Test stage started here. [358:14] This cyan curly brace finished here. So [358:17] I should have that stage here. Right? So [358:20] that was the problem. I did it wrong. [358:23] Now understand this is closing here. [358:26] Then there is a cyan curly brace. This [358:29] cyan curly brace where does this belong [358:32] to? This cyan curly brace belongs to [358:34] this. Where is this pink curly brace? [358:37] This pink curly brace belongs to the [358:40] stages itself. So our this stage the [358:44] last stage that we added the cleanup [358:46] stage must be in the stages itself. We [358:50] messed up the curly braces the last [358:52] time. I hope now it is corrected [358:55] thoroughly. So I will go to this screen, [359:00] add file, upload file, and then let's [359:04] choose the Jenkins file. I'll commit my [359:07] changes. What's wrong? I'll refresh. [359:10] Processing your file. Jenkins file is [359:13] here. This was committed now. Okay. So I [359:17] will go to my pipeline now. And then [359:20] let's build it now. Okay. This time it [359:24] is taking some time. So I'm hoping [359:26] things are falling in place. Let's go [359:30] here. Go to console output. Okay, it has [359:33] moved to this step which implies things [359:36] are steady. This console output we have [359:38] already seen multiple times. I want to [359:40] show you something different. Let's go [359:42] to stages and let's see this live or [359:45] rather click on this and let's see what [359:47] is happening. You can see all the logs [359:49] here and you see it is moving step by [359:52] step. Okay, the deploy stage has come. [359:55] We are in the deploy stage. Archived [359:57] artifact has failed. Now, why has this [360:00] failed? Doesn't match anything. No [360:03] artifacts found that match the file [360:05] pattern. Deploy configuration error. Why [360:08] is this? We are picking the artifact [360:11] from Oh, hold on. And here I have [360:16] artifact is this. And here let's run [360:20] this again because this is deploy build [360:24] number text doesn't match anything. How [360:27] can you say it does not match anything? [360:29] Let me quickly verify [360:32] because what I have done is day six has [360:36] this actual file. Okay. And we are doing [360:40] the same thing. I tested this in my lab [360:44] and it worked just fine. But I don't [360:47] have follow sim link here and I have [360:51] this in my first let me run this again [360:56] and then we'll run it without that sim [360:58] link thing. Um I don't think that will [361:01] cause a problem but I want to just check [361:03] it. I'm running this again and in the [361:06] meantime [361:08] let's edit our Jenkins file to not have [361:13] that thing. So I'll go to Jenkins file. [361:16] I'll click on edit and I'll not edit it [361:19] until my pipeline has moved to a certain [361:23] step. If I go to stages, it's nine. The [361:26] push is happening. So I will scroll down [361:29] here and then I want to remove this [361:32] thing. Fingerprint was true. Fingerprint [361:36] true. That's it. Let's see what is [361:38] happening with the job. Ninth one. The [361:41] deploy failed. Archive artifact. Okay, [361:44] this point is failing. So, let me [361:47] quickly commit these changes and if this [361:51] works, I'll have to see why this sim [361:53] link part was causing the problem. I [361:56] will now come again here and then I will [362:00] build. Now I'll go to flask pipeline. [362:03] I'll go to stages push stage. And [362:06] another thing I want to add is I could [362:09] have run my pipeline from a you know [362:13] from a stage as well. So maybe if my [362:16] file did not have a problem but problem [362:19] occurred somewhere else I could start my [362:21] pipeline from the push stage from the [362:24] deploy stage. So you have those options [362:27] and I'll show you once this build is [362:30] done. Now we are at the deploy stage. [362:33] Deploy info. Why is this happening? [362:36] Maybe it's the double quotes. Let me add [362:39] double quotes here. That's the only [362:42] thing. And then let's try to run this [362:45] again. [362:51] It was the double quotes, man. I was [362:53] using single quotes here. I had to use [362:56] double quotes. And I'll add in a video [363:00] ticker what was the actual problem. But [363:02] at least at this time double quotes [363:04] fixed the problem for us. We were using [363:06] single quotes before. So anyway, so the [363:10] text the test has succeeded. The cleanup [363:13] has happened. Workspace cleanup and post [363:16] actions. You see all these things. Build [363:20] 11 finished. Build 11 succeeded. You do [363:23] not see the failure notice because the [363:24] build has not failed. So if I go to the [363:28] previous build, build build 10 in post [363:30] actions, you would have build 10 failed [363:33] and of course the finished part. Now I [363:37] wanted to show you that right if I go to [363:39] let's say the latest build 12 11 that [363:42] has succeeded you can rerun the pipeline [363:46] from a certain stage. So restart from [363:49] stage and then you can supply from which [363:52] stage you want to restart that build. So [363:54] that is one thing that I wanted you to [363:56] know. So if you come to status, you have [363:59] this deploy info 11 and we named it to [364:03] so that it also has the build number. So [364:05] build 11 was successful. If I click here [364:08] and zoom in, you will see all the build [364:11] information. You see the first seven [364:12] digits of the commit ID. If I come here, [364:15] if I go to this private repository and [364:18] then you see the commit ID 95, it starts [364:22] with 95, ends with 33. So if I come [364:26] here, it starts with 95 and ends with [364:29] 33. The first seven digits. So you see [364:32] all the information that we added here [364:34] is available. I want to show you one [364:36] more thing. If I go to my terminal and I [364:39] go to where Jenkins home, if I go to [364:43] workspace and I press ls, you see there [364:46] is a workspace for our flask pipeline. [364:50] But that workspace will be empty because [364:53] we have performed a cleanup action as [364:56] well. So if I press ls, you don't see [364:59] anything in this workspace. Now let's do [365:02] one thing. Let's go one step back and [365:06] one more step back. If I hit ls, you [365:08] know all our jobs get stored here in [365:12] jobs directory. So if I go to this jobs [365:15] directory, press ls. I go to cd, the [365:18] name of our job was flask pipeline. If I [365:22] press ls again, I want to go to cd [365:27] builds and then ls. I want to go to cd [365:32] 111. I want to go to the build number [365:34] 11. If I press ls, I see an archive [365:38] directory here. And if I go to this [365:40] directory, you will see that this has [365:44] the file that we just generated cat [365:47] deploy do.file. Now why this is here [365:51] under under the archive directory? [365:53] Because we archived this using this step [365:57] here. We said that archive this file [366:00] which is why it was available for you. [366:03] Understand when you clean up the [366:05] workspace it is important before you do [366:08] that you make your files you make your [366:11] artifact available somewhere external [366:15] where you can use them or store them for [366:18] future use. Now I think if I go to my [366:22] browser now I go here and you see all of [366:26] and here was the fingerprint part that I [366:29] was talking about. We said fingerprint [366:31] is true. So you can view the [366:33] fingerprint. If I come here, if I click [366:35] here, it is telling me this is the [366:37] fingerprint. The file has been used in [366:39] the following places. Now understand if [366:42] this file is used again and again the [366:45] fingerprint will remain same and you can [366:47] track that history here. It is good for [366:50] traceability. Now finally let's do one [366:52] thing. Let's access our URL. I'll go to [366:56] 11 and then I will go to no let me go to [367:01] stages and then I want to go to post [367:03] actions and in post actions I not in [367:07] post actions I want to go to test and in [367:09] test I want to open this URL and you see [367:13] our app welcome to cloud with varos [367:15] built with flask shipped with Jenkins [367:17] running in docker so I think that is all [367:20] I wanted to cover that wraps up day six [367:24] you converted a working freestyle job [367:27] into a pipeline, wired in secure [367:30] credentials, produced an archived deploy [367:34] metadata and kept your workspace tidy [367:38] with cleanup. You now have a repeatable [367:41] Jenkins file foundation you can extend [367:44] with parameters, conditionals, and [367:46] parallel stages. Next time we will [367:49] iterate further with branchbased logic [367:53] and promotion patterns. If you have any [367:56] questions feel free to ask them in the [367:58] comment section and I will reply. If [368:01] this helped do consider liking, sharing [368:04] and subscribing. It will mean a lot to [368:06] me. Thank you very much. Hi, I am Vun [368:10] Jooshi and welcome to day seven of our [368:13] Jenkins basics to production course. [368:16] Today we will take our day six pipeline [368:19] and evolve it into multibranch pipeline [368:23] that autodiscovers branches and PRs [368:27] builds the right job be it the branch [368:30] job or PR job and keeps main releasable [368:35] in a trunkbased workflow. But before we [368:38] do all these demonstrations, we will [368:40] first understand what are multibranch [368:44] pipelines and why they are so powerful. [368:48] We will wire up an MVP to a private [368:52] GitHub repository, open PR, manually [368:56] rescan, and use our shared Jenkins file [369:00] to build, push, and deploy a dockerized [369:04] Python application. So without further [369:07] ado, let's get started. Multi-branch [369:11] pipelines. [369:12] It is a Jenkins job type that scans your [369:16] SCM and automatically creates a child [369:19] pipeline for every branch PR or tag. [369:24] Let's break this definition. Firstly, I [369:27] want to highlight that multibranch [369:29] pipelines are not a concept that is just [369:32] limited to Jenkins. You have these [369:36] things in GitHub actions and GitLab CI [369:39] built in. So in here in Jenkins you [369:43] create a multibranch pipeline job. But [369:46] when you talk about GitHub actions or [369:49] GitLab CI you have the liberty of [369:52] defining these things within the YAML [369:56] itself. So the concepts that we are [369:58] going to learn today are not just [370:01] limited to Jenkins. Now let's dissect [370:05] this definition. It's a Jenkins job type [370:08] that we understand that scans your SCM. [370:11] SCM is source code management and in our [370:14] scenario the SCM would be GitHub. So [370:17] what we are saying is it's a Jenkins job [370:20] type that will scan my SCM and [370:23] automatically is the keyword creates a [370:26] child pipeline for every branch PR or [370:32] tag. Now your repository can have [370:35] multiple branches. What we are saying [370:38] here is based on how many branches you [370:42] have or how many PRs are there in your [370:45] repository or what are the tags that [370:49] there are in your repository. It will [370:51] automatically create a child pipeline. [370:54] So with this definition what we [370:57] understand is multibranch pipeline is an [371:00] umbrella term and that job type may have [371:05] multiple pipelines within itself. So up [371:08] until now we were dealing with single [371:12] pipeline jobs. We had a pipeline job. We [371:15] defined the branch on which that [371:17] pipeline job will be applicable and that [371:20] pipeline job used to do its job. But [371:23] here what we are saying is we have a [371:26] multibranch [clears throat] [371:27] pipeline job that will be applicable for [371:31] all the branches or PRs or tags in your [371:35] repository. As simple as that. And all [371:39] of these things will be done by Jenkins [371:42] automatically. You do not have to define [371:45] the branches. You do not have to call [371:47] out the PRs. You do not have to call out [371:50] the tags. Now what are branches? What [371:53] are PRs? What are tags? We have seen [371:56] this before but a oneliner wouldn't [372:00] harm. So let's say we have a main [372:04] branch. In the main branch is our [372:07] primary code which is always releasable. [372:10] So now a requirement came in where we [372:13] have to introduce a loan feature in our [372:16] banking application. So what our [372:18] developers will do is they will create a [372:21] new branch. They will not touch the main [372:24] because the main is protected. They will [372:26] create a feature branch and they will [372:29] start working on this code. Understand [372:32] that when this feature branch was [372:34] created, we were on this stage. So [372:38] whatever the code was there for the [372:41] application up till this point will be [372:44] available to this feature loan branch. [372:47] Now our developers will start working on [372:52] the loan feature. That is what a branch [372:55] will do for you. It's a separate line of [372:59] work. It is your own copy of the code [373:02] where you make changes. Then what is a [373:06] PR? PR is let's say developers are now [373:10] happy with the code. They have performed [373:12] multiple pushes in the feature branch. [373:14] They have performed multiple testings [373:17] and now they are satisfied with the [373:20] entire code. So what they choose to do [373:23] is they choose to merge whatever they [373:26] have done into the main branch. And to [373:30] perform that merge they will have to [373:32] raise a PR. Why a PR Vun? Why can't I [373:36] push directly to the main branch? Often [373:39] in production, you will see main [373:41] branches or any of your release branches [373:44] are protected. You cannot directly push [373:46] onto those branches. So, what you do is [373:50] you raise a PR. PR is a pull request. It [373:53] can be thought of as a merge request. [373:56] When this PR is raised, a certain checks [373:59] are performed. And if those checks are [374:01] successful, the merge successfully [374:03] happens. And this is what you see here. [374:06] And what are tags then? Now understand [374:11] that tags are just releasable versions [374:14] of your application. For example, if my [374:17] application is running version 1.1, then [374:21] 1.1 becomes my tag because the commit [374:25] that is associated with version 1.1 is a [374:30] releasable version. Now let me explain [374:33] that to you in simple terms. For [374:35] example, let's say your main has 10 [374:39] merges. So 10 PRs that were approved. [374:42] Now each of those PR could represent a [374:46] version of your application. So tag [374:49] essentially is a version of your [374:52] application. So all those 10 comets that [374:56] you see here could represent a tag or a [375:00] handful of those comets can be thought [375:04] of as a tag. So again in simple terms [375:08] tag is just versioning version one this [375:11] tag version two this tag version three [375:14] this tag and it tags associate your [375:17] releases with the commits on which those [375:21] versions were decided or those tags were [375:25] decided. What we are saying here is when [375:29] you are dealing with a multibranch [375:31] pipeline, multibranch pipeline can [375:34] trigger a job can trigger a build based [375:38] on branches, based on PRs and based on [375:42] tags and as we progress with this [375:45] lecture we will learn more about these [375:48] things. So now that we have a highle [375:50] understanding of multibranch pipelines, [375:52] let's see these pointers. [375:55] Reposiatordriven [375:56] CI mirrors branching model autocreates [376:00] jobs per branch. So this is what we [376:03] understood. If you create 10 branches, [376:06] there will be 10 CIS that will be [376:09] created for you. There will be 10 CIS [376:11] that will run for you. And we will see [376:14] how this works in the demonstration. The [376:17] second point is Jenkins file first [376:21] builds only references containing [376:24] Jenkins file per branch or PR. So [376:27] understand this point. I could have 10 [376:30] branches. I may not want multibranch [376:33] pipelines to be running on a specific [376:35] branch. How do I do that? What I do is I [376:38] do not put any Jenkins file there. So [376:41] what I'm trying to convey to you is for [376:45] a branch to be selected for the CIA [376:49] there must be a file named as Jenkins [376:52] file. So if I have this feature loan [376:55] branch and I do not supply this Jenkins [376:57] file in this branch then Jenkins [377:01] multibranch pipelines will not be acting [377:04] on that branch. So multibranch pipeline [377:07] will always be looking for branches or [377:10] PRs or tags that have Jenkins file [377:14] available and this is the default name. [377:17] You have the liberty, you have the [377:20] option to choose a custom name as well. [377:23] But as I always say, you should be [377:26] sticking to the standards so that anyone [377:30] who looks at your work can simply judge [377:33] that yes, this is the Jenkins file. [377:36] Third pointer is fast feedback pushes [377:39] and PR updates trigger CI. Now imagine [377:43] as soon as a feature loan branch is [377:46] created locally and then as soon as it [377:49] is pushed to the remote, your CI will [377:52] run automatically because your Jenkins [377:56] job detected that there is a new branch [377:59] in the remote and how it detects how it [378:03] works is a different story altogether. [378:06] Right now I just want you to concentrate [378:09] on as soon as there is a new branch [378:13] available and that new branch has the [378:15] Jenkins file Jenkins will take action. [378:19] If you have a multibranch pipeline job [378:23] next is life cycle automation via web [378:27] hooks creates and retires job. Now it is [378:32] up to you how you want to pull your SCM. [378:36] Now maybe as soon as a new branch is [378:40] created, a web hook is delivered to [378:42] Jenkins and based on that web hook, [378:44] Jenkins will run the pipeline job on [378:47] your newly created branch. Or Jenkins [378:51] could be pulling your SCM every 15 [378:54] minutes, 60 minutes, 1 minute, 1 day and [378:58] so on based on your requirement. And if [379:01] it sees there is something new, there is [379:04] a PR or there is a branch or there is a [379:07] tag that has the Jenkins file, it will [379:10] run the CI the pipeline on that branch [379:15] PR or tag lower blast radius. And again [379:20] in life cycle automation you have web [379:22] hooks and you have polls. you can pull [379:26] the SCM and then there is a third option [379:28] that is you manually perform that scan. [379:31] So that is also there. The next point is [379:34] lower blast radius isolate builds keep [379:39] main clean and this is the beauty. So [379:42] what happens is whenever I have a [379:45] feature loan branch for example here my [379:48] job will only concentrate on this part [379:51] of the code main will always be [379:54] releasable and when I raise a PR for the [379:58] merge then we'll have new checks so that [380:02] the merge that happens here is also [380:06] releasable and the next point is scales [380:10] with teams it is obvious. So if I have [380:14] multiple teams, they can create multiple [380:16] branches and as soon as they create [380:19] their branch, as soon as they push their [380:21] branch to the remote, we know a CI will [380:24] kick in. As long as Jenkins file is [380:27] available in that branch, then [380:29] autodiscovers. We already understand it [380:32] will either pull or it will deliver a [380:36] web hook. that is autodiscovers and [380:39] prunes short-lived branches. So retires [380:43] jobs in this point and prunes [380:46] short-lived branches in this point. I [380:49] want to relate these things to you. So [380:53] understand you have a feature loan [380:56] branch. You perform your changes and you [381:00] are sure your changes work fine. Not you [381:03] are sure that there is automatic builds [381:06] that happen the pipeline that runs that [381:08] tells you that it is green your code [381:10] works you raise a PR and PR gets [381:14] approved and let's say it is merged once [381:17] that PR is merged do you really think [381:22] there is a need for this feature loan [381:24] branch no right you don't need this [381:27] branch once the work is done then you [381:30] can delete this branch because I no [381:33] longer need this. And understand as soon [381:37] as this gets deleted, my Jenkins will [381:40] know that I was working on a branch [381:44] called feature loan that had Jenkins [381:46] file, but I don't see that branch [381:48] anymore. So, let me also retire the job [381:53] that was created for that specific [381:55] branch. And that is what this means. [381:58] Retires jobs. And here prunes [382:01] short-lived branches feature. Once the [382:05] CI succeeds will be auto deleted. Now I [382:11] hope multibranch pipelines are [382:13] understood and in the demo you will [382:15] understand them more deeply. But here is [382:18] one more thing I want to mention. In [382:20] trunkbased CI/CD MVP is the glue feature [382:25] buck fix branches stay small like we saw [382:28] here. As soon as the work is done, you [382:30] run the CI and the PR gets approved and [382:33] the branch is removed. If there are [382:36] failures, then the developers will again [382:38] be working and then the same process [382:40] will repeat. But the crux is and we have [382:44] learned this in day two of our course. [382:46] When I talk about trunkbased CI/CD, I [382:50] want things to be super fast. My [382:52] developers, they push the changes, the [382:55] CI runs. If the changes are okay, the [382:58] merge happens and everything is hunky [383:00] and dory. So the failure or the success [383:04] happens frequently and fast so that the [383:08] main is always releasable. So let's read [383:12] this definition again. In trunkbased [383:14] CI/CD MBP multibranch pipeline is the [383:18] glue feature buck fix branches stay [383:21] small PRs get tested automatically and [383:25] main stays releasable. Now if you have [383:28] any doubts I would request you to go [383:30] back to day two and see what is [383:34] trunkbased CI/CD. Essentially in one [383:36] line the source of truth is the single [383:40] trunk main and then there are multiple [383:44] feature and bug fix branches that will [383:47] take over. Understand that in this [383:50] lecture we discussed so many workflows. [383:53] So if you have not watched day two, if [383:56] you are not watching this series in [383:57] sequence then if you already understand [384:00] these concepts it is all right. But if [384:02] you want to brush up, if you want to [384:04] learn maybe something new, then I would [384:06] request you to watch this series in [384:09] sequence. So in day seven, I'm back [384:12] here. So this is all about multibranch [384:15] pipeline. Now what I want to show you is [384:18] a production flow so that whatever we [384:21] have discussed in theory makes sense. So [384:24] once we discuss this production workflow [384:28] after that we will go to the demo so [384:30] that you first understand how the flow [384:33] looks like in production and then you [384:36] implement that flow with me. So the [384:38] heading for this is trunkbased CI/CD [384:42] multi- branch flow. So the diagram is [384:46] numbered. We will see what happens in [384:49] each step. So just to let you know green [384:52] color here represents the main branch [384:55] and main branch is the one that should [384:57] always be releasable and we are [385:00] discussing trunkbased CI/CD. So point [385:04] number one is let's say a requirement [385:07] came in wherein your TL was asked that [385:11] you have to change the font of the UI. [385:14] The UI is no more appealing to the [385:17] users. we have to use some sort of comic [385:21] science font or aial font to make it [385:24] beautiful. Your TL took that input and [385:28] assigned that task to a developer that [385:31] improve the UI font. So what will that [385:35] developer do? That developer will create [385:39] a new branch and understand since we are [385:43] introducing a new feature in a way we [385:46] are improving something in a way we [385:49] named it feature. If there was a problem [385:52] that we were fixing we would have called [385:55] this bug fix and then whatever bug we [385:58] were fixing. Here we are just making an [386:01] improvement. We are introducing a new [386:03] feature so to speak. So we named our [386:06] branch accordingly. We called it feature [386:08] forward slash UI font. That's step [386:12] number one. Now why did the developer [386:15] not modify this directly? Firstly, it is [386:17] not permitted because this is a [386:19] protected branch. And secondly, we all [386:22] know this is a best practice that you [386:25] create a new branch and then work on [386:27] that. This new branch whatever you see [386:30] here will hold all the code that was [386:33] available up till this point because it [386:38] is at this point that this branch was [386:41] created. Now if there is a new something [386:44] here a new comet here then this branch [386:48] will not have any information about it [386:50] because this branch was created after [386:54] this comet. Now there are ways to get [386:56] around that. You can you have features [386:59] like rebase, merge and so on. We'll see [387:02] them later but for now understand that [387:05] this branch will hold all the code up [387:07] till this point. Then step number two is [387:12] push branch job. Now let's say our [387:15] developer worked on the UI feature. Our [387:18] developer worked on locally and we all [387:21] know first the developer will work [387:23] locally and developer is now satisfied. [387:26] So now what will the developer do? [387:29] Developer will push this branch onto the [387:34] remote and this branch will already have [387:38] a Jenkins file and since this already [387:42] has a Jenkins file and the developer has [387:45] pushed it onto the remote and let's say [387:47] the remote is GitHub Jenkins will detect [387:51] that runs on each push. Now that could [387:55] be web hook or it could be a scan and [387:58] the scan could be a polebased scan or a [388:02] manual scan. Then what happens is [388:05] compile we all know what compile is. [388:09] Compilation is a process of converting [388:12] the source code to machine code and [388:15] machine code if the compiler based [388:18] languages are C, C++ or Go and to byte [388:21] code. bite code is applicable when you [388:23] are dealing with Java or C. So [388:26] essentially compilation is a process of [388:29] converting the source code into machine [388:33] or byte code. Again we have discussed [388:35] these things in detail in day one and [388:37] day two of these lectures. So do check [388:40] that out. Now only compiler based [388:43] languages will have the compilation [388:45] stage and it is not something that you [388:47] have to do manually. There are tools, [388:49] there are automation tools, there are [388:52] build tools available at your disposal [388:54] which can perform this compilation for [388:57] you. Like there is a maven for Java. [389:01] Then you have unit test. Now what are [389:03] unit tests? They run small pieces of [389:06] your code, maybe a function or a class [389:09] inside the test program. No server or [389:14] app deployment needed. So understand [389:16] this point. Unit tests can be performed [389:20] when the application is not running. [389:23] However, if I contrast this with [389:25] integration tests that we'll see later, [389:28] these tests warrants the application to [389:31] be in running state. So, unit tests can [389:34] be performed when the application is not [389:37] deployed. So, they can be performed on [389:40] your code. Now, Vun, but how will I [389:44] perform unit test? Again, I mentioned [389:47] that there are build tools available at [389:50] your disposal. These build tools, they [389:52] have certain plugins or add-ons that [389:55] will help you perform these unit tests. [389:59] And like I said before, there is Maven [390:02] in Java which you can integrate unit [390:05] test into. And then there are various [390:08] other tools like Maven will have this [390:10] JUnit CMake with Google test and there [390:14] are different tools that can perform [390:16] these unit tests. For example, JUnit is [390:19] for Java, Unity is for C, G test for [390:22] C++, G test is Google test. Then you [390:25] have pi test for Python. You have to [390:27] know these things at a very high level [390:29] at least at this stage if this is the [390:31] beginning of your CI/CD journey because [390:34] as you deal with specific projects that [390:36] deal with specific languages your [390:39] understanding will deepen because there [390:41] is no point memorizing these things if [390:44] you are not using it daily in your CI/CD [390:47] pipelines because we humans tend to [390:49] forget things that we don't work in [390:53] everyday life. So for now just take this [390:57] as a dump on onto you and once you [391:01] progress with your learnings these [391:03] things will be in your fingertips. Then [391:07] in this stage itself so first we set [391:09] compilation then unit test then there is [391:11] linting. What is linting? Linting is a [391:15] process of statically analyzing your [391:18] source code for any stylistic or [391:21] programmatic errors. Maybe there are [391:23] unused variables, there are suspicious [391:26] null checks or there is inconsistent [391:28] formatting. So linting will detect all [391:32] those things and will tell you. Then the [391:34] last one that I want to discuss is [391:36] coverage. Now let's say you have 100 [391:38] lines of code but the tool that you are [391:42] using for let's say code scanning only [391:46] scans 80 lines. So your coverage is just [391:51] 80%, and I should not use the word just [391:54] because for most of these checks to [391:57] pass, the coverage should at least be [392:00] 80%. So if you if your tool that you are [392:04] using for your code analysis, only [392:06] scanned 80 lines of your code, only [392:09] tested 80 lines of your code, then your [392:12] coverage is 80%. So code coverage is an [392:15] important parameter. A lot of pipelines [392:18] will fail if the code coverage is less [392:21] than 80%. So keep these things in mind. [392:26] Then package plus build container image. [392:29] Understand we are not deploying anything [392:33] right now. We are just performing [392:36] packaging plus building container [392:38] images. No deploy is happening. Now I [392:42] have written this based on what I have [392:46] seen in general but each pipeline is [392:49] different. Each project is different. So [392:52] do not limit your imagination. Maybe you [392:54] are performing some more tests in this [392:56] stage. What I am showing you is a [392:59] blueprint that you will often see in [393:02] production. Then you have publish [393:06] artifact metadata sbomb. [393:09] Let's dissect these terms. What is [393:11] artifact? We know anything that results [393:15] from a pipeline build is an artifact. So [393:18] if you are receiving a file, that file [393:21] is an artifact. If you are receiving a [393:23] container image, that container image is [393:26] an artifact. If you are receiving a jar [393:29] file after building a Java application, [393:32] then dot jar is your artifact. So in [393:35] this point, we are saying we will [393:37] publish the artifact. Now what is [393:39] publishing the artifact? A artifact like [393:41] container image can be published into [393:43] your image registry. An artifact like [393:47] jar can be stored in your artifactory. [393:50] Or if there is a file that resulted from [393:53] the build, maybe I have a file that [393:56] lists down the commit ID, the hash and [393:59] all these things, then that file maybe [394:02] needs to be stored in a central location [394:05] and that central location could be your [394:06] S3 bucket or your EFS file storage or it [394:10] can also be GitHub. So here we are just [394:14] saying you will publish the artifact and [394:17] metadata. So what is SBOM? SBOM is [394:21] software bill of materials. So your [394:25] application could be using multiple [394:28] things. For example, if I'm developing a [394:31] banking application, I might require a [394:34] calculator. Now I do not want my [394:37] developers to start writing the code for [394:39] a calculator because there are plenty of [394:43] well-known libraries for calculating [394:47] stuff for calculator. I my developers [394:50] would instead use that library. Now why [394:55] use a library? Because libraries they [394:58] ease your job. They reduce development [395:02] cost, improves reliability especially if [395:05] you are using a well-known library and [395:08] it saves time. So your developers if [395:12] they are working on the core banking [395:15] application they could be leveraging [395:17] plenty of libraries and their [395:19] application can have multiple [395:20] dependencies. So everything related to [395:23] your application your application [395:25] constitutes of multiple chunks. all [395:28] these chunks sbomb will list everything. [395:33] So the textbook definition of sbomb is [395:36] it is essentially a formal detailed [395:40] inventory of all the components [395:43] libraries and dependencies used to build [395:46] a piece of software. So it can be [395:50] thought of as an inventory. So this is [395:53] what happens in this stage and this [395:56] stage as you see here typically should [395:58] be finished within 5 minutes. Now why [396:01] the time constraints are important? [396:04] Understand that while this feature is [396:06] built, we want everything to be fast [396:09] here. When you are dealing with [396:10] trunkbased CI/CD, you do not want jobs [396:14] to run for 30 minutes, 20 minutes. You [396:16] want the entire pipeline to be completed [396:20] within the minimum possible time because [396:23] there will be other features on which [396:26] developers are working. Now if you are [396:28] working here then you do not want [396:31] multiple pipelines to be running in [396:33] parallel. You want these things to be [396:35] sequential. At least you should be [396:38] trying that things do not run in [396:41] parallel because if things run in [396:43] parallel then there could be merge [396:45] conflicts and so on. We will see a few [396:47] scenarios where we'll talk about this. [396:50] But for now, aim to trim your pipelines [396:54] to run at the minimum time possible. It [396:58] should consume the minimum time [397:00] possible. The third step is let's say [397:03] the branch job succeeded. Now why we are [397:05] calling this a branch job? Because your [397:07] developer worked on a branch and pushed [397:09] it. Okay, so we are calling that a [397:11] branch job. Next we have a PR job. Now [397:15] why a PR job? If the branch job [397:18] succeeded then what your developer will [397:20] do is your developer will open a PR and [397:24] as soon as a PR is opened a PR job gets [397:29] triggered but Warun why a PR job will [397:32] get triggered understand what we learned [397:35] here Jenkins multibranch pipelines will [397:38] scan your branch PR and tag so here it [397:44] ran a job when a branch was pushed Here [397:47] it will run the pipeline when a PR is [397:50] created. Now the tasks that are [397:53] performed in this and in here are [397:56] different. Understand? So here we are [397:59] performing some more testing. What we [398:02] are saying is triggers on PR open or [398:05] update. If a PR is open or PR is [398:08] updated, it will trigger a PR job. runs [398:12] light SAS and SAS is static application [398:15] security testing plus dependency scans. [398:19] Now dependency scans on codes and [398:22] manifest. So example if you are dealing [398:24] with Maven you will have pom.xml. If [398:27] you're dealing with Python you will have [398:28] requirements.ext. [398:30] What it will do is all these files they [398:33] have the dependencies of the application [398:35] or the packages that are required. [398:37] That's why it is called a dependency [398:39] scan. So what dependency scan will tell [398:42] you? It will give you a graph that this [398:45] is your application. This is the [398:47] dependencies and these dependencies have [398:51] these vulnerabilities. Now how the [398:54] vulnerabilities? vulnerabilities because [398:56] there are tools that can compare the [398:59] dependencies that your application uses [399:02] with a vulnerability database so that it [399:05] can tell you that this is how vulnerable [399:08] or how not vulnerable your application [399:11] is. Now you do not have to do this [399:14] manually. Okay, there are tools that can [399:17] be leveraged for this tools like sneak [399:19] trivi. You can integrate these tools [399:21] with your pipelines. If you learn about [399:24] dev sec ops in future then you will [399:27] understand more about these tools [399:29] because these tools are specific to [399:32] security vulnerability management and so [399:36] on. But as DevOps engineer I'm not [399:39] saying that you should not know about [399:40] these things. Maybe you should not know [399:43] about these things as deep as a dev sec [399:46] ops engineer. But at a very high level, [399:48] you should understand what these tools [399:51] are. But if you are a senior DevOps [399:53] admin, you will be expected to implement [399:57] these tools as well. Maybe you do not [399:59] know those things as deep as your [400:02] security engineers, your devs sec ops [400:04] engineers, but you are still expected to [400:07] configure these tools. The next one is [400:10] deploy ephemeral preview. I am not [400:13] saying deploy onto staging, development [400:16] or production. I'm saying deploy onto [400:20] ephemeral. Now why is this deployment [400:24] necessary? So whatever testing we have [400:26] performed thus far, be it your [400:28] compilation unit test, lints, coverage [400:31] and here SAS dependency scans, we have [400:34] performed these things in a static code. [400:36] But there are certain things that cannot [400:39] be done until and unless the application [400:43] is in running state. Which is why to [400:46] perform certain tests, we want the [400:49] application to be running. And here in [400:51] the PR job, we are deploying that [400:54] application onto an ephemeral hardware. [400:57] Now that ephemeral hardware or that [401:00] ephemeral part could be a different name [401:03] space in Kubernetes or if you're dealing [401:06] with traditional it could be an [401:08] ephemeral virtual machine where this [401:10] code is deployed. Why we are saying it [401:12] ephemeral? We are saying it ephemeral [401:15] because this will be tiered down if the [401:19] PR job completes successfully. If it [401:22] doesn't complete successfully, then you [401:24] may choose to not tear it down and [401:26] troubleshoot the problem and then make [401:28] changes in your code so that it gets [401:30] deployed onto the ephemeral environment [401:32] successfully. But for now, [401:36] understand that ephemeral environment is [401:40] supposed to be tiered down once the [401:42] application is deployed successfully or [401:44] if it is not deployed successfully, it [401:46] is your decision on how you want to deal [401:50] with it. Then what happens once the code [401:53] is deployed onto an ephemeral [401:54] environment? You perform integration and [401:57] smoke tests. What are these tests? These [402:00] tests will tell you whether the basic [402:02] functionality of the application works [402:04] fine or not. If you are in a banking [402:06] application, whether users are able to [402:09] check their account statement. So this [402:10] becomes a smoke test or quickly tell you [402:13] whether the application is accessible or [402:16] not. So these minimal tests which can be [402:18] performed quickly are integration or [402:22] smoke test. So these are performed once [402:24] these are successful then the code is [402:29] merged with the main and we'll look into [402:31] that. So this ephemeral environment is [402:34] tiered down upon PR closure or PR [402:39] completion. Now that the PR job has [402:43] completed successfully, we can [402:45] successfully merge this with main. Now [402:48] there are certain things that also [402:50] happen before this merge is performed. [402:53] So there are these tests which I already [402:55] told you. All required checks green that [402:58] must be there and let's say all of these [403:00] completed successfully including deploy [403:03] succeeded in the ephemeral environment. [403:06] Let's say that was also successful. Next [403:08] is enforce required checks. These are [403:11] the required checks plus up todate with [403:14] base. Now what is up to date with base? [403:18] That's what I just told you. Imagine [403:21] this branch feature UI font was created [403:25] at this stage. Correct? Now what if [403:29] there was a new commit here? This was a [403:33] new commit here. In that scenario, [403:36] this branch will not have the changes [403:39] that were part of this commit. And this [403:41] commit was resulted let's say from a new [403:44] feature on which a different developer [403:46] team was working. So we are at loss [403:48] because we are not working with the [403:51] latest code here. In that scenario the [403:54] PR it will not be merged with main [403:57] because we do not have the latest code [404:01] with us. Now in this scenario you could [404:03] rebase or merge with this new one and [404:06] then rerun your pipeline. That is why I [404:09] said your pipelines should be running [404:11] extremely fast because if you are [404:14] running your pipeline and your pipeline [404:16] is taking 10 minutes to run but there is [404:18] a different PR raised which just takes 3 [404:21] minutes to run then your pipeline is [404:24] already stale because your pipeline your [404:28] branch does not have the latest code [404:31] that is here. You are still working with [404:34] the old code. Hence enforce required [404:37] checks plus up to date with base is [404:41] important. If read fix push rerun PR job [404:46] and that part we already understand. Now [404:50] once this gets merged understand that [404:53] merge will result in a new comet in the [404:58] main pipeline in the main branch. And if [405:02] there is a change, Jenkins [405:05] will rerun the pipeline in the main [405:09] branch. If something is not connecting [405:11] right now, do not worry. In demo, all of [405:13] these things will make sense. So in the [405:16] fifth point, we are saying merge to main [405:19] main pipeline runs on merge reuse signed [405:22] artifact image. So like I discussed in [405:24] the previous lectures that you only [405:27] build once and then you promote the same [405:30] artifact. So the container image [405:32] essentially is only build once and then [405:34] it is pushed to ephemeral to stage to [405:38] production and so on and then deploy to [405:42] dev stage verify promote with approvals. [405:45] Now I told you that there is a single [405:49] line that is releasable. This is the [405:51] main branch. So you could be first [405:54] deploying onto the dev environment from [405:56] this main. understand the difference [405:59] between these branching strategies. [406:01] Okay. So if you were dealing with [406:03] gitflow based branching then you would [406:05] have dev branch, prod branch, prepro [406:08] branch and so on. But we are talking [406:11] about trunkbased CI/CD. In trunk based [406:14] CI/CD it is only the main branch that [406:17] holds the code and then this main branch [406:20] deploys to all the different [406:22] environments. your dev environment, your [406:24] stage environment or your prod [406:26] environment. Then the sixth step is get [406:30] host autodeletes source branch if [406:32] enabled. So once this merge is performed [406:35] successfully like we discussed before [406:37] this will be deleted. Jenkins will also [406:39] prune the job because it does not see [406:42] that branch anymore. [406:44] I hope this makes sense to you. Now we [406:48] will do a demo that will connect any [406:51] dots that remain to be connected. So [406:54] let's scroll down and this is what we [406:58] are going to do. So let me quickly [407:00] explain this to you what we are going to [407:02] do. Jenkins multibranch pipeline flask [407:06] docker demo. So what we'll be using is [407:08] we'll be using a private GitHub [407:10] repository. Multibranch pipeline will be [407:13] created in Jenkins and then we'll be [407:16] performing manual scans for now. Don't [407:18] worry as we progress we'll configure web [407:21] hooks and all those fancy things. It is [407:23] just one click and you can configure [407:25] those. No biggie. Right now we'll be [407:28] performing manual scans. So our step [407:30] zero would be to create an MVP job [407:33] multibranch pipeline job. We'll define [407:36] the job and then we'll set up the [407:38] credentials because we know we are [407:40] connected to a private GitHub [407:42] repository. Our Jenkins must have those [407:44] credentials. So we already have those [407:46] credentials saved from our past [407:47] lectures. We'll be using the same. Then [407:50] we'll be scanning our private repository [407:53] and our private repository must have our [407:55] Jenkins file for the scan to be [407:57] successful because Jenkins file must be [408:01] there. Then we'll run main baseline [408:04] build. Understand? I told you the [408:06] pipeline will detect the main branch [408:08] quickly and it will run the job. And the [408:11] same will happen when we move to the [408:14] next step of the process. There will be [408:17] a new pipeline job created automatically [408:20] for us when we create new branches. So [408:22] in step number one, then we'll be [408:25] creating a branch. In step number two, [408:27] we'll be dealing with a branch job. Here [408:30] we will be building an image pushing it [408:32] to the registry. We'll be deploying a [408:34] container on port 5000. We'll also have [408:38] a file generated deploy info which will [408:42] contain the details about that build. [408:44] We'll be archiving that file. If you [408:46] have been with me since day one, you [408:50] understand that the workspace that gets [408:52] created is temporary. You have the [408:55] option to clean up that workspace. But [408:58] if you archive something that gets [409:00] stored in the jobs directory in Jenkins [409:04] home and that place is there to store [409:08] your artifacts as well along with your [409:10] job. So here in this step we'll be [409:13] archiving deploy info. In step number [409:17] three, we will be raising a PR in [409:21] GitHub. So I should also tell what was [409:23] happening in step one. In step one, we [409:25] create a branch and then we create a [409:28] branch called feature UI and then we [409:31] edit the code. We make some changes. We [409:33] add a line in the UI like actual [409:36] development will work and then we will [409:38] push that branch. And as soon as you [409:41] push that branch, step number two will [409:44] kick in. And we have already discussed [409:46] what happens in step two. And once step [409:49] two is successful, we will move to step [409:52] three. And step three is we will raise a [409:55] PR in GitHub. And as soon as Jenkins [409:59] detects there is a PR, Jenkins will run [410:03] a PR job as well. In step number four, [410:07] we will see that we'll merge the PR [410:10] because the checks are green. We'll [410:12] merge that. And in the fifth point, the [410:15] main deploy will happen because here the [410:18] PR is approved in this step. Once the PR [410:21] is approved, the main will have a new [410:24] commit. The main will have the new [410:26] commit. Then Jenkins will detect it. [410:28] Jenkins will detect it and runs this CI [410:32] again. So it will detect the merge, [410:34] build, push, deploy, archive, test on [410:37] main and we will verify. This can be [410:39] thought of as a smoke test. We'll verify [410:41] that our application is accessible or [410:45] not. [410:46] And here in the sixth step we'll be [410:49] doing a cleanup in GitHub we'll [410:51] autodelete the feature branch and in MVP [410:55] in the Jenkins side the prune retired [410:58] branch jobs or workspaces. So whatever [411:00] was created temporary for example for [411:03] the new feature branch will be pruned [411:05] because the feature branch will no [411:07] longer exist in GitHub. So we'll be [411:10] doing all of these things in this demo. [411:14] So let's get started and before I do [411:17] that I'll take you to the GitHub [411:19] repository for this course. Now I have [411:22] already created this cloud with VJOS [411:26] private repository and we created that I [411:28] think in day five where we did the first [411:30] demo. I'll let you do that. If you do [411:32] not have this repository just create a [411:35] private repository and upload these [411:37] files. What are these files? You can [411:39] find in the repository for this lecture. [411:42] So this lectures repository is day [411:44] seven. In day seven you have a Python [411:47] directory. Just upload everything that [411:49] you see here in the private repository [411:52] that you create. Now what happens in [411:54] this? I want to quickly show that to [411:56] you. There is a docker file here and [411:59] that docker file is simply using the [412:03] python 3.11 slim base image. The working [412:07] directory is forward/ app requirements. [412:09] We are just installing one package that [412:13] is flask because our Python application [412:16] uses the flask framework. We are running [412:19] pip install flask and then we are [412:22] copying our code to the working [412:25] directory which is / app. We are [412:27] exposing our application on port 5,000. [412:29] Remember it is port 5,000 on which our [412:32] application listens. And as we all know [412:35] this expose is just for documentation [412:37] purpose. it will not open port 5,000 in [412:40] your application. You'll have to do [412:41] that. And we have done that and I'll [412:43] show that to you. Then we are finally [412:45] running the application. So this is our [412:47] docker file. Then we have our app. py. [412:50] This is our code. Our code is just [412:52] saying welcome to cloud with varos. Tip [412:56] is built with flask shipped by Jenkins [412:59] running in docker. And here we are [413:01] saying our application listens on port [413:04] 5,000. As simple as that. And this is [413:08] the main piece. This is our Jenkins file [413:10] which we have constructed. It has [413:13] various stages. It has a push stage [413:16] where we are pushing the built container [413:19] image. Then we are in this stage [413:21] deploying this container image. If I [413:24] come down then in this step itself we [413:27] are creating a deploy info file which [413:30] will have the build number, the image, [413:32] we'll have the repo name and the tag. [413:35] We'll have the commit ID, get commit ID. [413:38] We'll have the branch and so on. Then we [413:41] are finally deploying this image onto [413:44] our Docker host. And Docker host in my [413:48] scenario is my Mac machine. And then [413:51] lastly, we are performing a cleanup. [413:54] There are also certain post actions that [413:57] we are performing. On success, we are [413:59] echoing this. On failure, we are echoing [414:02] this. And no matter if it is success or [414:05] failure, we are echoing the job is [414:08] finished. It's a simple Jenkins file. [414:10] And if you do not right now know how to [414:13] create Jenkins file, I would urge you to [414:16] watch this series in sequence because we [414:19] learned this in day five, day six and so [414:21] on. So now that we have this [414:23] understanding, let's go to the Jenkins [414:27] screen. I'm in the homepage localhost [414:30] 8080. [414:31] In here I will just click on new item. [414:35] And what I want to name this is MVP [414:39] flask. And let me zoom in a little. Here [414:41] I will choose multibranch pipeline. [414:44] Creates a set of pipeline projects [414:46] according to detected branches in one [414:49] SCM repository. In one SCM repository [414:53] I'll click here. I will click on okay. [414:56] And display name I will leave blank. [414:58] Description I will leave blank. And here [415:02] is the important piece. I see branch [415:05] sources. Now what is the source of your [415:07] branch? I could choose git. I could [415:10] choose single repository and branch or I [415:12] could choose GitHub. Now why do I see [415:15] GitHub? You see GitHub under branch [415:17] sources? Because GitHub branch source [415:21] plug-in is installed. If you are using [415:24] any other source control then you should [415:28] check the branch source plug-in for your [415:31] git host. So I will click on GitHub and [415:34] this will give me all the options [415:37] related to GitHub. So what I will do is [415:39] first I will paste the URL. I'll go to [415:42] my private repository because hello URL [415:45] is important and that's how Jenkins will [415:48] be scanning my repository. So I will [415:51] copy this name. I will come here. I will [415:54] paste this here. And if I click on [415:57] validate, this won't be successful [416:00] because this is a private repository. I [416:02] must provide credentials, right? So, we [416:06] already added credentials for our [416:08] GitHub. If you do not have the [416:11] credentials right now, go to GitHub, [416:13] create a personal access token, and then [416:15] add those credentials. If you do not [416:17] know how to do it, I recommend you watch [416:19] the previous lecture. So, I'll click [416:21] here. [416:22] and then I'll click on validate and this [416:25] should succeed. Credentials. Okay, this [416:27] is what you want to see. Now I will [416:30] scroll down behavior. Now this one is [416:33] important. Discover branches. What is [416:37] your strategy? First one is exclude [416:40] branches that are also filed as PRs. Now [416:45] this is the recommended option. And what [416:47] this says is build the branch until PR [416:51] exists then build the PR job to avoid [416:55] the duplicate. So let's understand it [416:58] diagrammatically what we are saying. [417:00] I'll go to draw.io. [417:02] Okay. And what we are saying here is so [417:05] you have create you have pushed a [417:07] branch. If there are pushes in the [417:10] branch you keep building those pushes. [417:13] Okay. And that's understandable. Now [417:16] once my push is succeeded once my branch [417:20] job let's say is succeeded I would want [417:23] to raise a PR right and once there is an [417:28] opened PR you stop running the branch [417:31] job and that is what we want what we [417:34] want is if there is a branch job let's [417:37] say you pushed a branch and that branch [417:39] also had the Jenkins file so your [417:42] Jenkins will detect it and it will run [417:44] the branch branch job. That is what you [417:46] want. Let's say your branch job failed. [417:49] You pushed again and this time your [417:53] branch job failed again. So you pushed [417:56] the changes third time and this time [417:59] your branch job succeeded. Okay. Once [418:03] your branch job succeeded, you raised a [418:06] PR [418:07] and as soon as you raise a PR, your PR [418:11] job starts. You do not want when a PR [418:14] job is raised if someone else pushes to [418:17] this there should be a branch job run. [418:20] You don't want that. If a PR exists, you [418:24] do not run a branch job for me. As [418:26] simple as that. And that is what we [418:28] want. And understand when this PR job is [418:32] closed, the PR that you raised is [418:34] closed, you can again push and run the [418:36] brand jobs. But understand if this PR [418:40] job is successful most of the times you [418:42] would want the branch to get autodeleted [418:45] in that case it does not matter right. [418:48] So what you want is you want this [418:51] setting exclude branches that are also [418:54] filed for PR. So in this option what [418:56] will happen is build the branch until a [418:59] PR exists. As simple as that. Then build [419:03] the PR job to avoid duplicates. Second [419:06] one says only branch that are also filed [419:10] as PRs. Now this is saying build [419:13] branches only when they have a open PR. [419:17] Now be careful with this because it will [419:20] not build in generic pushes. So if you [419:22] are pushing to your branch it will not [419:24] build. It will only build when the PR is [419:28] raised. So essentially this part will [419:31] not be there at all. only this part will [419:33] be there because it is only running when [419:36] there is a PR job that exists. The last [419:40] one is all branches build every branch [419:43] simple but can double the builds when [419:47] PRs exist and that's understandable. So [419:50] I hope this makes sense and also I have [419:53] explained these pointers in the GitHub [419:56] repository as well. So go to that GitHub [419:58] repository and revise these concepts. [420:01] Next one is discover pull requests from [420:06] origin. Now let's see what are our [420:09] options here. So here we will choose the [420:11] recommended one and then discover pull [420:14] requests from origin. The selected one [420:16] is the current pull request revision. [420:19] Now this says only build my code. So for [420:25] example, [420:27] whatever code was here, just build that [420:31] code. So build the PR tip essentially. [420:34] The next option is merging the pull [420:38] request with the current target branch [420:42] revision. Now this says your code plus [420:47] the current main. But Wun, why is this [420:49] important? Understand? If I just build [420:53] this, the PR may fail because there [420:57] could have been a new commit here. If I [421:02] build this and then I try to merge this, [421:05] the merge will fail because it is not up [421:08] to date with the main because there has [421:10] been a new commit since this branch was [421:13] created. What I want you to do, Jenkins, [421:17] is I want you to test my code [421:22] plus I want you to see what will happen [421:26] when this merges with this the current [421:29] one the current main. So you check [421:34] with this option [421:37] merging the pull request with the [421:38] current target branch revision. So you [421:41] test my code with the current main [421:45] revision so that I know if I try to [421:47] merge with this one how will my code [421:51] behave. So essentially this is telling [421:54] you beforehand what will happen if you [421:58] try to merge this with the main right [422:01] now. It will give you the proper [422:03] perspective because it will take your PR [422:06] tip and the head tip. Last one is if you [422:10] have enough compute resources, I would [422:13] suggest to always use this both the [422:15] current pull request revision and the [422:18] pull request merged with the current [422:21] target branch revision. So what this [422:24] will do is this will verify both the [422:28] views. First it will run on your code [422:32] only this one. then it will run on your [422:35] code plus the current main tip which is [422:38] this one. Now why this is important? [422:41] Again I'm saying if this commit was not [422:44] there then you might not need any of [422:46] these things. But what if someone [422:49] performed a new commit and your branch [422:52] is no longer the latest. In that [422:56] scenario you would want to test what [422:58] will happen if I try to merge it right [423:01] now. And if you try with this one, my [423:04] results would be successful. And if you [423:06] know these will be successful, then you [423:08] perform your rebase and merge and [423:10] whatever you want. But this will tell [423:13] you beforehand what will happen if you [423:16] try to merge with the current main tip [423:19] with the latest main right now. And this [423:21] is the latest main. So this will only [423:23] come once the merge is successful. So [423:26] this is essentially your merge commit [423:29] once the PR is successful. So I'll [423:31] remove this. I hope all of these makes [423:35] sense to you. I will choose this one. [423:38] And if you go to GitHub here, I have [423:41] explain all of these things both current [423:43] and main. What will happen? Two builds [423:45] per PR update. Head and merge. Here you [423:50] will see a synthetic merge. And this [423:52] merge whatever happens is a synthetic [423:54] merge. It's a temporary merge because [423:57] understand a merge will have to be [424:00] performed to see what will happen if [424:03] this merges with this one. What will [424:06] happen now [424:08] for this to happen? It won't actually [424:10] merge your code. It will perform a [424:12] synthetic merge, a temporary merge to [424:15] see what will happen and it will tell [424:17] you the results for that. So I hope this [424:21] makes sense. Now let's go back to our [424:26] Jenkins job. I will choose this here and [424:29] discover pull request from forks. You [424:31] will not be dealing with forks when you [424:33] talk about enterprise private [424:35] repository. So I will not discuss that [424:37] but I will encourage you to read about [424:39] this as well in the GitHub notes. So [424:42] I'll keep this at default. I'll also [424:44] keep this at default. Here I will [424:47] property strategy all branches get the [424:49] same properties. I'll keep this as is [424:52] here. Build configuration by Jenkins [424:55] file. So this is important. Jenkins file [424:58] path. You can choose a new file as well. [425:01] Uh a custom file name as well. But I [425:03] will leave it to Jenkins file and it is [425:06] present directly in the root of my [425:08] repository. So that it is not inside any [425:11] path. I'll come down. Discard old [425:14] builds. I will let it be selected. Scan [425:17] multiple pipeline triggers. If you were [425:20] pulling your SCM, you could use this and [425:22] at which interval you want to pull that [425:25] you have different options you can [425:28] choose from here. So I will deselect [425:31] this because we will be running the [425:33] pipeline manually. Appearance this is [425:36] fine this is fine. Shared libraries we [425:38] have not discussed sharable libraries [425:41] right now. Pipeline libraries we'll see [425:43] that later but for now let's save this. [425:46] And I want you to concentrate and see [425:48] what happens. I'll click on save. And it [425:50] has automatically started scanning my [425:52] repository. This is so cool folks. Now [425:55] see checking branch main. Jenkins file [425:58] found met criteria scheduled build for [426:02] branch main. It automatically did this. [426:05] And if you see here, [426:08] let me show that to you later. One [426:10] branches were processed. Checking pull [426:13] request. Zero pull requests. It has [426:15] started detecting. Finished examining [426:18] this. Finished branch indexing. It took [426:21] 3.7 seconds. Finished. Success. Now [426:25] let's go here and no let's go to this [426:29] page. Here is MVP flask. This is what we [426:34] created. This is at the very top. This [426:36] is has a new icon as well for the [426:38] multibranch pipeline. If I click here, [426:41] it has sub jobs. And if you see here, if [426:45] I if I you see this one, this one has [426:48] some new options. Scan repository now, [426:51] scan repository log, multibranch [426:53] pipeline events. But this is the [426:57] pipeline job that it has created for the [426:59] main branch. Why I know that? Because [427:01] this is under the branch section and [427:04] here is a pull request section. So if I [427:07] go to branches section, you see how this [427:09] looks like. This looks like exactly like [427:12] the pipeline job we created in the [427:14] previous lecture. So that's what I said. [427:17] Mbp is an umbrella term. Then you can [427:20] have multiple jobs inside it based on [427:23] your branches and pull request. Now [427:25] since we only have main right now, you [427:28] only see main and we don't have any PRs [427:32] created which is why you see zero here. [427:35] I'll click on main and you see here is [427:38] the deploy artifact that we generated [427:41] and then here are the different stages [427:44] of our pipeline and if you see it has [427:47] taken action on main and ran the [427:50] complete pipeline the deploy is also [427:53] successful. So if I access local host on [427:58] port 5000 then I will see my application [428:01] is accessible. Welcome to cloud with [428:03] varos built with flask shipped by [428:06] Jenkins and running in docker and if I [428:08] go to hub.docker.com [428:12] and let me sign in quickly and till this [428:14] happens I want to take you to the CLI [428:17] and here let me run docker ps you see a [428:20] container was created a minute ago and [428:23] that is when we ran our pipeline. So if [428:25] I go to my browser again and here you [428:29] see cloud with VJO flask it was pushed 2 [428:32] minutes ago and if I go here the tag [428:35] that is being assigned if I go here [428:38] there is there was a latest tag there is [428:41] tag number one and one is the build ID [428:44] which is why you see this here. I hope [428:48] the dots are getting connected and this [428:51] will only connect if you have been [428:53] through the past lectures and you [428:55] understand what our Jenkins file is [428:58] doing. Now let's go back to Jenkins [429:02] screen and we have seen this. Now let's [429:06] do this flow. Now let's go to the [429:08] diagrams and I'll show you what we want [429:10] to do now. Now we want to create this [429:12] branch. Okay. So let's go to our CLI [429:17] first. I want you to know that I am in [429:20] Jenkins basics to production directory. [429:23] Firstly, let's go to day 7. Okay, I'll [429:27] go to day 7 and I'll clear my screen. [429:30] And here what I want to do is I want to [429:34] clone my private repository. Understand [429:36] that I won't be able to push to a [429:39] private repository until I provide [429:42] credentials. So in the first step we [429:45] will clone the repository and I can only [429:47] clone a private repository if I provide [429:50] the credential. So let me do that. I [429:53] will paste this here. Now do not worry. [429:55] I will be deleting this personal access [429:59] token later. For now I'm just showing [430:02] this to you so that you can also follow [430:05] along. Get clone origin. Now most of you [430:09] would have seen that origin is named [430:12] remote but I don't like that I want [430:15] things to be detectable. If I see this [430:18] name I should be able to identify to [430:21] which private repository this name point [430:25] to which is why I have named my origin [430:28] flask private. I have not kept it to the [430:30] default which is remote. Now if you go [430:34] to the GitHub repository of this [430:36] lecture, if I go here and if I go to the [430:42] top and then from here I go to first one [430:47] and if I scroll down then you would see [430:51] that all the instructions that I am [430:54] following you can find all these [430:56] instructions here. Okay. So just follow [430:59] along all of these steps with me so that [431:02] it solidifies your understanding. So [431:04] I'll go to the terminal. I'll press [431:06] enter. Now as soon as I do it, I think [431:10] my token has expired. So hold on. Let me [431:14] do one thing. I'll press Ctrl C. So more [431:17] learning for you. So I will go to my [431:20] GitHub repository and then let me do one [431:24] thing. I will scroll up. I will go here [431:28] and then I will go to settings. I'll [431:30] open this in new tab and then I will go [431:34] to [431:35] developer settings. [431:38] Here I will go to personal access tokens [431:40] tokens classic and here I will click on [431:44] create new token. Uh I'll choose the [431:48] classic one and here I will use my pass [431:51] key. I'll touch my finger and then I'll [431:55] call this Jenkins delete me because I'll [431:58] delete it after this lecture valerity 7 [432:02] days repo and then I will click on [432:05] generate token and then I will copy this [432:08] token and then I will paste this onto [432:11] the command where I will be running [432:12] this. I will paste this and then let's [432:17] close this. I will now paste that [432:21] command along with the new token here. [432:24] And this is cloning and the cloning is [432:26] successful. If I press ls now, I will [432:29] see cloud with var private repo. So [432:32] let's go inside this [432:35] and I'll again clear my screen. Now I [432:38] will run get remote. This will show me [432:42] that flask private is your remote. So [432:44] whatever push I will perform will be [432:46] performed into this repository. So if I [432:49] run get remote v, you run it with [432:52] caution because this will show you your [432:55] personal access token as well. So it [432:57] will show me where exactly we are [433:00] pushing. So if I run get branch L, this [433:04] will show me that I am currently working [433:07] on the main branch. understand that as a [433:10] developer I won't be working on the main [433:12] and I will not be able to push directly [433:15] on the main because main is often [433:17] protected. My main right now is not [433:20] protected because anyone can push onto [433:22] it but as we progress with this lecture [433:25] I will show you how to protect it. Now [433:29] let's do one thing. I will create a new [433:32] branch. I will write get branch feature [433:36] UI and then if I run get branch-l [433:41] we'll see that feature UI is created but [433:44] we are currently working on the main [433:46] branch because asterisk is against the [433:49] main branch. So what I want to do is I [433:52] want to switch to a branch called [433:54] teacher UI. Now a lot of folks also use [433:58] checkout but I prefer to use switch and [434:01] if you have any confusions a small [434:04] homework for you check out the [434:05] difference between git switch and get [434:08] checkout and tell that to me in the [434:10] comment section. So we have checked out [434:12] feature branch. So if I run git branch l [434:15] you will see the feature UI is now green [434:19] and it also has an asterisk. And if I [434:22] run ls, you see all our files are here. [434:27] Now we want to make changes into this [434:30] file like actual developers will do. vim [434:34] app. py. I'll press enter. I will go in [434:38] the insert mode. I will exit the insert [434:42] mode. Then I'll press the dollar sign to [434:45] move. At the last of this line, I will [434:47] go into insert mode. I will add a comma [434:51] here and then I will come here. I will [434:54] write UI here is equal to this is my new [435:01] UI. How do you like it? So this is what [435:06] I have added. Now your developers will [435:09] be actually modifying the code and doing [435:11] some fancy cool things. So do not let [435:14] that thing that the flow will be any [435:17] different. the flow will just be the [435:19] same. We are modifying the code of our [435:22] small little Python flask application in [435:25] actual production as well. The flow [435:27] would be same. I will hit escape and [435:30] then I will press shift ZZ to save this. [435:34] Now that this is saved, I will hit get [435:37] status. This will tell me that dear [435:40] friend, you have modified app. py do you [435:43] want to commit these changes? And yes, I [435:45] want to commit these changes. First, I [435:48] will use get add app. py. I'll press [435:51] enter. Then I will run get commit- m and [435:56] then I will write the commit message. [436:00] Now this commit message should always be [436:02] useful. So I will write updated UI to [436:06] have a new line. And once I have done [436:09] this, I will close this. I will press [436:12] enter. Now if I run get status, you [436:17] won't see that file here anymore. And if [436:19] I cat on our code, you will see that we [436:24] have the new line. Understand that the [436:27] modified code is right now in your [436:30] feature branch. It is not in your main [436:33] branch. So if I get switch main and I [436:37] cat app. py then you will see the old [436:41] code. Understand? And we are not [436:43] disturbing the main we are working on [436:46] the feature branch which is why only the [436:48] feature branch will have our code. So [436:52] once our feature branch has the modified [436:55] code we have committed that already. Now [436:58] what we can do is we can push this onto [437:01] our remote and we already know if I hit [437:04] get remote you see that flask private is [437:07] already there. Now what I will do is I [437:10] will write get push. Our credentials are [437:13] already added. We added that in the [437:15] cloning step. I will write get push [437:19] flask private. This is our origin. I [437:23] will name it appropriately flask [437:26] private. And where do I want to push it? [437:29] Which branch? And the branch is the [437:31] same. I'll have feature UI. So whatever [437:34] changes that we did were only there in [437:38] our local system. Those changes are not [437:42] there in our remote. Our remote only has [437:46] main right now. But now we want our main [437:49] to also have feature UI branch. So I'll [437:53] press enter. And this will start doing [437:56] the push. The push has happened. So if I [437:59] run get branch here, it will show you [438:02] that there are two branches which is [438:04] fine. But if I now hit get branches [438:10] a I think it will show me all the [438:13] branches it is also showing these are [438:17] available locally and if I talk about [438:20] remote has feature UI and main and [438:24] feature UI is there because we just [438:26] pushed it. If you would have run this [438:28] command before pushing then you would [438:31] see only main listed here. But now that [438:35] there are two branches, it is showing [438:37] both feature UI and main. And you see [438:40] here feature UI had recent pushes 2 [438:43] seconds ago. Compare and pull. We won't [438:45] be doing anything right now. I will hit [438:48] refresh. And if I check here, you see [438:52] feature UI is here. And if I go to the [438:55] code of feature UI, it will have the new [438:58] code UI. This is my new UI. How do you [439:02] like it? And that is what we wanted. But [439:06] if I go to our main branch, like I [439:09] showed you before, our main branch will [439:12] still have the old code which does not [439:15] have that UI line in it. Okay. Now there [439:18] is a main branch for which Jenkins has [439:23] already performed the build. So if I go [439:25] to this you see main it has already [439:28] built and we also tried to access and it [439:31] worked fine. What I want you to [439:34] understand is based on the diagram, what [439:38] we saw before was as soon as our private [439:43] repository was scanned, a job for this [439:46] main was triggered. And if I go to my [439:50] browser again, you see the branch name [439:53] main exists here. So what we just did [439:57] was we pushed our feature branch into [440:03] our repository. That is why you see a [440:07] main here and also the feature UI. [440:11] Understand that this branch job that you [440:15] see here, this should now trigger. Why [440:18] this should trigger? This should trigger [440:21] because we have a new branch and that [440:25] branch is called feature UI. The CI [440:29] should start for this branch. Now since [440:32] we are not using web hooks what I will [440:35] do is I'll come here and I will click on [440:39] scan repository. Now when it scans it [440:42] will see that there is a new branch and [440:45] that new branch has the Jenkins file. So [440:49] what I should now do is I should also [440:53] build that branch. And what I want you [440:56] to observe is you are not telling [441:00] Jenkins anything. MVP is taking action [441:04] automatically based on what it sees in [441:07] your repository. We are scanning it [441:10] manually right now. But imagine if you [441:12] were using web hooks here which we will [441:14] later in this course. You will not have [441:17] to do anything. All these tasks will be [441:20] automated for you. So if I hit refresh [441:22] now, you see there is a feature UI here. [441:27] Understand what we learned. The jobs can [441:30] be triggered. These pipelines can be [441:33] triggered based on PRs, branches or [441:37] tags. So branches we have already seen [441:40] for main and for feature. This pipeline [441:44] got triggered automatically. Essentially [441:47] child jobs were created for main and for [441:50] feature. And in the next step we will [441:55] raise a PR PR for what Warun we'll be [441:58] raising a PR for feature to get merged [442:02] into main. And we will see that once we [442:05] do that the pipeline also gets triggered [442:08] and this time this child pipeline will [442:12] not be part of this branches it will be [442:14] part of pull requests because we will be [442:17] raising a pull request for that. So now [442:20] this has completed. Understand that [442:23] right now we are using the same Jenkins [442:26] file for everything. But when you are in [442:29] production like we learned in this [442:32] diagram, you will have certain tests in [442:35] the branch job. You will have some [442:37] different tests in the PR job and so on. [442:41] As we progress we with this course we [442:44] will see those things. But for now I [442:47] wanted you to understand the multibranch [442:50] pipeline concept which is why I have [442:53] kept the pipeline simple. In upcoming [442:56] lectures of this course we will modify [442:59] our Jenkins file to do different tasks [443:03] at different stages. So for a branch job [443:06] there will be different tasks. For PR [443:08] there will be different tasks and so on. [443:11] For now we are doing the same tasks. So [443:15] I will go back here and this time I'll [443:19] go back to the browser and you see this [443:22] job has completed. So now if I go to [443:25] this tab you see what is being displayed [443:29] right now is the code that was in main [443:32] and main did not [laughter] have the UI [443:35] line. So what I will now do is I will [443:38] hit refresh. understand that whatever we [443:43] have been deploying all of these are [443:46] being deployed in my docker host and my [443:49] Jenkins file at the very last has a line [443:52] which first deletes the existing [443:54] container and then recreates the [443:57] container and this time when it [443:59] recreated the container it used the code [444:03] that was there in the feature UI branch [444:06] and the feature UI branch had the code [444:11] that had the UI line. So if I go to this [444:14] feature branch UI and I go to app py [444:19] this has the code which has UI. This is [444:22] my new UI. How do you like it? Which is [444:24] why you see it here. Understand that our [444:29] main branch still has the old code. So [444:33] if I go to my main branch, the code [444:36] there is still this. It does not have [444:39] that UI line. But I want you to [444:41] understand as we progress as we progress [444:45] with this lecture. We will reach this [444:48] stage where we will be merging to the [444:50] main and when we reach that stage you [444:54] will see that your main also has the [444:58] modified code. So we'll see that as we [445:01] progress. So for now you saw that a [445:05] branch job was automatically created for [445:09] feature UI. So let's go back to the [445:12] diagram and see what we want to do next. [445:14] So step two is done. Step three is [445:18] create a PR. So let's open a PR. So I [445:23] will go to my browser. I will go to this [445:26] repository cloud with VJOS private repo. [445:30] And here I see compare and pull request. [445:34] So let's raise a pull request. You can [445:36] either click here or you can go to the [445:39] pull request section and click on new [445:42] pull request. I will use this one. I [445:45] will click here. Add a title update UI [445:48] to have a new line. I want you to know [445:51] that have good titles which can be [445:55] easily identified because when you go to [445:58] the Jenkins UI under the pull request, [446:01] the job that will be created will have [446:04] the same name as the title of the pull [446:07] request. So keep that name helpful. [446:11] Updated UI to have a new line. I'll just [446:14] add cloud with VJO so that it can be [446:16] identified easily. You can have detailed [446:19] descriptions when working in production. [446:22] You must have detailed descriptions so [446:25] that your peers, your collaborators can [446:29] understand what this engineer was trying [446:32] to do. I will just write description [446:35] here so that we have something here. [446:38] Otherwise, you can even have markdown [446:41] language. So, you can have bold, italics [446:44] and so on. So I will click on create [446:47] pull request. Understand that right now [446:51] a pull request has been created. We have [446:54] not merged it. But what we learned when [446:58] a PR will be seen in the repository the [447:02] Jenkins will trigger. So it we have not [447:05] configured web hooks. So what I will now [447:08] do is I'll click on scan repository now. [447:12] And as soon as I click here, we should [447:15] see that it should identify that there [447:18] is a pull request created in the private [447:21] repository and I should take action. And [447:23] I see something is happening here. So [447:25] let's hit refresh. Understand this is [447:29] extremely critical. You see there are [447:32] two things here. But Wun, we just [447:35] created a single PR. Why do I see both? [447:39] Read carefully. This is the title we [447:43] gave. And in parentheses, this one has [447:46] head and this one has merge. And I want [447:50] to take you back to how we configured [447:53] what did we say in the PR section. [447:56] Discover pull request from origin. We [447:59] said we want you Jenkins to build both [448:03] the things. I want you to build my [448:06] feature branch and then I also want you [448:10] to perform a temporary merge wherein you [448:14] tell me what will happen if I merge this [448:19] PR with the main right now and that is [448:22] exactly what it is doing. It has created [448:25] a synthetic merge. It will not actually [448:27] merge your code. It's just a synthetic [448:30] merge, a test merge, a temporary merge [448:33] so to speak. So it will just give you [448:35] what will happen when in future you will [448:39] be merging that. And I chose this option [448:42] because this will tell me beforehand [448:44] before even I try to click on merge pull [448:48] request. It will tell me beforehand that [448:50] Warun your merge could have problems or [448:54] it could be successful. Both options are [448:56] there. So that is why you see you raised [448:59] a single PR and there were two jobs that [449:02] were created for you. So let's go back [449:05] here. Let's go to PR. But why has this [449:09] head job failed? Head job is the one [449:11] that is being run for on our code. That [449:14] is the feature branch. The PR that we [449:17] just erased from the feature branch. And [449:20] then this is for the synthetic merge. So [449:23] the PR had failed. Now there is a reason [449:26] for this failure and I along with you [449:30] want to dissect this so that you [449:32] understand. I deliberately chose this [449:34] option to show you how two jobs will [449:39] come here. So we have learned that [449:40] point. Now I want to tell you why there [449:43] is an error. So let's click here and if [449:46] I go to stages I can see that deploy [449:50] stage had a problem. So if I click here [449:52] and if I click here it is saying docker [449:56] error response from Damon conflict the [449:59] container name this already in use this [450:03] this this you have to remove or rename [450:05] that container to be able to reuse that [450:08] name but Vun why have we not received [450:11] this error thus far and the answer to [450:14] that question is let me take you back to [450:16] my private repository so that I can show [450:19] that to you because That's extremely [450:21] critical for you to understand. See my [450:24] code. If I go to day seven, I have this [450:26] Jenkins file here as well. If I I don't [450:30] have it here. Let's go to day six. Day [450:32] six has the Jenkins file. I'll upload [450:34] the Jenkins file in day seven as well. [450:36] So if you see my Jenkins file in the [450:40] deploy stage, what I am also doing is I [450:43] am removing this container. I'm saying [450:46] if there is a container with this name, [450:48] you remove that. But why did we face an [450:52] error? We faced an error because both of [450:55] these jobs ran simultaneously. They [450:59] tried to deploy a container [451:00] simultaneously and only one succeeded. [451:04] The one that finished a fractions of [451:07] seconds before and the second job did [451:10] not have time to run this command. It [451:13] happened so fast which is why you only [451:16] see one container being deployed. And if [451:19] I go to my CLI and I run docker ps, you [451:24] see there is only one container that was [451:26] created 3 minutes ago. The deployment [451:28] for this container failed. And for you [451:31] the job could fail for head or the job [451:34] could fail for the synthetic merge. It [451:36] does not matter. But now you understand [451:39] why one of these jobs failed. I also [451:43] want you to clearly know that we are [451:47] deploying onto the same docker host in [451:50] actual production. Maybe the branch job [451:53] will have different steps and it will be [451:55] deploying to somewhere else. Maybe the [451:58] PR job will be deploying to a different [452:00] nameace in Kubernetes or they are [452:02] deploying onto the same nameace with [452:05] just different names for the pods. [452:08] understand that these things are project [452:10] specific. In this lecture, I want to [452:13] concentrate on MBP, multi-branch [452:16] pipeline, how the deployment part is [452:18] done, what different steps each could [452:22] have. We will see as we progress with [452:25] this course. So now that we have the [452:27] understanding why this failed, the PR [452:30] the second job failed. Let's go to the [452:33] GitHub UI and see what messages do we [452:35] have here. So I'll hit refresh here and [452:37] then it says one failing three [452:40] successful check and we know which one [452:42] is failing. The PR head is failing. So [452:46] which is what we see here. The PR job [452:49] the head that has failed. And if I come [452:52] here so it is saying the branch job [452:54] succeeded. The PR merge job also [452:57] succeeded. This get guardian security [453:00] checks. I did not configure that. This [453:02] is automatically available by GitHub. [453:04] What this will do is it will scan your [453:06] repository to see if there are any [453:08] secrets that you have committed by [453:10] mistake. And you all must know that it [453:12] is a best practice to never ever commit [453:15] your sensitive information in your [453:18] GitHub repository even if it is private. [453:22] So now we see an option to merge this [453:27] pull request. We could merge it. What I [453:30] want to do is I want to do this cleanly. [453:35] So let's do one thing. I will go here. [453:37] I'll go to configure. Now that we have [453:39] learned what happens when both of these [453:41] are selected, let's just change this to [453:44] the current pull request derivation. I [453:46] will save this. Okay. And then I will go [453:50] to my GitHub repository. And this time I [453:55] will hit get branch. Let's see. We are [453:59] in the feature UI branch. I will hit vim [454:02] app. py. And this time I will add one [454:06] more line here. I will go at the very [454:10] last. I could have used the dollar sign [454:12] as well. And let's write here we [454:15] modified it again. Okay, I'll save this [454:19] one. I will write get add app. py. Then [454:24] I will write g committ-en m added one [454:29] more line to the UI and then I will [454:33] close the double quotes. So I'll press [454:35] enter. I will write get push flask and I [454:40] want to push this to the feature UI [454:42] branch. I'll press enter and this has [454:45] done the push. Now if I go back to the [454:48] CLI, I see if I hit refresh, I see there [454:52] is a pull request. Understand what we [454:55] learned. When there is an open PR, the [454:59] branch job will not run. Which is why I [455:02] don't see a new job here. I just pushed [455:04] it to the feature branch, but I don't [455:07] see a branch job. But what I see is a PR [455:11] job. Why a PR job? Because we chose that [455:14] option. understand everything is [455:17] connected. So if I open this to the new [455:20] tab, I go here, I go to configure and if [455:24] you see this, we said that exclude [455:26] branches that are also filed as PRs. So [455:30] we have a PR open. We did not close the [455:33] PR, which is why you see a pull request [455:36] RAM. So if I refresh this now, you see [455:40] this is my new UI. How do you like it? [455:43] we modified it again. So if I go here [455:46] again, I want to do one more thing. Now [455:49] we said that there is also an option via [455:53] which we can delete the feature branches [455:56] once the merge is successful. So if I [455:59] click on merge pull request right now [456:02] this will not delete my feature branch [456:05] automatically because I have not [456:07] configured that. So let's also configure [456:09] that. So I will go to settings here and [456:13] in the settings I will scroll down and [456:16] here you see these things are for pull [456:19] requests. So what I will choose here is [456:21] automatically delete head branches. [456:23] After pull requests are merged you can [456:27] have head branches deleted [456:28] automatically. So in our scenario as [456:31] soon as we click on merge the pull [456:34] request our feature branch will be [456:37] deleted automatically. I want to show [456:40] you one more thing. So while we are at [456:43] it, let's also protect our branch. Now I [456:46] am using a freeware version of GitHub. I [456:49] don't have an enterprise version which [456:50] is why I can just enable that feature [456:54] but I cannot enforce it. So if you have [456:56] the pro version, of course you can [456:58] enforce it and I'll show you how to [457:01] configure it. But then it will not be [457:03] applicable for me because I'm running a [457:05] freeware version. So I'll click on add [457:07] classic branch protection rule here. I [457:10] want that protection rule to be [457:11] applicable for main and I want require a [457:16] pull request before merging. When [457:18] enabled all commits must be made to a [457:21] nonprotected branch. In our scenario the [457:23] nonprotected branches feature UI or any [457:26] feature or buck fix branch that is [457:29] created. Our protected branch is only [457:31] one which is we have specified here. You [457:35] can specify multiple branches. You can [457:37] have different patterns. Maybe you are [457:40] calling your main release or production. [457:43] It depends on your naming strategy. For [457:48] us, it is main. I will uncheck required [457:51] approvals. And then as you can read [457:53] non-protected branches and submitted via [457:56] a pull request before they can be merged [457:58] into a branch that matches this rule and [458:01] for us it is main. I will scroll down [458:04] and I will encourage you to read all of [458:06] these options and then you can read it [458:08] here. Your protected branch rules for [458:10] your branch won't be enforced on this [458:12] private repository until you move to [458:15] GitHub team or enterprise organization [458:18] account. So I want you to keep this in [458:21] mind. So let's go to the pull requests [458:23] and in this screen I will merge it. I'll [458:29] go here and now I will scroll down. I [458:32] will click on merge pull request. Uh you [458:35] have to have a proper commit message [458:37] here. I will click on confirm merge and [458:40] I have done the merge. Pull request [458:43] successfully merged and closed. And [458:45] let's notice two things. First is we [458:47] want the feature to be deleted. We [458:49] enabled that thing. So it should be [458:51] deleted automatically. And second I want [458:54] to see a branch job here because [458:57] understand the main just had a new [459:00] commit. So if I go to my draw.io, we [459:03] have merged to main. So we are here. [459:07] There is a new commit in the main [459:09] branch. Jenkins should automatically [459:11] scan that and do the deployment of the [459:15] new container for us. And we know that [459:18] container will also display the same [459:20] message. So let's go here. I will hit [459:22] refresh. I'll have to scan a repository [459:26] now. and it will detect that main just [459:29] had a new change, a new commit. Last [459:32] success was 20 minutes ago. Let's hit [459:35] refresh. Oh, it's something is already [459:38] running. Uh if I go here and you see [459:42] main has started again. It is in [459:45] progress. So if I go here, a build two [459:49] has started. And if I go to [459:53] hub.docker.com, docker.com we will see [459:56] the resultant image here itself. So if I [459:58] hit refresh the code has not yet [460:00] deployed because the pipeline is still [460:03] running but I want to show you the other [460:05] thing that we did. So I will scroll up I [460:09] will go to my repository one branch our [460:12] feature UI branch folks is automatically [460:17] deleted. So if I go here, I don't see [460:20] anything. But I want to show you an [460:24] interesting thing. Now if I go to app. [460:26] py, this has our new code. Now [460:30] understand the main did not have the new [460:34] code until and unless a successful merge [460:39] was performed. Now that our merge was [460:43] successful, [460:44] you see there is the modified code here [460:49] UI. This is my new UI. How do you like [460:52] it? We modified it again. So I hope all [460:56] these dots are getting connected. And [460:58] this will have the same message even if [461:02] this deployment succeeds. And it has [461:04] succeeded. Our Docker hub should have an [461:08] image with the build number because that [461:11] is what we specified in our Jenkins [461:14] file. We said our images should have [461:18] this tag. So image and tag. This tag is [461:21] the build number. So if I go here, if I [461:25] sign in, I don't know why I get signed [461:28] out again and again. I think I clicked [461:30] on keep me signed in. Anyway, so if I go [461:33] here, if I go to tags, you see there is [461:36] a latest tag because hello, that is what [461:39] we specified. We were pushing two tags. [461:42] One is the latest tag and there is [461:44] another tag and this tag had the build [461:47] number. Since the build number for this [461:50] one is two, which is why you see the [461:53] pushed image has two as a tag. I hope [461:58] all the dots are getting connected. So [462:01] what we saw is if I go back to our [462:04] diagram in step zero we created an MVP [462:08] then we saw the main was automatically [462:11] considered for the pipeline the job ran [462:13] for the main branch then we created a [462:16] new branch we pushed that new branch to [462:19] our GitHub repository and then what we [462:22] saw is a branch job was triggered then [462:26] once we raised a PR a PR job was [462:29] triggered and Then once we performed the [462:32] merge, our code was merged with the main [462:36] and then finally a cleanup was [462:38] performed. Our feature branch was [462:40] deleted from GitHub. In MVP also the [462:43] retired jobs were pruned. And if I go [462:46] back here in the Jenkins, if I go to [462:49] this one, you see last success was 3 [462:52] minutes 36 seconds back. And there are [462:55] zero pull requests now because all those [462:58] pull requests are now pruned because [463:01] there are no existing pull requests in [463:04] my GitHub repository. And right now my [463:08] repository only has the main branch. So [463:10] if I click scan repository now, nothing [463:13] will happen because there has been no [463:15] new commit and we have not pushed any [463:18] new branch. I hope all the dots are [463:22] connected. That wraps up day seven. We [463:25] set up multibranch pipelines, configured [463:28] branches, PR discovery, ran builds via [463:32] manual scans, and used our Jenkins file [463:35] to build, push, and deploy the Flask app [463:38] on port 5000. Then we opened a PR, [463:43] merged to main, rescanned, and verified [463:46] the running container and updated code. [463:49] In the next lecture, we will talk about [463:52] Jenkins agents. If you have any [463:55] questions, feel free to ask them in the [463:57] comment section and I will reply. If [464:00] this helped, do consider liking, sharing [464:03] and subscribing. It will mean a lot to [464:05] me. Thank you very much. Hi, I am Vun [464:09] Jooshi and I welcome you to cloud with [464:12] VJosh. In this lecture, we are going to [464:15] look at Maven from a DevOps engineers [464:18] lens. We will start with compiled versus [464:22] interpreted languages. Then see why [464:25] build tools exist and where Maven fits. [464:29] From there we will install Maven create [464:33] a project with demo 1 and really unpack [464:37] pom.xml [464:39] life cycles, phases, plugins and goals. [464:43] Then we'll also look into repositories [464:46] and what are their roles. We will also [464:49] touch on parent and superom briefly so [464:54] you see how real enterprise projects are [464:57] wired. Finally in demo two we will run a [465:02] full flow build with Maven in Jenkins [465:05] and run the app as a docker container. [465:09] By the end of this lecture you will be [465:12] able to read any pom.xml XML understand [465:16] what Maven is doing in the CI and fix [465:21] common Maven build issues with a DevOps [465:24] mindset. With that said, let's get [465:28] started. [465:29] So before we talk about Maven, I think [465:32] it makes sense for you to first [465:36] understand programming languages at a [465:38] very high level because these are the [465:41] languages for which build tools are [465:44] used. For some of them build tools are [465:47] used and for some of them build tools [465:51] are used but they are used from a [465:53] different lens. So to understand that [465:56] deeply let's first understand compiled [466:00] versus interpreted languages. Now [466:03] compiled languages are the languages [466:06] which require translation. So for [466:09] example, if I have the code with me, [466:13] that code first must be converted into a [466:18] machine readable format and that machine [466:21] readable format could be converting that [466:24] source code into machine code or into [466:28] byte code. So when you are converting [466:31] the source code into machine code, this [466:34] is primarily done for languages like C, [466:36] C++ and Go. If I talk about converting [466:40] the source code to byte code, then this [466:43] is generally done for Java, C and so on. [466:49] Now I am using the word converting but I [466:52] should use the word compiling. The [466:56] source code for compiled languages [466:59] requires compilation. So when I talk [467:02] about machine code, it is CPU native. So [467:06] if I talk about languages like C, C++, [467:09] Go, any code that is written in that [467:12] language, the application for that can [467:15] be directly deployed into your operating [467:19] system. Of course, you will need certain [467:21] libraries, but when I talk about modern [467:24] applications, they have everything [467:27] inbuilt. So, they just need to be [467:29] installed onto your OS and once that is [467:33] done, they can run just fine. But when I [467:37] talk about compilation which results in [467:40] the conversion to byte code in that [467:43] scenario you require a runtime and that [467:48] runtime for Java language is JVM. Now [467:53] JVM is not the only runtime that can be [467:57] used for Java. anything that converts [468:01] the source code into JVM byte code and [468:05] the languages for that are also Kotlin, [468:08] Scala and Groovy. JVM as the runtime can [468:12] be used for that. Now this is too much [468:14] of information. I understand that you [468:17] will need all this information because [468:19] while you are working with these [468:21] languages, you will have to understand [468:24] how they run and how the compilation for [468:28] them work. Now we will see this briefly [468:32] as we progress with this lecture about [468:34] JVM. So for now just understand in [468:38] compiled languages you translate the [468:41] source code. You compile the source code [468:44] from a human readable format to a format [468:48] that is understandable by machines and [468:51] that format is either machine code or [468:55] byte code. So compiled languages [468:58] requires build stage and that is [469:01] obvious. So if I want my language to be [469:05] converted into to be compiled into [469:07] machine code or byte code then I require [469:11] a build stage and in that build stage I [469:15] will perform that conversion. Hence when [469:18] I talk about compiled languages they [469:21] will always require a build tool and one [469:25] function of that build tool will always [469:29] be compiling and hence compiled [469:33] languages always require a build tool [469:37] that will do compilation for them. Next [469:40] is faster execution. CPU direct compiled [469:45] languages are often faster when compared [469:48] with interpreted languages. Code hidden [469:52] binary only. That is true. So once your [469:55] source code gets compiled, it gets [469:58] compiled into dotclass or dot jar. And [470:02] there is packaging also. And we'll [470:04] dissect these terms as we progress with [470:06] this lecture. But the resultant artifact [470:09] that you deploy onto your operating [470:11] system or you deploy that onto your [470:14] Kubernetes platform that code is hidden. [470:18] You just see the binary and you cannot [470:21] perform that reverse conversion. [470:24] Examples of compiled languages are Java, [470:27] C++, Go, Rust, etc. Now, this is not a [470:31] comprehensive list. There are many more [470:33] languages and I will let you see what [470:36] those are. Then let's talk about [470:38] interpreted languages. These languages [470:41] do not require any compilation. So when [470:44] I talk about build tools, they are still [470:47] needed for interpreted languages but [470:50] they do not have to perform compilation. [470:52] But Warun if I do not have to perform [470:56] compilation then why do I need a build [470:59] tool? Why do I have a point here that [471:02] says may still benefit from build stage? [471:05] That is because you still have to do [471:08] dependency management and dependency [471:11] management is not an easy job like we [471:14] will see as we progress with this [471:16] lecture. So for now understand that [471:19] interpreted languages they also require [471:23] a build tool. They are typically slower. [471:26] Now previously they the interpretation [471:28] you was performed line by line by the [471:32] interpreter. So these were really slow [471:35] but now modern advancements they are not [471:39] that slow but they are still slow when [471:41] compared with compiled languages code [471:44] visible. So if you have a Python script [471:47] or a JavaScript you would essentially [471:50] share the whole code if you want to ship [471:53] that application. Anyone who wants to [471:56] deploy that application, you will have [471:58] to share that code. Which is why it is [472:00] said the code is visible. If you have [472:03] worked with AWS Lambda, then you would [472:06] see that if you are using Python to [472:08] write your automation, you have to share [472:10] the entire script. There is no way to [472:12] share a binary, which you would do when [472:15] I talk about compiled languages. Now [472:17] before I wrap up this section, I want to [472:20] quickly discuss what are the build tools [472:23] that are often used. So if I talk about [472:26] Java, the build tool that is very famous [472:30] for Java is called Maven. And the second [472:33] one that you could also see in [472:35] production is Gradel. And as we progress [472:37] with this lecture, we will learn more [472:40] about Maven because the lecture is [472:42] called Maven for DevOps. Then for C and [472:47] C++ there are other build tools. But for [472:50] now I want you to understand that when I [472:53] talk about compiled languages they [472:55] require a compilation stage. And for [472:58] compilation to be performed there must [473:01] be a compiler that is performing that [473:03] role. If I talk about Java that compiler [473:06] is Java C. And the artifact format. What [473:10] is an artifact? artifact is anything [473:14] that results from a build. For example, [473:17] if you are building a container image [473:19] using docker commands, then the [473:23] resultant container image is an [473:26] artifact. Similarly, if I talk about [473:28] programming languages, if I'm compiling [473:30] something, anything that results from [473:33] that, that could be class file, dot jar [473:35] file, war file, dot ear file. When I [473:39] talk about Java, this is called an [473:42] artifact. So, anything that results from [473:44] a build is an artifact. Let's read what [473:48] is written here. Build tools don't just [473:51] invoke compilers. Now, with this [473:53] definition, you could be thinking that [473:55] build tools are just used for [473:57] compilation. But it is not like that. [474:01] They do so much more. They handle [474:04] dependencies. And this is an important [474:06] task. For example, if I'm writing a [474:08] banking application, then I would of [474:12] course need a calculator to calculate [474:14] the interest rates or to calculate the [474:16] loans and so on. But I will not ask my [474:21] developers to write the code for [474:23] calculator. What they would do is they [474:26] would use a well-known library so that [474:30] they do not have to write the code for [474:32] calculator again. And that library when [474:35] used becomes a dependency. So a crisp [474:39] definition for dependency is anything on [474:42] which your application depends. Now that [474:45] could be a library, that could be a [474:47] configuration file or an IP address. So [474:51] anything on which your application [474:54] depends is a dependency. Now when I talk [474:58] about build tools, they will handle [475:00] dependencies for you. For example, if my [475:03] application depends on a calculator [475:05] library, then my build tool will fetch [475:08] that library for me. And not just that, [475:11] maybe that calculator library in turn [475:14] depends on another library. So that will [475:17] also be taken care by my build tool. And [475:19] this is called transitive dependency. [475:22] One dependency depends on another [475:24] dependency. That is called a transitive [475:27] dependency. Not just compilation and [475:30] handling dependencies. Build tools will [475:33] also do packaging, testing and artifact [475:36] publishing for you. Vun, what is [475:38] artifact publishing? Packaging is [475:42] packing your application source code [475:44] into jar.class orwar. And these are just [475:48] formats of your artifact. So if you are [475:51] working with a Java application you [475:52] would typically see jar being used but [475:57] war and ear could also be seen. War is [476:01] for web apps ear is for enterprise. Then [476:05] packaging is that but what is artifact [476:07] publishing? Understand that when you [476:10] generate your container images you are [476:12] uploading those container images [476:14] somewhere right and that somewhere is [476:17] typically your image registry. It could [476:18] be docker hub or it could be any [476:20] cloudnative offering that you are [476:22] leveraging. When I talk about artifacts [476:25] that are result of build container [476:27] images are one. Second could be dot jar [476:30] and dotwar. These artifacts can also be [476:34] published and they are often published. [476:36] But warun where are they published? [476:39] Where are they uploaded? For container [476:41] images we have image registries [476:44] container registries. But when I talk [476:46] about these artifacts, these are often [476:49] stored in solutions like Sona type nexus [476:54] JROG artifactory or you could also be [476:58] leveraging any artifact storage provided [477:01] by your cloud provider. So for now what [477:04] I want you to take away is build tools [477:08] don't just invoke compilers. They handle [477:10] dependencies, packaging, tests and [477:14] artifact publishing. With that [477:16] understanding, let's see what I want to [477:19] discuss next. Why and what of build [477:23] tools automate source to artifact [477:25] conversion reliably. We have seen this. [477:28] Resolve dependencies including [477:30] transitive chains. For example, if my [477:33] application depends on A but A depends [477:36] on B and B depends on C and C depends on [477:39] D and E, then my build tool will take [477:43] care of everything, not just immediate [477:45] dependency but also any transitive [477:49] chains, lock versions for reproducible [477:52] builds. Now what happens is when I want [477:55] you I want you to understand this or [477:58] contrast this with containerization. So [478:01] I'm saying lock versions for [478:03] reproducible build. So whenever you are [478:06] creating a container image you use a [478:09] docker file and in that docker file you [478:12] provide the version of the base image [478:15] that you wish to use and every time you [478:18] build the container image that version [478:21] is used. Similarly when I talk about [478:24] build tools they also lock versions. So [478:28] they have a file often times where you [478:31] provide these are the versions of [478:34] dependencies that I wish to use and you [478:38] must install the same version while I am [478:42] building my software or application and [478:45] not just compiled languages even [478:48] interpreted languages. If I talk about [478:51] Python, Python, you use a package [478:54] manager called pip. And pip is the one [478:57] that is taking care of your [478:58] dependencies. With pip you have a file [479:02] called requirements.ext. [479:04] With Python, you use a file called [479:07] requirements.ext. [479:09] And this file will have all the versions [479:12] of the dependencies that you wish to use [479:15] along with your application. Hence the [479:18] build tools once you provide your inputs [479:22] will freeze the version that your [479:25] application requires for the [479:27] dependencies. Hence we are saying lock [479:30] versions for reproducible builds. So [479:34] whoever is building your application or [479:37] whoever is building your software, they [479:39] already know that the application that I [479:43] am building right now requires these [479:45] dependencies and this is the version of [479:48] these dependencies. Hence wherever that [479:51] application is built, it builds and [479:54] performs the same way. Compile or [479:59] prepare code consistently. And we have [480:01] already discussed this. Not just this, [480:05] build tools will also run tests for you. [480:08] And these tests could be unit tests or [480:10] integration tests. Llinters. Linting is [480:13] a process in which you check your code [480:16] for errors, bugs, style issues before it [480:20] runs. And it will also point out any [480:22] unused variables. And then there are of [480:24] course security scans. So your build [480:27] tools are capable of performing all [480:31] these things. Now there are plenty of [480:34] build tools available. In this lecture [480:36] we'll be discussing Maven. But [480:38] understand if you are using any other [480:41] build tools, they may offer similar [480:44] functionality for different languages. [480:47] For example, Maven is primarily used for [480:50] Java. Although it can be used for other [480:53] languages but primarily it is used for [480:55] Java. Hence, Maven is capable of [480:59] performing all these things. Now, your [481:02] build tool may not be able to perform a [481:05] certain functionality. But understand [481:06] that at large, at a very high level, [481:10] build tools are expected to perform the [481:13] functionalities that we are discussing [481:15] in this lecture. But this lecture is [481:18] specifically for Maven. So whatever I'm [481:21] saying for build tools that is [481:24] applicable for Maven for sure for other [481:27] build tools you may have to check the [481:30] documentation. Next is package and [481:33] publish reproducible artifacts and this [481:36] we have already discussed packaging and [481:38] publishing. Packaging is converting your [481:41] source code into jar.war or any other [481:45] format. So if I if you look at here see [481:49] maybe out.exe file maybe ordll files [481:54] whatever is an artifact whatever [481:57] packaging format your languages support [482:00] the build tools will be able to package [482:03] your source code in that format and then [482:06] publish. As we already know, publish is [482:08] essentially uploading your artifact into [482:11] a central repository so that other team [482:15] members can also use it. And the next [482:18] one is eliminate environment drift and [482:21] manual errors. And this is [482:23] self-explanatory. If you have defined [482:25] everything in your build tools, [482:27] specifically the versions, then it [482:30] eliminates environment drifts and manual [482:33] errors. Why elim eliminate environment [482:35] drifts? because your application is [482:38] going to run the same way in all the [482:40] environments because you have defined [482:43] the versions that your application is [482:46] supposed to use for dependencies. Hence [482:49] everything is streamlined. So all these [482:52] functionalities are performed by your [482:55] build tools. Now here is a typical build [482:58] automation steps. First is download [483:01] resolve dependencies then compile. If I [483:04] talk about interpreted languages, this [483:06] step won't imply for them. But for [483:09] compiled languages, there will be a [483:11] compiled stage. Then you will be running [483:13] tests. Now tests could be unit tests. [483:16] Unit tests implies checking certain [483:18] parts of your application individually. [483:20] It could be checking a class or a method [483:23] and so on. Integration test requires [483:25] your application to be running. So you [483:28] check whether the functionality is [483:31] working between let's say two smaller [483:33] chunks of the application. So if I'm [483:34] dealing with a banking application maybe [483:37] I want to see whether from the account [483:39] statement section I can go directly to [483:41] my account details. So all these things [483:45] are part of integration testing. Then [483:48] there are static checks and scans. We [483:50] have already discussed lint s. SAS is [483:53] static application security testing. [483:56] Then you have license checks. You have [483:58] sbombs and we'll see that as we progress [484:00] with this lecture. At the very last you [484:03] package jar war binary and then there [484:05] could be one more step which is deploy. [484:08] And deploy is same as publish. Here when [484:11] we are talking about build tools [484:13] deployments typically does not mean [484:16] deploying onto kubernetes or onto a [484:19] virtual machines. I'm saying typically [484:21] because there are build tools for [484:23] example even Maven you can use this to [484:26] deploy onto a Tomcat server but that is [484:28] generally not seen these days but my [484:32] point is there could also be deploy but [484:35] that deploy is essentially uploading [484:38] your artifact onto a central repository [484:42] like we have repository for container [484:44] images you have repositories for your [484:47] artifact storage as as well. So I hope [484:51] you have a clear understanding of why we [484:54] need build tools and what are build [484:57] tools. With that solid understanding, [485:00] let's see what I want to discuss next. [485:04] What is Maven? Maven is Apache's build [485:09] and project management tool for JVM. [485:12] Understand I am not saying Java, I am [485:16] saying JVM. Now I told you the source [485:19] code when I talk about Java the source [485:22] code is converted into JVM byte code. So [485:26] any language for which the source code [485:28] is converted into JVM byte code and [485:31] those languages are Kotlin, Scala, [485:35] Groovy and Java. So all these four [485:38] languages you can definitely use Maven [485:42] as a automated build tool for these [485:45] languages. So Java is a language, JVM is [485:49] a runtime. So all these languages like [485:52] Java, Scala, Kotlin, Groovy, they will [485:55] require a JVM runtime underneath to run [485:59] properly. And you would have heard these [486:02] terms when you are deploying Java [486:04] applications. The machine must have JRE. [486:07] JRE is Java runtime environment and [486:09] that's directly related to JVM. So the [486:13] next point is widely used for Java, [486:17] Kotelane and Scala project but [486:20] predominantly you will see Maven being [486:23] used for Java. Interesting fact you can [486:27] even use Maven for C, C++, Python and so [486:31] on. However, you do not do that because [486:34] Maven specializes in Java. And why you [486:38] can do that? Here you see a point. [486:40] plugins handle testing and all these [486:43] things. So, Maven has plugins. As long [486:47] as there is a plug-in for C, there's a [486:50] plug-in for C++, you can use Maven for [486:54] those languages. Having said that, there [486:57] are specialized tools, specialized build [487:00] tools that are used for C, C++, Python [487:03] and other languages. However, Maven can [487:07] also be used but it is not recommended [487:10] you do that. Then pom pom.xml [487:15] is the single source of truth. Now as we [487:18] go ahead with this course we will [487:19] understand that pom.xml is the single [487:23] source of truth. So every dependency [487:25] that your application uses the details [487:28] about your application that is available [487:31] in pom.xml. [487:33] Next point is conventions over [487:36] configuration keep projects simple and [487:40] consistent. Now Maven says that we [487:43] adhere to conventions over [487:45] configuration. So you as a Maven user [487:48] will not have to configure everything. [487:51] We as Maven will give you a proper [487:54] structure of the project. Now what is [487:56] the structure of the project? I mean the [487:59] project structure, the directory [488:00] structure. Maven defines that structure [488:03] and as we progress we will see that [488:06] structure. Now I want you to understand [488:09] if you would have to configure [488:12] everything you know where source lives, [488:14] how to compile, how to run tests, where [488:16] to put outputs, what to name them, which [488:20] repos to use and in what order to run [488:24] steps. That is the configuration you [488:27] avoid by following conventions. Which is [488:30] why we say maven is conventions over [488:34] configuration. Next is plugins handle [488:38] testing, coverage, policies, packaging, [488:41] etc. Now if I talk about Maven, there is [488:45] a plug-in that is doing all the tasks [488:48] for you. Compiling, there is a compiler [488:51] plug-in. If you are doing unit test, [488:54] there is a surefire plug-in. If you are [488:56] doing integration tests then you have a [488:59] failsafe plug-in. If you are doing [489:02] coverage then you have Jakokco. Now [489:05] coverage is the code coverage. So for [489:07] example if you have 100 lines of code [489:09] and you are only testing 80 lines of [489:11] code then your code coverage is 80%. So [489:14] you have a plug-in that can do that for [489:17] you and that plug-in is Jakokco. So if [489:19] you are checking for policies you can [489:22] use enforcers. If you want sbomb, there [489:25] is a plug-in called cyclone DX for that. [489:27] Now see, I am mentioning a lot of [489:30] plug-in names. As DevOps engineers, if [489:33] you are in your learning phase, starting [489:36] phase, do not get intimidated by these [489:39] terms. As you progress with your [489:41] learnings, these terms will start making [489:44] sense to you. And not just when you deal [489:47] a project that has Java, all these terms [489:50] will be frequent in your daily language, [489:53] daily communication. So you will [489:55] remember these terms. But do not think [489:58] that there are so many plugins, there [489:59] are so many tools. It takes time to [490:03] learn. It takes time for a plant to [490:06] grow. Keep patient. Be at it. Be [490:10] consistent. and all these terms, all [490:13] these tools will start making sense and [490:17] you will be able to connect all these. [490:19] So just be at it. Next is produces [490:24] reproducible artifacts integrates [490:27] cleanly with CI/CD pipelines. As we have [490:30] discussed before, you define versions at [490:34] a single place and that single place in [490:37] Maven is pom.xml XML and as long as [490:41] anyone is using that pom.xml along with [490:44] your source code, they will be able to [490:47] generate exactly the same artifact as [490:50] you are in your personal laptop. They [490:52] will be able to do that in the CI server [490:56] or in their personal laptops as long as [491:00] they have your source code and the [491:03] pom.xml. [491:05] The next one is Ant versus Maven versus [491:09] Gradel. Now Ant is legacy now. It was [491:12] used before Maven existed. Now in ANT, [491:16] you had to configure everything and you [491:18] have to write your steps manually and so [491:20] on. So it was configuration. But in [491:23] Maven it is conventions over [491:25] configuration. And then there is Gradel. [491:29] Gradel provides you flexibility. Now 80% [491:32] of the Java projects I have dealt with [491:35] were using Maven. So I wanted to check [491:38] what are the stats. So I looked at Chad [491:41] GPT and Chad GPT said that 60 to 70% of [491:45] Java projects are built using Maven and [491:48] 30 to 40% of Java projects are built [491:51] using Gradel. Now Gradel provides more [491:54] flexibility and it is predominantly used [491:57] with Android. So with Android [492:00] applications you would often see gradal [492:02] being used but in my experience I have [492:04] personally dealt with Maven more at [492:07] least in 80% of the Java projects that I [492:11] have been part of. Now we understand [492:15] what is Maven. So I do not want to [492:18] bombard you with another theory part. [492:20] Now let's go to the CLI and install [492:23] Maven. So what I will do is I will go to [492:27] my browser and I already have created [492:31] one EC2 instance. This is T3 micro and [492:34] it is based on Ubuntu latest Ubuntu. And [492:37] let's power this on. So I have not [492:40] configured anything. I have just created [492:42] the instance and I've stopped it. Now we [492:45] will copy the address and SSH into this. [492:49] So if I go to my terminal, if I run ls [492:53] here, I not this screen, this one. Yeah, [492:56] if I run ls here, I'm in the SSH folder. [492:59] I already have the keep required for [493:02] this instance here. And I will be using [493:04] this key to login into my instance. So [493:07] let's go back to the browser. I will [493:11] click on connect. I will simply just [493:14] copy this SSH command. I will paste this [493:17] command here. I've already given this [493:19] appropriate permissions and let me show [493:22] that to you. If I run ls-la [493:25] then you would see that this key that I [493:28] am using it has strict permissions only [493:31] read and write are with the owner and [493:34] the owner is vun Johi yours truly. So I [493:37] will just copy that again and I will [493:40] paste it here. I will press enter and [493:43] this will connect me to this instance. [493:46] Now I would suggest you to rename this [493:50] instance to Maven like I have done. I [493:53] will you can check the command to do [493:55] that in the GitHub repository. It is [493:57] host name. Ctl can be used to change the [494:00] host name. I've called it Maven for [494:02] simplicity. Now let's first install [494:06] Java. So I will go to my browser and [494:10] then here I will go to the GitHub notes [494:12] for this lecture. I will go to [494:14] installing Maven here and then here is [494:18] the download guide for Maven. I will [494:21] open it. I will zoom in a little. So we [494:25] will be installing the latest version of [494:27] Maven which is 3.9.11. [494:31] This one and it says that Maven 3.9 plus [494:35] requires JDK 8 or above to execute. [494:39] Okay. So let's keep that in mind. What I [494:42] will do is I will right click here and I [494:45] will copy the link. Copy link address. I [494:49] will copy this oft.gz. [494:52] Now I have specifically chosen to [494:56] install this into an EC2 instance so [494:59] that everyone can follow along with me. [495:02] So I would encourage you to spin up an [495:04] instance and follow along with me. So I [495:07] will go to my terminal and here first I [495:11] will run sudo apt update and this will [495:15] update my index. Once this is done I [495:19] will write sudo apt search I will search [495:24] for open jdk. Now when I do this search [495:29] I see jre I see JDK. But in here what we [495:35] want to do friends is we want to install [495:38] JDK not JRE. JRE is Java runtime [495:43] environment and it is installed when you [495:46] want to run Java applications. But we [495:49] are builders, we are DevOps engineers. [495:52] What we want to do is we want to build [495:54] Java applications and then run it. So [495:57] hence we will be installing open JDK. [496:01] JDK is Java development kit. This along [496:06] with the JRE also includes the compiler [496:09] which we will be using while compiling [496:12] our application. So let's do one thing. [496:15] I will be using version 17 for Java. So [496:20] if I run this, hold on. If I run 17 [496:23] here, if I add 17 here, then it will [496:27] tell me all versions of Open JDK that [496:31] are 17. So what I want to install is [496:36] this one. Open JDK this one. So what I [496:39] will do is sudo apt install. I will add [496:43] that and then I will add here Y for yes. [496:47] And this will install Java on my [496:51] machine. So once the installation is [496:53] done, I can run java - version to check [496:58] the version of java that is installed. [497:00] So if you see we have java 17.0.16 [497:05] installed. And if I run java c-en [497:09] version, java c is java compiler, it [497:12] will show me that the compiler version [497:14] that you are using is 17.0.16. [497:17] And if you remember when we were [497:19] discussing our build tools, we said that [497:23] the compiler that is often used for Java [497:26] is Java C, which is why you see Java C [497:29] was automatically installed when we [497:32] installed JDK Java development kit. Now [497:36] I want to install Maven. So I could have [497:39] just written sudo apt install Maven Y. [497:45] Now if I do that the life cycle of Maven [497:48] that I install will be governed by apt [497:53] okay the package manager for my Ubuntu [497:56] system. But if I install Maven using a [498:00] binary like I showed you we copied the [498:02] path of the binary. If I use a binary to [498:06] install Maven then I own the life cycle [498:09] of that tool. So the install, the [498:12] upgrade, the roll back, the cleanup, [498:15] everything will be done by me manually [498:18] or I could automate that process. So [498:20] often when you are dealing with these [498:23] external tools like Maven, Sonar Cube, [498:25] you would see that these are also [498:27] installed in /OP. Now /OP is for [498:31] optional packages. So you would [498:33] typically see that these tools are [498:35] installed there. But I will leave that [498:38] decision to you. I am using a binary so [498:41] that I can also teach you a few more [498:43] Linux concepts while we are learning [498:45] Maven. So I will not run this command [498:48] and in this lecture itself when we move [498:51] to the last demo we will be using this [498:54] command to install Maven. But for now I [498:57] want to show you the binary method and [499:00] why we are running Java is why we [499:03] installed Java first is Maven requires [499:07] Java. So if you see here JDK8 or above [499:10] is required. So if I copy this link [499:12] again, I will copy link address. I will [499:15] go back to my terminal and I will clear [499:17] my screen for better visibility. I will [499:20] cd into opt then I will write sudo wget [499:26] and this is the file I want to get. So [499:29] if I run ls you see that tar.gz is [499:32] downloaded. I will use sudo tar - x for [499:37] extract, v for verbos, z for gzip and f [499:41] for files in this file. So that our [499:44] tar.gz is extracted. So once I run ls I [499:48] see that the file is extracted. Now what [499:50] I can do is I can delete this one [499:53] because I don't need it. I will run sudo [499:56] rm rf. It is apache and there is a [500:00] hyphen bin. I'll press enter and I will [500:02] run ls. Now this is deleted. I just have [500:05] this here. So what I can now do is I can [500:08] cd into Apache. I what I will now do is [500:12] I will rename this for simplicity. So I [500:15] will use move mv Apache and then I will [500:18] call it just Maven and I will also have [500:21] to add a pseudo here. I'll add pseudo. I [500:24] will rename this file. Once it is [500:26] renamed I can go into Maven. And if I do [500:30] ls, I can see that there is a bin [500:33] directory and inside bin directory there [500:35] is the executable that we want to run. [500:39] So if I go to if let's say I add bin mvn [500:45] - version then I can see the version of [500:50] maven. But if I just run mvny-en [500:54] version, this command will fail because [500:57] my shell will not be able to recognize [501:00] mvn. Previously, it ran because I [501:04] provided the complete path of my [501:06] executable. Complete path of my [501:09] executable. In Linux operating system, [501:12] if I run SSH, SSH works for me. It works [501:16] because SSH binary is available in the [501:20] path that my OS is searching for [501:24] executing any binary. So if I run which [501:26] SSH, it will tell me that SSH is located [501:31] in USR bin. And if I echo the path [501:36] variable which tells Linux operating [501:39] system that you have to search these [501:42] paths in order to find any binary. So if [501:46] I press enter, you see user bin is here [501:49] and user bin is also here. So all these [501:53] paths that are searched by the OS are [501:56] separated using a colon here. So for SSH [502:02] we are searching this but for MVN MVN is [502:06] available in /op maven and we don't see [502:11] that here. So we should be adding that [502:15] path in our path variable so that when [502:19] we execute mvn it can run without us [502:24] specifying the full path like this. So [502:27] let's do it. So what I will do is I will [502:30] go to my users bash rc file. So I will [502:34] write vim and then I want to go to bash [502:39] rc here. And then once I'm here I'll [502:42] press shiftg to come at the very last. I [502:45] will press i to go in the insert mode. [502:48] And then what I will do is I will write [502:51] export. And this is how you define [502:54] environment variables. And then here I [502:57] will write path is equal to now what is [503:01] the path for me? I do not want to remove [503:04] that path all together. What I want to [503:07] do is I want to append the existing [503:10] path. So what I will say is my dear [503:12] friend you first take the path that you [503:15] already have right now and what I want [503:19] you to add is I just want you to add [503:21] /op/maven [503:25] /bin in that path and then I will close [503:30] this and once I'm done with this I will [503:33] hit escape I will press shift zz to save [503:36] this now I will source this file so that [503:39] it refreshes and then my shell knows [503:44] about it. I will write bash rc. Now this [503:46] is source. Now if I type which mvn now [503:50] it is telling me your maven is located [503:54] here. And if I run maven - version now [503:58] it will tell me the version of maven as [504:01] well because now my system recognizes [504:04] the maven binary because we have added [504:07] the path for this binary in our path [504:11] variable. So if I echo path now you see [504:16] I have to put a space here you will see [504:19] that opt maven bin is available here. So [504:23] I hope this makes sense. Now we have [504:26] installed Maven. Now let's go into [504:31] building our project. The first project [504:33] that we want to build. I'll clear my [504:35] screen. I will come to my home directory [504:40] and ls. I don't have anything here. So I [504:43] will go to my browser and I will go to [504:45] my GitHub nodes. And then this is the [504:49] demo that I want to do. I'll zoom in a [504:51] little. I will open this now. See [504:55] firstly I told you that Maven is [504:58] convention over configuration. So what [505:00] we want right now is we want a structure [505:03] and a simple source code. So what we [505:06] will be doing in this lecture is we will [505:08] be using something that is provided by [505:11] Maven to us. So if I scroll down you see [505:14] that you can create your own first [505:17] project and you can use this command to [505:21] do that. So what I will right now do is [505:23] I will just copy this command. I will [505:26] copy a format that is much more easy for [505:30] you guys to understand. So let me go to [505:33] my terminal and here I will paste it. [505:37] Now before I run this I want to explain [505:40] this to you. MVN archetype generate. [505:44] Firstly what is archetype? Now archetype [505:48] is project template. Now project [505:50] template will give you the directory [505:52] structure. It will also give you the [505:54] pom.xml. You do not have to write [505:56] pom.xml from the very scratch. There are [505:59] multiple archetypes available. You can [506:02] use based on your requirement. So this [506:04] will provide you a skeleton that you can [506:08] use for your maven and that's pretty [506:10] awesome. You don't have to write pom.xml [506:13] from the very scratch. So we are using a [506:17] very basic archetype that is generate. [506:20] So what is an archetype? An archetype is [506:23] a project template. So you start with a [506:27] runnable skeleton. Now what are the [506:30] fields that we are defining here? Hyphen [506:32] D can be thought of as part of the [506:34] syntax. Then we are providing the group [506:37] ID. Now group ID we will discuss this in [506:41] detail much later but for now just [506:43] understand it's the reverse DNS of your [506:47] website. For example, if my website is [506:49] cloudwithvos.com [506:51] then my group ID will be [506:54] com.cloudwithvos. [506:56] Artifact ID is my app. Now understand if [507:00] cloud with varosh has multiple projects [507:04] then how do I identify those projects [507:07] because there is a group under which all [507:10] those projects reside but which project [507:14] I want to use that is what is provided [507:17] by artifact ID. So group ID is the let's [507:20] say the grouping and artifact ID is [507:23] which entity in that group you want to [507:26] deal with. Here I'm just saying my app. [507:30] Then archetype artifact ID. Now this is [507:34] the artifact ID of the archetype that [507:37] you wish to use. So like I told you [507:40] there are multiple archetypes available [507:43] and the artifact ID for that one is [507:46] Maven archetype quickart. This is the ID [507:50] that we wish to use. Now there could be [507:52] multiple other archetype artifact ids [507:55] which would result in a different pom [507:58] file or a different source code but the [508:01] source code that we will be using is [508:04] part of the archetype quick start then [508:08] this is the version of this 1.5 and this [508:10] is pretty much self-explanatory [508:12] interactive mode is false so I will [508:15] press enter and once I do that this will [508:19] create a project structure for me. So if [508:22] I press ls now you see that my app is [508:26] there. Now I will cd into my app and [508:30] I'll clear my screen. I'll press ls. You [508:33] see two things here. src and pom.xml. So [508:36] let's run tree command. I don't have [508:38] tree installed. So sudo apt install tree [508:43] y. Now what this will do is this will [508:45] install tree and tree is a fantastic [508:48] utility that will tell you the directory [508:51] structure. So if I clear my screen, I [508:54] run tree here, it will tell me that this [508:57] directory has pom.xml [508:59] and this also has source and source has [509:03] this hierarchy and test has this [509:06] hierarchy. So firstly I want you to know [509:09] that you as DevOps engineers will not be [509:14] writing pom.xml from the very scratch. I [509:17] hope that relieves you a little. And [509:20] then the source part also it is not you [509:24] who will be writing this src directory. [509:27] This contains the actual code. So main [509:31] here contains the code and test here [509:34] holds the tests that will be performed [509:38] here we have created it but in actual [509:41] production applications these things [509:44] will be part of your SCM. So when I say [509:47] your pom.xml XML and SRC will be in SCM. [509:51] The first thing that should come to your [509:54] mind is these files should not have any [509:57] credentials, any secrets, any keys [510:00] because even if you are in a private [510:04] repository, you should never commit your [510:07] secrets onto SCM. So first thing is you [510:12] will be storing these in your version [510:14] control. So it could be your GitHub, [510:17] GitLab or any other version control [510:20] system. Now we have briefly discussed [510:24] about pom.xml. [510:26] pom.xml [510:27] is the main Maven file for your project. [510:31] It is the file that is the single source [510:36] of truth for Maven. It tells Maven your [510:39] project's identity, libraries, [510:42] dependencies, build steps. Maven reads [510:46] this file to build, test, package and [510:51] deploy your app the same way every time. [510:54] So understand this file is extremely [510:57] critical. So let's do one thing. I will [510:59] cat into pom.xml. [511:03] So this is as expected in XML format. So [511:07] I want to explain this file to you line [511:11] by line so that you can easily [511:14] understand what happens in pom.xml as [511:17] DevOps engineers. Now you would be [511:20] thinking Vun if we are not writing this [511:22] file we do not have ownership of this [511:25] file then why should we understand this? [511:28] You want to understand this because at [511:30] times most of the build failures will be [511:34] directly related to pom.xml. [511:37] Essentially if your build is failing [511:40] there is something tricky or something [511:42] going on in your pom.xml. [511:46] So until and unless you as DevOps [511:49] engineers understand pom.xml XML because [511:52] the CI/CD ownership is with you. You are [511:56] the one who will be integrating [511:58] different plugins in the pom.xml. [512:01] Your developers will be writing the most [512:03] of it. But if you are using a plug-in [512:06] for security testing, if you are using a [512:08] plug-in for something else, those [512:10] plugins would be added by you in the [512:13] pom.xml. So you want your pom.xml XML to [512:17] be error-free and you as DevOps [512:19] engineers can only ensure that once you [512:22] understand the pom.xml file to begin [512:26] with. So here it is in a black and white [512:30] format. So what I have done is I'll take [512:32] you to Visual Studio Code. I have copied [512:35] this pom.xml exactly like you see here [512:38] to this screen so that I can explain the [512:41] pom.xml. Now before I start explaining [512:44] pom.xml XML. I feel it is imperative you [512:47] first understand XML. So I will just [512:50] take 5 10 minutes to tell you basic XML [512:54] and then we can start with pom.xml. So [512:58] XML, JSON and YAML are all textbased [513:03] format for structuring and exchanging [513:07] data. Though XML is a markup language [513:12] with tags. Now what are tags? We'll see [513:14] that later. While JSON and YAML use key [513:19] value style structures and you are [513:21] wellvered with YAML because most of the [513:24] DevOps tools use YAML and if you have [513:27] done CKA uh and if you search my channel [513:31] you can see JSON path lecture if you [513:33] don't know JSON you can learn more about [513:35] it there. But for now we know what XML [513:39] is. Now let's understand this structure. [513:42] Now, XML uses something called elements. [513:46] So, what you see here, this is an [513:48] element. This is an element. This is an [513:52] element. Then, Waron, what is an [513:53] element? Element is something that [513:56] starts with a tag and then closes with a [513:59] tag. So, for this element, this is the [514:04] starting tag and this is the closing [514:07] tag. for this element. This is the [514:11] starting tag and this is the closing [514:14] tag. Now closing tag has a forward slash [514:18] in it. That is how you identify it. Now [514:22] this is one element. Vun why is this one [514:25] element? So see this element starts with [514:29] project. This is the starting tag and [514:32] this ends with project which is the [514:35] closing tag. So entire this is also an [514:39] element. Then vun how do I [514:40] differentiate? So project here is a [514:44] parent element and then whatever you see [514:46] here the indentation follow the [514:49] indentation this this this all of these [514:52] are child element. Similarly, if I talk [514:55] about this properties element, then [514:58] whatever you see inside it is child [515:02] element of the properties parent [515:05] element. So this is pretty basic and you [515:09] would have seen something similar when [515:11] we talked about YAML or JSON as well. [515:14] You see this starting tag is simple but [515:19] this starting tag has these things as [515:23] well in them. Now anything that is there [515:26] in the starting tag these are called [515:28] attributes. Now this project tag has [515:31] three attributes. Now you do not have to [515:34] understand this deeply because it's just [515:36] touch and go. You uh you will rarely be [515:39] playing with these. These things are [515:41] pretty much this constant in uh you know [515:45] pom.xmls. [515:46] I'm just telling this to you for your [515:48] understanding. And what you see in here [515:51] this is called text. So this element has [515:56] this text under it. Now this is all you [516:00] want to know about XML. Now with that [516:02] understanding let's start discussing [516:06] form.xml. Now this is the basic [516:09] structure that you would see. This is [516:11] the root element. Whatever you see at [516:13] the very top, this is called the root [516:15] element. And that's understandable. This [516:16] one is the root element. And then you [516:20] have these things model version will [516:22] pretty much always be 4.0.0. What I have [516:26] done is I have added comments in this [516:29] XML. And the format in which you add [516:33] comments in XML is this. So any comments [516:36] that you want to have can be under this. [516:39] It's pretty funky but that is what it is [516:43] for you. So you have to add comments [516:46] like this. And you can see here pom [516:48] model version fixed at 4.0.0 for Maven 3 [516:52] and 4. So that's why I said it will [516:54] pretty much always be four in whatever [516:57] projects you are dealing with. Then the [516:59] most important part is G A V. And we [517:04] call these GV coordinates. So when we [517:07] talk about any location on our mother [517:10] earth, we talk about coordinates. So if [517:13] you want to reach there, these are the [517:15] latitude and longitude and if you reach [517:17] there, you will reach that specific [517:19] destination. So essentially coordinates [517:23] are address for a specific location on [517:27] earth. Right? Similarly, if I talk about [517:30] GAV coordinates in Maven, these uniquely [517:35] identify a dependency or a project. We [517:39] will dissect this definition as we move [517:42] ahead. Now, GAV is used to uniquely [517:47] identify a project or dependency whether [517:50] it is in your local or remote. What is [517:55] local and remote? to understand. If you [517:59] talk about container images, at times [518:01] the container images are available in [518:03] your nodes and at times they are [518:06] available in image registries and then [518:08] you pull it from there. Correct? Same [518:11] concept here. You can have your [518:13] artifacts or your dependencies locally [518:17] or you can have them in a remote [518:21] location. So GAV is uniquely identifying [518:26] your project or dependency. But Vun why [518:30] are you using project and dependency [518:32] interchangeably? [518:33] Understand my Maven project could depend [518:37] on another Maven project and if my [518:40] project my Maven project depends on [518:42] another Maven project then how do I [518:45] reference that another project? I [518:48] reference that using GAV because I know [518:52] for a fact that these GAV coordinates [518:55] are unique for any project or any [518:59] dependency. For example, if there is a [519:01] MySQL dependency, it will have a group [519:04] ID. It will have a artifact ID. And once [519:07] you provide these things along with the [519:10] version only, then your Maven instance [519:14] will be able to resolve that or fetch [519:17] that. So if you do not provide version [519:21] for your dependency for example, then [519:24] Maven will not fall back to latest. It [519:27] will not fetch it. So providing these [519:30] coordinates, [519:31] GAV coordinates, project coordinates are [519:35] of utmost importance. So now that we [519:38] understand this, let's dissect G, A and [519:43] V. So here the group ID we have already [519:47] discussed it's reverse of your DNS. So [519:50] here the example that we are using it's [519:52] com.mmy company.app. Then artifact ID is [519:56] module name. Now see understand it this [520:00] way your organization could be dealing [520:04] with multiple microservices correct in [520:08] that scenario what you can do is let's [520:11] say you are talking about application [520:13] one so let's copy this so that you [520:16] understand it in a nice way so I will [520:18] copy this I will come here I'll paste it [520:22] here let's take care of the indentation [520:25] as well let's do one thing Let's call [520:27] this com dot cwj [520:33] app. Now this is for app one let's say. [520:38] Now if this is for app one, app one [520:42] could be consisting of multiple [520:44] microservices and then I could name that [520:47] artifact ID app 1/ [520:52] payments and then another one I could [520:54] name app one hyphen [520:57] orders. I want you to clearly understand [521:02] that when you are dealing with an [521:05] application, your application could be a [521:08] single application that only has one [521:10] pom.xml like this one or you could be [521:14] dealing with an application that has [521:16] multiple pom.xmls. Why multiple pom [521:20] doxmls? Because each micros service will [521:23] have its own pom.xml. [521:27] like I showed you here. If you are [521:28] dealing with this, then you will have [521:30] one micros service with this, another [521:32] micros service for orders, another [521:34] micros service for uh let's say wish [521:36] list and this holds true not just for [521:40] microservices but for also your pom.xml. [521:44] So the number of microservices is [521:46] directly proportional to the number of [521:48] pom.xmls that you have had. Now the [521:52] group ID this depends on how you want to [521:57] keep it. Maybe you have it for per [521:59] application or maybe you have it for [522:03] your entire application stack. If your [522:06] applications are not microservices then [522:08] you could have just app here and then [522:12] one pom.xml could be for app one. [522:15] Another pom.xml could be for app 2 and [522:18] so on. So create that mental picture in [522:21] your head. So now that we understand [522:24] both of these, let's talk about version. [522:28] Version is simply the version of your [522:31] application. But Warun, why do I see a [522:33] snapshot here? Now versions can be of [522:36] two types. They can be releases or they [522:40] can be snapshots. When we are talking [522:42] about releases, releases, think of them [522:45] as production. They are immutable. they [522:48] do not change. But when I talk about [522:50] snapshots, snapshots are that is work [522:54] under construction. So the development [522:56] is still ongoing whenever I am talking [523:00] about snapshots. So in essence when [523:04] someone is developing it, understand [523:08] that they will be pushing it as well. [523:10] Then how do I ensure other developers [523:13] that are using this snapshot they always [523:16] have the updated snapshot with them? Now [523:20] that something is governed by something [523:23] called update policy. So I'll take you [523:26] back to the GitHub repository and if I [523:30] scroll here you see snapshot versus [523:33] releases. The refresh rules for these [523:37] snapshots are defined in update policy. [523:40] Now that can be always, daily, you can [523:43] specify an interval or never. If you [523:46] build your application at a regular [523:48] interval in a day, then I would suggest [523:50] you to choose always and daily. If you [523:53] just build once a day and these things [523:57] are subjective. I will let you choose [523:59] what is best for your use case. Now [524:03] let's go back to our Visual Studio Code. [524:07] Now let's discuss these. Now name is a [524:10] display name. Now this is typically same [524:13] as the artifact ID but you can have a [524:16] different name like this. What it will [524:19] do is this name will appear in all your [524:22] reports and logs. So ideally you should [524:26] keep this same as artifact ID. But if [524:28] your artifact ID is not descriptive [524:31] enough then you can have a different [524:33] name here. URL. Now this does not [524:36] necessarily mean a publicf facing URL. [524:39] This URL could point to your repository [524:43] or it could point to your documentation [524:45] or it could point to the wiki. So if you [524:48] are dealing with let's say a [524:49] microservices design you this pom.xml [524:52] XML belongs to a payment micros service [524:55] then within your organization you will [524:57] have the link of the private let's say [525:00] GitHub repository that holds the code [525:03] and pom.xml for this specific micros [525:07] service. Now let's scroll down what you [525:11] see here is properties maven properties [525:14] buildtime placeholders not OS [525:17] environment variables. Now do not mix [525:19] this with the con concept of environment [525:21] variables but they function kind of in a [525:25] similar way. So what happens is whenever [525:28] you are defining your dependencies you [525:31] have the right to not define the version [525:35] in the dependency section but ask the [525:38] dependency section to refer the [525:41] properties section when they want to [525:43] check the version of a certain [525:45] dependency. Now Vun why is this useful? [525:48] So imagine in production implementations [525:51] your pom.xml could have hundreds of [525:54] dependencies. It is futile. It is not [525:57] worth your time to search for those [526:00] dependencies and maybe change the [526:02] version. What you can do is you can [526:05] directly come to the properties section, [526:07] check the version that you wish to [526:10] change and change it here because anyway [526:13] the dependencies are referencing the [526:15] properties section to fetch the version. [526:18] So let me show this to you. I'll go to [526:21] the browser and then I will scroll down. [526:24] Here you see properties and you see what [526:26] I have done is these two properties are [526:30] built in. So for example there is no [526:33] dependency that is referencing these [526:35] properties explicitly implicitly under [526:38] the hood Maven is using them but for you [526:42] to understand what I have done is I have [526:44] added two more lines here you see I have [526:46] added two more properties one is spring [526:50] boot version 3.3.4 [526:52] and then there is a JUnit version which [526:55] is 5.11.0 zero. Now, how do I reference [526:59] this? In the dependency section, if I [527:01] scroll down, you see in the version [527:04] section, I want you to clearly [527:06] understand that whenever we are talking [527:08] about a dependency or a project, what is [527:12] the thing that uniquely identifies them? [527:14] It is the group ID, the artifact ID and [527:17] the version. This is the GV coordinates. [527:20] So, keep this in mind. Version, [527:24] supplying version is important. If you [527:26] do not specify any version here then [527:29] this will fail because Maven will not [527:32] automatically choose the latest version. [527:34] It will just fail. So you must provide [527:37] the version of the dependency that you [527:40] wish to use. So here we are saying when [527:42] you want to check the version for this [527:44] dependency you go to the properties [527:46] section and check for spring boot [527:48] version. And here it is spring boot [527:50] version. If you want to check the [527:52] version for this dependency, you go to [527:55] junit.ver version and here is junit.v [527:58] and it will pick 5.11.0 [528:01] based on what we have defined in the [528:04] properties section. So you would often [528:06] see that properties section is used [528:08] heavily in production implementations. [528:11] Now let's go back to our visual studio [528:14] code and understand what is dependency [528:18] management and dependency. Right off the [528:21] bat, I want you to clearly understand [528:24] that dependency management is not [528:27] downloading any dependencies for you. It [528:30] is just there to tell the version of the [528:34] dependency that you have not specified [528:37] in the dependency section. So for [528:39] example, if you have not specified the [528:41] version for a dependency in the [528:43] dependencies section itself like you see [528:46] here there is no version here version [528:48] tag here then it will check with the [528:51] dependency management section. So that [528:53] is what dependency management is doing. [528:56] So in a single line. So dependencies [528:58] what actually gets pulled into your [529:00] project. Dependency management is the [529:03] version rule book used only when [529:07] dependency doesn't specify its own [529:10] version. So here you see both of these [529:13] dependencies they have not specified its [529:16] version which is why dependency [529:19] management section is referenced. Now, [529:22] now you might be thinking, Vun, I see [529:26] JUnit Jupiter API here and I see JUnit [529:30] Jupiter PAMS here, but I do not see an [529:35] artifact ID with a same name. I just see [529:38] that you are referencing something [529:41] called JUnit bomb. You are importing [529:45] something called JUnit bomb. It does not [529:49] have these names. Then how will this [529:52] work? Here I want to explain the [529:55] difference between Sbomb and Maven bomb. [529:58] So you see it's a JUnit bomb. Here Sbomb [530:01] is something that we discussed in our [530:03] Jenkins lectures, CI/CD lectures where [530:06] we learned that SBOM is software bill of [530:09] material. This will hold everything that [530:12] your software is using. Any library, any [530:14] dependency that your software is using [530:17] is part of SBOM. And sbomb is good [530:20] because it tells you that these are the [530:22] libraries that you are using and some of [530:24] these libraries require licenses. So you [530:27] may have to pay someone to use these [530:29] libraries. So sbomb will tell you things [530:32] like this whether there are any [530:33] vulnerabilities in the dependencies that [530:35] you are using in the libraries that you [530:37] are using. So sbomb is good for that. Do [530:39] not confuse sbomb with maven bomb. Maven [530:43] bomb says that whatever I am downloading [530:48] this has everything about that specific [530:52] thing. So here junit bomb will have the [530:57] version information. I'm not saying it [530:59] is a dependency. It is downloading some [531:01] dependency. It is downloading a [531:02] dependency with a specific version. No, [531:04] I'm saying junit bomb will have the [531:08] versions of everything JUnit. So here we [531:13] are using JUnit Jupiter params and here [531:17] we are using JUnit Jupiter API. So this [531:21] JUnit bomb will have the versions of all [531:26] these dependencies and we know for a [531:28] fact if a dependency is not defining a [531:31] version it will not be downloaded. What [531:33] it will do is this dependency will check [531:36] the JUnit bomb and it will find the [531:38] version that is to be used for JUnit [531:41] Jupyter API and the version that is to [531:44] be used for JUnit Jupiter params and it [531:47] will use it. So a bomb is essentially a [531:51] collection of all the versions that are [531:54] specific to that something and that [531:57] something here is Junit. Now another [532:00] thing I want you to understand is the [532:03] type is pom and here the scope is test. [532:07] Now this type is fine a pom we are [532:10] calling it pom because it just has the [532:12] version information. The default is jar [532:15] here type for dependencies or for [532:17] dependency management. But we are saying [532:20] pom because this junit bomb will just [532:23] hold the versions not the actual [532:24] dependencies. We have discussed this at [532:27] stretch. Okay. Now the scope is just [532:30] import and the scope here is test and we [532:33] will now see what scope means. So let's [532:37] go back to our GitHub notes and again I [532:40] would encourage you to go and read the [532:42] GitHub notes. Whatever I am discussing [532:45] is there in the GitHub notes. You see [532:47] here dependency management bombs. It is [532:51] the versions version rule book. So if I [532:54] scroll down we have already discussed [532:56] these. Now I want to discuss scope. Now [532:59] see when you are building a project you [533:02] will be performing different things. You [533:04] will be compiling, you will be testing, [533:06] you will be running the application and [533:08] so on. Now you don't want that whatever [533:12] was generated during the test should be [533:15] available in at runtime or during [533:19] compilation. Right? because it is [533:21] unnecessarily increasing the attack [533:23] surface because you must choose the [533:27] narrowest scope that fits to reduce size [533:32] and attack surface. You choose the scope [533:35] that is needed for that specific [533:39] dependency. Now what it is a switch that [533:42] defines which class paths the dependency [533:45] appears on. bundling depends on project [533:48] packaging and plug-in. So let's see what [533:51] I mean by this. Let's see this with an [533:53] example. So if I am using a dependency [533:57] called JUnit Jupiter params and JUnit [534:01] API here I have defined the scope as [534:04] test which is a good practice because I [534:07] don't want this dependency to be [534:11] available at runtime. I don't want that. [534:15] And if I would have not defined this [534:17] scope, the compile scope would have been [534:20] taken. And in compile scope, the [534:22] dependencies available in the compile [534:24] part as well, in the test part as well, [534:25] in the runtime part as well. I don't [534:28] want that at all. I just want the [534:32] dependency to be available for the stage [534:35] it is needed. If I'm performing testing, [534:37] then it should be only available for the [534:40] test class path. If I scroll down, I [534:42] would again encourage you to read all of [534:46] these. Now, let's discuss the last part [534:49] here. This last part is about plugins. [534:53] Now, when I talk about Maven, it is [534:57] plugins that are doing the actual work [535:00] for you. So, be it your build, be it [535:02] your compile, test or package, whatever [535:06] work is being done is done by plugins. [535:10] Now in this section in plug-in [535:13] management section you just pin the [535:16] version of plugins that are to be used. [535:19] So you see if I'm talking about [535:22] compiling then the version of the [535:24] compiler that should be used is 3.13. If [535:27] I'm talking about tests shorefire [535:30] plug-in the version is this. If I'm [535:33] talking about packaging packaging to jar [535:36] format then the plug-in that I want to [535:38] use is this. And the version that should [535:40] be used is 3.4.2. [535:43] As simple as that. Now you have a good [535:48] understanding of pom.xml. [535:51] With this understanding, let's see what [535:54] I want to discuss next. Now next is [535:57] another theory session. I'm sorry for [535:59] that. But see, until unless you [536:02] understand the theory, the demos will [536:04] not make sense. So we will have to do [536:07] that as well. So I'll scroll down and I [536:10] want to discuss build life cycle phases [536:14] plugins and goals. Maven follows a life [536:18] cycle made of ordered phases. At each [536:22] phase plugins goal perform the actual [536:26] work. Okay. Now like I told before [536:30] plugins are the ones that are actually [536:33] doing all the leg work for you. So life [536:37] cycle is divided into three parts. Okay. [536:41] So if I talk about life cycle, there is [536:43] a default life cycle, there is a clean [536:45] life cycle and there is a site life [536:47] cycle. Now each life cycle has different [536:50] phases. This one has 23, this one has [536:54] three and this one has four. So if I [536:56] take you to the documentation, if I go [536:58] to GitHub documentation and then I [537:00] scroll down. So there are 23 phases in [537:05] the default life cycle but we only [537:09] discuss the ones that are mostly used. [537:12] So if you want to see all the phases [537:14] then you can refer this link and this [537:16] will show you all the phases that are [537:19] there in the default life cycle. So [537:21] there are so many phases. What I want [537:24] you to understand is let me go back to [537:27] draw.io. [537:29] If you run the test phase, it will also [537:32] run the compile phase and validate [537:35] phase. If you run the deploy phase, it [537:38] will run all the phases that are before [537:41] deploy. So it works like this. If you [537:45] run verify, then it will run all these [537:47] phases before verify. So it is if you go [537:51] to the documentation, if you run test [537:53] phase, then all these will be run by [537:57] default. So all the previous phases of [537:59] the phase that you are running will be [538:01] run when you run a phase. I hope that [538:05] makes sense. Now first phase is [538:08] validate. Now again I want you to know [538:10] that life cycle and phases these are [538:13] logical concepts. Okay? These are just [538:16] concepts. They are not doing the actual [538:18] leg work. Actual leg work is done by [538:21] plugins. And we'll see how that works. [538:24] So this is self-explanatory. Validate [538:26] phase it will validate your pom.xml. [538:29] Compile phase will compile your main [538:32] source code intoclass. Then there is [538:34] test. It will run tests. Package it will [538:37] package it into jar war. Integration [538:39] test will in execute integration tests. [538:43] Verify run post package checks and [538:46] quality gates. Install. [538:48] These two are important and these are [538:51] relevant to the architecture that Maven [538:55] uses for storing information. Now we [538:58] have not discussed about repositories [539:00] yet but for now just understand that [539:03] there could be a local repository and [539:06] then there could be a remote repository. [539:08] With that understanding let's understand [539:10] these definitions. Install will publish [539:14] the artifact. Publishing the artifact [539:16] implies uploading the artifact to the [539:19] local repository. The for example, if I [539:21] have installed Maven onto my Ubuntu EC2 [539:25] instance, then it will just upload or [539:29] publish the artifact in the local [539:31] repository. And where that local [539:32] repository is, we'll see that later. But [539:35] for now just understand install part [539:38] deals with local repository and deploy [539:42] part deals with remote repository. So in [539:45] deploy part you publish the artifact to [539:47] a remote repository. Then we talk about [539:50] the clean life cycle. As the name [539:52] implies it is required for cleaning. Now [539:55] vun why do I need cleaning? and you need [539:58] cleaning and this cleaning is relevant [540:01] to the artifact that is being generated. [540:04] Now understand when you already have the [540:07] source code you can use Maven commands [540:10] to generate an artifact out of that and [540:13] once that artifact is generated and you [540:16] have done the upload of that artifact [540:18] you don't want that artifact to stay [540:20] there right that can be deleted because [540:22] if you want that again you can again [540:24] generate that artifact because you [540:26] already have the source code and how [540:28] this is all done we will see that but [540:31] for now understand that cleaning deals [540:34] with cleaning of the artifacts that are [540:37] generated. So artifacts as in dot jar or [540:40] war and it is specifically also telling [540:42] you the directory. So right now when we [540:45] saw the project structure for my app you [540:49] saw there was pom.xml [540:51] and there was src. So if I run ls you [540:54] saw pom.xml and src. Whenever you do a [540:58] build, whenever you do a packaging for [541:01] example, you would see that a new [541:03] directory gets created called target and [541:06] this target directory holds the artifact [541:10] for you. So that is what we are saying [541:12] here. Remove outputs from previous [541:14] builds. Now there are certain tasks that [541:17] you can perform pre-clean and then there [541:20] are certain tasks that you can perform [541:22] post clean. So these are just [541:23] referencing them. Then there is the last [541:28] life cycle called the site life cycle [541:30] which has four phases. Now this deals [541:33] with the site documentation about the [541:36] site generation. So there are different [541:39] phases here. Pre-sight site posts site [541:42] site deploy. I have written about them [541:45] in the GitHub repository. For us DevOps [541:48] engineers these two phases are of utmost [541:51] importance. [541:52] though we should also know about this. [541:55] So for now I will let you read about [541:58] this and let's see more about these [542:02] phases. Now what did I tell you? These [542:04] phases are not the ones that are [542:07] actually doing the leg work. So if I go [542:09] to my terminal and I clear my screen I [542:13] will just run maven [542:16] validate. Okay. And I'm running maven [542:20] validate because validate is a phase. [542:23] Understand? I am running a phase. I'll [542:26] press enter. And this will validate my [542:29] project structure and my pom.xml and [542:31] will give me an output which is okay. [542:35] And as you know if I now run maven [542:39] compile it will run the compile but it [542:41] will also run validate because the [542:43] previous phase automatically gets [542:45] executed. So this is understandable but [542:47] I also said that these phases are not [542:50] actually doing any leg work. Leg work is [542:53] actually done by your plug-in. So let's [542:56] go to the GitHub repository and I want [542:59] to show that to you. If I scroll down, [543:02] yeah, this is the chart that I want to [543:05] show you. If I talk about the default [543:08] phase. So whenever you are writing MVN [543:11] compile for example then behind the [543:15] scenes the plug-in that is being used is [543:18] Maven compiler plug-in and now I want [543:22] you to correlate. So if I go to my [543:25] visual studio code and if I scroll down [543:28] we defined the version of Maven compiler [543:32] plug-in as well here 3.13 just for your [543:36] information so that you can correlate [543:38] these things. So if I go back [543:41] whenever I'm running mvn compile under [543:45] the hood this plug-in is being used and [543:48] a plug-in can have multiple goals. When [543:53] you run mvn compile it is the compiler [543:57] plugins compile goal that is being [544:01] triggered. So this is the only thing [544:04] that you want to take away from this. [544:07] When you are running a phase, a phase [544:10] will be supported by a plug-in. And that [544:15] plug-in can have multiple goals and when [544:18] you run MVN compile, the compiler [544:21] plugins compile goal gets executed. So a [544:27] plug-in a tool Maven can use and that [544:31] plug-in could be compiler testr runner [544:33] packager deployer and you have seen [544:35] these plugins here right shorefire [544:37] plug-in now if it is maven compiler [544:40] plug-in it will be just called compiler [544:42] if you have maven shorefire plug-in it [544:44] will just be called shorefire for [544:46] simplicity and if I come here if I run [544:50] let's say mvn test or let's say mvn [544:54] package if I run mvn package package [544:56] then Maven JAR plug-in gets used and the [545:00] JAR plugins JAR goal is invoked again if [545:06] you are running MVN test compile then [545:10] also the compiler plug-in is used but [545:12] this time compiler plugins test compile [545:16] goal is used so understanding this is [545:18] very important when you are dealing with [545:21] plugins plugins can have multiple goals [545:24] and based on which phase you have run [545:26] that goal is execute. And now someone is [545:29] thinking we have just run mvn compile. [545:33] So let's run this command as well. So [545:36] let me go here and I will run ls and [545:39] let's run mvn compile. And if I run mvn [545:44] compile and press enter you know that [545:47] when compilation runs a previous phase [545:51] automatically gets run and that previous [545:53] phase is validate. So if I go to my [545:56] diagrams, if I'm running mvn compile, we [545:59] know validate automatically runs. And if [546:01] you have to see the extensive list here, [546:03] if I go here, if you run compile, all of [546:07] these things are getting done. Okay, but [546:10] we have not explicitly mentioned them [546:12] because we are only dealing with the [546:14] most used phases. But understand when [546:16] you are running compile, all these are [546:19] also getting executed. But the [546:22] interesting part that I want to show you [546:24] is if I go to this if I run [546:29] mvn compiler compile then only [546:34] compilation will be done. The previous [546:37] phases will not run. Why? Because I am [546:41] directly invoking a goal. I'm directly [546:44] executing a goal. I'm saying just [546:47] compile. But if you run mvn compile it [546:50] will also validate it will also run all [546:54] the other phases like you saw before. So [546:56] what I will do is if I run ls now you [546:59] see there is a target folder created. So [547:03] if I run mvn clean now what it will do [547:06] is mn clean this a spell error mvn clean [547:10] it will just delete the target folder. [547:14] So if I press ls it is just so show [547:17] showing source again. So let's run mvn [547:21] compiler [547:22] compile. And if I run this, this will [547:26] not perform any validation. This will [547:29] just run this goal. And it will be [547:32] evident from the output you see. And if [547:36] you see this output just compiler is [547:39] here, right? And if I scroll above, if I [547:42] see the output for MVN compile, you see [547:46] resources is also there because it is [547:49] explicitly showing that if I go to my [547:52] browser, if I go here, then you see [547:55] there is something related to resources [547:58] that also runs when you run the phase. [548:02] But this time we only ran a goal which [548:06] is why you only see compile here. So [548:08] understanding these nuances are [548:10] important because this somehow decreases [548:14] the time your pipeline takes to run. For [548:17] example, if I were to run MVN install, [548:22] then all these phases will run. But if I [548:25] am sure that MVN install will work just [548:28] fine, then why not just run a goal with [548:33] a plug-in that will just publish the [548:35] artifact to local repository. So [548:38] understand why there are features why [548:41] there are functionalities available you [548:43] would be using them based on your [548:46] requirement. So for example if you run [548:49] mvn deploy it will run all the previous [548:51] commands but what if you run mvn deploy [548:54] deploy it will only deploy it will not [548:58] perform all these steps. So keep these [549:02] things in mind. Now that we have this [549:05] understanding, I want to take you back [549:09] to a demo. So I will go to my terminal. [549:13] And here we have already seen few [549:15] commands. We have seen mvn validate. We [549:18] have seen mvn compile. And now what I [549:22] want to show you is mvn. So first let's [549:25] run tree. I'll run tree. And this shows [549:29] me that target has classes. And you see [549:32] this hierarchy. This is done based on [549:36] whatever was there in our pom. So app my [549:39] company com and if I go here app my [549:43] company com. So this directory structure [549:46] is taken care by maven itself. We did [549:49] not have to do anything there. So here [549:52] what I will now run is I can run java. I [549:56] want to run this application. So I will [549:58] run java class path. Now where is my [550:02] class located? So I know that it is [550:04] target and in target we have classes and [550:08] inside classes what I want you to do is [550:12] I want you to run this com dot my [550:16] company [550:18] dot app and here I want to run dot app [550:24] this one okay I am specifying a target [550:27] and then I will press enter and there is [550:29] a spelling error it should be company [550:32] And if I press enter, you see hello [550:35] world. And that is what this application [550:38] is supposed to do. And what I can also [550:41] do is I can run MVN package. And what [550:44] will MVN package do? See, we do not see [550:46] a jar file yet. We see a dot class [550:49] because what we ran before was MVN [550:52] compile. What MVN compile will do for [550:55] you is it will change your source code [550:58] into class. But what MVN package will do [551:01] is we learned it here package will [551:03] convert your source code into jar war or [551:07] whatever format you define for us it is [551:10] jar. So if I run mvn package it will [551:13] create dot jar file for me. So let's run [551:17] this and once this is run I'll run tree [551:18] again and you see in the target itself [551:21] there is my app one. You see what format [551:24] this is adhering to. We have defined [551:27] this in our [551:29] GAV itself. So if I go here, this is the [551:34] 1.0 - snapshot. 1.0 [551:39] - snapshot. And this my app is coming [551:42] from here. I hope all this is making [551:46] sense to you. Now if I want to run this [551:48] jar, what I can do is I can write java - [551:53] cp. And this time my target is [551:57] previously our target was classes but [552:00] this time my location of this is in in [552:04] the targets directory I want to go to my [552:07] app dot jar and then I will just specify [552:12] the same thing com dot my company [552:17] dot app dot app. Now this might feel a [552:22] little off to you if you are not from [552:24] development background but whenever you [552:27] are running a Java application this is [552:29] how you run Java application. So it is [552:33] once you start working you will [552:35] understand these things. If I press [552:36] enter this is saying hello world. Now [552:40] let's run mvn install. Now what MVN [552:44] install would do is it will publish the [552:47] artifact onto your local repository. But [552:52] Warun where is the local repository and [552:55] it is here. It is pointing to your local [552:59] repository. It is saying that whatever [553:02] artifact you have generated right now [553:05] that artifact has been published onto [553:09] the local repository. If I would have [553:12] run mvn deploy then it would have [553:15] published the artifact onto a remote. [553:18] But have we defined a remote yet? We [553:20] haven't. Right? So if I here run ls and [553:24] I supply this then of course you would [553:28] see that there is a snapshot file there [553:31] and along with the snapshot it also has [553:35] pom and some xml metadata as well in [553:40] this location. So for now mvn install we [553:43] have seen. Now let's run mvn deploy. mvn [553:47] deploy will fail because we have not [553:50] defined a remote where this artifact [553:53] could be published. Now I want to show [553:56] you a few more things. So mvn clean as [554:00] we know will clean up our target [554:03] directory. So if I run ls I don't see [554:05] target anymore. I usually in projects [554:09] what you would see is two phases are [554:12] called simultaneously. So mvn clean will [554:15] be called first and then install will be [554:18] called. So what it will do is it will [554:20] first clean the target directory and [554:23] then it will run mvn install. So this is [554:26] a step-by-step process. First mvn clean [554:28] will run then mvn install will done run. [554:32] And we know for a fact that mvn install [554:35] will again recreate this / target [554:38] directory. So if we run this then if I [554:41] press ls you would see that /ashtarget [554:45] is again created for you and mvn clean [554:48] as we know will again remove this. Now [554:51] we can also run mvn site this will [554:55] generate the site data for you and if [554:58] you run tree after running this let's [555:00] first run ls then I will run tree so [555:03] that you can see what all things are [555:05] generated with the site. So ls if I do [555:09] tree then you see in the target the side [555:13] data is also created for us. Now I will [555:16] let you explore this. Now that we have [555:18] gotten this understanding let's go and [555:21] understand repositories. So if I go here [555:24] what is a repository? Repository stores [555:28] artifacts [555:30] and metadata. And when I showed you the [555:32] local repository just now, you saw that [555:35] there were files with extension.com. [555:38] There were files with the name metadata [555:40] in it. And there were also artifacts. [555:43] What were the artifact? The artifact [555:46] that we saw was ajar snapshot version. [555:50] And that is what our palm was intended [555:54] to generate. So repository is a place [555:57] where we store artifacts. That point is [556:00] clear. On the right of your screen, what [556:03] you see is a local repository and then [556:06] remote repositories. [556:08] Local repository is maintained in the [556:12] machine that has Maven installed. So in [556:15] my demo my EC2 instance had Maven which [556:20] is why a local repository was there and [556:23] local repository is in the M2 directory [556:26] that we already know from our previous [556:29] demo. But Wun why are remote [556:32] repositories needed? Now understand when [556:35] you are downloading certain [556:36] dependencies, your application may [556:38] require dependencies that are not [556:40] available within your machine which are [556:42] not available in the local and why they [556:45] are not available in the local. [556:47] Understand that the local repository is [556:50] supposed to cache whatever dependencies [556:53] are needed by your project. So first the [556:57] dependencies must be fetched from [556:59] somewhere remote and once they are [557:01] fetched they can be cached into your [557:04] local repository. Your local repository [557:06] is there to speed up your builds. Now [557:11] you might be thinking what are the [557:13] examples of remote then remote could be [557:16] Maven central. Now, Maven Central will [557:18] hold the most popular, you know, public [557:21] dependencies that can be leveraged by [557:23] anyone. And similarly, your project [557:26] could also be using that the example [557:28] that I gave you before. For example, if [557:31] I'm using a library that has the logic [557:34] for calculator, then that logic can of [557:37] course be found in Maven Central. So, [557:40] let me show that to you. So if I go to [557:42] my browser and I write Maven central [557:46] here then I should see MVN repository [557:50] central. So yeah this is the repository. [557:52] So if I search for let's say my SQL then [557:56] I will see my SQL connector is this one. [557:59] Let's see my SQL connector for Java. So [558:01] let's click here. And if I click here it [558:04] is saying that category is JDBC drivers. [558:07] And if I go to any of the versions that [558:10] I wish to use, let's go to 8.0.33. [558:14] Then it will tell me how can I use this [558:17] dependency. So if I add this in my [558:19] pom.xml, [558:21] then it will fetch this specific [558:24] dependency from maven central. Now will [558:26] it always fetch from from maven central? [558:29] It depends on what are your precedents. [558:33] Now precedence we will see in a little [558:35] while. But for now I hope you got the [558:38] idea. So let's go back here. Now remote [558:41] can be of multiple types. I will tell [558:43] you the most popular remotes that I have [558:46] seen. First is Maven central. You pretty [558:49] much will always have that and this is [558:51] something that is configured in your [558:53] super pom. What is super? We will see it [558:56] in a little while. Then you have vendor [558:59] repos. Let's say you are using a custom [559:01] application. Now custom application will [559:04] also have let's say custom dependencies [559:06] which that vendor has specifically [559:08] created for that application. Now that [559:10] you are using that application and [559:12] building that application, you would [559:13] want to use those dependencies as well. [559:16] So such dependencies are fetched from [559:20] vendor repositories. You could be using [559:23] vendor repositories or could not be [559:24] using vendor repositories. It depends on [559:27] what is your use case. Whether you are [559:29] using a custom application, building a [559:31] custom application whose pom fetches [559:34] dependencies from somewhere remote. So [559:36] that is one option from somewhere remote [559:38] that is a vendor repository. Then the [559:41] third type you would see is company [559:43] managed and this is something that you [559:45] will see 100% of the times when you are [559:47] dealing with production implementations. [559:50] Companymanaged repositories are the [559:53] repositories that are company specific. [559:56] So for example you know that these five [559:59] dependencies most of my project use then [560:02] you will upload those dependencies in [560:04] your company managed repository. Now [560:07] company managed repositories may have [560:09] access to internet or they may not have [560:12] access to internet depending on how [560:15] tightly governed your environment is. If [560:19] for example your company managed [560:20] repository doesn't have access to [560:22] internet then all the dependencies that [560:25] are required by your application you [560:28] will pre-download those dependencies [560:30] onto your company managed repository. [560:34] Now what are the examples of company [560:35] managed repositories? You would have [560:37] heard of sonata type nexus. Sona type [560:40] nexus is a popular choice. Then another [560:43] popular choice is JROG's artifactory. If [560:46] you are onto a cloud provider, you could [560:50] be using the native repository provided [560:53] by your cloud provider. So in summary, [560:57] repositories can be categorized into two [560:59] local and remote. Local is something [561:02] that is available in the instance where [561:04] Maven is running and local will cache [561:06] any dependencies that were downloaded [561:08] from outside so that your next builds [561:12] can run in a speedy way. So caching [561:15] enables faster repeatable offline [561:17] builds. Then remotes can be of three [561:21] types in general. There is central [561:23] repository that is configured via [561:25] superom. Then you have vendor repos. Now [561:27] these are optional depending on your [561:29] sort of deployment. But you would pretty [561:31] much always see company managed [561:33] repositories. [561:35] Now that we have this basic [561:37] understanding, let's go to the next [561:39] slide so that we can deeply understand [561:42] what a production grade architecture [561:45] looks like. Here I have just given you [561:47] the definitions of local and remote. [561:50] Let's see how production implementations [561:53] may look like. So this is the production [561:55] flow with match all mirror. So images on [561:58] the left can be ignored for now. Let's [562:01] concentrate on this diagram. Like the [562:03] previous diagram, we have the same [562:05] number of components. There also we had [562:07] a total of four repositories. Here also [562:10] we have a total of four repositories. 1 [562:13] 2 3 and four. Now this one is local. [562:17] What we have configured here is we are [562:20] telling our Maven for any dependency [562:23] that you want to resolve, you reach out [562:26] to the mirror and mirror will know what [562:31] to do. So for example, let's say I'm [562:33] fetching a dependency. If mirror has it, [562:36] mirror will hand over that dependency to [562:39] Maven. If let's say mirror doesn't have [562:41] it, then you as the administrator of [562:45] this mirror would have defined how to [562:50] get the dependencies that you do not [562:52] have stored. So that mirror could be [562:54] reaching out to vendor repos or it could [562:56] be reaching out to central repository to [563:00] resolve that dependency. But understand [563:04] in this scenario your mirror must have [563:07] means to resolve that dependency. Now [563:10] why I'm using the word must is because [563:13] of this asterisk. Let's see why this [563:16] matters. So what we are saying is any [563:20] request that is generated you just send [563:22] it to the mirror maven and mirror will [563:24] help you out. Now mirror if it has it [563:28] will give the dependency otherwise it [563:30] will fetch it from here. But if you have [563:33] not configured your mirror to fetch the [563:35] dependencies from central and vendor [563:37] repos then your command will fail. Maven [563:42] will not reach out to Maven central if [563:46] you are using an asterisk here because [563:49] what you are essentially telling Maven [563:51] is mirror has all my answers. You reach [563:54] out to mirror and then mirror will reach [563:56] out to the required repos to fetch the [563:59] dependency. But if you have not [564:01] configured your mirror like that and you [564:03] are using asterisk then the dependency [564:05] resolution will fail. Now I want to [564:08] explain two more things. There are two [564:10] things. There is downloading [564:12] dependencies and then there is uploading [564:14] artifacts. So for downloading [564:16] dependencies, we just learned what the [564:18] architecture may look like. But what [564:21] about uploading dependencies? Uploading [564:24] your artifacts. How do you do that? And [564:27] you might be thinking, why do I want to [564:28] upload my artifacts? Veron, you want to [564:30] upload it because other projects could [564:32] be using your artifact. Maybe the [564:35] project two that Vun is working on [564:37] requires the artifact that project A has [564:40] generated. Again understand that these [564:43] artifacts what are these? These are JAR [564:45] files. These are war files. These are [564:47] ear files. And what are these [564:49] dependencies? These dependencies are [564:50] also these JAR files. These ear files. [564:53] These war files that your project might [564:56] need. So keep your mind open to [564:59] understand these things. So the answer [565:01] to why publish and why publish is when [565:04] you run MVN deploy your artifact is [565:08] published onto a remote repository. So [565:10] why publish? You publish jars so that [565:13] other applications other services or [565:16] other teams can use them as dependency [565:20] in their project and they can reliably [565:23] pull them from remote repositories. And [565:25] the concept is same as container images. [565:27] you build once push the container image [565:30] to a container registry image registry [565:32] and anyone who needs can pull and use [565:36] that image. Similarly, we have the [565:38] concept of artifacts here. Artifacts are [565:42] uploaded and anyone who wants to use [565:44] that artifact can use it. Now, upload [565:47] could also be of two types. You know, [565:50] you have two versions release and [565:52] snapshot. So both of these repositories [565:55] they are contained in the Nexus or JROG [565:57] itself. I've shown them outside so that [566:00] you can bifurcate it easily. So any [566:04] release version is added here. Any [566:06] snapshot version is added here. And [566:09] these both are part of this your Nexus [566:11] or JROG itself. So now let's move to the [566:14] left and see how these things are [566:17] configured. Firstly I want you to pay [566:20] attention to this part. What I'm saying [566:22] here is in home directory of the user [566:25] you have M2 folder and there you have [566:27] settings.xml. So if I go to my terminal [566:30] and I go one step back and if I hit ls- [566:34] a you see there is an M2 folder. If I [566:38] run ls on M2 directory I don't see a [566:42] settings.xml. So you will have to create [566:45] this settings.xml. Now another option is [566:48] this setting. If I create settings doxml [566:51] here, that will be specific to this [566:53] Ubuntu user. But what if I want a global [566:57] configuration? Firstly, I wouldn't [566:59] suggest having a global configuration [567:01] because if you have a global [567:02] configuration, then all your users can [567:06] access your global configuration. So for [567:08] example, right now I have user Ubuntu. [567:10] But next let's say I create a user [567:12] Romesh. That user Romesh will also have [567:15] access to this settings.xml. XML and you [567:19] might not want that. Hence, what I [567:21] suggest is only your Maven user or your [567:26] CI server should have access to [567:28] settings.xml. So, you create this onto [567:32] the user that you will use for Maven. So [567:35] here our user is Ubuntu which is why you [567:39] see in my diagram I have created this [567:41] settings.xml XML in the home directory [567:45] of Ubuntu user and I have defined a [567:48] mirror here. So what I am saying is I [567:50] have an asterisk here. Anything any [567:53] dependency you want to resolve you send [567:56] it to the mirror and the mirror will [567:58] have the information on what to do with [568:01] that. Then if you see my pom.xml [568:05] in pom.xml I'm defining my deploy [568:09] targets. I'm saying that if you want to [568:11] deploy releases, you go to Maven [568:14] releases. This one nexus.example.com [568:18] repository Maven releases. If you are [568:20] deploying snapshot versions, then you go [568:22] to the same path but at the very last [568:25] replace with snapshots and you will be [568:28] storing them here. Now understand that [568:32] pom.xml will be part of your source [568:35] control. So it will be pushed onto your [568:37] you know GitHub, GitLab and your version [568:40] control systems. Hence you do not want [568:42] to store credentials into your pom.xml. [568:45] So the recommendation is you store your [568:48] credentials in your user in your user's [568:51] home directory that is supposed to be [568:53] used for maven and that is exactly what [568:55] we have done here. If we would have [568:57] defined this globally then all the users [568:59] would have access to this file and we do [569:01] not want that. Hence I recommended have [569:05] this details have the mirror and the you [569:08] know credentials for the remote [569:11] repository for your mirror repository in [569:14] your settings.xml which is part of the [569:17] home directory of the user that is used [569:19] for Maven. I hope this is clear. Now I [569:22] want to go to the GitHub repository and [569:25] show you a few more things. So if I go [569:28] here, this is what we have already [569:32] discussed. Here you have what is a [569:35] repository and here you can see and I [569:37] have given all the recommendations that [569:39] you can use here. Mirror, declared [569:42] remote, what is mirror and we have [569:44] defined that here as well. Deploy versus [569:47] resolve whatever we have discussed just [569:49] now I have defined it here as well. Now [569:53] C this is the picture that I have in the [569:56] diagrams A B this is also what I have in [569:59] the diagrams. Now this is C. So if you [570:03] are not using match all mirrors then you [570:05] would supply your remotes like this. You [570:09] have defined the first remote is this [570:10] and second remote is this and they are [570:13] checked in this order. Now you see I [570:16] have mentioned the order here. Why this [570:18] order matters? So firstly whenever a [570:20] dependency resolution is taking place [570:22] your Maven will first look at the local [570:26] if it is not in the local repository [570:28] then it will go to mirrors. Now if you [570:30] have configured a specific mirror now [570:32] you would configure a specific mirror if [570:34] you also intend to define your remotes [570:37] like this. Okay, then it will go to the [570:40] mirror and then it will go to the [570:42] declared remotes in order. And if none [570:44] of those have those dependencies, it [570:46] will go to central maven central. But [570:49] the scenario where your mirror has an [570:53] asterisk. This order is local will go to [570:56] mirror. If mirror doesn't have an [570:58] answer, it will not go to maven central [571:01] automatically. You should configure your [571:03] mirror in such a way that if mirror you [571:05] don't have it then the dependency can be [571:07] fetched from the vendor repose or from [571:09] the central repository. So I hope that [571:12] makes sense and you understand this [571:15] order part two. Now let's go to the [571:19] browser and see what I want to discuss [571:22] next. I want to discuss superom and [571:25] parent. So firstly super pom. Now we [571:29] have been using archetype to generate [571:32] our source code and the structure. But [571:35] who is providing this structure? Who is [571:38] providing the access to the Maven [571:42] central? We have not configured our [571:44] Maven to reach out to Maven center. But [571:46] it is automatically depending uh [571:49] downloading dependencies as we have [571:50] seen. We have not defined any reporting [571:53] structure or we have not defined any [571:56] sensible defaults. So all these things [572:00] are coming from a superpom. [572:03] Every pom.xml you write implicitly [572:07] inherits from Maven's superpom. So it [572:10] can be thought of as a default pom [572:13] bundled inside Maven. It defines global [572:16] defaults. For example, all of these. So [572:19] now you know how important is super pom [572:23] and why it is used. But what is parent [572:26] pom? So before defining parent pom I [572:29] want to give you an example. Let's say [572:31] you have an application that uses 10 [572:33] microservices. Now if you have 10 [572:35] microservices you would have 10 [572:38] pom.xmls. Now let's say each micros [572:41] service that your application uses has [572:43] 50 dependencies. So 50 dependencies [572:46] implies you have a total of 500 [572:48] dependencies. And if 50 dependencies per [572:51] microser, let's say 30 of those [572:53] dependencies are same for all the [572:56] microservices. Now understand [572:59] GAV parameters coordinates are must for [573:03] all your dependencies. Now does that [573:06] make sense to define all the versions [573:10] for you know 50 dependencies that are [573:13] used across your application in 10 [573:16] microservices. you will have to have you [573:18] know each dependency will have a version [573:20] number not not sane right and even if [573:24] you have what you can also do is you can [573:26] have a properties section in your [573:27] pom.xml XML and that property section [573:30] can define the version and then your [573:33] dependencies can refer that that also [573:35] works but if you have to change any [573:37] version let's say if you have to change [573:39] the version of one of the dependencies [573:41] then you will have to make changes at 10 [573:43] different places in 10 different poms [573:45] that's understandable but what if I [573:48] could centralize every version that is [573:51] needed into a central file called parent [573:54] pom that is what parent pom does for you [573:57] parent pom pom will parent pom can be [574:01] used as a version repository. So it can [574:03] hold all the versions that are needed by [574:06] your dependencies and your child poms [574:08] all those 10 microservices can refer to [574:11] this parent pom to check the versioning [574:13] information for any of the dependencies. [574:15] Now how that makes your life easier? If [574:19] you have to change a version, you just [574:22] change it in the parent palm and the [574:24] child palms can automatically refer that [574:27] parent pom. So let's see how that works. [574:30] And parent pom is just another pom you [574:33] own that child projects inherit from. In [574:35] production, it is used to centralize [574:37] dependency management like we just saw. [574:39] Shared versions, bombs, pin plug-in [574:42] management, set common repositories, [574:44] properties, enforce organizationwide [574:47] rules, Maven enforcers. These are [574:49] advanced concepts. I'm not discussing [574:51] them. But as you progress with your [574:54] Maven learnings, I will let you explore [574:57] these as you move with enterprisegrade [575:00] projects. Now we have an understanding [575:03] of parent pom and super pom. I want to [575:05] show you the structure of parent pom. So [575:07] here is the parent pom cloud with varos [575:10] parent. So if I go to my child pom I am [575:13] saying that spring boot starter all your [575:16] versions come from parents spring boot [575:19] bomb. So as we have discussed before [575:22] bomb here is the type bomb and we are [575:26] fetching this and this will have the [575:28] details about all the versions that are [575:32] associated with spring boot. I hope this [575:36] makes sense to you. Now I want to go on [575:40] to demo two build with Maven in Jenkins [575:43] and run as a docker container. So [575:46] whatever we have been running right now [575:48] we have been running manually. Now what [575:50] I want to do is I want to deploy the [575:53] same application. I want to create a [575:55] docker file and then I want to [575:57] containerize it and then I want to run [575:59] that container in my docker host. So [576:03] let's see how we are going to do it. [576:05] Firstly, I have a Jenkins container [576:09] running in my Docker host. And that [576:12] Jenkins container I created as part of [576:14] my Jenkins basics to production course. [576:18] You can create a Jenkins container by [576:20] following the steps that I have defined [576:22] here. If you are interested in day [576:25] three, I believe we discussed the [576:26] complete installation of Jenkins. And in [576:29] day five, we configured DOD that is [576:32] Docker outside of Docker. And if you [576:34] want to learn more about it, you can [576:36] watch day five and day three of my [576:39] Jenkins series. So I will be using a [576:42] container that I already have created. [576:44] So if I go to my terminal, I'll go here [576:47] and I will just close this. I will just [576:50] shut this instance down. P sudo init. [576:53] And then I will come here. So whatever [576:56] we are doing in this demo will be done [576:58] in the Jenkins container itself. We do [577:00] not need anything else. So I will run [577:03] docker ps. You see there is a Jenkins [577:05] container. I will write docker exec [577:10] and hyphen u root. I will tell you what [577:13] I'm doing. I will write Jenkins and then [577:16] I will run bash. Firstly I [snorts] want [577:19] you to know that I am going to install [577:23] Maven in my Jenkins. Now why do I need [577:25] to install Maven? Understand Jenkins can [577:28] only run Maven specific commands if I [577:31] have Maven installed in my Jenkins [577:34] server. I could have installed Maven in [577:37] my Jenkins container or I could also use [577:39] a Maven plug-in in Jenkins but plug-in I [577:43] don't want to use right now. I will be [577:45] using Maven directly installed in my [577:48] Jenkins container so that my Jenkins [577:50] container can run Maven specific [577:53] command. So I will run apt update first [577:57] and once this is done I will run apt [578:00] install maven y. I will run maven [578:06] version. This will tell me that I am [578:08] using version 3.8.7 [578:11] of maven. Remember when we installed the [578:13] binary we had 3.9.11. [578:17] This one has a previous version because [578:20] the repositories that are added for my [578:23] apt in this container may not have the [578:26] latest version right now. And as we [578:29] progress apt update will have the latest [578:33] available version. But these are always [578:35] slower. That is why I said if you want [578:37] the latest and greatest and if you want [578:39] complete control, you should install it [578:42] using a binary. Now that we have Maven [578:45] installed, let's exit our container. And [578:48] if I run docker ps, I see that Jenkins [578:51] container is still running. So let's go [578:52] to our browser and see what I want to do [578:54] next. So Maven version, these commands [578:56] we have done as part of step one. As [578:59] part of step two, we have to create a [579:01] freestyle job called Maven job. So let's [579:04] do it. So I'll access my container on [579:06] local host port 8080. And then I will [579:09] click on new item. Here I will name it [579:12] Maven job and this is a freestyle job. I [579:16] will click okay and I'll scroll down. [579:19] Add build step. I want to execute shell. [579:23] That's what I'm saying. And what is that [579:25] you want to execute warun? So let's see [579:28] that in our notes. So if I come here [579:33] this is exactly what I'm going to do. I [579:36] am telling step by step what Jenkins you [579:39] should be doing. Understand Jenkins is [579:42] our automation server. Previously [579:44] whatever commands we were running we [579:47] were typing those commands onto our EC2 [579:50] instance that had Maven installed. But [579:52] now we are integrating Maven with our [579:56] CI. So whatever steps we want to do, we [579:59] will tell our CI server and because our [580:02] CI server has Maven installed, it will [580:05] run all those steps successfully. So in [580:08] the first step, we are doing what we did [580:10] back in our virtual machine. We are just [580:12] generating a sample code. Then we'll be [580:15] cding into my app and then we'll be [580:18] running clean package. So first we'll be [580:21] cleaning and then we'll be creating a [580:24] jar file. So that is what the output [580:26] will be. Hyphen B is for batch mode [580:29] non-interactive good for CI logs. Then [580:32] build the jar clean work space. Skip [580:35] tests for speed. So right now we want it [580:39] to be fast which is why we are skipping [580:41] tests. And this I have kept deliberately [580:44] here so that you know you can use [580:47] certain flags with the phases with the [580:50] maven phases to skip certain things and [580:54] that is how you skip tests and there are [580:56] various things you can skip and I will [580:59] let you explore those things yourself. [581:02] Then I am creating a docker file. I am [581:05] using JRE now. I'm not using JDK. I'm [581:08] using JRE because I just need the [581:11] runtime. I do not need Java compiler [581:14] here because the compilation part will [581:17] be done by Maven in this step itself. [581:20] And then it will generate this artifact [581:23] which I am copying to my /app/app.jar [581:29] as app.tjar. [581:31] So in this step I'm saying use this [581:34] image working directory is / app. [581:36] Whatever next commands you run will be [581:38] applicable on this command. Then I'm [581:40] saying copy in from the target folder. [581:43] Remember when we were running mvn [581:46] package and then we ran ls there was a [581:49] target folder created and that target [581:51] folder had our artifact here as well. [581:54] The same thing it is just we are running [581:56] these things in our Jenkins container in [581:59] an automated fashion. Then lastly we are [582:03] running the command that we ran before [582:05] java class path we are specifying [582:08] forward/appjar [582:10] because here is our artifact and then we [582:13] are saying com.mmy company.app.app [582:16] and this is what we did before as well. [582:19] Then we are creating a docker ignore [582:21] file. You can see what we are doing [582:24] here. And then in the fourth step we are [582:27] creating a container image. We are [582:29] associating a tag called local because [582:31] we are not pushing this container image [582:33] anywhere. We are just building this [582:35] locally. At the very last we are echoing [582:38] container output below docker run. We [582:41] are running a container from that image. [582:43] So essentially what we are saying is you [582:46] run the container and as soon as it runs [582:49] you remove it once it has done its job. [582:52] Understand that our container is just [582:55] supposed to say hello world and [582:57] containers do not run 24 + 7. They just [583:01] stop when they have done what they are [583:04] supposed to do. Our container here is [583:06] just supposed to say hello world and [583:08] once it has said that it will stop and [583:11] we are saying don't stop it you just [583:13] remove it which is why we have added [583:15] hyphen rm flag. Now when you are dealing [583:18] with web applications you want to run [583:20] them 24 + 7 and which is why you have [583:23] engineext process or any other web [583:25] process which continues to run which is [583:27] why those containers run 24 + 7. So here [583:32] is what we are going to do. So I will [583:34] scroll up. I'll copy this. I'll go to my [583:36] job and I will paste this here. I will [583:39] save this and then I will say build now. [583:43] And this will start building this. So if [583:45] I go here I can go to console output and [583:49] this will show me what are the steps [583:51] being performed and these are pretty [583:53] much the same which were performed when [583:55] we were working with our virtual [583:57] machine. So the point that is [584:00] interesting is this is where the docker [584:03] image was created and if you see here if [584:07] I scroll down naming the docker image [584:10] this one was done done and here docker [584:14] run - rm -ame is this and this is the [584:18] image that you should be using and here [584:20] you see the output hello world. Now [584:25] instead of container being in stopped [584:27] state we have removed it. So if I go to [584:29] my terminal and I run docker ps here you [584:35] only see this even if I add a you only [584:38] say see this Jenkins container. But what [584:41] I can now run is I can run docker images [584:45] and this will show me all the images in [584:48] my machine. And here you see cloud with [584:51] vjo java hello local and this was [584:54] created a minute ago by whatever we have [584:57] defined in our Jenkins job. So I hope [585:01] all these things are making sense to [585:05] you. That is a wrap for this lecture. In [585:09] this lecture we took Maven from basics [585:12] to production. You saw the difference [585:15] between compiled and interpreted [585:17] languages. Why we need build tools and [585:21] what Maven actually does under the hood. [585:24] We created a project, walked through [585:26] pom.xml, understood life cycles, phases, [585:29] plugins, goals, and looked at [585:32] repositories, super pom, parent pom, and [585:35] finally wired everything into Jenkins [585:38] plus Docker. If this helped you see [585:41] Maven the way a DevOps engineer should, [585:44] do like the video, drop your questions [585:46] in the comments. The GitHub repo and [585:48] commands are linked in the description [585:51] so you can replay the demos on your own [585:55] machine. I will see you in the next one. [585:57] Thank you. Hi, I am Varun Jooshi and I [586:00] welcome you to cloud with VJosh where we [586:04] discuss cloud DevOps and everything in [586:07] between. In this lecture, we are going [586:10] to look at dev secc ops from a DevOps [586:13] engineers lens and understand how [586:17] security actually fits inside a real CI [586:21] pipeline. We will start with why and [586:24] what of DevSec Ops then break down key [586:27] terms you hear everywhere. SAS, dash, [586:31] code quality, SCA and Sbomb. From there [586:35] we will jump into what Sonar cube really [586:39] is, its role in modern pipelines and [586:43] walk through an end toend production [586:46] grade dev sec ops flow. Then we will get [586:50] hands-on. We will install Sonar Cube on [586:54] an EC2 instance. Walk through the Sonar [586:57] Cube console and architecture. Then [587:00] provision a separate EC2 for Jenkins and [587:04] install Maven and Docker. Finally, we [587:07] will put everything together and run a [587:10] full dev sec ops demo with Sonar Cube, [587:14] Maven and Jenkins. So you see exactly [587:19] how code quality, code security checks [587:22] and CI triggers in real life. The GitHub [587:27] repo linked in the description has [587:29] detailed explanations and stepbystep [587:33] demos for everything we cover. Replay [587:37] the labs on your own setup. And if it [587:41] helped, please do start the repository. [587:45] With that being said, let's get started. [587:48] Now before I start this lecture, I first [587:51] want to visit the GitHub repository for [587:54] this lecture so that I can tell you what [587:57] are the prerequisites for this lecture. [587:59] So I will go here. I will zoom in a [588:02] little YouTube standalone lectures and [588:06] from here I'll click on lectures and [588:09] then I will go to sonar cube that is [588:12] lecture for today. Now if I scroll down, [588:16] I'll use the clickable table of contents [588:18] to go to the prerequisites. And here you [588:21] see there are three prerequisites. The [588:24] first one is modern SDLC. Now we have [588:28] discussed this deeply in a different [588:30] series, a series where I am covering [588:33] Jenkins and CI/CD in greater depth. So [588:36] we covered modern SDLC and also CICD. [588:43] I would request you to watch both of [588:46] these lectures because this will give [588:49] you the foundations [588:51] and once your foundations are clear [588:54] whatever we are discussing in this [588:57] lecture and understand that dev sec ops [589:00] sonar cube all of these are extensions [589:03] of DevOps and until and unless you [589:06] understand the life cycle of DevOps you [589:09] will not be able to make sense of this [589:12] lecture. [589:13] So in this lecture we cover modern SDLC [589:16] and if I give you a glimpse of it this [589:19] this is it and here you see we have [589:22] covered SDLC then we have covered [589:25] different programming languages and then [589:27] we have also covered how are these [589:31] different when deployed onto different [589:34] environments. So how they behave when [589:36] you deploy them onto containerized [589:38] versus non-containerized environments. [589:41] So this lecture is of utmost importance [589:44] and then the next lecture is again the [589:47] CI/CD lecture. Here I explained [589:51] different terms. We started from the [589:54] very basics that how git works a very [589:58] highlevel view of that and then we moved [590:00] on to different continuous practices [590:03] that you would see branching strategies. [590:06] Then we looked at different production [590:08] workflows which you will see this is git [590:11] flow based. Then we discussed what's the [590:13] promotion strategy. Then we moved on to [590:16] trunkbased CI/CD pipelines. So watch [590:20] these two lectures. And then the third [590:23] prerequisite for this lecture is you [590:25] must understand Maven. Now I have a [590:29] detailed tutorial on Maven in this [590:32] channel itself. The links for that is [590:36] here. Why this lecture is needed? In the [590:39] demo section of today's lecture, what we [590:42] will do is we will be running a [590:44] fullfledged production grade pipeline [590:46] demo wherein we will be integrating [590:50] Maven with our pipeline and Maven has [590:54] plugins that we will be using and those [590:57] plugins are the ones that help us [591:00] integrate Maven with Sonar Cube. Hence, [591:03] if you do not understand Maven, which is [591:06] a popular build tool used for Java [591:10] language, then this lecture will not [591:13] make sense. So, watch all these three [591:16] lectures and if you have already done [591:18] so, then let's get started with this [591:22] lecture. So the first thing that I want [591:25] to discuss is why dev secc ops late [591:29] checks cause release failures and [591:32] rework. Now let's say you have normal [591:36] CI/CD pipelines. You do not have [591:39] security integrated with your pipelines [591:42] yet. Late checks cause release failures [591:46] and rework. Now imagine if you are using [591:49] a security tool and that security tool [591:52] is running when you are deploying the [591:55] application or right before when the [591:58] application is deployed and it finds [592:01] certain vulnerabilities. Will it make [592:05] sense to again start from the very [592:07] scratch and remove those [592:10] vulnerabilities? It does not make sense. [592:13] That is why whenever we are discussing [592:15] about CI/CD pipelines and especially [592:19] security in CI/CD pipelines, we always [592:23] say shift left. And what does shift left [592:27] imply? If this is the flow and we are [592:31] saying shift left, what that means is if [592:34] you are performing security analysis in [592:37] deployment stage, then move it to [592:40] testing. If you are performing in [592:42] testing then you should be performing [592:44] them in build too. If the security is [592:47] part of build then it should be part of [592:49] development too and so on. Hence what we [592:54] are saying is whenever we talk about [592:57] security we should be taking care of [593:01] security in all the phases of SDLC of [593:06] the DevOps life cycle. Wun but how do I [593:11] take care of security in the requirement [593:13] gathering phase there is no tool that [593:16] you would use in the requirement [593:17] gathering phase but what my intent is [593:20] whenever I'm gathering requirements for [593:22] my application I should be always [593:25] thinking about security if my [593:28] application teams come to me and tell me [593:30] that varun we would like to have our [593:34] application deployed onto containers [593:36] then my mind should automatically start [593:39] working that how do I perform security [593:42] testing not just for my container images [593:46] but also for my running container. So my [593:48] mind should be working in that direction [593:52] from the phase of requirement gathering [593:55] itself and then in each phase security [593:59] should be given utmost importance. [594:02] Previously it was not done like that. [594:06] Security was part of late checks. But [594:09] for the past 5 10 years industry has [594:12] come to realize that without security [594:16] their pipelines or their workloads are [594:19] futile. Hence now everyone is taking [594:23] care of security. They are adhering to [594:26] shift left approach. Next is manual [594:30] reviews are slow, inconsistent and [594:33] unoduditable. And that is obvious. If [594:36] you are not performing your security [594:38] scans as part of your CI/CD pipelines, [594:42] then you are essentially doing that [594:44] manually. And when you do something [594:46] manually, what happens is the [594:49] integration doesn't work as smoothly. [594:53] Let's say you are performing static [594:55] scans of your container images. But what [594:59] I want is my pipeline should not move to [595:03] the next step until and unless that scan [595:08] is successful. But how do my pipeline [595:12] know about that? My pipeline will know [595:15] only about that when I have integrated [595:19] that security tooling with my pipelines [595:22] or I have an automated way via which I [595:27] can tell my pipeline that you must fail [595:30] if that security scan fails because on [595:35] its own a security tool will just tell [595:38] you that this does not work or this has [595:40] vulnerabilities but you as devops Ops [595:44] engineer must ensure that if something [595:47] is failing or if something is detecting [595:49] vulnerabilities then your pipeline [595:53] should also fail or there must be [595:57] something logged into your pipeline that [596:00] this something has vulnerabilities. Now [596:02] you take care how [snorts] you are [596:05] configured to take care of it and who [596:08] will configure it? It is us DevOps [596:10] engineers who would be configuring it. [596:14] Last one is hidden risks accumulate in [596:17] code and dependencies. I want you to [596:20] understand that your application is not [596:24] just the application code. Your [596:27] application could be depending on [596:29] several other things. Maybe it depends [596:31] on Google for authentication. Maybe it [596:34] is dependent on third-party libraries [596:37] for some functionality. If I am [596:39] designing a banking application and my [596:42] banking application requires a [596:44] calculator, I will not ask my developers [596:47] to write code on how to do calculations. [596:51] I will use an existing third-party [596:54] library for that. And when you start [596:58] using libraries for your entire [597:01] application stack and these they do not [597:03] have a specified count. For example, if [597:05] I have a banking application, I might be [597:08] using hundreds of libraries and it is [597:11] impossible [597:13] for me to keep track of all those 100 [597:16] libraries and as soon as your [597:19] application progresses, there are new [597:22] versions of libraries also available. [597:25] So, you must be taking care of all these [597:28] things. So if you do not have security [597:30] integrated within your CI/CD pipelines, [597:33] then these dependencies [597:37] possess hidden risks. Now that we have [597:41] this basic understanding, let's [597:43] understand what is dev Sec. [597:54] So if you are already a DevOps engineer [597:57] and you have some sort of security in [598:00] your CI/CD pipelines then your pipelines [598:04] already adhere to the principles of dev [598:07] sec ops. Now I'm not saying your [598:10] pipeline is as mature as it should be. [598:14] What I'm saying is if you have [598:17] introduced some component of security [598:20] then you have already integrated [598:22] security with your CI/CD pipelines or [598:25] with your DevOps flows. Another thing I [598:29] want you to clearly understand that [598:32] whenever I'm talking about dev sec ops, [598:36] DevOps, SDLC, [598:38] these are not tools, okay? These are not [598:41] products that you can buy. These are [598:44] methodologies. These are practices that [598:48] you must apply when you are designing, [598:51] building, testing, deploying and even [598:55] operating an application. So there are [598:58] of course different tools that are [599:01] available at your disposal to achieve [599:03] the dev secc ops culture or to achieve [599:06] the devops culture but in itself these [599:10] are just methodologies. Now which tools [599:13] you should be using for dev sec ops [599:16] which tools you should be using for [599:17] devops it depends on your application [599:21] requirements. So what DevOps SDLC they [599:24] do is they provide you a cultural [599:28] framework that this is the way things [599:31] should be done and as long as you adhere [599:36] to these principles you are free to [599:40] bring in any tools that you want. So in [599:44] single line dev secops is integrating [599:48] security tools with your DevOps [599:52] practices. As simple as that. It is a [599:55] software engineering approach that [599:58] integrates security practices into every [600:01] phase of SDLC and the DevOps life cycle. [600:05] And you can see SDLC defines the what [600:09] and why for your application and the [600:11] DevOps life cycle focuses on how to [600:15] deliver reliably. Now DevOps focuses on [600:20] how to deliver reliably. What is the [600:23] focus of DevSec Ops? Dev Sec Ops focuses [600:27] on how to deliver it reliably and [600:31] securely. Because when you integrate [600:33] security with your DevOps life cycle, [600:37] you are essentially stepping into the [600:40] dev sec ops realm of things. Security [600:44] integrated across plan code, build and [600:47] deploy. That is what we discussed. [600:50] Automated checks in CI/CD on every push [600:54] or PR. So whenever you are creating a PR [600:58] let's say to merge your feature with [601:01] your main then there are automated [601:05] security checks that are taking place. [601:07] And what are these security checks? [601:10] Where do they plug in in a production [601:13] flow? We will look into all of these [601:15] things in a little while. This gives us [601:19] clear ownership across developers [601:22] security and operations. Now you could [601:24] be working in a team where is a [601:26] different cloud team, there's a [601:28] different DevOps team, there's a [601:30] different dev sec ops team and then [601:32] there's a different security team in [601:35] general. So if you have integrated dev [601:39] sec ops tools into your pipelines [601:41] appropriately, then you will have clear [601:45] ownership. For example, if let's say [601:48] your container scan is failing, I for [601:51] sure know that this must be looked by my [601:54] dev sec ops team. So this provides clear [601:58] ownership. Now that we have this [602:00] understanding, I want to discuss key dev [602:04] sec ops terms and concepts that you will [602:07] hear in your day-to-day communication. I [602:11] want to start with SAS. SAS is static [602:16] application security testing. Now, as [602:19] soon as I hear static, I know that these [602:23] are the security testings that are being [602:26] performed onto a sitting code and that [602:31] sitting code could be in my IDE or that [602:35] code could be inside a binary or an [602:39] executable. So static scan of source [602:43] byte code or binaries for [602:46] vulnerabilities that is what SAS will [602:49] do. Now the topic of this lecture is [602:52] sonar cube. So if I go to visual studio [602:54] code here and I go to extensions here [602:58] and then I search for sonar cube you [603:02] would see that there is an extension [603:04] available that will perform SAS for me. [603:08] And if you can see it mentions the same [603:12] in detail here. And there are multiple [603:15] others tools. If I write check marks, [603:18] check marks will also have a tool that [603:21] will do SAS for me. Beat vulnerabilities [603:24] with more secure code. There is another [603:26] extension. Check mark SAS is a check [603:29] marks ID extension that enables scanning [603:31] and retrieving results and so on. So [603:34] there are tools that can perform [603:38] SAS for you. Again when I talk about [603:41] SAS, DAST, Esbomb, [603:45] SCA, code quality checks, these are [603:47] again methodologies or practices. These [603:51] are not tools. There are tools that can [603:54] perform SAS. There are tools that will [603:56] help you with dash. There are tools [603:58] which will help you with these, these [604:00] and these. So for now the clear [604:04] understanding I want you to have is [604:07] these are just umbrella terms. There are [604:11] different tools that work under this [604:13] umbrella to achieve these. Now let's [604:17] talk about SAS. We already know it is [604:20] static scanning. It will find SQL [604:23] injections, hard-coded secrets, insecure [604:26] APIs and so on. SQL injections are maybe [604:30] your application has an input stream and [604:32] in that input stream the malicious user [604:36] or the malicious machine tried to modify [604:39] your back end and your back end is your [604:43] database which is SQL then where it runs [604:47] early in CI and integrates with ids. We [604:51] have already seen that there are [604:53] different extensions that are available [604:55] for ids and we are also saying that they [604:58] run early in CI. Now imagine when you [605:01] are building let's say a Java [605:02] application that Java application will [605:05] result in an artifact and let's say that [605:07] artifact is intar format. So you would [605:11] want something to scan that artifact for [605:14] any vulnerabilities. That is where your [605:18] SAS tool fit. They scan your code, your [605:22] binaries for vulnerabilities. Now, the [605:26] first two tools in all the sections are [605:28] the most popular ones. So, you would [605:30] often see sonar cube and check marks in [605:33] production environments. If your [605:35] application is containerized, then you [605:38] could also see sneak and trivi as part [605:42] of your SAS tools. Then let's talk about [605:45] dash. DAST is dynamic application [605:48] security testing. Now dynamic means it [605:52] is performed on applications that are [605:55] running and these tests are typically [605:59] performed using HTTP. So you will have [606:02] an end point on which your application [606:04] is exposed and that is the entry point [606:07] for these tests. Tests running [606:11] application by simulating real [606:13] attackers. So it is looking to exploit [606:16] your application to see what are the [606:19] loopholes, what are the parts that can [606:22] be exploited by malicious actors, [606:26] targets, runtime issues such as XSS, [606:30] injection, authentication, bypasses. [606:32] Where will you run DAST? You will run [606:35] dash predominantly in your staging or [606:38] pre-pro environments because you do not [606:42] want a full-fledged dash to be running [606:45] alongside your production application. [606:48] That does not make sense because [606:50] production application has your real [606:53] users accessing your application. You [606:55] don't want to overburden your [606:58] application by performing a full-fledged [607:01] dash in production. So what you would [607:03] typically see is dash is performed [607:07] majorly extensively I would say in [607:10] prepro or staging and when I talk about [607:13] prod environments you will still see a [607:16] passive or minimal dash running there [607:19] are different tools first is oasp zap [607:22] burp suite these two are the most famous [607:24] ones and if you have containerized [607:27] workloads you would also see sneak dash [607:31] is being used for dash fast and there is [607:34] sneak that is also used for fast. So [607:38] understanding the difference fast is for [607:41] your static code dash is for your [607:44] running code. Then let's talk about code [607:46] quality checks. Now the first thing that [607:48] I want want to mention here is code [607:52] quality talks about the quality of code. [607:56] It is not talking directly about [607:58] anything security. It is talking about [608:02] the quality of code. How is your code [608:05] written? So if I I have I want to [608:07] explain this to you in general terms. If [608:10] I want to go from point A to B and I can [608:13] go from point A to B straight. But there [608:16] is another way to reach point B via C D [608:19] E. If I take that longer route, does [608:21] that make sense? No. Right? If I want to [608:24] go from A to B, I'll go directly. If I [608:27] am taking the longer route that decides [608:29] the quality I am using. So similarly if [608:32] I talk about software if you have [608:34] multiple duplicates in your code. If you [608:37] are doing the same things and calling [608:40] different functions for that then all [608:43] these things deteriorate the quality of [608:47] your code. Another thing is maybe your [608:50] code has defined multiple variables [608:53] which are not being used anywhere. So [608:55] that says the quality of that code is [608:58] not good. If something can be achieved [609:01] in 10 lines of code and you are using 50 [609:04] lines of code then that is a code [609:08] quality issue. So in summary code [609:12] quality checks maintainability, [609:14] complexity, duplication, naming and dead [609:18] code. Now second point is code quality [609:21] checks helps reduce technical debt and [609:26] prevent future regressions. Now [609:28] technical debt is shortcuts in code or [609:32] design which are taken to move fast now [609:36] and that makes the code harder [609:38] [clears throat] and slower to change [609:40] later. So these are technical debts. [609:43] Then you have code quality checks helps [609:47] prevent future regressions. Now what is [609:50] regression? Regression is to return to a [609:52] previous and less advanced state. So how [609:55] code [snorts] quality helps here is it [609:58] helps in keeping code clear, tested and [610:02] well reviewed so old bugs do not [610:05] reappear. Now again I want you to know [610:08] that code quality just deals with the [610:11] quality of your code. It does not [610:14] directly pertains to security. It [610:18] pertains to how well your code is [610:21] written. And understand that it is [610:24] important that your code quality is good [610:28] because you can have certain rules in [610:32] your sonar cube or code quality tools [610:35] which will prevent lowquality code from [610:39] going into production. So where you [610:41] would perform code quality IDE CI or [610:45] linting analysis and quality gates. Now [610:48] what is linting? linting are static [610:50] checks which will flag style formatting [610:53] or simple correctness issues before code [610:57] runs. Now where you would perform code [610:59] quality checks, you would perform them [611:01] in your IDEs and also in your pipelines [611:05] and we'll see exactly at which stage of [611:08] pipelines do these methodologies fit [611:12] into dependency scanning or SCA. Now SCA [611:16] is software component analysis. Now with [611:19] the expanded form you can imagine that [611:22] it will tell you all the things your [611:24] software is comprised of. So for example [611:28] if I have a banking application and my [611:30] banking application has 100 libraries. [611:34] I'm using 100 libraries in that banking [611:37] application. What will SCA do is it will [611:40] tell me what all libraries I am using [611:44] and then it will tell me are there any [611:48] vulnerabilities in my library. So it [611:51] scans third-party libraries for known [611:55] vulnerabilities and licensing. Now since [611:58] it is a software component analysis, it [612:01] will also give me the licenses that are [612:05] being used. Now I want you to know that [612:07] I could be using 100 libraries and maybe [612:10] 50 of those libraries are open- source [612:12] but rest of them they have a price [612:14] associated with them. So dependency [612:17] scanning tools will tell you that. [612:19] Understand they are called dependency [612:21] scanning tools for a reason because they [612:25] work on the vulnerabilities in your [612:28] dependencies. Understand that SAS was [612:31] used for your static code. Dependency [612:35] scanning is used to find out the [612:37] vulnerabilities in your thirdparty [612:40] libraries that your application is [612:42] using. And this was used to find [612:45] vulnerabilities in your code. Matches [612:49] dependency versions against CVE and [612:52] license databases. Now what is CVE? CVE [612:55] is common vulnerabilities and exposure. [612:58] And there is a database central database [613:00] for these things. What dependency [613:03] scanning tools will do is it will [613:05] compare the application versions that [613:08] you are using with this CVE database to [613:11] tell you that these these these are the [613:14] libraries that have vulnerabilities and [613:17] you as application developer or DevOps [613:20] engineer will have to fix those [613:23] vulnerabilities. And next one is where [613:26] you would run this during build and CI [613:29] continuous dependency scanning runs. [613:32] There are tools available for this. [613:34] Sneak and trivia are the most popular [613:36] ones. Now at this stage I want to tell [613:40] you that SBOM is an extension of SCA. [613:45] Okay. But fundamentally their primary [613:49] purpose is different. So when I talk [613:52] about SCA or dependency scanning, the [613:55] primary purpose for them is to list out [613:59] the vulnerabilities that are there in [614:02] your third party libraries. The primary [614:05] purpose of SBOM is to give you a machine [614:09] readable inventory listing components, [614:13] versions, and licenses. Understand? [614:16] Dependency scanning was actually [614:19] performing a scan and telling you about [614:22] vulnerabilities in your third party [614:24] libraries. Whereas SBOM is just giving [614:28] you an inventory. It is just telling you [614:30] that these are the 100 libraries that [614:33] you are using and this is the version of [614:35] it and this is the licensing for this [614:37] and these are rest of the components [614:40] that your application is using. So it is [614:42] just fetching the inventory. So for [614:45] example, if I talk about AWS cloud, I [614:48] could have a script that fetches me all [614:51] the EC2 instances that are running. But [614:54] I could also have a script that fetches [614:57] the instances and also deletes the [615:00] instances that are in powered off state. [615:03] So these two things are different. [615:05] Similarly, if I talk about dependency [615:08] scanning, it is actually finding the [615:11] vulnerabilities for you in your third [615:13] party libraries whereas Esbomb is just [615:16] giving you the version of those [615:19] libraries in short and then you see [615:22] licensing is there in both of this. Now [615:26] this is an overlap of course but I said [615:29] primary function of SCA is scanning [615:32] third party libraries and primary [615:34] function of SBOM is to give you machine [615:37] readable inventory of the third party [615:40] libraries version that you are using and [615:42] the expanded form for SBOM is software [615:46] bill of material generate sbombs in CI [615:49] and attach to artifacts enable [615:52] postrelease rescans when new CVS are [615:56] published. Now understand if you didn't [615:59] have ESBOM then you will have to run the [616:02] scan on the running application or onto [616:06] the artifact that was generated. For [616:08] example, if you are working with a Java [616:10] application, then when there was new [616:13] releases in the CVE, when the CVE [616:16] database was updated, you would have to [616:18] compare your application with the new [616:21] database to check out if there are [616:23] vulnerabilities in your application, [616:26] right? You will have to compare that [616:27] with your jar or maybe if you are if you [616:30] have a running application, then you [616:32] will have to check there. But if you [616:35] have sbomb and sbomb is already there in [616:38] your let's say artifact registry then [616:42] you can directly compare that CVE [616:44] database with your sbomb and get to know [616:48] whether there are any vulnerabilities [616:50] with your running application or not. [616:52] That is how sbomb helps. But again I [616:56] want you to know that if your [616:57] application at runtime is installing [617:01] certain softwares or using certain [617:03] libraries then sbomb will not detect [617:07] them. SBOM will only detect those things [617:11] that were generated at the time when [617:14] artifact was generated because sbombs [617:17] are typically generated when you are [617:20] creating an artifact. For example, if if [617:22] you are creating a container image, you [617:24] can associate an SBOM with it. Or if you [617:26] are creating a dot jar, you can [617:28] associate an SBOM with it where [617:31] generated at buildtime like I just [617:33] mentioned and attached to artifact or [617:36] images. Next is formats. Now, SPDX, [617:39] Cyclone DX tools that you use are sift [617:42] and cyclone DX plug-in. Now these are [617:45] the formats in which sbombs are [617:47] generated and these are the tools that [617:50] generate these sbombs. Now in summary [617:54] SAS is static code checks, IDE plus CI, [617:58] SCA is dependency scanning, SCS software [618:01] component analysis. Then you have SBOM [618:04] inventory of component software bill of [618:06] material. DAT is dynamic application [618:09] security testing. It is performed during [618:13] your runtime. It provides you with [618:16] runtime security. Code quality is equal [618:19] to maintainability checks ID plus CI. [618:22] Now I want to emphasize on one thing [618:24] again that code quality is not directly [618:29] security related. It just tells you how [618:32] is the quality of code. So keep that in [618:34] mind and I will tell you why this [618:36] distinction is important as we progress [618:39] with this lecture. Now I want to discuss [618:42] Sonar Cube. What is Sonar Cube? It is a [618:46] continuous inspection platform that [618:49] automatically [618:51] analyzes source code for bugs, security [618:55] vulnerabilities, [618:57] code quality issues and test coverage, [619:01] enforcing consistent standards across [619:05] CI/CD pipelines. So essentially whatever [619:08] key terms that we discussed before we [619:12] mentioned SAS we mentioned code quality [619:15] we did not mention code coverage but we [619:18] did mention both of these. So if I move [619:20] to these concepts, SAS and code quality [619:26] is something that Sonar cube will help [619:30] you with and how it helps. We will see [619:33] it practically as well. But let's first [619:36] understand that what is code coverage. [619:39] So at a very high level we know that [619:42] Sonar Cube will help us with three main [619:44] things. Code coverage, fast and code [619:48] quality. So let's quickly understand [619:50] what is code coverage. Now let's say you [619:52] have written 100 lines of code. And when [619:56] you are testing you are only testing 80% [619:59] of it. Then your code coverage is 80%. [620:02] So as simple as that code coverage is [620:06] how much [snorts] of your code is tested [620:09] to work properly. Again mark my words. I [620:14] said how much of your code is tested to [620:18] work properly. I did not mention [620:21] anything about security. So if your code [620:24] is supposed to perform oneplus 1, if it [620:28] can perform oneplus 1 properly and you [620:31] have tests defined to test that then [620:34] your code coverage is 100%. But oneplus [620:37] 1 testing is not security. right here as [620:40] well. If you are performing code [620:42] coverage, it will tell me how much of [620:45] your code is tested to work properly. It [620:48] is not telling me that your 80% code is [620:52] fine when it comes to security. It is [620:54] not saying that. Next is CI runs tests [620:58] coverage reporters generate reports cube [621:02] imports. Now this is a critical [621:04] distinction. Now let's understand this [621:07] first. So narcube I said has three main [621:10] functionalities. This functionality [621:13] is directly related to security. Okay, [621:17] this is security static application [621:20] security testing. But when I talk about [621:23] code quality or code coverage now these [621:26] are not direct security related things. [621:30] Sonar cube can perform code quality [621:33] because it is scanning your code. When [621:36] it comes to code coverage, tonar cube [621:39] does not perform the code coverage [621:42] itself, what it does is it depends on [621:47] other tools to provide that report to [621:49] it. And once that other tool has [621:52] provided that report to sonar cube, then [621:54] sonar cube can decide whether this code [622:00] adheres to your code coverage rules or [622:03] not. For example, if I have set my code [622:05] coverage to 80%. And there is an [622:08] external tool that is sharing the code [622:10] coverage report with Sonar Cube, then [622:12] Sonar Cube can judge whether your code [622:15] is adhering to the standard that we have [622:17] set or not. And based on that it can [622:21] also fail the pipeline but in itself [622:25] sonar cube cannot generate coverage [622:27] report. So that is what we are saying [622:29] here. CI runs tests. Coverage reporters [622:34] generate reports. Sonar cube imports. [622:38] Next is enforce minimum coverage on new [622:41] code via quality gates. That is what I [622:43] just discussed. So when you are working [622:45] with sonar cube, what you can do is you [622:47] can create something called quality [622:49] gates. Now quality gate is something [622:52] that stands as a guard. So if your code [622:56] coverage is let's say not 80% then that [622:59] gate will not open and if that gate will [623:02] not open then your code cannot move to [623:05] the next step and next step could be the [623:06] staging stage. So it cannot proceed [623:09] further and you can have your pipelines [623:13] fail if the sonar cube quality gate [623:16] condition is not met and we will see [623:19] this in the demo section. Common tools [623:22] are Jakoko. Jakokco the expanded form is [623:25] Java code coverage. So you will use that [623:28] for Java. There is an bull that you can [623:30] use for JavaScript. There is coverage.py [623:32] that you can use for Python. Now I want [623:35] to discuss SAS from a different lens. [623:38] Now why different lens? Because here I [623:41] want to define certain terms for you and [623:43] these terms are simple and once I give [623:46] you a oneliner you will have this in [623:49] your memory for a very long time. So [623:50] let's see what it is. First is static [623:53] code analysis to find vulnerabilities, [623:56] security hotspots and security sensitive [624:00] bugs before production. Firstly, what is [624:04] a vulnerability? Now vulnerability is a [624:08] security flaw that an attacker can [624:10] exploit. As simple as that. Now your [624:13] software can have multiple [624:15] vulnerabilities and then those [624:17] vulnerabilities can have different [624:18] categories. But if I have to define it [624:21] in one line then vulnerability is a [624:23] security flaw that can be exploited. [624:26] What is a security hotspot? Security [624:29] hotspot is something insecure that is [624:33] there in your code. Now that something [624:35] insecure [624:36] is not impacting the functionality of [624:39] the application but it is a security [624:42] risk. For example, if I have secrets in [624:45] my code, that is not impacting the [624:49] functionality of my application. My [624:50] application would still work fine, but [624:53] it is a security hotspot. It is a [624:55] security flaw. So, in summary, security [624:59] hotspot is a code that might be risky [625:03] and needs human review. Now, why I say [625:06] might be risky? Now there is a chance [625:09] that you are internally using a private [625:13] CA to sign your certificates and that is [625:16] how your application is meant to be. In [625:19] that case that security hotspot can be [625:23] ignored. Next is security sensitive [625:27] bugs. Now when I talk about bugs, bugs [625:31] have typically the highest priority to [625:33] be fixed. Now if you have a bug, bug [625:36] implies something that causes [625:40] application to not behave as expected. [625:43] So fixing bugs have the highest [625:45] priority. Here we are saying SAS can [625:48] also detect security sensitive bugs [625:52] before production. Next is categorizes [625:55] findings by type and severity. So sonar [625:58] cube can do all these things that we [626:01] discussed here and then it can also [626:03] categorize finding by types and severity [626:06] for prioritization runs in PR's CI to [626:10] shift security left and this is what we [626:14] will see in the demo. Now let's discuss [626:17] code quality. Code quality at a very [626:19] high level we know tells us whether the [626:21] code is well written or not. Now first [626:24] is detects code smells and duplication. [626:27] Now what is code smell? Code smell is a [626:30] poorly written code that makes changes [626:33] harder. Duplication as we all know [626:36] duplication is you are repeating the [626:38] same part again and again. You could [626:40] also create a function for that. So that [626:42] is duplication. So sonar cube will [626:45] detect these things primarily about [626:48] maintainability, readability and [626:50] technical debt. So sonar cube can help [626:53] with code quality because sonar cube [626:56] will tell you how good is your code when [626:59] it comes to maintainability and [627:01] readability. Last one is enforcable via [627:05] quality gates to prevent regressions. [627:07] Now all of these things that Sonar Cube [627:11] is somehow administering, Sonar Cube has [627:15] the ability to have a gate that if these [627:19] things are not met, then your quality [627:22] gate will fail or pass. For example, [627:25] let's say if your code coverage is not [627:27] 80%, then I won't open the gate. Hence, [627:31] the quality gate that we mentioned here [627:34] briefly will fail. I know I have given [627:39] you a theory bombardment. You have had [627:43] different terms thrown at you and I [627:46] understand it could be overwhelming for [627:49] few of you. So that's why what I now [627:52] want to do is I want to describe an end [627:55] toend production dev sec ops life cycle [627:59] so that you know what fits where and [628:02] when you know what fits where you can of [628:05] course at least participate in any dev [628:08] sec ops discussion with ease and I [628:11] assure you if you understand this flow [628:14] you can have security related [628:17] discussions with your dev seccops teams. [628:20] in ease and this will pave the way for [628:24] any of you who aspire to be a dev sec [628:27] ops engineer. Now let me zoom this [628:30] properly so that whatever I want to [628:32] convey is in the same screen. Now the [628:35] first line is end to end production dev [628:38] seccops flow Java plus Maven. So in this [628:42] flow we are dealing with Java language [628:46] along with Maven which will be used as [628:49] the build tool for Java. Now the [628:52] pipeline that you see here is platform [628:55] agnostic. So GitLab, Jenkins or GitHub [628:59] actions or whatever you are using this [629:02] flow is platform agnostic. This is [629:05] essentially the methodology the thought [629:08] process that is behind DevSec. The key [629:12] point in dev sec ops as we understand [629:14] now is security. How do we integrate [629:18] security at each stage? So that is what [629:21] we will be doing in this workflow. The [629:24] first point is check out from private [629:28] repository. Now that is what will happen [629:30] always. Your production code will be [629:33] stored in a private repository and from [629:36] there you will be checking that code [629:38] out. Now if I talk about a containerized [629:41] application, your repository will have [629:44] source code that is obvious. If I talk [629:46] about Java, it will be your src and then [629:49] pom.xml [629:51] and it will also have the docker file. [629:54] Docker file will be used to create a [629:56] container image and then it will be [629:58] pushed onto a central image registry and [630:01] then containers will be created using [630:04] that image. Second point is triggered by [630:07] push or pull request. Now you will never [630:10] in my opinion see any production [630:13] pipeline getting triggered on a push. [630:16] Typically your main branches or your you [630:18] know production branches are protected. [630:22] So you can only raise a pull request. [630:24] For example, you are working on a [630:26] feature branch and once you have done [630:28] your work, what you would do is you have [630:30] performed your pushes, what you would do [630:31] is you would create a pull request and [630:35] as soon as your pull request is created, [630:38] your pipeline will trigger. The second [630:41] step is build plus test. Now before I [630:47] start this discussion, I want to talk [630:50] about testing in general. So if I talk [630:53] about testing, there are two parts to [630:55] testing. There is functional testing and [630:58] then there is security testing. And [631:00] often people will just say testing and [631:04] you based on the communication will have [631:06] to understand whether that person is [631:08] talking about functional testing or [631:10] security testing. And as you spend time [631:14] in this industry, you will understand [631:16] these things. No biggie. But I wanted to [631:19] give this distinction to you right here [631:21] so that it eases the terminologies that [631:25] are thrown at you. Okay. So testing [631:27] there is functional testing. Now [631:28] functional testing implies whether my [631:32] application is working or not working as [631:35] expected. Okay. It does not deal with [631:38] security at all. Second is security [631:42] testing. Now security testing is [631:44] something that deals with security. So [631:46] you will have methodologies like SAS, [631:49] DAS, SCA, SBOM here and security testing [631:53] also deals with supply chain. So if I [631:55] talk about SBOM, what is SBOM? It will [631:58] tell you what your software is comprised [632:00] of the versions. SCA will tell you [632:03] vulnerabilities that your libraries will [632:05] have. So they also deal with supply [632:08] chain. So two things functional testing [632:11] and security testing. So when I talk [632:14] about code coverage, code coverage is [632:17] part of functional testing. It is not [632:19] doing anything security. So I hope this [632:22] distinction is clear. Now that you [632:24] understand this, let's move to the right [632:27] of the screen. So I'll make this here. [632:30] And then second point is build plus test [632:34] CI phase. Now first thing is see if I [632:36] talk about Java there are first testing [632:40] frameworks that must be defined by your [632:43] developers in the code. First is those [632:46] testing frameworks and JUnit and testNG [632:49] that you see here are those testing [632:52] frameworks that must be there in the [632:55] code and then what all are the tests [632:59] that are to be performed using that [633:02] framework must also be defined by your [633:04] developers only then these testing tools [633:08] will work. So for example, if I have a [633:10] Maven plug-in and that Maven plug-in is [633:13] let's say called Shorefire, that [633:15] shorefire plug-in will only be able to [633:19] perform unit tests if my developers have [633:22] defined a testing framework like JUnit [633:26] and under that they have defined what [633:29] tests are to be performed. So it is [633:31] important for you to understand this. [633:34] Now there are two types of testing that [633:36] I have mentioned here unit and [633:38] integration and I think I should have [633:39] explained it in this screen itself. [633:42] Functional testing checks overall [633:44] whether the function your application is [633:46] working or not. And then there are some [633:48] bifurcations. Unit testing will check [633:51] certain classes in your program certain [633:54] sections in your program whether they [633:56] are working or not. So unit testing is [633:58] you check different chunks of of your [634:02] application separately. Integration [634:04] testing as the name implies integrate [634:06] things. So it will check whether the [634:10] process as a whole is working properly [634:13] or not. Maybe you want to check if I [634:15] click on account statement whether it is [634:18] taking me directly into my account [634:21] summary page or something of that sort. [634:23] whether the components in your [634:25] application are integrated properly or [634:27] not. Smoke testing is simple testing. It [634:30] will just tell you whether the [634:31] application is accessible or not. So if [634:34] you have used tools like curl and [634:36] postman, they are used for smoke [634:38] testing. So this is this. Now when I [634:40] talk about this second stage, we talked [634:43] about unit testing and integration [634:45] testing. So if I talk about Maven, Maven [634:48] has different plugins that can be [634:50] leveraged to perform unit testing and [634:54] integration testing. Surefire is the [634:57] plug-in that is used to perform unit [635:00] testing, but it can only perform unit [635:02] testing if you have defined a framework [635:04] in your code along with the tests that [635:08] are to be performed. So fail safe is a [635:10] Maven plug-in that is used to perform [635:13] integration testing. But integration [635:15] testing again can only be performed if [635:18] you are using a testing framework like [635:21] JUnit. Along with that you have also [635:23] defined the tests that are need to be [635:25] performed. So the first section of this [635:29] is defined by your developers in the [635:31] code. The second section the plug-in [635:33] that you have to use the plug-in that [635:35] are called is typically with your DevOps [635:38] engineers and we have discussed this in [635:41] our Maven lecture but I will also touch [635:43] this in the second point. Now second [635:45] point is once these things are defined [635:48] second is you would typically run MVN [635:50] verify. Now if I take you to my browser [635:54] and if I go here, we know that MVN [635:58] verify is a part of the default life [636:01] cycle in Maven. Now if you have not seen [636:03] Maven lecture, pause this right now and [636:05] watch that otherwise this will not make [636:07] sense to you. So if I am running the [636:09] verify phase then all the phases that [636:12] are before verify 23 phases are there in [636:15] the default life cycle. All of these [636:18] will run. This is important to [636:20] understand. So when you are running [636:22] verify, you see there integration test [636:25] will also run. If I scroll up, test [636:29] compile will also run. Test will also [636:32] run. So verify will at the very last [636:36] will also package your application and [636:39] once your application is packaged, it [636:41] can also run integration test. But again [636:44] I want you to understand one thing [636:46] extremely clearly that unit tests can be [636:50] performed in an application that is not [636:53] running. But when I talk about [636:55] integration tests, integration tests [636:58] cannot be performed in a stopped [637:00] application. And why is that? So unit [637:03] testing is testing a particular class or [637:06] a particular section of your code. But [637:09] when it comes to integration, it is [637:11] checking your running application. For [637:13] integration testing, you want the [637:16] application to be running. So that's the [637:18] crux of it. So when you run MVN verify, [637:22] all these things are performed and you [637:24] also have a package like you see here. [637:27] JAR is with you. So let's go back to the [637:30] diagram again. And now Jakokco records [637:34] code coverage for Sonar Cube. Now there [637:37] are plugins that have performed these [637:40] tests. Now there is Jakokco plug-in that [637:43] will collect this report and this report [637:47] will then be delivered to Sonar Cube. [637:50] That is how Sonar Cube can perform code [637:53] coverage for you. Now again I want to [637:56] clarify one thing like I mentioned [637:58] integration testing would warrant your [638:01] application to be in running state. But [638:03] Vun, we have not deployed your [638:05] application anywhere. How is it possible [638:07] to perform integration testing? So if [638:10] you have been part of our Maven lecture, [638:11] I think we have discussed that there as [638:13] well. There is an ephemeral environment [638:16] that is created. In that ephemeral [638:18] environment, the application is deployed [638:20] and then integration testing is [638:23] performed. Now let's talk about the [638:26] third part that is run SCA and generate [638:30] Sbomb. And as you can see we are using [638:33] tree to run SCA and to generate sbomb we [638:37] are using sift again you see fail is [638:41] equal to stop pass is equal to continue. [638:44] If this fails we can fail the pipeline. [638:49] Again whenever we are discussing this [638:52] fail or pass okay in these three steps [638:55] primarily I want you to clearly know [638:58] that the tool will just give you the [639:01] report whether to fail the pipeline or [639:03] not is your responsibility. Now of [639:07] course there are ways via which I can [639:09] integrate sonar cube with my pipeline so [639:12] that if sonar cube quality gate is [639:14] failing my pipeline also fails but that [639:17] is something that I will configure. So I [639:20] want you to know that if you want your [639:23] pipeline to fail upon any of these [639:25] failures then you must be configuring [639:27] your pipeline to fail when these tools [639:31] are not giving you expected results. Now [639:33] what are expected results? For example, [639:35] maybe for your application critical [639:38] vulnerabilities should not be there. But [639:40] if high severity vulnerabilities are [639:42] there, you can still pass the pipeline. [639:46] But for a different banking application, [639:48] they don't want a single vulnerability. [639:52] So for them even a low vulnerability is [639:56] a vulnerability and it should fail the [639:59] pipeline. So you should understand these [640:02] things. If SCA is scanning a library and [640:05] that library does not have a valid [640:08] license in that case as well you fail [640:11] the pipeline but you fail the pipeline [640:13] with logs. You tell them that you are [640:15] using ABC library and ABC library does [640:18] not have a valid license. So understand [640:21] you can fail the pipeline based on the [640:24] inputs received from the tools that you [640:27] have configured at each stage. This [640:30] stage this stage and this stage another [640:32] thing I want to highlight is here I am [640:35] just concentrating on security. If your [640:38] Maven or test integration if your [640:40] pom.xml is not written clearly then also [640:43] this could fail. If you are not running [640:45] proper commands for containerizing your [640:48] application then also this could fail [640:50] but I am discussing about security [640:52] related failures right now nothing else [640:56] here run SCA you have sneak dependency [640:59] check during build generate sbomb and [641:02] persist with artifact persist with [641:04] artifact is the key point here I want [641:08] you to clearly understand that at this [641:11] stage itself we already have the [641:13] artifact Fact we have run MVN verify and [641:17] if I go back to my browser you know MVN [641:20] verify will also run MVN package and if [641:23] we have run MVN package we already have [641:26] the dot jar file with us. So the [641:29] artifact is already available and if we [641:32] have the artifact then of course we can [641:36] have sbomb with us and the next is sonar [641:40] cube analysis. And you see here this is [641:43] the icon for sonar cube. Here we have a [641:46] quality gate. We are saying if sonar [641:49] cube analysis if we if the quality gate [641:52] fails then you stop the pipeline. If it [641:56] passes then you continue. Now the first [641:59] point is imports Jakoco coverage [642:02] reports. Remember here we generated our [642:06] Jakokco reports and here Sonar cube is [642:09] taking an input. Now only when this [642:12] coverage report is shared Sonar cube can [642:16] fail or pass the pipeline based on [642:18] coverage. If you do not have Jako core [642:20] or any other means to provide Sonar cube [642:22] with this code coverage report then [642:24] Sonar Cube could care less in its own. [642:26] it cannot perform these code coverage [642:28] analysis. Second is sonar cube will [642:31] perform SAS and code quality checks. [642:35] Next is enforces quality gates before [642:38] packaging and we have seen this here is [642:42] the part where we are packaging our [642:44] application. Now I want you to [642:46] concentrate here. Okay, build I am [642:48] saying docker image mvn jar jar. I [642:52] should have run mvn package here, right? [642:55] But I'm running running mvn jar jar. Why [642:59] is it? And do I even have to run any [643:01] maven command here? So let's first [643:04] understand this. So if I run mvn package [643:08] then I will have an artifact with me. [643:11] Right? But why to run mvn package? [643:14] Because if I run mvn package all these [643:16] phases will run again. I don't want [643:19] that. I only want a jar file. So what I [643:22] can do is if you have been through the [643:23] Maven lecture the phase for package is I [643:28] can just run MVN jar jar. This will just [643:31] invoke the jar plug-in with the jar goal [643:35] and this will just generate the jar file [643:38] for me and that is what I want. Right? [643:41] Then I should run mvn jar jar. But [643:44] another thing that comes to my mind is [643:46] why do I run mvn jar jar? I already have [643:49] the artifact because I ran MVN verify [643:53] here. Now until and unless any of these [643:56] stages are changing my artifact in some [643:58] way, I don't have to run anything here [644:00] again related to Maven, [clears throat] [644:02] I can just use the artifact that was [644:04] generated here. And this brings me to an [644:07] extremely critical point. The first [644:10] responsibility of DevOps engineers or [644:13] DevSec Ops engineers, the lines are [644:15] blurred now. Everyone is expected to [644:17] know security. So I'll continue saying [644:19] DevOps. The point is the primary goal of [644:23] DevOps engineer is to reduce the time it [644:26] takes for the CI/CD pipeline to run [644:30] while ensuring whatever things that are [644:33] needed are performed. So if you do not [644:36] have to run MVN jar jar and if it [644:39] decreases the runtime of your pipeline [644:41] by even 1 second then remove it. Don't [644:45] keep things that will delay your [644:46] pipelines unnecessarily. [644:49] So here I will select this MVN jar jar. [644:53] Come here and I will strike this out. So [644:56] we are not running MVN jar jar anymore [645:01] and we have understood the reason why we [645:04] are not running and then we are building [645:06] an image. We are tagging the image for [645:09] registry upload. In the next stage that [645:12] is push image, we are pushing the image [645:15] onto a central image registry and I have [645:19] taken ECR here that is elastic container [645:21] registry from AWS and understand along [645:25] with container image as well. You can [645:28] upload sbomb. I do not have a demo for [645:32] this but I will let you explore how do [645:35] you push both image and sbomb and any [645:38] OCI compliant image registry like docker [645:40] hub elastic container registry or other [645:43] native registries provided by different [645:45] cloud providers they would support the [645:48] upload of sbomb along with your [645:50] container image. So feel free to explore [645:54] this point further. Next point is deploy [645:57] to staging. Now understand we are first [646:01] deploying to staging and what you should [646:04] now think is oh my application is now [646:07] running so there must be dashed [646:10] performed. So we have discussed code [646:13] quality and SAS is performed here SCA [646:16] and SBOM is performed here and now we [646:20] want to perform DS. So automated [646:23] application deployment that's fine. [646:24] Prepro tests run on deployed endpoints. [646:27] That's also fine. Here is the actual [646:30] magic that is happening. We are running [646:33] dash and smoke tests. Now what is [646:36] dashed? We know dast will point at your [646:40] HTTP endpoints the entry points to your [646:43] application and it will test whether [646:45] anything in your application can be [646:48] exploited or not. and OASP zap or burp [646:51] suite for dynamic checks. You would use [646:54] curl postman to perform smoke test for [646:57] app readiness. Now again like I told you [647:00] before a full-fledged dash will run in [647:03] the staging and then once the code is [647:06] promoted from staging to production [647:10] there is a quality gate. So first the [647:13] code will only proceed if the dash has [647:17] successfully completed and if the dash [647:20] fails the pipeline fails and that is [647:22] something that you will have to [647:23] configure. OASP zap will not know that [647:26] that it has to fail your pipeline you [647:28] will have to write the logic for that. [647:30] If OASP zap is saying application has [647:33] critical vulnerabilities I cannot move [647:36] forward then you must have your [647:38] application pipeline also fail. Now [647:41] let's say your application is hunky and [647:44] dory and it is promoted to production. [647:47] Now in production you would be using [647:49] canary or whatever your rollout strategy [647:52] is. It could be blue green, it could be [647:54] canary, AB testing. There are plenty. So [647:56] you could have any of those. And then [647:59] here you will be running only minimal [648:02] dash and that is also called passive [648:04] dash. And at the last step here you see [648:07] OASP zap and there is continuous [648:10] monitoring. OASP zap is there because [648:12] you could be running minimal dash plus [648:15] you will have your VAF. VA waf is your [648:17] web application firewall for your [648:19] application and different observability [648:21] tools. Now I want you to clearly know [648:24] that DAST is not an antivirus or [648:28] anti-malware. Okay, you will still need [648:30] an anti-malware or antivirus in your [648:33] virtual machines or in your containers. [648:36] What DAT is doing, DAST is checking for [648:40] any exploits that your running [648:42] application has and [648:45] these are tested using attackers. How [648:48] attackers would exploit your [648:49] application? They will enter your [648:51] application using the entry point of [648:53] your application. It is typically your [648:55] HTTP endpoint and that is how dash is [648:59] performed. But again a full-fledged dash [649:02] is performed in the staging. But when I [649:05] talk about actual production then a [649:08] minimal dash is performed. Now this is [649:12] the entire dev sec ops flow. Now I [649:16] promise you I promise you there is [649:19] nothing outside this in dev secop. There [649:23] could be different tools that are [649:25] available at your disposal but at a very [649:28] high level if you have understood these [649:30] terms I assure you you can talk to [649:34] anyone security centric about dev sec [649:38] ops how flow works. Now of course there [649:41] are deeper integrations for that you [649:44] will have to study more but at a very [649:46] high level you are at a very good place [649:49] to watch any dev sec ops project and [649:53] understand what is happening and you all [649:56] know I always aim for making the [650:00] foundations strong and foundations can [650:03] only be made strong once you know in [650:06] detail what these things are doing [650:08] because Doing demos in my opinion is [650:11] easy. What is hard for most of the [650:15] people is understanding the underlying [650:17] concepts. And once you understand the [650:19] concepts, then doing demos are of course [650:22] extremely easy. Now with that being [650:26] said, I have a request. So if you are [650:28] enjoying the channel and learning [650:31] something new from me, I would really [650:33] encourage if you could like the video. [650:36] You know, leave a comment. Even if it is [650:38] just a high, it helps. YouTube likes [650:40] that and share it with your colleagues. [650:43] I have recently opened channel [650:45] memberships at a very nominal cost. If [650:49] you want to support the channel, please [650:51] consider joining. Now with that small [650:55] advertisement, let's start with the next [650:58] section. Sonar cube architecture. Now [651:01] before I discuss this, let's go to the [651:03] GitHub documentation and let's first [651:06] create some Sonar cube. Let's do some [651:10] labs because too much theory will bore [651:13] you, right? So let's go to lab one. So [651:16] we have done this production flow and [651:18] you can visit the GitHub notes. It has [651:21] details of whatever we are discussing. [651:23] Okay. So now let's do sonar cube lab. So [651:28] I'll go to my AWS console. I already [651:31] have logged in and I'm in Ohio region. [651:34] I'll click on launch instance here. I [651:37] will Okay, good. This is what we all [651:40] wanted. So we are signed in. I'm in the [651:42] launch instance section. Let's call this [651:46] Sonar Cube. I want to use Ubuntu. Let's [651:49] use 24.04. [651:51] Then here, see I have AWS credit. So if [651:55] I click here you see I have some credits [651:57] and if you have created a new account [651:59] you would also get credits for close to [652:02] 180 days and under that free credits you [652:05] are not entitled to use any of these [652:07] instances. You can only use a certain [652:09] section of instances for and they have [652:12] done enough. So the maximum instance [652:14] size you can have is 2 vCPUs and 8 GIB [652:17] and this is more than enough for most of [652:20] the DevOps tools that you would be [652:22] encountering. I'm not talking about a [652:23] IML but for DevOps these are sufficient [652:26] and for our use case I will be choosing [652:29] C7 flex large. Now you could maybe get [652:34] away with T3 small for Sonar Cube and [652:38] Jenkins but I would encourage you to use [652:41] this if you can. So I'll click here and [652:44] I have already created a key pair. That [652:46] key pair is called cloud with VJO SJ [652:50] Ohio. I scroll down. I want to use an [652:54] existing security group. So I have two [652:56] security groups here. One is called [652:58] Jenkins and one is called launch wizard [653:00] one. So I will use launch wizard one for [653:03] sonar cube and we'll use that one for [653:05] the Jenkins EC2 instance. Now the rules [653:08] that this security group has are pretty [653:11] simple. I'm just allowing port 9,000 and [653:14] port 22. So if I go here, this is the [653:19] default. Yeah. Launch wizard one. So [653:22] Narcub SG the security group name is [653:24] launch wizard one. So if I scroll up the [653:27] inbound rules for this one is I'm [653:29] allowing port 22 for SSH and 9,000 is on [653:33] which sonar cube UI listens. And [653:36] similarly we'll see Jenkins when we come [653:38] to Jenkins. Then I'll go back here and [653:41] rest looks good to me. This one I will [653:44] change it to 20 GI and I will click on [653:48] launch instance. I'll go to instances [653:51] and while this is initializing I will [653:56] click here instance actions and I'll go [653:59] to image and templates launch more like [654:02] this and this one I will call Jenkins [654:07] okay and yeah this one is fine this one [654:11] is fine this is also fine this is also [654:14] fine here the security group that I want [654:16] to choose is Jen Jenkins, right? And if [654:21] I show you the Jenkins security group, [654:24] it also has the same thing. So, it is [654:28] allowing port 22 and port 22 is being [654:31] used by me to SSH into Jenkins and 80080 [654:35] will be used to access Jenkins UI. Now I [654:38] want you to know that when Jenkins [654:41] interact with sonar cube and you know it [654:43] will because Jenkins will be you know [654:46] uploading the reports that are generated [654:48] from Jakokco to sonar cube. It will be [654:53] uploaded on port 9,000 and we have [654:55] already allowed port 9,000 in our sonar [654:59] cube instance. So that should be fine. [655:01] So here I will also click on launch [655:03] instance. Now let's go to instances here [655:06] in the Sonar Cube instance. I'll just [655:08] quickly refresh this one. And in Sonar [655:11] Cube, I'll click on connect. Oh, it's [655:13] still initializing. So let's allow it [655:15] some time. The checks have passed. Now [655:18] let me click on connect. I'll [655:20] [clears throat] go to SSH client. And [655:22] here I will just copy this one. So if I [655:25] go to my terminal, I have already [655:29] downloaded the key. I'm in SSH folder. [655:32] And this is the key that I want to use. [655:35] And if you see, I only have 60 [655:38] permissions for this key. Now, if you [655:40] don't have that, fix the permissions. [655:42] And then you can directly access. Oh, [655:45] let me I'll have to recopy it. I'll go [655:50] here. I'll copy this. And then I will [655:52] paste this here. Now, I'll press enter. [655:54] And this should work. And now I see are [655:57] you sure you want to continue [655:58] connecting? So this is I will use to fu [656:02] that is trust on first use. I'll say yes [656:05] and I will then clear my screen so that [656:08] we have more real estate. Then I will go [656:11] to my GitHub notes and we will follow [656:14] rest of the steps from there itself. Now [656:17] here I said 22.04 [656:19] but you can use the latest version as [656:21] well. So if I scroll down we have [656:24] already done this. uh we have allowed [656:27] these ports. We have connected and now [656:29] we want to change the host name. I'm [656:31] changing the host name so that it is [656:33] easy for us. I'll paste this here and it [656:37] will change the host name for me. Then I [656:40] want to also change the time zone so [656:42] that we can see the logs and relate to [656:46] them properly. And if I now type date, [656:49] you will see that it is showing me [656:51] today's date. [656:54] And then if I scroll down now I want to [656:57] install Java. Now you can install Java [657:00] 21 as well but I have been working with [657:02] Java 17 for a very long time and I [657:04] always click on Java 17 but you can use [657:08] Java 21 as well. There is no problem [657:09] with that. I will copy this one and then [657:12] I will paste this. And I think Java 17 [657:14] is deprecating in March 2026 if I'm not [657:18] wrong. You press here and then I'll [657:21] press enter. So what we are doing is [657:23] understand that here we are installing [657:27] JRE. JRE is Java runtime. We are not [657:31] installing JDK. We are installing Java [657:35] JRE because Sonar Cube is an application [657:39] that uses Java. We are not developing or [657:43] building any application in this server [657:45] which is why we are installing JRE. Now [657:47] why this is important? This is important [657:49] because when we go to Jenkins there we [657:52] will be installing JDK. Why JDK? Because [657:55] there we'll be building our application [657:59] using Java compiler. Hence we'll be [658:02] installing JDK at that point. Now the [658:06] fourth step is install Postgress SQL and [658:10] create DB for sonar cube. Now when you [658:13] are dealing with sonar cube you can you [658:16] know install it without a database and [658:19] underneath I mean not without a database [658:22] but without an external database and it [658:24] would be using H2 database internally [658:27] and H2 database is fine for testing and [658:30] development purposes but I wanted this [658:33] demo to be as close to production as [658:36] possible which is why we will be [658:39] configuring a postgress SQL Now for this [658:43] demo I am configuring Postgress SQL in [658:46] the same VM that is running Sonar cube [658:49] but in production you would often [658:51] configure it onto a different VM or you [658:54] could also be using a service like [658:57] Amazon RDS that is a cloud managed [658:59] service for that. So let's start with [659:02] installing Post SQL. So I will copy this [659:06] and this will install Post SQL along [659:08] with common extensions. I will paste [659:10] this here. Let it run and then I will go [659:13] back here. I will enable Postgres SQL so [659:18] that whenever there is a reboot this [659:20] service automatically gets started and [659:23] then next we will start post SQL. Yeah, [659:26] let's start post SQL and then we can [659:30] quickly check the status as well. Yeah, [659:33] status looks fine. It is active. Now [659:36] let's do one thing. What we will do is [659:39] we will go into psql and configure the [659:43] database. So understand that right now [659:46] we are in shell which is performing [659:48] actions onto this virtual machine. But [659:52] what we want is we want to perform [659:54] actions in the postgress SQL instance [659:57] that we have installed. For that we'll [660:00] have to use this super user and we'll be [660:03] using psql to do that. So we'll do this. [660:06] We'll type it here and then we are in [660:09] that UI. Now we will create a login [660:13] role. Understand that what we will do [660:16] here is we will create a database and [660:19] then we will supply a owner for that [660:22] database. The database name would be [660:25] sonar cube and the owner for database [660:27] would be sonar user. So create role this [660:30] this is use a strong password for me [660:33] both the user and the password will be [660:37] sonar. So I will copy this from here and [660:39] then I will paste it here. So this will [660:43] create the role for me. This will create [660:46] the database sonar cube and the owner [660:48] will be sonar. For this one I will paste [660:52] it here. And then next is we'll grant [660:55] all privileges to sonar user on this [660:58] sonar cube database. I'll paste this [661:01] here. And it's saying create database [661:04] sonar cube owner sonar syntax error near [661:07] create. Oh, I think I copied both the [661:10] lines. Is it? I copied the comment. [661:12] Sonar cube does not exist. I think this [661:15] command I didn't Oh, I didn't copy the [661:19] semicolon in sonar as well. I didn't [661:22] copy. Oh, just wait. This is part of the [661:25] syntax. Semicolon. Oh, yeah. Create [661:28] role. Oh, this command also. Oh, this [661:31] did not give an error. Why? Anyway, the [661:33] role is now created. So, you have to [661:35] copy the semicolon as well. Semicolon. [661:38] I'll paste it here. And database is [661:41] created. Now, I will grant access and [661:45] then we can quit it using this. I'll [661:47] quit it from here. And then I can run [661:51] the verification. And so if I run this, [661:53] I should see the role. We saw the role [661:56] is here uh sonar. And then if I run [662:01] this, this should list the sonar cube [662:05] database. So sonar cube database. Yeah, [662:09] it is here. And you see access [662:11] privileges is with sonar. So this is [662:14] fine. And then this should not return [662:17] anything for us right now. I'll paste [662:20] it. I'll just quit it and then I'll [662:22] paste it here. I'll press Ctrl A to come [662:25] at the very front [662:28] and then here I will write Sonar. Sonar [662:32] is my password. I'll press enter and did [662:35] not find any relation. This is fine [662:37] because there is nothing connected to [662:38] this database yet. Nothing connected as [662:41] in sonar cube instance is not talking [662:43] with this sonar cube database that we [662:46] have just created. So I'll clear my [662:47] screen. I'll go at the top. Now here are [662:50] few kernel tweaks that you have to do. [662:52] So I'll just copy them and paste them. [662:55] You have to do them as is. Do not worry [662:57] about them. And here also I will do some [663:01] tweaks that are mentioned here. So you [663:02] just have to follow the GitHub notes and [663:04] you should be at a good place. Now in [663:06] this step what we are doing is we are [663:08] creating a dedicated user for sonar cube [663:13] and that user we are calling sonar. So I [663:15] will copy this. I will paste this here. [663:17] Now whatever actions that we will [663:19] perform so we'll be downloading a sonar [663:21] cube binary. We'll be installing sonar [663:23] cube. Then what we'll do is we'll give [663:25] all the rights to that directory to this [663:28] sonar user so that we run sonar cube [663:32] service using the sonar user because [663:35] that's a best practice. So if I'm [663:37] running sonar cube service I would want [663:38] a different user for that not be using [663:42] Ubuntu user or the root user for that. [663:45] Now comes the important part that is [663:47] download and install. So I'll just [663:50] quickly open this link to show you [663:52] something. So if I zoom in, there are [663:55] different editions of Sonar Cube that [663:57] are available at your disposal at [664:00] learning stage. Community build edition [664:03] is the one that you should be using. It [664:05] covers all the important things that you [664:08] will be dealing with as part of DevSec [664:10] Ops. And if in enterprise environments [664:13] you would typically see enterprise [664:15] edition or data center edition being [664:19] used. And you can go to this page and [664:21] check what all functionalities are [664:23] available. What are the differences [664:25] between all of these versions. Here we [664:28] are using the latest release of [664:30] community build version. So I will go [664:33] here and then I will cd into the [664:36] temporary directory because that is [664:38] where I will be downloading the binary [664:41] for this. So I will just copy this [664:43] command and this will download this [664:45] specific version for it. Now do not use [664:49] it 2 3 months down the line. Check the [664:51] video upload date and if there is a new [664:54] release you should be using that. Now I [664:57] am downloading it as sonarcube.szip. [665:01] As you can see, hyphen o is [665:03] sonarcube.zip. [665:05] And once this is downloaded, what we [665:07] will do is we'll download unzip software [665:11] as well. Unzip package as well to unzip [665:14] it. I'll come here. Let it complete. I [665:16] will paste it. I'll go back to the notes [665:19] again. And here there are simple [665:22] commands. So what what we are doing [665:24] first we are unzipping sonar cube.zip. [665:26] Then we are moving sonar cube to opt and [665:31] like we have discussed in maven lecture [665:33] when you are installing certain binaries [665:34] third party binaries you should be [665:37] installing them in the optional [665:40] directory opt directory and we are [665:42] calling this sonar cube current while we [665:45] are copying it and that's also fine and [665:47] then like I told you what we are doing [665:50] is for this directory we are changing [665:54] the ownership of both user and group to [665:58] sonar user. So I will copy all these [666:02] three commands. I will go here. I'll [666:04] paste them. I'll press enter. So if I do [666:08] ls-l on optar, [666:12] I'll have to add sudo here. Sonar cube [666:15] current. Then you see everything has [666:18] changed to sonar sonar. That is what we [666:20] wanted. Now eighth point is configure [666:23] sonar cube to use post gray SQL. Now see [666:28] what we want is we want sonar cube to be [666:31] using the database that we have just [666:34] created. So we'll have to point our [666:37] sonar cube application to the correct [666:40] database. Understand that we have [666:42] installed both of these things onto a [666:44] single VM. Hence, sonar cube application [666:48] can refer to the sonar cube database [666:50] with just local host. If this were a [666:53] different instance, then you would have [666:55] to type the host name or whatever means [666:58] you have defined to communicate with [667:01] that instance. It could be IP address as [667:03] well, but in production it is [667:04] recommended you always use DNS names for [667:08] communication. Now, I will copy this. I [667:11] will what I'm doing is I am editing this [667:15] file opt sonar cubeq current [667:17] configuration sonar.properties. Now [667:19] whenever you want to change the behavior [667:21] of sonar cube then this is the file that [667:23] you will be editing and I'm editing this [667:26] file as sonar user. You see hyphen u I'm [667:29] using sudo so that from Ubuntu user I [667:32] can invoke diim using sonar user. So [667:36] I'll copy this. So I'll go here and here [667:39] you see when I said H2, H2 is there. It [667:44] is not recommended for production. You [667:46] can see this line here. It is [667:47] recommended for tests but not for [667:49] production. Then you have different [667:51] options here. We have configured a [667:53] different database server for us. So [667:55] what we will do is remove this hash. [667:58] Here I will be providing the username [668:00] and password that should be used. So our [668:03] username was sonar and the password was [668:06] also sonar. Again this is for accessing [668:11] the database not the Linux instance. So [668:13] if I go here what we are doing here is [668:17] we are accessing our database using the [668:20] sonar user. So you see we created a [668:23] sonar user and a password was also sonar [668:26] like we created. So we are supplying [668:28] this here not the Linux user that we [668:31] created. So let's scroll down and here [668:34] we have supplied this. Now if you were [668:37] to use an external database you have [668:39] three options. Okay. So you can use [668:42] Microsoft SQL server or your SQL Azure [668:47] or you can grow Postgra which we are [668:49] using right now. You can also use [668:52] Oracle. And if I talk about here you can [668:55] also use Amazon RDS with Postgray SQL as [668:59] well. So since we are using this we'll [669:01] have to perform the modifications in the [669:05] SQL section. So I am already in insert [669:08] mode here. What I will do is I will [669:11] first uncomment this and you see this is [669:14] the property JDBC postgraq [669:17] localhost is fine because you know my [669:20] instance is the database instance is in [669:22] the same VM. Now what I will do is at [669:25] the very last I will just provide the [669:28] port number and the port number from my [669:30] database is 54 32 [669:34] I think. Hold on. [669:36] Yeah 5432 sonar cube. 5 432 and then [669:43] sonar cube. That is what we wanted. Now [669:48] what I will do is escape shift zz to [669:52] save this. I will go at my GitHub [669:55] documentation. So what we have done thus [669:57] far is we have downloaded the binary. In [670:00] that binary we have moved it into the [670:02] opt directory and then we have done some [670:05] changes in the sonar.properties file. [670:08] But understand right now sonar cube [670:10] service is not running at all because we [670:14] have not created the sonar cube service. [670:16] So what we will now do is we will create [670:19] a service for sonar cube and the service [670:22] that we will be creating will reside in [670:25] this path. So we are calling it sonar [670:27] cube.service and it is not just this. If [670:30] your machine is running SSH you would [670:33] see that there is also a SSH.service [670:36] created the these are called unit files [670:38] and we'll be creating this now. So let [670:40] me copy this. I'll paste it here. Okay. [670:43] and I'll go into insert mode and I will [670:46] paste this block in this. Now what we [670:49] are saying here is we are creating a [670:51] service called sonar cube.service. If I [670:54] run pseudo systemctl stop sonar cube [670:58] service then you should invoke this. If [671:01] I am running start then you must invoke [671:04] these scripts and these scripts as you [671:07] know we have kept the binary in this [671:09] path. Hence we are running it from there [671:12] itself. So I'll copy this and then I [671:14] will paste it here. Escape shift zz. So [671:18] we have created a unit file. Now as soon [671:21] as I do this I can perform a system ctl [671:25] damon reload and my Linux system will [671:28] detect it. So if I now run systemctl [671:31] status [671:33] sonar cube and press enter then it will [671:36] tell me that it is disabled but it is [671:38] detecting that sonar cube service and I [671:40] do not have to write dot service here [671:43] because that is automatically appended. [671:45] Now I will copy this. [671:48] I'll paste this here. I'll enable the [671:50] service and then let's copy both of [671:53] these to start the service and then [671:55] check the status of it. I'll press enter [671:59] and now the service is running. I'll [672:03] press Q. It will exit. Now I will come [672:06] here. See I have defined what all things [672:09] we are doing. We you can read it from [672:11] here. Now if your sonar cube is not [672:13] starting properly then you can use these [672:16] things to troubleshoot. But for now we [672:19] are done. So let's go here. I will go to [672:21] instances. I will copy this. I will [672:26] write http [672:28] colon slash slash I'll add that IP [672:31] address here and here I will write 9000 [672:35] because that is on which sonar cube [672:37] listens on and taa here is sonar cube [672:41] for you sonar cube is starting now till [672:44] the time it starts I want to show you [672:47] the output for that verification command [672:49] now which previously did not give us [672:52] anything so we ran And this during [672:55] verification and this did not result in [672:58] anything. So if I go to my CLI now I [673:02] paste it here. I press enter. I have to [673:06] provide the password not the placeholder [673:09] for the password. So let me write sonar [673:12] here. I'll press enter. And you see it [673:14] is now listing the tables for me. So it [673:17] works now. So I'll quit this. I will [673:20] clear my screen and let's go to our [673:24] browser and then let's go to sonar cube. [673:27] So admin admin is what is the default [673:30] password and login. So write admin admin [673:34] I'll login into this and change password [673:38] screen. That's fine. Old password is [673:40] admin. New password is something very [673:44] difficult to guess. I'll enter my [673:46] password and then I'll confirm it and [673:48] I'll press update. Welcome. I'll do [673:52] later. Got it. And now this is the Sonar [673:56] Cube UI for you. Let's do a quick walk [673:59] through of the UI. If I go to quality [674:02] profiles, you see quality profile are [674:05] collection of rules to apply during an [674:08] analysis. For each language there is a [674:11] default profile and that prof def [674:13] default profile is called sonar way. All [674:17] projects not explicitly assigned to some [674:20] other profile will be analyzed with the [674:22] default. Ideally all projects will use [674:25] the same profile for a language. Now [674:29] quality profiles are collection of rules [674:32] and there are different quality profiles [674:35] created for different programming [674:38] languages and even for different [674:41] components. So if you see Azure resource [674:44] manager also has a quality profile. [674:47] Cloud formation AWS also has a quality [674:51] profile. Same with docker flex go. Now [674:56] let's search for Java here. And if I go [674:59] here I you see there are 537 rules. Now [675:03] quality profiles are collection of [675:05] rules. Now this quality profile has 537 [675:08] rules. So if I go to this profile I can [675:11] see what are those rules and what is the [675:15] categorization of those rules. So if you [675:17] see security related, reliability [675:20] related, maintainability related and I [675:22] told you when I talk about [675:24] maintainability and reliability, we are [675:27] dealing with code quality in general. So [675:30] if I click on this [675:33] active rules, it will take me to this [675:35] rules tab. Now it is listing all the [675:38] rules for me. And you see it has also [675:41] defined severity for these rules. So if [675:44] I go to high it will tell me all the [675:46] rules that are created for high priority [675:51] high severity. Now I want you to [675:53] understand one thing that you as DevOps [675:57] engineer if you are not from a [675:59] programming background you might not [676:01] understand a lot of these things that [676:03] are going on the screen and you don't [676:06] need to understand them deeply. Firstly, [676:09] you will use a quality profile for a [676:12] programming language that you are using. [676:14] And it is often that you could be [676:17] creating a custom quality profile. And [676:19] custom quality profile, let's say if [676:22] Sonar Way for Azure Resource Manager has [676:25] 31 rules, maybe the one you create could [676:29] only have 25 rules. It depends on what [676:32] your project demands. And all these [676:35] rules you will be creating based on [676:38] discussions with developers not DevOps [676:41] alone will be responsible for creating [676:45] quality profiles. Here is quality gate. [676:49] Now quality gate as I told you you can [676:52] have multiple permutation and [676:55] combinations within a quality gate. For [676:57] example I have a quality gate which [677:02] should fail. I am saying the quality [677:04] gate should fail. I'm not saying the [677:06] pipeline should fail. Now upon failure [677:09] of quality gate, your pipeline is [677:11] failing. That is something that you have [677:12] to configure. So here I'm saying my [677:16] quality gate should fail if my coverage [677:20] is greater than or equal to 80%. So [677:24] anything lower than 80% should be [677:27] failing. And then not just this, you can [677:30] create custom quality gates and we will [677:33] see that as we progress with this [677:36] lecture. Now this is the basic of the UI [677:41] of sonar cube. Now that we have this [677:43] understanding, let's go to the sonar [677:47] cube architecture and then we will start [677:50] with our demo. So let's go to my [677:53] diagrams and then let's talk about sonar [677:56] cube architecture. Server stores results [678:02] and runs rules. Scanners analyze code in [678:06] CI and push reports to the server. So [678:11] there are two components. There is a [678:13] server and then there is a scanner like [678:15] a server and a client. So sonar cube [678:19] server is something that we just [678:21] installed. So we installed the sonar [678:24] cube application. We also installed a [678:27] different database and that database [678:30] is running our sonar cube database. And [678:33] then this is the part we have not [678:36] touched yet. Server stores the result. [678:38] So whatever you defined here the [678:41] database for this will be storing [678:44] whatever the results of the scans are [678:46] whatever rules are being created all of [678:48] those things are stored here projects [678:52] measures settings users so if you're [678:54] creating any users those users are [678:56] stored there your projects your default [678:58] settings all those things are stored in [679:01] the sonar Q server and now that we have [679:06] had a proper bifurcation So within the [679:09] sonar cube server we have a sonar cube [679:11] application and a different sonar cube [679:14] database. So you can read here what all [679:16] things are being stored in the database. [679:19] So when we are discussing about sonar [679:21] cube server understand that I'm talking [679:23] about both the sonar cube application [679:25] and the sonar cube database that thing [679:28] is clear. So central service storing [679:32] analysis results metrics and issues [679:35] provides web UI for browsing projects [679:38] triaging issues. So if I go here you see [679:41] there is a UI and in this UI you can [679:44] create multiple projects. You can create [679:46] projects locally or you can set up new [679:50] projects. And then it is saying runs [679:53] compute engine evaluation rules, [679:55] metrics, quality gates. And this we have [679:58] briefly screen in the UI and we'll be [680:01] seeing this in detail when we move to [680:03] the demo. Next is stores data in DB [680:08] projects, measures, settings and users. [680:10] So all of these things are stored in the [680:13] DB itself. exposes HTTP API for scanners [680:18] CI and integrations. Now understand [680:21] sonar cube scanner is sitting somewhere [680:24] else and your CI server is sitting [680:27] somewhere else. How do you enable [680:29] integration between these two for that [680:32] you have something called sonar cube [680:34] scanner. Now sonar cube scanner what it [680:36] will do is it will run in your CI server [680:40] and then whatever reports it is [680:42] receiving it will send those reports to [680:45] your sonar cube server. So next let's [680:48] discuss sonar cube scanner. It can be [680:50] also be thought of as the client or the [680:53] agent. Client that analyzes code and [680:56] produces an analysis report. So this is [680:59] the one that is actually scanning the [681:01] code. Now when I say scanning the code, [681:03] it is performing SAS and code quality. [681:07] Understand that code coverage is not [681:09] performed by this scanner. This scanner [681:13] just receives the report and then [681:16] delivers it to the sonar cube server. [681:19] When it comes to code coverage, next is [681:22] runs in CI agents, build servers or [681:25] developer machines. That is fine. Reads [681:28] config. collects sources, tests and [681:31] coverage reports. We have already [681:32] discussed this stateless and scalable. [681:35] So this is an important point. Maybe you [681:38] have Jenkins in your environment and [681:41] also have GitHub actions. You can [681:44] configure a same sonar cube server but [681:47] can have multiple sonar cube scanners. [681:50] This point is talking about that common [681:53] variants maven gradal generic sonar [681:56] scanner. Now that we understand this, [681:59] let's configure first a Jenkins instance [682:03] and then let's start with our demo. So I [682:07] will go to my browser. I will go to [682:10] instances. I will go to Jenkins here. I [682:14] will click on connect and I will copy [682:17] this. I will go to my terminal. I will [682:20] go to this second tab of this terminal. [682:22] I will paste this. Press enter. go to my [682:27] let's first press yes here then I'll go [682:30] to my GitHub notes here and then I will [682:33] scroll down scroll down scroll down [682:37] further down [682:40] and we have briefly discussed quality [682:42] profiles quality gates sonar cube server [682:46] sonar cube scanner now let's do the [682:49] Jenkins lab so we already have the [682:52] appropriate security group associated [682:54] with our Jenkins server. Let's scroll [682:56] down. Let's do the naming now. We'll [682:59] change the host name to Jenkins because [683:01] we like to adhere to the best practices. [683:04] I'll go here. Then I will [683:08] change the time zone. I'll press enter. [683:12] I'll scroll down. Now here you see I'm [683:16] installing JDK. [683:18] Install JDK includes JRE the runtime [683:21] plus Java C and dev tools needed by [683:24] Maven which is why we are installing JDK [683:27] here. I will press I'll paste it here. [683:29] Press enter. Go here. Now these steps [683:34] are taken directly from the official [683:36] documentation of Jenkins. So there is no [683:39] need to explain these points. We are [683:41] just installing Jenkins. I'll copy these [683:44] and I will come here. Let it complete. [683:48] I'll paste it here. Press enter. Now [683:51] this will install Jenkins. Once Jenkins [683:54] is installed, I will run this system Ctl [683:56] associated command so that we enable [683:58] Jenkins. We start it and then we'll [684:01] check the status of it. I'll come here. [684:04] Let it complete. It has completed. I'll [684:06] paste it here. And then let me go back [684:09] to the documentation again. So now the [684:12] installation is complete. I will also [684:15] run this command so that we know the [684:17] password for Jenkins. I'll just paste [684:20] that command here. And you do not have [684:23] to remember this. When we log into the [684:25] console, you will see this line there as [684:27] well. I'll go to my browser. I will come [684:30] here. Instances. I will copy the IP [684:33] address of my Jenkins instance. I'll [684:36] come here. http [684:39] colon slash slash. I'll add this and [684:43] 8080 [684:45] is the port on which Jenkins listens by [684:47] default. I'm in the sign in page. Now it [684:50] is showing. You can find the password [684:52] from here. So we have already run that. [684:54] So I'll copy the password and I will [684:56] paste it here. I will click on install [685:02] suggested plugins here. Install [685:04] suggested plugins. I encourage you to do [685:06] the same. And this will do the [685:09] installation. till that time let's do [685:11] one thing we can install Maven in our [685:14] Jenkins instance so what I will do is [685:16] I'll copy this and it's simple and in [685:20] our Maven lecture we installed this [685:22] using binary in production you would see [685:25] that often binaries are used to install [685:29] versions so that their life cycle can be [685:31] managed outside of the package managers [685:34] like a here I'll paste it here press [685:36] enter this will install Maven and it [685:38] will also give me the version it has [685:40] installed. It will install 3.8.9 [685:43] and let's check the Maven version. MVN [685:46] 38. Yeah, 387. Anyway, now let's scroll [685:49] down. Now, see, I want you to know why [685:52] we are installing Maven, why we are [685:54] installing Docker. The architecture of [685:57] Jenkins that we are using, our Jenkins [685:59] controller and the Jenkins executor are [686:02] both the same. So our controller is the [686:04] one where we are defining the jobs and [686:06] controller is the one which is also [686:08] acting as an executor because it is also [686:10] running our jobs. In this scenario, you [686:12] would want your Jenkins to have Maven [686:16] and Docker installed. You want to [686:19] install this because you want your [686:21] Jenkins server to run Maven and Docker [686:25] specific commands. I want you to [686:28] understand that Jenkins at a very high [686:30] level is just an automation server, [686:32] right? But that automation server [686:35] requires tools. If you are asking it to [686:38] do something, if you are asking Jenkins [686:40] to do Maven specific jobs, then it must [686:42] have Maven installed. If you are asking [686:44] it to run Helm specific commands, then [686:47] your Jenkins should have Helm installed. [686:49] If you are asking it to run cubectl [686:51] commands, then Jenkins must have cubectl [686:55] installed. But having said that, I also [686:58] want to mention one more thing. In our [687:01] architecture both the controller and [687:03] executor is the same. In production [687:06] implementations you would have Jenkins [687:09] controller and that Jenkins controller [687:11] will be bifurcating the jobs will be [687:13] dividing the jobs onto different agents. [687:17] For example, Docker specific jobs will [687:20] be running onto Docker agents. Agents [687:23] that have Docker installed. Similarly, [687:25] Maven specific jobs or Maven specific [687:29] tasks will be assigned to agents that [687:33] are Maven specific that have Maven [687:36] installed and so on. And as we move on [687:39] to advanced demos, as I upload advanced [687:42] demos in my channel, you would see that [687:44] we would be leveraging all those things. [687:46] And not just for DevOps, but also for [687:49] dev sec ops. when I'll be uploading [687:52] advanced demos you can always refer to [687:55] those because after this lecture you we [687:58] you would have gotten a solid [688:00] understanding of dev seccops flow in [688:03] general now we have Maven installed [688:06] let's install docker again all the steps [688:08] that I'm using are documented in the [688:11] official documentation of docker itself [688:14] I'm just adhering to that first step is [688:16] if you are working with a VM that is [688:19] being used for other purposes I would [688:21] suggest you to remove this so that you [688:24] have a fresh installation. I just [688:26] freshly created this VM. So I'll just [688:28] move to second step here. I will paste [688:30] that here and then I will come here. I [688:34] will add docker repository to apt [688:36] sources and then finally I will run apt [688:40] update and I will install the things [688:43] that are required for docker. I'll paste [688:47] it here. It is installing. While it [688:50] installs, let's see what we want to do [688:52] next. Now, what I want to do is what I'm [688:54] doing is I want my Jenkins to be running [688:59] docker commands. Now, for Jenkins to run [689:02] docker commands, it must be part of [689:05] docker group only then it can run those [689:07] commands. And while I'm on this screen, [689:09] I'm also making Ubuntu, which is my [689:12] current user via which I have logged in [689:13] into my Jenkins server also part of that [689:16] docker group. And then I'm restarting [689:18] the Jenkins service. So I will copy this [689:21] first. Let me see whether that has [689:24] completed because I don't want to [689:25] restart Jenkins in middle it. It is [689:28] username. Let's say let's create a new [689:30] user here. Cloud with vos. Yeah, it's [689:34] already created here. And I'll give it a [689:37] strong password. I will confirm my [689:40] strong password. Full name is Vun [689:43] Jooshi. email is abc at the ratexyz.com. [689:48] Save and continue. And here is the [689:51] Jenkins URL. So I'll save and finish [689:54] this and then I'll click on start using [689:56] Jenkins. And there in my CLI I was [690:00] supposed to run that command, right? [690:03] I'll go here and then let's run all [690:06] these three commands. I'm not sure. I'll [690:08] come here. I'll paste those commands [690:10] here. I'll press enter. Okay, this has [690:13] run. Now let's run docker version. [690:16] Docker versions should run from my [690:18] Ubuntu user as well. And it is working. [690:21] So now we have Jenkins server ready. And [690:24] that Jenkins has Maven and Docker [690:27] installed. And again I could have also [690:30] used plugins instead of directly [690:32] installing these tools onto my Jenkins [690:35] CLI, but that is a discussion that we [690:37] did in our Jenkins CI/CD series. So for [690:40] now let's clear our screen and now let's [690:43] start with the demo. Now before I start [690:46] with the demo I want to give you a quick [690:48] glimpse of what we are going to do. So [690:51] this demo is sonar cube with docker [690:55] maven and Jenkins. End to end we'll be [690:58] injecting bad code. We will be running [691:01] sonar scan. We'll be enforcing quality [691:04] gates. And then we'll be building a [691:06] container image and deploying a [691:09] container. So the first step that we [691:11] will do is we'll create a private [691:13] repository. We will upload our code in [691:16] that private repository. Then we will [691:18] let Jenkins do rest of it. Jenkins will [691:21] first check out the code from the [691:24] private repository. Then we will have [691:26] Maven integrated with Jenkins. Maven [691:29] will be compiling the software and it [691:32] will be there will be Jakokco plug-in. [691:34] We'll be modifying our pom.xml here. [691:37] We'll be adding a Jakco plug-in. That [691:40] plug-in will be collecting the test [691:43] reports and we'll be delivering those [691:45] reports to our Sonar Q. Then the Sonar Q [691:48] will be performing the scans here. And [691:51] here those scans imply SAS and code [691:56] quality scans. Here sonar gate decision. [691:59] Sonar will tell whether to allow this [692:02] pipeline or to fail this pipeline. It [692:04] will only allow the pipeline if the [692:07] quality gates are green. It will deny [692:10] the pipeline to move forward. If quality [692:13] gates are failing and once the quality [692:16] gates succeed, we will be creating a [692:19] container image and we will be finally [692:22] running a container from that container [692:25] image. So that is the complete flow and [692:29] in production the flow that you would [692:32] see would be similar to this. In [692:34] addition to this you would be seeing [692:37] dash, SCA, sbomb and so on. So let's go [692:43] to our [692:45] GitHub nodes and see what we want to do. [692:48] So the first step is we want to create a [692:52] private GitHub repository and then we [692:55] also want to you know create a token. [692:58] Now why we are creating a token? We are [693:00] creating a token because this is the [693:02] token that our Jenkins will be using to [693:06] interact with our GitHub repository. So [693:10] I will copy this name. I will go to my [693:13] GitHub repository here. Let me scroll at [693:17] the top here and then I can go to the [693:20] homepage. Here I can go to repositories [693:25] and then I can click on new repository [693:28] and here I can give a name to this [693:30] repository. Let's call this cloud with [693:32] vos private repo and here I want this to [693:37] be private and I will click on create [693:40] repository. That's it. A repository is [693:42] created for me. Now I want to create a [693:46] personal access token because that is [693:48] how my Jenkins will access my private [693:52] repository. So here I will go to [693:55] settings and then in settings I want to [693:58] go to developer settings and in [694:00] developer settings I want to go to [694:02] personal access tokens. Here I'll go to [694:04] tokens classic and then I will click on [694:08] generate new token. I want to generate a [694:10] classic token here. I can call this [694:14] Jenkins GitHub integration. [694:18] I want the names to define exactly what [694:20] I'm doing. Please pardon the long name. [694:23] Now I will have repo selected and then I [694:26] will click on generate token. I'll be [694:28] deleting this token after this lecture [694:31] after uploading this lecture. There is [694:33] no point remembering this. So what I [694:34] will do is I'll copy this. You will be [694:36] only shown this once. So copy it and [694:39] paste it somewhere safe. So I'll go to [694:41] my [694:43] Visual Studio code and here I'll go to [694:46] this file and I will paste my GitHub [694:49] token here. Then I will go to my [694:53] browser. So we have generated the token. [694:56] We have created the private repository. [694:58] Now what I will do is I'll go here to my [695:01] steps again and I'll scroll down. Now [695:05] see now is the step where we are [695:07] creating a Maven project. So this is the [695:10] project that where we will be running [695:12] all our security scans. Before I run [695:18] this step, I will be using my Jenkins VM [695:21] to run these steps. But if your laptop [695:24] has Maven installed and git installed, [695:27] you can run it from your own laptop as [695:30] well. I am using my Jenkins server so [695:32] that the larger audience can follow [695:35] along with me. I'll be cding into the [695:38] temp directory and then we have already [695:40] seen this. We are using Maven archetype [695:44] quick start with version 1.5 here. So we [695:47] have discussed this bit by bit in our [695:50] Maven lecture. So I'll not spend any [695:52] time discussing this. I'll copy this. [695:54] I'll go to my Jenkins VM. I'm on Jenkins [695:58] VM. I'll paste all of these here. I'll [696:01] press enter. I want you to clearly know [696:04] what we are doing. We are replicating an [696:07] actual production implementation. We are [696:10] treating our laptop, our Jenkins server [696:13] in this scenario as a developer laptop. [696:16] So developer, what would developer do? [696:18] If I press ls, developer will have the [696:22] code like this and then the developer [696:25] will be pushing this code onto a private [696:28] repository. Those are the exact steps [696:32] that we will be doing. But before we do [696:35] those steps, I want you to know the [696:37] archetype that we are using here, the [696:41] quick start archetype that we are using. [696:44] This does not have the Jakokco plug-in [696:47] defined. And the code that is there just [696:51] says hello world. But I want to show you [696:55] how vulnerabilities, how code smells [696:57] will be detected and how Sonar Q will [697:00] show them. Hence what we will do is [697:02] we'll modify a few things so that we can [697:07] see how sonar cube shows these things to [697:10] us. So firstly we will be adding a [697:14] jakokco plug-in and you know the plug-in [697:16] information we put it in the pom.xml. So [697:19] I'll edit my pom.xml. [697:21] I'll go into insert mode. I will scroll [697:25] down where plugins are defined. And I [697:28] see that you see plugins is a plural [697:32] here. Here is the first plug-in. This is [697:35] the plug-in element. And this is the [697:38] starting tag. This is the closing tag [697:40] for plugins. This is the starting tag. [697:43] And at the very bottom, you would see [697:46] there is a closing tag. This is the [697:48] closing tag for plugins. Now, this is [697:51] the closing tag for this plug-in. So [697:54] I'll go into insert mode here and then I [697:57] will add my plug-in at the very last. [698:00] I'll press enter. And now I will go to [698:03] my browser from where I will copy the [698:07] plug-in that I wish to add. Now this [698:09] plug-in you see is for Jakokco Maven [698:13] plug-in. This will have Jakokco. Now [698:16] Jakokco once we have Jakokco it can [698:19] share the code coverage reports with [698:22] your sonar cube. Understand that your [698:27] surefire and failsafe plugins will be [698:30] performing unit and integration tests [698:33] but it is Jakokco plug-in which will be [698:37] creating the coverage reports. Now I [698:40] will copy this and then I will go here [698:44] and I will paste this here. This has [698:46] disturbed the formatting a little and [698:48] we'll fix that later. So I'll save this. [698:51] Now if I run ls you have src and [698:55] pom.xml. Let me also quickly install apt [699:00] install tree so that I can show you the [699:03] structure. I should have added a pseudo [699:06] as well. pseudo apt install tree. So if [699:09] I run tree now you can see there is [699:11] pom.xml XML and src has main which has [699:14] your main code and then test has the [699:17] tests defined. So what I will do is I [699:21] will add this as part of our code. So [699:24] what I will be doing is I'll be [699:26] modifying this code. So right now this [699:30] code just says hello world and for our [699:36] practice that will not suffice. So I'll [699:38] go here. I will press D 10D that will [699:42] delete 10 lines. So if you press DD it [699:45] removes a single line. If you press D10D [699:48] it will remove 10 lines and 20 will [699:50] remove 20 and so on. You get the idea. [699:54] I'll copy this and then I'll paste it [699:56] here. So here what we are doing is we [699:59] are adding code smell. We are adding a [700:02] security vulnerability. You remember [700:04] what we called this? We called this [700:06] security hotspot right and then we are [700:09] also defining a bug and so on. So let's [700:12] close this and we have polluted our code [700:17] now. So we are good to perform the [700:20] testing. Now what is the next step? [700:23] Let's go there. I'll scroll down. Now [700:26] let's perform the get commit. So I'll [700:29] clear my screen first. I'll run ls. You [700:32] see our code is ready to be pushed. I'll [700:34] write get init. Get init will initialize [700:38] this directory. I'll press enter. [700:40] Initialized an empty directory which is [700:43] okay. Get add. I want to add everything [700:47] that you see. And then I will run get [700:52] commit and then hyphen m message could [700:55] be initial commit with form and code. [701:01] Right? I have to spell it right. And [701:03] that's why copy pasting is good. I'll [701:05] press enter. Now [701:09] let me just quickly also do this. User [701:12] email is fine. [701:14] Let's add the name. I will add Vun [701:18] Jooshi here. And now what I can do is [701:21] let's get branch V. Now oh we have to [701:26] run get commit. Hold on. Get commit. And [701:31] then if I now run g branch-env, you see [701:34] this is the master branch. And in GitHub [701:38] we have something called main. And I [701:40] like the name main better. So let's do [701:44] one thing. So here is the repository [701:46] that we created. Okay. So this one if I [701:50] go here you would see that this one has [701:53] main. So let's do one thing. I will just [701:56] quickly rename this. I'll write get [702:00] branch - m main. So we have this main [702:05] name here and then what we can do is I [702:09] can go to my GitHub nodes. I will [702:13] quickly go here and yeah we have to add [702:15] this repository. Now you would have [702:18] mostly seen that this is usually called [702:21] origin and that is the name that is [702:24] quite popular when I talk about remote. [702:26] So I usually name my remotes in a way [702:30] that I can recognize them. So which is [702:32] why I have not written get remote add [702:34] origin and then written that URL. I have [702:37] written get remote add private repo [702:40] because I want to call this private [702:41] repo. So once you have done this now I [702:44] want to push my code to this private [702:46] repository. understand that I cannot [702:49] push to a private repository until and [702:52] unless I provide some authentication [702:54] information and that is the [702:56] authentication information I have to [702:59] provide next. So if I go to Visual [703:01] Studio Code, if I go to my browser, I [703:04] have to copy this. I will paste this [703:08] onto Visual Studio Code. I will copy [703:12] this personal access token and I will [703:15] insert this token [703:18] here. Okay. Then I will copy this. What [703:21] I'm saying is get remote set URL private [703:24] repo this one so that I can freely push [703:28] to this private repository. I'll paste [703:30] it here. I'll press enter. If I run get [703:34] remote v. Now you can see that that [703:37] repository is having this URL that we [703:41] put just now. Once you have done this, [703:44] run this command with caution. Okay, you [703:46] should not be showing your personal [703:48] access tokens. Get push private repo and [703:53] then I want to push this to the main [703:55] branch. So I'll press enter and this has [703:59] done the job. So if I go here to my [704:02] browser and to the repository that we [704:05] just created. If I hit refresh here then [704:09] you see our files pom.xml and srcr here. [704:14] Maven is associated with maven wrapper. [704:17] You don't have to know much about it but [704:19] understand that this file should not go [704:21] in. Ignore anyway. So src and pom.xml [704:27] are now here. Now let's see what we want [704:29] to do next. Create Jenkins freestyle job [704:33] using execute shell. I am using a [704:36] Jenkins freestyle job in this demo for [704:39] simplicity. Once I have advanced demos [704:42] in my channel, they will be using [704:45] Jenkins pipeline. And if you are [704:47] interested in knowing Jenkins pipeline [704:49] or the pipeline syntax or Jenkins [704:53] multibranch pipelines then I already [704:55] have detailed demos in my channel which [704:59] will show you that. But for now let's [705:02] stick to Jenkins freestyle job. And [705:04] anyone who already knows Jenkins would [705:07] know how to convert these freestyle jobs [705:11] into a Jenkins pipeline. and maybe when [705:14] we do advanced demos later in this [705:16] channel, we will be covering dev sec ops [705:20] using Jenkins file. So for now, let's [705:25] create a freestyle job named my app [705:28] sonar job. So I'll go to my Jenkins [705:31] screen. Here is my dashboard and my [705:33] Jenkins is ready. The first thing I want [705:35] to do is change this to dark mode [705:38] because this is blinding cloud with var [705:41] Josh my password is this tab keep me [705:45] signed in and here what I can do is I [705:49] can it was in settings and here [705:53] appearance and I want this appearance to [705:55] be dark it's done oh cancel save and [705:59] then I can go to Jenkins here and let's [706:02] create a job and I want to call this job [706:05] my app sonar job. It's a freestyle job. [706:08] Okay. [706:10] Now here I will now before I forget let [706:15] me do one more thing. Okay. I'll go to [706:18] this pom.xml. It is not formatted [706:20] properly. Right? You have this last [706:23] plug-in is everywhere. And what I will [706:26] do is I'm deliberately doing this so [706:28] that you can also see. I'll go to XML [706:30] formatter. And here what we can do is [706:34] familiar with this one. Yeah, this one. [706:36] I'll paste it here. I'll click on format [706:39] beautify. And this has fixed our [706:42] plug-in. Okay. Previously it was you see [706:45] all over the place. Now it has fixed it. [706:47] So I will quickly copy this. I will go [706:52] here. I will paste it here. Oh, I'll go [706:55] to edit mode. And then I'll select this [706:57] all and I'll paste this. And I'll click [706:59] on commit changes. I'm committing the [707:01] changes from the UI and refactor pom.xml [707:05] for improved structure. You see how [707:08] copilot how fast it was. Anyway, so I'll [707:11] click on commit changes and we have the [707:14] correct structure of pom.xml now. So [707:17] let's go back to our Jenkins job here. [707:19] Source code management get. Now it is [707:22] asking us to provide the repository URL. [707:25] How do we provide that? Let's close this [707:27] one first. Let's go to private repo. [707:30] Let's go here. Let's quickly copy this. [707:34] Yeah, I'll copy this. I'll paste this [707:36] here. Now again, you know that we [707:39] require credentials whenever we want [707:42] Jenkins to access our private [707:44] repository. And credentials is the place [707:46] where you configure it. Now it is saying [707:48] fail to connect. Why failing? Because [707:50] this is a private repository. I'll click [707:52] on add here. Jenkins. I will global [707:56] unrestricted username with password. The [707:59] scope will be global. Global implies any [708:02] of the jobs can also use it. Otherwise, [708:04] a different one is system with system [708:07] only Jenkins and nodes can access it. [708:10] Username will be cloud with varos for [708:13] me. You have to enter your username and [708:16] password. I will go to Visual Studio [708:18] Code. I have my [708:21] personal access token here. I will paste [708:24] it here. ID I can call it GitHub. GitHub [708:29] token seems all right to me. And then I [708:32] will click on add. [708:35] And as soon as I select this, this red [708:37] should go because uh Jenkins will verify [708:40] that indeed the person has provided [708:43] correct credentials for this private [708:46] repository. And here I don't have a [708:48] master anymore. So I'll just write main [708:50] here. I'll scroll down. What will be the [708:52] trigger of the pipeline? For my [708:54] pipeline, I'll directly click on build [708:56] now. So the trigger will be manual here. [708:58] Now environment, I also have to supply [709:03] some details about sonar cube, right? [709:06] And along with details as an IP address, [709:10] I should also have some sort of [709:13] authentication for sonar cube, right? [709:15] And right now in this step, we are [709:18] defining that authentication. So I'll [709:20] click on use secret text or file. I will [709:24] click on add here. I'll click on secret [709:28] text. But Vun, we have not generated [709:30] anything for sonar cube yet. How do you [709:34] intend to provide this here? So we'll be [709:36] doing that. But first I want to explain [709:39] what are bindings. Now bindings are used [709:42] to inject secrets securely into the [709:46] build environment. So you never hardcode [709:49] tokens or paste them into commands. As I [709:53] told you in the next step, we'll be [709:56] adding certain build steps. And there [709:58] will be execute shell. In execute shell, [710:01] I don't want to enter these credentials, [710:04] which is why I'm adding these [710:05] credentials as bindings. And here for [710:08] GitHub, we added it within the git [710:11] section, the source code management [710:13] itself. So these credentials will not be [710:16] displayed in plain text in console [710:19] outputs and that is what we want. That [710:22] is why we are using bindings here. Now [710:25] we must have a project created in sonar [710:29] cube and that project will have [710:31] everything associated with our [710:33] application. Correct? And our [710:35] application if I show this to you is [710:38] called my app. And that is if I go here. [710:42] Yeah, this is called my app. My hyphen [710:46] app and this is important because GAV [710:49] parameters are important when we talk [710:51] about Maven and Sonar Cube integration [710:54] and you will see this shortly. Why I go [710:56] here I can set up a diff a CI here but [711:03] for now let's create a local project. [711:06] Now display name I'm calling it my app. [711:09] And if I hit on thisformational message, [711:13] the project key is unique identifier for [711:15] your project. It may contain up to 400 [711:18] characters. These these these are [711:20] allowed with at least with this [711:21] non-digit. I will click my app. And now [711:24] I'll write my app and then we'll press [711:27] next. Set up new code for project. Now [711:30] what is this? I want you to understand [711:33] that let's say you have 1,000 lines of [711:37] code. You have run sonar cube analysis [711:41] for those thousand lines. The next time [711:44] when you run the analysis, do you want [711:47] to run the analysis for those thousand [711:49] lines again or would you want to run the [711:52] analysis for only things that you have [711:55] changed? I hope you have answered I only [711:57] want to run it for the things that I [711:59] have changed. Understand what I told [712:01] you. Your primary purpose is to ensure [712:05] your pipelines run in as less time as [712:09] possible covering all what is needed and [712:13] to keeping that in mind you want your [712:16] security scans to complete as soon as [712:19] possible. If they are doing the entire [712:21] scans again and again isn't that futile. [712:26] So what this step is asking you is what [712:29] do you define as new code and the [712:33] options that you have are previous [712:35] version and this is the one that we are [712:37] using. So any code that has changed [712:39] since the previous version is considered [712:42] the new code. So let's say a scan was [712:44] run for your thousand lines and it found [712:47] two vulnerabilities in and now in the [712:50] new code what you did is you fixed those [712:52] vulnerabilities. So it will only run on [712:56] the part that has changed. It will not [712:59] run it from the very beginning. So [713:02] that's the crux here. And then you have [713:04] multiple options here. You can supply [713:06] number of days. Any code that has [713:08] changed in the last x days is considered [713:10] new code. So you have various ways you [713:14] can define this. For our demo, we will [713:17] choose previous version. And then we [713:20] will click on create code. And if you [713:22] see here follow the instances default [713:25] default is also the previous version. So [713:27] you can click on create project now. Now [713:29] the project is created but we do not yet [713:34] have any credentials credentials that we [713:39] will use to access sonar cube from our [713:43] Jenkins instance. So for that we'll have [713:46] to do two more things. First is let's [713:49] create a user. I will go here security [713:54] and I will click on users. Let's create [713:58] a new user and then with that user we [714:01] will associate a token and that token [714:05] will be used by Jenkins. So I will click [714:09] on create user and the login name for [714:13] this user I want genkins and then I will [714:16] also write sonar so that I know this [714:20] user is genkins sonar and then I will [714:23] also add my app. Why I'm adding my app? [714:25] Because I know that I created this user [714:28] specifically for this app or for this [714:30] project. Now you will not be using user [714:34] created for Vun or user created for [714:37] Prakash or user created for Kanea in [714:40] here. You will be using a different user [714:43] that is specifically created for [714:45] integration between Jenkins and Sonar [714:47] Cube. So that is what I have done here. [714:50] Now the password that you provide here [714:53] is not needed for us because we will not [714:55] be using this password to access Sonarq. [714:59] We'll be using token. But since it is [715:01] asking it, let's provide that password [715:03] as well. And here the name will be [715:06] Jenkins Sonar and ABC XYZ can be the [715:11] email address. So I'll create this user [715:14] and then if I come here, I can create a [715:18] token. Now token I will call Jenkins [715:22] sonar and then you can set an expiry as [715:24] well. So I'll click on generate and now [715:27] I will copy this token. And this is also [715:30] shown only once. So it is better if you [715:33] save it somewhere. So I'll paste it [715:35] here. And then I will close this. Now [715:38] what I want to do is we have a project [715:42] created. We have a user created. We also [715:45] have a token associated with that user. [715:48] But what I also want to do is I want to [715:50] give that user system administrator [715:53] privileges. Now that user is a sonar [715:56] administrator. Now I'll click on done. [715:58] And third thing that I want to do is I [716:01] want to define a quality gate. I'll [716:04] click on create quality gate. I'll call [716:06] this my app quality gate. I'll click on [716:12] create. And you see whatever was there [716:15] in the default quality gate those things [716:17] are already here. And I also want you to [716:19] see one more thing projects. Every [716:21] project not specifically associated to a [716:24] quality gate will be associated to this [716:27] one by default. So there is this quality [716:30] gate that is available here by default. [716:33] So we'll go here now. See if your [716:36] quality gate fails you will be reaching [716:39] out to your fellow developers and tell [716:42] them that these are the problems you [716:44] will have to fix it. It is not you who [716:46] will be fixing those problems as DevOps [716:49] or dev sec ops. your point of concern is [716:53] ensuring the scans and all these things [716:56] are running properly and in as less time [717:00] as possible. So what I will now do is I [717:04] will delete all these rules because our [717:08] code is not designed for these complex [717:13] rules for now. So I'll delete all these [717:16] rules and then I will add a new [717:19] condition. I will say run the condition [717:21] on overall code not just the new code. [717:24] This is important overall code and then [717:27] quality because I have a tiny code and [717:29] we'll be running this multiple times. [717:32] Okay. So I want it to run on the overall [717:34] code quality gate fails when coverage [717:38] and see you have so many options unit [717:40] test errors unit test failures. I will [717:42] let you explore these options. But here [717:46] I want to choose coverage and I'm saying [717:48] if coverage is less than 80 then you [717:52] fail this quality gate. I will click on [717:56] add condition. Now it is asking me which [718:00] are the projects you want to associate [718:02] with my app quality gate. For me this [718:06] are these are the projects that are not [718:08] yet associated. So I want this to be [718:10] associated. So I will be clicking here. [718:13] Now if I go to all, if I go to with I go [718:18] here, I go here. If I go to with, now [718:21] you see my app is here. So now [718:25] we have all the things in place. Now [718:28] let's go to Jenkins and configure other [718:32] things. So I will come to my Jenkins [718:35] screen and variable I want to use is [718:38] called sonar token and we'll be using [718:41] this variable in our next step and I'll [718:43] show you how specific credentials. Now [718:47] it is asking for credentials. So we have [718:48] not added those credentials in Jenkins [718:50] yet. Let's click on add Jenkins and this [718:55] is fine. And here I want to choose [718:58] secret text. Now you would be thinking [719:00] but Vun in GitHub you chose username and [719:03] password in here you are just adding [719:06] secret text. Understand that GitHub [719:09] expects both username and password [719:12] although these tokens are unique but [719:14] when it comes to sonar cube it is just [719:16] expecting that token and that token is [719:20] unique. So I will type that secret here. [719:23] So let me go to Visual Studio Code. I [719:26] will copy this. I will paste this here. [719:30] So whatever ID I give to the token in [719:33] the CLI or in the Jenkins console, I [719:36] will be using that ID to call these [719:38] credentials. So if I click on this [719:39] question mark, it will give you a gist [719:42] about it. An internal unique ID by which [719:45] these credentials are identified from [719:47] jobs and other configurations. So let us [719:51] give it an identification called sonar [719:53] cube- token. This is a good one. So if I [719:56] scroll down I can click on add here. Now [720:00] it has selected that token for us. If I [720:03] scroll down I will click on build steps. [720:06] Add build steps here. And then I will [720:10] choose execute shell. I will also expand [720:14] this. Now let's go back to our GitHub [720:16] notes. I'll scroll down. I'll scroll [720:19] further down. We are in the demo, right? [720:22] Yeah. Now this is the part that I will [720:26] be pasting onto there. So let's go [720:31] through this. First we are listing [720:34] whatever is there in our workspace. And [720:37] what will be there in our workspace? Our [720:39] first step was get. It will be [720:43] performing a get checkout first. So [720:45] whatever is cloned will be shown to us [720:49] when we run this first step. Then we are [720:53] defining what is the sonar IP. Now sonar [720:57] IP this will be the IP address of your [721:00] sonar cube instance. So you will we'll [721:03] take it that from the AWS console. Now [721:06] the third is Maven build plus test plus [721:11] coverage plus sonar cube scan. [721:15] Understand what I told you in the [721:17] architecture part. There are two things [721:20] to sonar cube. There is a server and [721:24] then there is a scanner. Now for the [721:28] scanner part, we are using a Maven [721:31] plug-in that is performing the scanning [721:35] for us. That is playing the role of the [721:38] scanner here. But for certain other [721:41] languages, you will have to install a [721:43] Jenkins plug-in called Sonar Cube [721:47] plugin. And this is important. So I'll [721:49] just quickly show that to you as well. [721:51] If I open this in new tab, if I go to [721:56] plugins here, I Oh, see it's saying Java [722:00] 17 is end of life March 2026. So, we'll [722:04] have to I'll start using Java 21 from [722:07] now onwards. I'll go to plugins here. [722:09] I'll go to available plugins and then [722:12] here I can check for Sonar Cube. So you [722:15] will be installing this sonar cube [722:17] scanner plug-in for certain advanced use [722:20] cases. For now we are just using Maven [722:25] plug-in to perform the functionality of [722:29] scanner. So what we are saying is we are [722:31] first running MVN clean which will just [722:34] delete the target folder. It will [722:36] perform the cleanup and we have [722:38] discussed this plenty of times. If I [722:40] talk about clean, clean has three [722:44] phases. We are running this phase. It [722:46] will remove all files generated by the [722:49] previous build. That is what this will [722:51] do. Then we are saying verify. Now again [722:56] understand this is extremely important. [722:59] We are running verify. And if you run [723:02] verify then all the previous phases are [723:06] run. So verify will also run the package [723:09] integration test and all of these. So if [723:12] you have run MVN verify then you already [723:15] have the artifact. Keep this important [723:18] point in mind. So first we ran the clean [723:21] phase then we ran the verify phase and [723:24] after that we are invoking a maven [723:27] plug-in named sonar and within that [723:30] plug-in we are calling the role sonar. I [723:35] hope this is understood. Then we are [723:38] saying sonar project key is my app. That [723:41] is the project key we defined back in [723:44] the sonar cube UI. Remember they must be [723:49] same. Then we are providing sonar cube [723:52] host URL. This is the URL on which sonar [723:56] cube is listening. So we have [723:57] variableized it and the variables value [724:01] is being picked up from here. Next is [724:05] sonar cube token. So from where are you [724:08] picking the token? So we have created a [724:11] binding which is why we can call it like [724:13] this sonar token. Why we can call it [724:16] like this? Because I have defined that [724:21] variable like this sonar [724:24] token. Try to create that mental picture [724:26] in your head right now. And then what we [724:29] are saying is package. Now this is the [724:32] important piece. You do not have to run [724:35] any of these. Why you don't have to run? [724:38] Because we already have the artifact at [724:40] this stage. Then why do we want to run [724:44] this? Hence, we will not be running [724:47] Maven package or [724:51] MVN jar jar. You could have also thought [724:53] that we can run that but that is not [724:55] needed because MVN verify has already [724:58] created the artifact for you. And then [725:02] in step five we are creating a docker [725:04] file. Understand that this would have [725:07] resulted in a target folder and that [725:09] target folder would have the jar file. [725:12] And what we are doing here is we are [725:15] just copying that jar file and naming it [725:19] app.jar. We are using an image that has [725:24] Java runtime environment JRE because we [725:28] just want to run an application that is [725:31] based on Java. We are not building a [725:33] Java application. Hence, we do not need [725:36] the Java compiler. And then finally [725:39] entry point is this. We are running the [725:41] application Java - jar app.jar is the [725:47] artifact that I want to run. So this is [725:50] my docker file. Then using this docker [725:53] file I am building my container image. [725:57] I'm creating my container image. And [726:00] then in seventh step what I am doing is [726:03] I am removing my Java app. Now why I [726:07] have added this step? I am removing [726:09] forcefully removing this container. Now [726:11] if you run this multiple times then this [726:14] will give you an error stating the [726:16] container with this name already exists. [726:18] So whenever I'm kicking in my pipeline, [726:21] I have added this step so that your [726:24] pipeline doesn't fail for this reason [726:26] that this container already exists. Now [726:29] I want to let you know one more thing. [726:32] If you are building image like this, the [726:35] build context will also have your source [726:37] code and your pom.xml. And you do not [726:41] want that in production implementations. [726:43] you want your build context should only [726:45] have files that are needed for docker [726:47] build. I just wanted to mention this so [726:50] that you have clear idea. What I will [726:53] now do is I will copy this entire block. [726:56] I'll go to my Jenkins UI. I will paste [727:00] this here. I will go to AWS screen. I [727:03] will copy my sonar cube IP address. I'll [727:07] copy this. I'll come here. I'll go to [727:11] the variable section. And here I will [727:14] define the value of my variable. Another [727:17] thing that I want to do is I don't want [727:20] to run this MVN package unnecessarily [727:23] because I already have the artifact with [727:25] me. And I will also comment this one [727:28] out. This looks fine to me. And yeah, we [727:34] have added the token. My app is here. [727:36] Credentials we have provided. I think we [727:39] are good to apply this. And let me [727:42] quickly check in the GitHub notes if [727:44] there is something that we need to do [727:46] after this. Yeah, this is good. So let's [727:49] go here. I will click on save and let's [727:52] click on build. Now this will trigger [727:55] the pipeline. Now let's see what [727:57] happens. Let's click here. Let's go to [728:00] console output so that we can see what [728:02] is going on. Okay. Okay. The pipeline [728:06] has failed. And the reason for failure [728:09] is quality gate status failed. View [728:13] details on this. Now before I show you [728:18] the UI, I want to show you something [728:20] else here. I think I didn't reach here [728:24] and I directly moved on to next steps. [728:26] What we are saying here is quality gate [728:30] weight is equal to two. We are saying [728:35] you wait for the quality gate. If the [728:38] quality gate is failing, you fail the [728:42] pipeline. That is what we have done. The [728:45] quality gate, we were waiting for the [728:47] quality gate to either succeed or fail. [728:50] It failed and we have failed the [728:53] pipeline. That is what you see here. Now [728:56] let's copy this and then I will paste it [728:59] here and then yeah let's go to my app. [729:04] So this will show me the details about [729:07] my app project and it says failed new [729:11] code is no because we were we selected [729:14] overall code and it is showing all the [729:18] things that we discussed security [729:20] hotspot one. So let's start with all of [729:23] these. So issues security hotspot code [729:26] measures activity. Let's do a walk [729:29] through of all of these. If I go to [729:31] issues, it will tell me about all the [729:34] issues that are there in my code. Now [729:37] again I'm saying not all of these things [729:40] would be understood by you but it is [729:42] okay as long as you you know understand [729:45] them from a very high level. So issues [729:48] is segregating things based on these are [729:50] for pom.xml. Okay. And these things are [729:54] for your main code itself. App dot on [729:56] Java. And then you have test.java [730:01] has these things. Now again let's go to [730:05] security hotspots. If I go to security [730:08] hotspots, we can see that there was one [730:11] password and it has said password [730:13] detected in this. And then we can go on [730:16] code. It will show you the entire code. [730:20] Now I want you to know why this pipeline [730:23] failed. The coverage is 0%. Now what [730:27] does coverage mean? Coverage imply test [730:30] coverage, right? And I told you that the [730:32] testing requires developer to define a [730:36] testing framework and then define test [730:40] cases that how do you want to test this, [730:42] how do you want to test that and then [730:44] the plugins that Maven has can perform [730:48] that testing but the developer must [730:51] define the framework and the tests to be [730:54] performed. In our scenario, we didn't [730:56] have any of those. We just had the [730:58] Jakokco plug-in. So what Jakokco plug-in [731:01] received was the coverage is 0%. And [731:04] that is what we see on our screen. If [731:07] you see source code, the coverage is 0%. [731:10] I expand this 0%. [731:13] So what I am trying to convey to you is [731:18] your developers must define test cases. [731:21] And why this failed? because we have a [731:24] quality gate that wanted [731:27] more than 80% or equal to 80%, but we [731:31] are giving it 0%. Hence our quality gate [731:35] failed. Now what we want to do is now we [731:39] want to rerun the pipeline and we want [731:42] to sorry this was the quality gate but [731:45] uh it's saying the same thing if [731:47] coverage is less than 80% you fail it. [731:50] Now what we want to do is we want to [731:53] rerun our pipeline but this time we want [731:56] to have it run successfully. Now in [732:00] actual production you would be reaching [732:02] out to your developers and telling them [732:04] that fix these problems only then the [732:06] pipeline will succeed. But in here [732:09] we are the problem creator and we are [732:12] the problem fixers. So what we will do [732:14] is we will just delete this condition [732:17] altogether so that it does not check for [732:21] this and then I will come to my Jenkins [732:24] UI [732:26] and then I will rerun this pipeline [732:29] build now and this time let's see how [732:33] this works and then I can go to console [732:37] output [732:39] and this time it should ideally succeed. [732:43] So it would be running all the steps [732:45] like we have defined in our execute [732:49] shell section. And you see it has run [732:53] everything. And this error is because [732:56] there was no container with name my Java [732:58] app because previous run did not reach [733:01] this stage at all. It failed in that [733:04] section itself in the quality gate [733:07] itself. Here then we are creating the [733:10] container. Now after creating the [733:12] container we are also deleting the [733:14] container once it has done what it is [733:17] supposed to do. I hope you have an end [733:20] to end understanding. Let me quickly see [733:23] if I want to cover anything else. So I [733:26] have covered everything that I wanted to [733:28] cover. So that's a wrap for this [733:30] lecture. In this lecture we took dev [733:33] seccops from concepts to production. You [733:36] saw why dev secops exists. what each [733:39] security tool category means and how SAS [733:43] dash code quality checks SCA and SBOM [733:47] fit into real CI pipeline. We installed [733:52] Sonar Cube, explored its architecture, [733:55] set up Jenkins with Maven and Docker, [733:58] and finally ran a full Sonar Cube plus [734:01] Jenkins dev secc Ops workflow end to [734:04] end. If this session helped you [734:07] understand dev sec ops the way a DevOps [734:09] engineer should do like the video, drop [734:13] your questions in the comment and share [734:15] it with anyone who wants to level up [734:18] their cloud and DevOps skills. I will [734:21] see you in the next one. Thank you very [734:24] much. Hello everyone, I am Varun Jooshi [734:26] and I welcome you to cloud with VJosh [734:29] where we build realworld DevOps and [734:32] cloudnative systems through hands-on [734:35] production focused demos. Today we are [734:39] building a full production grade dev sec [734:42] ops pipeline the kind that will help you [734:45] in interviews and in real world project [734:48] work. We will combine Jenkins, Sonar [734:52] Cube, Trivy, Amazon ECR, Amazon EKS, [734:56] Arbback, image scanning and build a [735:00] rollouts. We will cover installation and [735:03] configuration of Jenkins controller on [735:06] EC2. We'll be setting up a secure asset [735:10] based Jenkins agent. Install sonar cube [735:14] with postres SQL and configure quality [735:17] gates. build the CI flow, git, trivi [735:21] file system, maven sonar, image scan, [735:24] ECR push and so on. We will be [735:28] provisioning a multi-AZ Amazon EKS [735:31] cluster. Configure AM configure AWS O [735:36] config map arbback and deploy with [735:39] rollout checks. Run the entire flow from [735:42] code push to a secured production grade [735:46] deployment. All commands, manifests, IM [735:50] policies and the Jenkins file are [735:53] available in the GitHub notes so you can [735:55] follow along easily. So let's get [735:59] started. Like in life, this lecture also [736:02] has some prerequisites. So let's quickly [736:05] visit them and after that we'll get [736:08] started. I'll go to my browser and here [736:10] I am in my YouTube channel. So firstly [736:14] you should have good understanding of [736:18] Maven and Sonar cube as we will be using [736:22] these tools extensively in our demo. [736:25] There are certain Jenkins concepts and [736:29] other Kubernetes concepts that you must [736:32] know as we progress with this lecture [736:34] but all of those things are already [736:37] covered in my channel. I will link [736:40] appropriate videos where required so [736:44] that you can learn without interruption. [736:48] With that being said, let's go to the [736:50] diagrams and see what are we going to [736:53] deploy in this session. So demo end [736:58] toend production dev sec ops pipeline. [737:02] See here the orchestrator that will be [737:05] orchestrating our entire CI/CD will be [737:08] Jenkins. We will have a Jenkins [737:11] controller and the tasks that are [737:14] related to our projects will be [737:16] performed by a Jenkins agent so that [737:20] there is proper role separation and the [737:23] controller just handles the logic the [737:26] control plane and there is an agent that [737:30] is actually running the job for you. Now [737:34] let's see what all stages or what all [737:37] steps will be there in our pipeline and [737:41] have a sky by view of all the things. [737:45] First is check out private git [737:48] repository. So like in production [737:51] implementations I'm sure your [737:52] application is also hosted onto a [737:57] private repository. Now that repository [738:00] could be on GitHub, GitLab or any other [738:03] offering but in this lecture we will be [738:06] leveraging GitHub as our private [738:10] repository. Now if you have a private [738:13] repository then if you want to access [738:16] that repository you will need [738:18] credentials to do that and that is [738:20] exactly what we are going to do in the [738:24] very first stage. Now to give you a [738:27] background of the application that we [738:29] are deploying, we are deploying a Java [738:32] application. A very simple Java [738:34] application that just says welcome to [738:37] cloud with VJOS. [738:39] Hence [snorts] [738:40] in the repository what we will start [738:43] with is pom.xml [738:46] along with the source code and as we [738:49] progress there will be more things that [738:52] will be added to that repository. more [738:54] things like the Docker file, the Jenkins [738:57] file and so on. But we will start with [739:00] the source code and pom.xml. [739:04] And that is exactly what your developers [739:07] will be doing. Your developers will be [739:09] pushing the code along with pom.xml into [739:13] a private repository and from there you [739:16] DevOps folks will start your work. Now I [739:19] will be mentioning DevOps because as you [739:22] know the line between DevOps and DevSec [739:25] Ops is now blurred. So everything [739:28] including security is slowly and [739:31] steadily moving in the DevOps side of [739:35] things. In the second step we will be [739:38] performing a TV scan and this TV scan is [739:43] SCA. And if you remember, we learned in [739:46] our Sonar Cube lecture what exactly is [739:49] SCA? What will SCA do? SCA will scan for [739:53] vulnerabilities [739:55] in your thirdparty libraries. For [739:58] example, we know that in Java, we define [740:02] all the dependencies or the third-party [740:04] libraries in our pom.xml. [740:08] So, Trivy will be scanning this pom.xml [740:11] XML to see whether there are any [740:14] vulnerabilities in the dependencies that [740:16] we have defined or not and TV is not [740:20] just SCA it will also check our OS [740:23] packages. It will also check whether [740:26] there are any secrets that are openly [740:29] available the well-known patterns. For [740:31] example, maybe you have AWS secrets in [740:35] your source code and you should never be [740:38] doing that. Trivy will be able to detect [740:41] that. So the first step is scanning SCA [740:44] and if this fails our pipeline will fail [740:48] and that is what we want in production. [740:50] We want that if something is not right [740:53] with my code. If my code or my [740:55] dependencies they have any [740:58] vulnerabilities I want my pipeline to [741:01] fail. Then next we have build plus test. [741:06] In the build stage, we'll be leveraging [741:08] Maven to build our code and then we will [741:12] be performing scans using a plug-in [741:16] called Sonar. And as you see in the next [741:20] stage, we have Sonar cube analysis which [741:23] will be performing SAS that is static [741:27] application security testing. And in [741:30] case if any of these things fail, fail [741:33] as in if your code has vulnerabilities [741:37] and your quality gates define that I [741:41] cannot pass this code if there are any [741:43] vulnerabilities or if your code coverage [741:47] is less than let's say 80%, then I will [741:51] not allow the pipeline to move forward. [741:53] I will fail this pipeline again. If you [741:57] have not watched the Maven and Sonar [741:59] cube lecture, these things will be a [742:02] foreign language for you. So I request [742:04] you to first go through that so that all [742:08] things in this demo make sense to you. [742:11] And in the next stage as we know we'll [742:14] be doing ECR login plus we'll be [742:16] building the image. So if you want to [742:19] build the image then you must have tools [742:21] that are required to build the image [742:23] such as if you are using docker you [742:26] should have docker installed to build [742:29] your container images then ECR login. So [742:33] if you want to at some stage push your [742:36] container image to an image registry [742:40] then of course you will have to login [742:42] into that image registry. And in our [742:45] demo we are using Amazon ECR to host our [742:50] container images. Hence in this stage [742:53] what we'll be doing is we'll first be [742:55] performing ECR login. Then we'll be [742:58] building the image and in subsequent [743:00] stage next to next stage we'll be [743:02] pushing that container image to AWS ECR. [743:06] At this stage I want you to understand [743:09] that we will also be performing try [743:12] image vulnerability scanning. Wun we [743:16] have already performed SCA here. We have [743:19] already performed SAS here. SAS is which [743:23] is checking your code for [743:25] vulnerabilities. [743:27] SCA was checking your dependencies for [743:30] vulnerabilities. SAS is checking your [743:33] code for vulnerabilities. Understanding [743:36] the difference is the key here. And then [743:39] here the Trivy image scan is performing [743:44] the scan on your image. But Wun, we have [743:47] already done these things. Why do I need [743:49] to scan my image? I want you to clearly [743:52] know that your docker file uses a base [743:56] image. In this step, we are scanning [743:59] your base image for any vulnerabilities. [744:03] And again, in this step as well, if [744:06] there are high or critical [744:08] vulnerabilities, if there are any [744:10] findings like this, then we are failing [744:12] the pipeline. I want you to clearly know [744:15] that there are three stages where we [744:20] essentially are failing the pipeline. If [744:23] our desired thresholds are not met, this [744:27] is the key point here. For example, [744:30] maybe for application one, 70% code [744:33] coverage is all right. But for [744:36] application two, at least 90% code [744:39] coverage is important. So for you DevOps [744:43] folks, it becomes imperative that you [744:46] configure these quality gates, these [744:48] failures, these successfuls based on [744:52] your app requirement and then as we [744:55] discussed here we'll be pushing to AWS [744:58] ECR and then finally we will be [745:01] deploying it to Amazon EKS and I believe [745:04] this is the step where everything will [745:07] come together your AWS knowledge, your [745:09] authentication knowledge your [745:11] authorization knowledge because we will [745:13] be leveraging principle of least [745:16] privilege. So our Jenkins which will be [745:19] deploying resources onto Amazon EKS must [745:23] authenticate and must only be able to do [745:28] things that we allow it to do nothing [745:30] more nothing less. So with that critical [745:35] understanding [745:37] let's get started. So in this demo the [745:40] first step would be to configure our [745:44] infrastructure. So we'll be configuring [745:46] three things and straight away we'll be [745:49] doing that. First we'll be creating a [745:53] Jenkins controller. Second, we'll be [745:56] creating a Jenkins agent. Third, we'll [745:59] be establishing the communication [746:01] between the Jenkins controller and [746:03] Jenkins agent. And that is where SSH [746:05] comes into picture. And then at the very [746:08] last, we'll be configuring sonar cube [746:10] with an external database which is [746:13] something that you would see in [746:15] production implementations. So with that [746:17] said, what I will do is I'll go to my [746:19] GitHub repository. I'll go to my browser [746:21] first. Here I have this repository [746:24] Jenkins basics to production. I have [746:26] added project one here. All the steps [746:30] inch by inch, word by word are mentioned [746:33] in my repository. So you can follow [746:35] along with me and if it helps do do [746:40] start the repository it helps me a lot. [746:43] Now [snorts] first step is install [746:46] Jenkins controller. So let's go there. [746:49] And you see I am using C7 iflex.large [746:53] instance. I'll be using this instance [746:56] for all the three components. My [746:58] controller my agent and sonar cube. This [747:01] provides two vCPUs and 4 GB of RAM which [747:06] I feel is sufficient for these demos. [747:09] Now if you want to lower the instance [747:11] type then you can maybe use T3 small but [747:15] I have used C7i flex.l large throughout [747:20] this lecture and this falls under the [747:23] AWS free credit. So if you have those [747:26] credits, if you don't have those [747:28] credits, then I would encourage you to [747:30] create a new AWS account with all the [747:33] details and then AWS will give you I [747:36] think 120 free credits to begin with and [747:38] they will also give you 50 more if you [747:41] you know clarify certain criterias for [747:43] them. Anyway, I'll let you figure that [747:46] out. For now, let's go to the console [747:49] and let's start creating instances. I'll [747:52] click on launch instance here. This one [747:55] I want everyone that is following this [747:58] lecture to be very specific with [748:00] nomenclature because nomomenclature will [748:03] help you a lot when you are dealing with [748:06] a production grade issue. So Jenkins [748:09] controller is the name that I'll give to [748:11] this machine. This is the name tag not [748:14] the host name of the machine. The host [748:15] name we'll be changing once the machine [748:18] is configured for us. I'll be choosing [748:20] Ubuntu. Instance type here I will change [748:24] this to C7i flex large which is free [748:28] tier eligible it will give me two vCPUs [748:30] and 4 GIB of memory GBtes. So I'll click [748:34] here key pair I have already created a [748:38] key pair and key pair is called cloud [748:40] with var sjen Mumbai. I want you to know [748:44] that these key pairs are regional in [748:48] nature. So if you are dealing with an [748:50] AWS account and you deploy resources to [748:53] multiple regions then I encourage you to [748:56] always have the region name in your key [749:00] bear naming. Another point I want to [749:02] highlight I am calling this account [749:05] cloud with varoo sj which is why I have [749:08] it here so that if I have multiple key [749:11] pairs in my machine I know that which [749:14] key pair to be used for which account. [749:17] This gives you a clarity. Hence, I [749:19] highly highly encourage you to use [749:22] naming that helps you understand when [749:26] you look at this 6 months down the line [749:28] because I assure you, you won't [749:30] remember. Now, if you do not have a key [749:33] pair, just click on create key pair [749:36] here. The algorithm that you should be [749:38] choosing is ED25519. [749:41] You could choose RS as well but this one [749:43] is a latest and greatest one which you [749:45] should be using anyway. So I already [749:47] have this key pair created and I have [749:50] selected that. I'll scroll down. I will [749:52] be using the default settings here but [749:55] here I will be choosing an existing [749:58] security group and the security group I [750:01] already have three security groups [750:03] created. I will be choosing the security [750:06] group that I have created for Jenkins [750:09] controller just to give you a quick [750:11] glimpse of it. I'll open this in new [750:13] tab. I will go to security groups and [750:17] here I will select the Jenkins [750:19] controller. Now the inbound rules I have [750:23] is port 22 is allowed from anywhere and [750:26] this is something that I am using and [750:28] you can be restrictive with it. You can [750:30] only allow your source IP. But I have [750:33] configured these for simplicity. In [750:36] actual production implementations, you [750:38] would see that only what is needed is [750:42] allowed. And if you see the latest [750:44] trend, this trend of setting up jump [750:46] hosts or sshing into machine is slowly [750:50] moving away when you are on AWS [750:52] specifically because you would be using [750:55] systems manager which provides you other [750:57] means to login into your machine rather [751:00] than using SSH. So you would be [751:02] typically using those things here, not [751:06] the traditional jump hosts or bastion [751:09] hosts that you would have used before. [751:12] Second, I am allowing port 8080. Now [751:15] 80080 is the port on which Jenkins [751:19] listens. So I am allowing that and again [751:22] I'm allowing that for the entire world [751:24] which you might not be doing in [751:26] production but I'm doing this for [751:29] simplicity. Then we have done this. I [751:32] also want to give you a walk through of [751:34] all other security groups because we are [751:36] already in this screen. And then we have [751:38] Jenkins agent and Jenkins agent I am [751:42] only allowing port 22 here only as such [751:46] it is the Jenkins agent that is [751:50] listening on port 22 from everyone since [751:53] we will be connecting to our Jenkins [751:55] agent from Jenkins controller over port [751:58] 22 that port 22 is already allowed in [752:02] our Jenkins agent so that Jenkins [752:04] controller can connect to it. Now [752:06] previously JNLP was used Java network [752:12] JNLP Java network [752:16] hold on I've forgotten the name JNLP [752:19] Java network launch protocol yes Java [752:21] network launch protocol was used but now [752:24] we are using normal SSH connections and [752:28] that was used because you know if your [752:30] agent was sitting behind the firewall [752:32] and so on but now it is preferred to use [752:36] SSH based agent and we'll see this [752:38] briefly as we progress. Then we have the [752:41] sonar cube SG. Sonar cube SG again 22 is [752:45] used so that I can SSH into Sonar cube [752:48] and then I have allowed 9,000 so that I [752:52] can access the UI of Sonar Cube. So this [752:55] is for SSH and this is for UI. Again [752:58] when Jenkins interacts with sonar cube [753:01] it interacts on port 9,000 itself which [753:04] is why you do not have to allow anything [753:06] else. Again in production [753:08] implementations you only allow what is [753:11] needed. Do not allow anything else. [753:14] Principle of least privilege everywhere [753:17] you go. Now launch instance here I will [753:20] come I'll change this to 20GB. Now when [753:23] we'll be configuring agents, we'll have [753:25] that 30GB and I'll explain that to you [753:28] later. But for now, let's keep this 20 [753:30] GB and then I will click on launch [753:33] instance. I'll go to instances [753:36] and I see this is pending. And while [753:39] this is pending, let's also create an [753:41] instance for Sonar Q. We'll just [753:43] creating this that instance. We'll be [753:45] configuring it after we have set up the [753:48] controller and the agent. So let's do [753:50] that later because I see this one has [753:52] already started. So I'll click on [753:53] connect. I'll copy this and then I will [753:57] go to my terminal and in my terminal I [754:01] will cd into SSH directory. I'll press [754:06] ls and with ls you see I have so many [754:09] keys here private keys here. But I want [754:12] you to know that this is the one that we [754:15] will be leveraging. And I know this [754:17] because I have done the naming properly. [754:19] And of course at the AWS screen, I chose [754:22] this. And before I paste this, if I do [754:24] ls-la, then you would also see that I [754:28] have given appropriate permissions. I [754:30] just have 600 for the this key pair. So [754:34] you would also prefer 600 for your [754:38] private keys. I'll paste that here. And [754:41] oh, I have to copy that again. Let me go [754:44] there. I'll copy this and then I will [754:47] paste it here. And this is asking me do [754:50] you trust this? Now this is called trust [754:52] on first use. We'll briefly discuss this [754:55] later as well when controller is making [754:57] connection to the agent. So I'll press [755:00] yes here. And this is added to my known [755:03] host file. Now if I connect to this [755:05] machine any other time it will not ask [755:08] me to you know type yes or no because [755:11] the fingerprint is added in known hosts. [755:15] I trust the destination I am accessing [755:18] because it is indeed the destination I [755:22] want to access. So I will go to my [755:24] GitHub notes now and here let's start [755:28] configuring. So the permissions part [755:30] I've also mentioned here. The first [755:32] thing is we'll be setting the host name [755:34] because we don't want these generic host [755:37] names. We want particular host names [755:40] that tells us what that thing is doing. [755:43] So I will copy this. I'll paste it here. [755:46] Exec bash will just refresh so that we [755:49] see the changes right away. And then I [755:52] want to change the time zone. understand [755:55] that this command is changing the time [755:58] zone of the Linux operating system of [756:01] this machine. But when we install Java [756:04] the JVM part, we still have to change [756:07] the time zone for the JVM part so that [756:10] when you work with the UI, you see the [756:13] correct timestamps in your logs to in [756:16] your UI too. So I will again go back [756:19] here. Now we'll be installing Java 21. [756:23] This is JDK. Now understand that there [756:27] are two options that you have. One is [756:30] JRE and another is JDK. JRE is preferred [756:34] when you want to just run Java [756:37] applications. JDK is preferred when you [756:40] want to also build the Java application. [756:42] So JDK is used predominantly by us [756:46] DevOps and developers. So that we can [756:49] also do compilation and all other things [756:52] that JDK comes with. But why are we [756:56] installing JDK in the controller? Vun, [756:58] we are not building anything in the [757:00] controller. We will be building things [757:03] in our agent. So agent must have JDK. [757:06] That part we understand. But why do I [757:09] need to install JDK in my controller as [757:12] well? The reason for that is Jenkins [757:14] core and plugins depend on JDK tools. [757:18] JDK ensures that you have maximum [757:21] compatibility, stability and plug-in [757:24] supports because we will be leveraging [757:26] different plugins and there could be a [757:28] plug-in which also requires JDK. Hence, [757:31] in the controller as well, I will [757:33] recommend you to install JDK. So I'll [757:36] copy this and then I will go to my [757:40] terminal and I will paste it there and [757:44] this will install JDK. Then is the part [757:48] where we add Jenkins repository. So [757:52] these steps will add the Jenkins [757:54] repository and then once the repository [757:57] is added we can install Jenkins. But [758:00] before we do that, we are also running [758:02] pseudo apt update so that our indexes [758:05] have the information about the latest [758:08] version. So I will copy this thing and [758:10] then I will come here. Let it finish and [758:14] once it finishes I'll paste that onto [758:16] here. I will paste this here. I'll press [758:19] enter. And all these steps that you see [758:22] here are taken from the official [758:23] documentation. And you can see the [758:26] reference link here too. Now is the part [758:28] that I was telling you. Now we'll be [758:30] setting the JVM the time zone in the JVM [758:35] as well. So that is what we'll be doing. [758:37] Now you could use this. It will directly [758:40] do the changes and we'll let you know. [758:42] So let's take this one or you can [758:44] manually just add this part in your you [758:47] know Jenkins service. I will copy this. [758:51] This is essentially modifying the you [758:53] know unit file that is created for [758:56] Jenkins. So I'll go here. I will paste [758:59] it here and then it has done its job. [759:02] And before I yeah let's do these things. [759:06] So first we are doing a demon reload. [759:09] Why a dammon reload? Because we have [759:10] done some changes in the unit file. So [759:12] it is always good to you know do a demon [759:14] reload. Then we are restarting Jenkins. [759:17] And finally we are checking the status [759:19] of Jenkins. Now I don't know why this [759:21] these steps are repeated. Uh let's just [759:24] copy this and then I will also run one [759:26] more command which is like enable [759:28] Jenkins so that even if we reboot our [759:31] system Jenkins service automatically [759:34] start. So it is active and running. I [759:36] will hit Q here and I will paste that [759:39] command and it will enable Jenkins and [759:41] then I want to check the status of [759:43] Jenkins again. It says that Jenkins is [759:47] up and running. So let's go to our [759:49] browser. I will go here. I'll go to [759:52] instances. This instance is already [759:55] selected. I will copy this thing and I [759:58] will paste it here. And then I will also [760:01] add 8080. And I will explicitly mention [760:05] that this is HTTP not HTTP. I'll press [760:08] enter. And voila, we have the Jenkins [760:13] screen in front of us. It is asking that [760:16] you can check the initial password here. [760:19] So we have already ssed into our Jenkins [760:22] server. So we'll go here. I will write p [760:24] sudo cat. I will paste that path here. [760:27] And here is [snorts] my password. So [760:30] I'll copy this thing and then I will [760:33] paste it here. I will hit continue. I'll [760:36] press never here. And then I will be [760:39] installing the suggested plugins. Now [760:42] why I'm choosing this? because it will [760:44] by default install everything that we [760:48] need initially. Now I want you to know [760:50] that in my Jenkins series I have [760:52] explicitly covered what all these things [760:54] are and why we need them. But for now [760:57] just a quick overview we won't be using [761:00] gradal and or you know LDAP these things [761:05] we are not using. We would be of course [761:07] using the dark theme and then git of [761:10] course we would be using. So understand [761:12] at this stage itself that the git [761:15] plug-in is already installed and is [761:19] again a build tool for Java. Gradal [761:22] again an advanced build tool for Java [761:25] especially if you are using Android [761:26] applications. Pipeline graph view we [761:28] need and I'll show this to you while we [761:31] will see how our pipeline is behaving [761:33] and then there are a few more. So we'll [761:36] let this finish until that time let's [761:39] see what we want to do next. Now this [761:43] can finish in the background. Till this [761:45] point we have already done what we [761:47] wanted to do. Now let's start [761:50] configuring our Jenkins agent. Now again [761:53] in Jenkins agent I will be assigning [761:56] 30GB of storage because when I assigned [761:58] 15 GB of storage I was constantly [762:01] getting disk issues. This is because all [762:03] our all your tools that you will be [762:06] installing in agent requires some [762:08] storage. And as we progress, you will [762:10] see why. But at this stage itself, I [762:14] want you to know that Jenkins is just an [762:17] orchestrator for you. So for example, if [762:21] you have to run any commands that are [762:23] specific to Maven, if you have to run [762:25] any commands that are specific to Sonar [762:27] Cube or to Trivy or to AWS or to docker, [762:32] then you must have those tools installed [762:35] because Jenkins is just an orchestrator. [762:38] It will help you move from point A to [762:42] point B. But to reach there, you must [762:45] provide Jenkins with all the tooling [762:47] that it needs. Now since it is our [762:50] Jenkins agent which will be running all [762:52] these jobs for us, we must ensure that [762:56] we provide that agent with all the tools [762:59] that are required. Hence all these tools [763:02] may download you know privy may download [763:05] CVEes to their local database first and [763:08] then if I talk about Maven if you have [763:10] watched the Maven lecture you know there [763:13] is a local repository created and that [763:15] will also take some storage which is why [763:18] I would suggest you to assign 25 to 30 [763:21] GB to your agent because agent will be [763:24] doing the actual leg work hence agent [763:27] must be provided with sufficient [763:30] resources for compute and memory C7 if [763:34] flex large should suffice. So I'll come [763:36] here I will select this Jenkins [763:39] controller action image and templates [763:43] launch more like this and this time I [763:47] will call this Jenkins agent rest I am [763:51] satisfied with most of the things I'll [763:53] change the security group first I will [763:57] choose select existing I will remove [763:59] this controller one I will choose the [764:02] agent security group Jenkins agent SG [764:05] and Then here I will choose 30GB and [764:08] then I will press launch instance. Now I [764:11] want you to know one more thing very [764:13] clearly. There is a difference between [764:16] security group name and the name. This [764:20] name is the name tag. This is the name [764:24] of the security group itself. Now if I [764:27] go to the tags, let's select this agent [764:29] SG. And if I go to tags, I don't see any [764:32] tags here because I have not defined the [764:35] name tag. That is why I don't see [764:37] anything or I do not have any other tags [764:40] as well. But this is the security group [764:43] name itself. This is not a tag. So this [764:46] is a nuanced difference. Most of the [764:49] folks do not understand. There is a [764:51] difference between a tag and the actual [764:54] security group name. And this you know [764:57] this changes if I move to my instances. [765:00] So if I go here the name that you [765:03] provide while creating the instance is [765:06] actually the name tag. So if I move to [765:09] tags you see the name tag is added and [765:12] the value for that tag is Jenkins [765:15] controller. So understand this nuanced [765:17] difference. I'll go to my agent VM now [765:20] and here I will click on connect. Then I [765:23] will copy this and then I will go to my [765:26] terminal and what we have is it has [765:29] named this one Jenkins controller which [765:31] is what we wanted. I will press CtrlT so [765:35] that I get a new tab. I will zoom in a [765:38] little here. I will cd into SSH because [765:41] that is where my private key is. I will [765:44] paste that command. This will work [765:46] because this key is in my working [765:50] directory in my current directory. Now [765:52] if you have this somewhere else then you [765:55] may have to provide that path not may [765:57] you will have to provide that path. I'll [765:59] press enter again. T O FU trust on first [766:03] use. I will say yes and then I will [766:06] clear my screen. Again we'll follow the [766:08] steps from our documentation. That [766:10] documentation is golden. Not just for [766:13] this demo but as you are working in [766:16] production grade projects. You can [766:18] always come and refer this for creating [766:21] or dismantling any infrastructure. So [766:25] here let's move to our controller step. [766:30] This is SSHbased Jenkins agent on EC2. [766:34] We are here. Let's run this command. [766:37] Here we are setting up the host name. [766:39] Host name is Jenkins agent which is [766:42] explanatory. Then we'll set the time [766:45] zone. [766:46] And then we'll scroll down again. We'll [766:49] install Java 21. And here as well we [766:52] need JDK. And we know why we need JDK [766:56] because it is the agent who will be [766:59] building the apps. If you have Maven, [767:02] Maven will require Java underneath and [767:05] Maven to perform the compilation for [767:07] you, to perform the packaging for you, [767:09] to perform the verification for you. It [767:12] will require Java 21, not 21. It will [767:17] require Java development kit that is [767:20] JDK. Now I want you to clearly know that [767:25] you will not use random users to perform [767:28] tasks. You would want that all your [767:31] Jenkins tasks are performed by a user [767:34] that you explicitly create for Jenkins. [767:37] So I will not be running my Jenkins [767:39] tasks using this Ubuntu user. I will [767:42] create a separate user Jenkins so that [767:46] that user is the one that is running [767:49] commands for me. So I will copy this. I [767:52] will paste this here. And you have [767:54] detailed comments in my you know GitHub [767:57] repository so that you know what we are [767:59] doing here. M says I want the home [768:03] directory for this user. S says the [768:07] shell for this user will be bash that is [768:10] the born again shell. Now our Jenkins [768:14] agent till this stage is kind of ready [768:19] because Jenkins agent just needs Java [768:23] installed. That's it. But we have not [768:26] told the controller that this is the [768:29] Jenkins agent. Now to do that thing we [768:33] first have to you know have some sort of [768:36] connection between the controller and [768:39] the agent. So first I want you to [768:43] understand that connection. So I'll take [768:45] you to my diagrams and then I will take [768:48] you here. Now again I have discussed the [768:52] mutual authentication that happens in [768:54] SSH in a greater detail if I talk about [768:58] my Kubernetes TLS section. So if you [769:01] want deeper understanding then you can [769:03] always watch that. But here I will [769:06] explain that to you very briefly. So [769:09] here is our Jenkins controller that is [769:12] running on an EC2 instance and here is [769:14] the Jenkins agent which is also running [769:16] onto an EC2 instance. So we have [769:18] configured both of these. Now we want to [769:22] SSH from our Jenkins controller to our [769:26] Jenkins agent. To do that, first we will [769:30] be generating a key pair using SSH key [769:33] gen utility and we'll be generating this [769:36] key pair in the Jenkins controller [769:38] itself. Now once the key pair is [769:41] generated, what we will do is we know [769:44] that my key.pub which is the public key [769:48] of the key pair that we generated. So [769:50] this is the key pair we generated and we [769:52] generated this key pair in the SSH [769:56] directory itself. Now you would be [769:58] thinking why is this var lib Jenkins? I [770:02] want you to know that if I talk about [770:04] Jenkins, the default Jenkins home is vib [770:09] Jenkins. But the user that we manually [770:12] created in our agent, the home for that [770:15] user will be /home/jenkins itself. That [770:18] is what we expect. If I create a user [770:20] var, then the default home would be [770:23] /home/vosh. [770:25] Similarly, if I talk about Jenkins, what [770:28] it has defined in its configuration is [770:31] the home for me will be vib Jenkins. As [770:36] simple as that. Now, since its home is [770:39] this, our SSH directory will exist here [770:43] and then here is where we'll generate [770:45] our key pair. Now, if this directory [770:48] does not exist anywhere, we will create [770:50] that. But for understanding we are [770:53] saying that we will be creating these [770:55] key pairs in this directory. And once [770:58] this key pair is created we know that [771:02] this public keys information this public [771:05] keys data must be available with the [771:08] Jenkins agent in the authorized_keys [771:12] file. So what we will do is we'll be [771:15] creating this key pair. Then we'll be [771:17] copying this key onto our Jenkins agent. [771:21] As simple as that. Once we have done [771:23] that, Jenkins controller can SSH into [771:27] Jenkins agent by using this private key. [771:31] and Jenkins agent. The SSH HD demon that [771:35] is running in the Jenkins agent can [771:38] identify that indeed the private key [771:41] presented by the Jenkins controller is [771:43] associated with the public key that is [771:46] already there in my authorized keys [771:50] file. As simple as that. So we'll be [771:54] doing this now. Few bullet pointers. [771:57] Jenkins controller acts as SSH client. [772:00] We know that because this is the one [772:02] that will be initiating the [772:04] communication. Second initiates [772:05] communication and the one who initiates [772:08] communication is called the client and [772:11] towards [772:13] the entity for which the connection is [772:16] initiated is called the server. So this [772:18] is the client, this is the server and [772:20] then lastly runs build via SSH agent. So [772:25] whenever we areuling a job and running [772:27] that job, Jenkins controller will be [772:30] asking this Jenkins agent to run that [772:34] job. So Jenkins agent acts as SSH [772:37] server, receives public key, executes [772:40] pipeline steps. We know that. I hope [772:42] this understanding is clear. Now let's [772:45] go to the GitHub repository and while I [772:47] do these things, you will understand [772:49] them in a better way. Now generate SSH [772:54] key on the controller. Now I want you to [772:57] clearly focus on one thing. What I'm [773:00] doing is I will be generating these keys [773:03] and I will be accessing this as the user [773:06] Jenkins. Now what is the significance of [773:10] this? Now the Jenkins user created in [773:13] our controller will be the one who will [773:16] be generating these keys. I don't want [773:18] to use that Ubuntu user. I want to do [773:20] things like we do in production [773:22] implementation. So I'll copy this one. I [773:24] will go to my terminal. I will go to my [773:27] controller. And here I will clear my [773:29] screen for better visibility. I'll paste [773:31] it here. And you see the key part. We [773:35] have you know logged in into this one [773:38] because we have typed pseudo su-en [773:41] jenkins. We have logged in as Jenkins [773:44] user. If I press ls, you see these are [773:48] all the details about my Jenkins home. [773:50] So these are all are the things that are [773:53] available in Jenkins home. Now if you [773:55] have watched my previous lectures on [773:58] Jenkins, you will understand what all [773:59] these things are. But for now I will [774:02] just write ls-la. [774:04] You see SSH is not yet created here. So [774:08] let's do that thing. So I will do mkdir. [774:13] SSH [774:14] and then I will cd into SSH. I will do [774:18] ls. I don't see anything here. Now what [774:21] I will do is now I will be running this [774:24] command because this will make sense [774:26] once you have SSH directory. So I will [774:30] copy this. I wouldn't have to you know [774:33] enter in here because I am using [774:36] absolute paths here not relative paths. [774:40] So I will press enter and this has [774:43] started the process of creating this [774:46] key. Now I want you to understand this [774:47] command as well. This is just a comment [774:49] that we are adding and we are saying the [774:52] algorithm that I want you to use is [774:55] ed2519- [774:58] f is this is where I want my key to be [775:02] located. So I will press enter. I will [775:05] don't I will not assign a passphrase. I [775:07] will press enter again. And if I do [775:10] ls-la, [775:11] you see that I have a public key here. [775:16] And this public key needs to be [775:18] transferred to our Jenkins agent. Now [775:22] again, I will just be catting this file [775:26] and copying the content and pasting it [775:29] there. But you can also use SSH copy ID [775:33] and so on. There are plethora of options [775:36] to copy a file from one Jenkins server [775:38] or from one Linux host to another Linux [775:41] host. Since this is not not a Linux [775:43] course, I won't go into telling you SCP [775:47] commands and SSH copy ID. For [775:50] simplicity, we'll just be copying the [775:52] contents and doing our job. So, we'll [775:56] copy it in a little while. But first [775:58] let's see what we want to do in our [776:01] Jenkins agent. Copy the public key to [776:05] agent VM. So here we are saying this [776:07] just copy the file and on the agent SSS [776:11] session already open switch to Jenkins [776:13] user. Understand that we just created a [776:17] Jenkins user in our agent as well. So [776:20] what we want to do is we want to [776:22] leverage this Jenkins user to perform [776:25] anything Jenkins. Now I want you to know [776:28] that when controller accesses our agent [776:31] VM, it will be using this Jenkins user. [776:34] So if I want to SSH into a destination, [776:38] I must provide a username that exists in [776:41] that destination. So the destination for [776:44] my Jenkins controller is this Jenkins [776:46] agent. And I am saying when you access [776:48] your destination you must use the user [776:51] Jenkins which is why we have created [776:53] this user Jenkins so that everything [776:55] Jenkins is done via the Jenkins user. So [776:59] I'll clear this for better real estate [777:02] and then I will just create a file here. [777:05] So what I'm doing is I'm first logging [777:08] in using the Jenkins user. So sudo su-en [777:13] jenkins and once I am here with the [777:16] Jenkins user in the Jenkins agent VM [777:20] then I will run this one. I want to [777:23] create a directory SSH and then I want [777:26] to create a file that is called [777:28] authorized key. So I'll press enter and [777:31] voila I am in the file. Now I will come [777:33] to my controller. I will cat the Jenkins [777:37] agent key dotpub. Here I'll press enter [777:41] and this is my public key. I will copy [777:44] this public key from here and then I [777:46] will paste this public key here. I will [777:50] press escape shift zz to save this file. [777:54] Now let's see what we want to do next. [777:57] Here is we are setting up the [777:59] permissions. Like I told you 60 is good [778:02] for you know these files these key files [778:05] so that only the user has the [778:08] permissions not others and groups and [778:11] then here I'm doing it same for the [778:14] directory SSH is a directory [778:17] authorized_keys [778:18] is a file I'll copy this and then I will [778:21] just paste it here and I'm on the agent [778:24] VM that is where I want to run this I'll [778:27] press enter and this has done its job. [778:31] Now let's go here. Now we have to test [778:35] this connection from the controller. [778:38] Test the connection. So what I'm doing [778:40] is pseudo su as Jenkins user. I want to [778:43] SSH into my Jenkins instance and I want [778:46] to run the command host name. So let's [778:49] see whether this works or not. So when [778:52] we run this command, we should see [778:55] Jenkins agent, right? I'll copy this [778:57] one. I will go to my terminal. I will go [779:01] to my controller. I will paste this [779:03] here. And here understand that our key [779:07] is located in Jenkins home in the [779:10] controller. And in the controller, the [779:12] Jenkins home is where lib Jenkins. So [779:16] I'll go here and here is the placeholder [779:19] for agents public IP. I'll go to my [779:23] console. I'll go to my browser and here [779:27] I want the public IP of my agent which [779:30] is this. I will come here. I will paste [779:33] that here. I will press enter and I want [779:37] you to pay clear attention here. This is [779:40] showing the fingerprint and saying do [779:42] you trust this as in it is asking you to [779:46] verify whether this fingerprint is for [779:48] the server or not. Now I am not going to [779:52] do that verification for now. That is [779:54] why I am just pressing yes. This says [779:58] trust this on first use and then [780:00] continue to trust this until this [780:03] identity changes. So if you let's say [780:06] are not using an elastic IP and if you [780:09] just stop this Jenkins agent and then [780:12] start it tomorrow or start it after a [780:14] couple of seconds then this public IP [780:17] will change and when this IP changes you [780:20] will again have to do this because the [780:24] identity that you trusted here has [780:27] changed. So understand these nuances of [780:30] SSH it is of paramount importance. I'll [780:34] press enter. I'm trusting this and it [780:36] has shown me the host name. So SSH works [780:41] and we are good. Now let's go to the [780:45] notes and see what we want to do next. [780:48] You have sshed from controller to your [780:51] agent. And that works. But how does the [780:55] controller know that where we ssed is [780:59] actually an agent? We don't know that [781:02] the controller doesn't know that. So we [781:04] will have to tell the controller that [781:07] the agent where the jobs will run is [781:11] this. And that is what we will do now. [781:13] So we will go to the Jenkins UI. [781:15] Finally, we will create a user cloud [781:18] with VJOS. Let's call it that. And [781:23] password. Let's give it a secure [781:25] password. I will give the same password [781:28] here. name is Varun Jooshi and the email [781:32] is cloudwithvosthegmail.com [781:36] and if you have any feedback if you want [781:38] to share anything personal then feel [781:41] free to communicate with me on this [781:44] email address cloudwithvos at the [781:47] rategmail.com [781:49] I will click on save and continue now [781:52] this is the Jenkins URL again when you [781:55] are dealing with production [781:56] implementations you would have a DNS [781:59] name for your Jenkins URL and that DNS [782:02] name will obviously point to an IP [782:05] address and that IP address even if it [782:07] changes although it wouldn't but even if [782:09] it changes for your specific use case or [782:12] for your specific implementation then [782:14] the DNS name will always remain the [782:16] same. Hence it is recommended what you [782:19] provide in Jenkins URL should be [782:21] persistent because if you are running [782:24] certain post actions post actions as in [782:26] once your pipeline runs you do this you [782:29] could be sharing these things these URLs [782:32] with your teammates or with your let's [782:35] say team leads who when click on this [782:38] URL will directly be taken to the logs [782:41] page or to the Jenkins URL which is why [782:44] it is important that you provide a [782:47] functional URL that remains persistent. [782:50] Now, because if this URL changes, you [782:52] will have to update this URL in the UI. [782:56] It is doable, but it is recommended you [782:59] choose a persistent name here. So, [783:02] always access your things using DNS [783:05] names, not IP addresses. I'll click on [783:08] save and finish. Start using Jenkins. [783:12] And firstly before I go blind I will be [783:15] changing this appearance to dark mode so [783:18] that we can work safely. Yes I have [783:21] changed it to dark mode. Now if I go [783:25] here I just click on settings. I can [783:28] click on set up the agent from here as [783:31] well or I can click on nodes and [783:34] configure my agent from there. So I will [783:37] click here and here I will click on new [783:41] node. What is the name of the new node? [783:44] Let's call it Jenkins agent. You can [783:47] give it any name. This is a permanent [783:49] agent which is fine because this is the [783:51] one that it will be there forever in at [783:55] least our scenario forever as in till we [783:58] do this pipeline. And I'll press create [784:00] here description you can add it number [784:03] of executor. What this means is how many [784:06] jobs can this you know agent run in [784:09] parallel. Now since I have vCPUs [784:13] two vCPUs I can choose two here so that [784:17] we can have parallelism here. But if you [784:20] do not want jobs to run in parallel you [784:23] can have one. If your agent has let's [784:24] say 20 CPUs then you can even have 20. [784:27] It doesn't have to be onetoone with [784:29] vcpus because this decision is [784:31] subjective and will depend on what sort [784:34] of deployment you have. So for now I [784:37] will just choose one exeutor here [784:39] because we for sure will only be running [784:41] one pipeline at once and number of [784:44] executors one should suffice and I also [784:48] want to go to the Jenkins our repository [784:52] so that I can see the steps here as [784:55] well. So name Jenkins agent type [784:57] permanent agent. I've written everything [784:59] here so that you can work in peace. Here [785:04] Jenkins remote root directory. Now here [785:07] you would specify the directory where [785:09] you your workspace will be created. So [785:12] workspace as in when you are running a [785:14] job when your agent runs the job it will [785:17] create a workspace and workspace is like [785:18] a scratch directory. So for that we have [785:21] created the home directory of Jenkins [785:23] user. We would want to create that. Now [785:25] again [785:27] extremely important permissions are of [785:30] paramount importance. We have created a [785:32] user Jenkins right in our agent VM and [785:37] we are providing the home directory of [785:39] the Jenkins user itself. So whenever our [785:42] controller accesses our Jenkins agent [785:45] using the username Jenkins which is [785:47] already there in our Jenkins agent, it [785:50] will by default have access to its home [785:54] directory. So it is important that you [785:56] take care of permissions. It's not that [785:57] you can provide any path here. Provide [786:00] the path for which the Jenkins user has [786:03] full access. So I'll write this here. [786:06] Then labels. See you can call it [786:08] anything but I like to be descriptive. [786:12] Hence I will provide this name Docker [786:14] Maven TV. Now this will have other tools [786:17] as well like cubectl or maybe [786:21] eksctl or definitely AWS CLI but for now [786:25] we are providing it this label. I'll [786:29] come here. I will give that label here. [786:31] Usage use this node as much as possible. [786:34] This is fine. Launch method is launch [786:36] agent by connecting it to the [786:38] controller. And here is the part where [786:42] I'll change it to launch agents via SSH. [786:46] And then it is asking me to provide the [786:50] details. Again, it is asking me to [786:53] provide the details of the agent. Here I [786:57] could provide the public IP address of [786:59] my agent. But that is not how you would [787:02] do things in production. In production, [787:04] you would supply the private DNS name [787:07] for your agent so that your controller [787:10] doesn't have to walk miles to reach your [787:14] controller on a public IP. And when you [787:16] are using a public IP, the traffic from [787:18] AWS first goes to the internet, then to [787:21] your instance. And we don't want that. [787:24] This is an internal traffic. We want [787:27] this to stay internal. Hence, we will [787:30] provide the private IP address of the [787:33] Jenkins agent here. I want you to [787:36] clearly know that you would be using [787:39] private DNS names in production. So, you [787:42] could be using this name. But for this [787:45] demo, I will be using this private IPv4 [787:48] IP of my agent for my controller to make [787:52] connection to it. So, I'll come here and [787:55] then I will provide this IP here. Now it [787:59] is asking for credentials. Now [788:02] credentials as in how do I connect to [788:05] this host and remember we are connecting [788:09] by supplying our private key. So we'll [788:12] have to provide the contents of our [788:15] private key to this UI so that when [788:19] controller is connecting to the agent it [788:21] can do so seamlessly. Right folks. So [788:25] here I will click on add Jenkins. This [788:29] is global credentials. This is fine. And [788:32] here [snorts] is the part where I will [788:35] choose SSH username with private key. It [788:39] just if you understand the concepts [788:42] these things will come automatically to [788:45] you. We have created public and private [788:48] keys. Hence it is obvious that we will [788:50] be choosing SSH username with private [788:53] key here. Here it is asking us for the [788:56] ID and the ID that we can give here is [789:01] Jenkins agent which works fine for us. [789:04] Username we know what is the username. [789:07] Username is Jenkins that we just [789:11] created. private key on passphrase. We [789:14] have not given any passphrase that is [789:16] fine but enter private key directly. And [789:20] here we can click on add and here we [789:24] have to add our private key. So let's go [789:27] back to our terminal and copy our [789:32] private key. So we are already in the [789:34] Jenkins controller. We are already [789:36] logged in as Jenkins and then we are [789:39] already in SSH. So if I press ls you see [789:43] our private key is this one. So what I [789:47] can do is I can write cat Jenkins key [789:50] press enter and this is our private key. [789:53] Now your private keys must be private. [789:56] Do not do screen sharing or do not share [789:58] them with anyone. These are meant to be [790:02] private. So I will copy this. I will [790:05] come to my Jenkins UI. I will paste this [790:08] here and then I will click on add here. [790:12] Now this is added and here I can choose [790:15] those credentials. This part is done [790:18] host key verification strategy. Now I [790:22] will be choosing the first one known [790:24] host file verification strategy. And [790:26] what is this strategy? You remember I [790:28] have been telling you whenever I'm [790:30] trying to access something for the very [790:32] first time, it is asking me to press [790:36] yes. If I accept the fingerprint, if I [790:40] trust that indeed the server that is I'm [790:44] communicating to me has this fingerprint [790:47] and we have been adhering to trust on [790:50] first use. Again I would tell you that [790:52] in our Kubernetes TLS sessions I have [790:57] demystified this in greater detail. If [791:01] you want to look at it go and check that [791:04] out. But for now what we have to do is [791:08] if I choose known host file verification [791:12] strategy which is what you would be [791:13] choosing in production scenarios and [791:16] then keep this agent online as much as [791:19] possible. I'll click on save. Okay, now [791:22] let's see what happens if I go to my [791:24] Jenkins agent and I go to my logs. It is [791:27] trying to launch the agent. First is you [791:29] can see these logs here. Let me zoom in [791:31] a little. It is trying to launch our [791:33] agent which is all right. But if I go to [791:35] my logs here, it says key exchange was [791:39] not finished. Connection is closed. [791:41] Trying in 15 seconds. There are nine [791:43] more retries. So it is trying to access [791:47] that but there is no one who can say yes [791:52] which is why it is again and again [791:54] failing. Understand we did press yes [791:59] once but that yes was only provided for [792:03] this public IP but the identity changes [792:08] if I am authenticating to my agent using [792:11] the private IP address of my agent. [792:15] Hence I must first press yes with the [792:20] private IP and that is what we are going [792:23] to do in this stage. Everything is [792:25] connected guys. Now I'll go here. I will [792:28] again copy this private IP address of my [792:31] agent. I will provide that IP here. And [792:34] I want to show this to you in parallel. [792:36] So if you go here, the Jenkins [792:39] controller is still trying to connect to [792:41] our agent. Okay, here. Now I'll press [792:45] enter. Now this is again saying the same [792:47] thing. Do you trust? Because this [792:49] fingerprint is based on the identity of [792:52] the destination. The identity changed [792:55] because our Jenkins controller is using [792:57] a private IP which is why it is asking [793:00] us to provide these credentials. I will [793:03] say yes to say yes to the fingerprint. [793:06] I'll say yes. Press enter. It is added. [793:09] If I go back here, it should now connect [793:13] because in the known host files in my [793:16] controller, I have an entry for Jenkins [793:21] agent and it says launch failed. Let's [793:26] wait for some more time and see how it [793:28] works. So, let's go here again and let's [793:31] click on launch agent again and let's [793:34] see if this works this time or not. Let [793:37] me also see if I've missed any steps. [793:41] Okay, this is fine. [793:44] This is also fine. Yeah. Yes. So, we [793:47] used this strategy and I mean we scanned [793:51] it with this. You could also use SSH key [793:55] scan to update the fingerprints. Let's [793:57] see if it worked now. Log says launch [794:00] failed. SSH connection closed. Let me go [794:05] to configure [794:07] home Jenkins. Use this node as much as [794:10] possible. Launch node is credentials is [794:15] Jenkins [794:17] and [794:19] known host file is file verification [794:22] strategy. This is fine. And yeah, this [794:27] is correct. [794:30] log [794:32] launch failed me launch again but if it [794:36] works from the CLI it should work from [794:39] there as well if I do ls- l if [794:44] [clears throat] it is working from here [794:47] then it should work from there as well [794:49] yeah ls-l is also working lf a is also [794:54] working so if I go again [794:59] let me see If I have [795:02] configured the username correctly, it's [795:06] username is here. [795:08] Credentials. [795:12] Where have I configured the username? [795:16] Well, let me just reverify these [795:18] credentials, guys. I will go to [795:21] credentials. [795:23] I'll go to system here. Global [795:26] credentials. Jenkins agent [795:30] update username Jenkins is fine [795:35] and then key is also fine a replace let [795:39] me let's just see if it is working now [795:56] and this This is strange. Let me try to [796:00] home Jenkins. This is fine. [796:04] Agent IP 3464. [796:08] This is also fine. [796:11] And here I was trying to update the [796:13] credentials. Let me let me [796:17] you know try to enter that again. I will [796:20] come to my terminal. I will ls cat [796:26] jenkins agent key. I'm not sure why it [796:29] is saying the format is not supported. [796:33] SSH authentication successful connected [796:36] online. I really have no clue what [796:40] happened. I just entered that private [796:43] key again and it started working and [796:45] previously the error was also quite off. [796:47] It was saying that algorithm was not [796:50] supported and we used the supported [796:54] algorithm [796:56] anyway. So now that our agent is [797:00] successfully connected, let's see what [797:03] are the next steps. So here install [797:06] Jenkins controller. We are done with [797:08] this. Done with this. We are done with [797:12] this. Yes, we are done with this as [797:16] well. [797:17] Yeah, we are done with this too. [797:20] Configure agent UI. Done with this. Now [797:23] we want to install sonar cube. So I'll [797:26] go to instances again. I will click on [797:28] the controller one action image and [797:32] templates [797:33] launch mode like this. And here I will [797:37] supply [797:39] sonar Q. And then I will come [797:44] down. This is fine. Ubuntu is fine. This [797:49] is fine. Existing security group. Yes. [797:52] But this time I will choose the sonar [797:55] cube security group. 20GB is fine for [797:57] this. And then I will click on launch [798:00] instance. [798:01] I'll go to instances [798:04] and then I will choose sonar cube and [798:08] then I will wait for connect to appear. [798:11] And till that time let's go to our [798:13] documentation. Same thing C7 I flex [798:17] large 20GI then the security groups we [798:20] have already seen and we will be [798:23] starting from these steps. So I'll go to [798:26] my instances, click on connect, copy [798:30] this. I'll come to my terminal and here [798:34] I will press CtrlT. So the first tab is [798:39] for controller, the second is for agent [798:41] and the third one is for sonar cube. So [798:43] in the flow in the sequence like we [798:47] configured, I'll cd into SSH. I will [798:50] zoom in a little here. I will paste that [798:53] command here. Press enter. I will say [798:56] yes for the fingerprint. And now you [798:58] know why this exists. I'll press yes. [799:01] And I'll clear my screen for some real [799:03] estate. And then I will go to my [799:05] browser. I will go to instances. I'll go [799:08] to my documentation. I will start with [799:12] setting up the host name. I'll paste it [799:15] here. Then I will change the time zone. [799:19] I will confirm the changes are [799:21] implemented. Asia, Kolkata. I'm in India [799:24] and then I will copy this and this is [799:27] the key part here and you press enter [799:29] first. We are installing JRE here not [799:33] JDK. Why is that? This is because Sonar [799:37] Cube requires Java runtime to work [799:41] properly and we are not building any [799:45] application here which is why we are [799:47] good with just JRE. I'm using version 21 [799:51] throughout of Java be it JDK or JRE and [799:55] then I will scroll down and here is a [799:57] typo you would see Java 17 because Java [800:00] 17 was the one I used before anyway. So [800:03] let's scroll down install post gray and [800:05] create DB for sonar. Now this is a [800:09] one-time activity. What we are saying is [800:12] the sonar cube instance that we are [800:14] running we want the database to be [800:17] hosted onto postgres SQL. Now ideally [800:20] what you would see in production is [800:21] there is one virtual machine that is for [800:24] sonar cube and then there is another [800:26] virtual machine that is a DB machine and [800:29] DB VM will hold your databases. Now we [800:34] are not going to this level of [800:36] bifurcation. What we are doing is we [800:39] have a virtual machine created but [800:41] within that sonar Q virtual machine we [800:43] are creating a database for sonar cube. [800:47] So in production you would be using a [800:50] different database and then you would [800:52] tell sonar cube how to reach that [800:54] database. In this specific scenario [800:56] which we are using, I will just say [800:59] local host. But if you are hosting your [801:01] database separately, then you provide [801:03] the identification information of the [801:06] database instance that is hosting the [801:08] sonar cube database. And here we are [801:11] installing postresql with common [801:13] extensions. We will enable postresql and [801:17] then we will start postrql. So I'll copy [801:20] all these commands. I will go to my [801:22] terminal. I will paste those commands in [801:25] the Sonar QV VM. Press enter and this [801:28] will start installing and running [801:30] Postgress SQL. Now I will go to Psql. [801:35] Psql is the CLI for Postgress SQL [801:39] itself. So I will copy this and then I [801:43] will paste it here. Now you will see a [801:46] Postgress CLI. Now I will be creating my [801:51] database along with a password. I will [801:54] be creating a user sonar with password [801:57] sonar. And why I'm doing this? So that [802:00] when my application tries to access the [802:03] this database, it can provide the [802:06] authentication information which I am [802:08] creating right now. So I'm creating [802:10] sonar user and that sonar user will have [802:14] sonar password. These are DB users. So [802:18] I'll copy this. Remember to copy the [802:22] semicolon as well. So I'll paste it [802:24] here. It will tell me create role. And [802:27] then I will create a database. And I'm [802:29] calling that database Sonar Cube. And [802:31] who will be the owner of that database? [802:34] Sonar user will be the owner for that [802:37] database. So I'll paste this here. Hold [802:39] on. I didn't copy it properly. So I'll [802:42] copy this and then I will paste it here. [802:45] And this will say create database. Now I [802:48] will go to this section which will grant [802:51] all privileges on database sonar cube to [802:54] sonar user. Again the password that I'm [802:58] also using for this is sonar. So [803:00] username is also sonar. Password is also [803:02] sonar. And quick verification. And we [803:06] can run this verification. So first [803:08] let's do one thing. We will quit from [803:10] the postress CLI. I'll enter this. And [803:14] this is telling me yes sonar is there. [803:17] And in the second step we'll verify that [803:19] the sonar cube db indeed exists. And yes [803:23] it exists because I can see it here. So [803:27] I will quit this one. And then this will [803:30] not result in anything because right now [803:33] we do not have anything in our database. [803:36] So instead of strong password here [803:39] placeholder let me write sonar. And this [803:42] will tell me that did not find any [803:44] relations which is all right at this [803:47] stage. Now what we will do is we'll be [803:50] tuning in certain sonar Q parameters. [803:52] Now this all things are there in the [803:55] documentation. I don't want to bore you [803:57] with what I'm doing but understand that [804:00] you will be tweaking certain kernel [804:02] level parameters for sonar cube to run [804:05] properly. And then as I always do create [804:10] a user for sonar so that anything that [804:15] sonar is doing sonar cube is doing that [804:18] sonar user is being used. So I will copy [804:22] this and then I will paste it here. [804:24] Again in the comments you can read [804:26] thoroughly what we are doing creating a [804:28] dedicated user for sonar with its home [804:31] directory at oparq. Now there are [804:35] various download options for sonar cube. [804:38] I am using a community build. You have [804:40] different editions in production based [804:43] on what your requirements are. You can [804:46] choose the version that you need. So for [804:49] now I will cd into temp. I will download [804:53] the sonar cube zip. I'll call it sonar [804:55] cube zip. And I'll be downloading [804:57] version 25.11.0. [805:01] And I think this is the patch release or [805:03] something. And then we'll be installing [805:06] unzip first because we are downloading a [805:08] zip package. And then we will move it to [805:12] this directory. And then we'll be [805:14] changing the ownership for this [805:17] directory. So I will copy this and then [805:20] I will go to my terminal. I'll paste all [805:23] these commands here and then I will [805:26] scroll further down and then we are [805:29] saying p sudoy-enu with user sonar we [805:33] want to edit the sonar properties and it [805:36] is in the sonar properties that we [805:39] provide the username and password that [805:42] we created a while back for our postgray [805:46] instance understand that here we'll be [805:49] telling our sonar cube application to [805:52] reach out to localhost port 5432. But if [805:56] you host your database separately, you [805:58] will add the identification, the DNS [806:01] name or the IP address which is not [806:03] recommended for your database instance. [806:07] I will copy this thing from here and [806:10] then understand that we are editing it [806:12] via the sonar user. So I will press here [806:16] and then I will press enter and here is [806:21] the file. I'll press I to go in the [806:23] insert mode and if you see user [806:26] credentials are to be provided here I [806:28] will uncomment this first and then in [806:32] the place of password I will write sonar [806:35] and username I will also write sonar. [806:38] And here is the critical piece. Now I [806:40] wanted this to be production-like which [806:42] is why I am not using the default [806:45] embedded H2 database that is you see [806:48] default here which could have been used [806:51] for development work or for you know lab [806:54] demonstration but I wanted this to be as [806:57] close to production as possible which is [806:59] why I am using postress SQL. You also [807:02] have option to use Oracle or Microsoft [807:05] SQL. And all these three options that [807:07] you have including postress SQL are all [807:11] relational databases. [807:13] Again, you could use a self-managed [807:17] database here or you could be using a [807:20] managed service by cloud provider. For [807:22] example, you could have also used Amazon [807:24] RDS for Postgress SQL here. Here first [807:29] since we are using Postgress SQL I will [807:32] uncomment this and then I will have to [807:35] provide the path here. So till local [807:38] host it is fine. Then I will add the [807:41] port number which is 5431 [807:45] 5431 [807:47] slash I'll go to my documentation and [807:49] then I just have sonar cube forward [807:52] slash sonar cube. Now I will hit escape [807:56] shift ZZ to save this file. Now that [807:59] this file is saved, I have to create a [808:03] unit file. Now what is a unit file? Unit [808:06] file defines a service. So we are [808:09] creating a unit file named sonar cube. [808:14] Now this is just a small configuration [808:17] file that tells OS how to start, stop [808:21] and manage a service. So if you see here [808:24] we are telling it how to start, how to [808:26] stop and how to manage other things [808:29] about the service. So we are saying if [808:31] it fails you restart it. Then it also [808:35] tells us what it depends on. So this is [808:39] a Linux topic. So I will leave it here. [808:41] Here we are creating a unit file. So I [808:45] will come here. I will paste it here. [808:48] Then I will copy this content. I will go [808:51] into insert mode. I'll paste it here. [808:53] Escape, shift, zz to save this. And then [808:57] I will scroll down. Then damon reload is [809:00] fine. It will read all the files in the [809:03] unit file location, which is this one. [809:07] You know this location. And there are [809:09] other locations as well, but that is [809:12] outside of this scope. Then we'll enable [809:15] sonar cube. We'll start cube. And we'll [809:18] finally check the status of sonar cube. [809:20] Now if you go to the GitHub [809:22] documentation, I have written all the [809:25] steps that are to be performed and what [809:27] is the significance of those steps. Of [809:30] course, I cannot cover everything in the [809:32] video because if I do that, that will [809:35] become a course in itself. So that is [809:39] why I like to keep my documentation [809:42] detailed so that most folks can get all [809:45] of their answers within the GitHub [809:47] repository itself. But if you need more [809:51] information then of course at the very [809:53] last of my documentation you would find [809:56] references which are typically official [809:58] references which you can read to gain [810:01] more understanding but for now our sonar [810:05] cube service is running and we all are [810:08] happy. So let's go to our browser and [810:11] then I will sign in again. Why did it [810:14] sign me out? Anyway I will sign in [810:17] again. I am in my instances. I'll go to [810:20] sonar cube. I will copy the public IP [810:22] address of sonar cube. I will write http [810:26] and then forward slash slash. I'll enter [810:29] the IP. Sonar cube listens on port 9,000 [810:32] by default. I'll press enter and it is [810:35] not working. Congratulations. [810:38] Is the IP correct? Yes, the IP is [810:41] correct. Uh why it is not loading? Have [810:44] I associated the right security group [810:48] with it? [810:49] Yes, I have. Port 9,000 is allowed. Is [810:53] it adding HTTPS by default there? What's [810:58] the problem? Let me check the status of [811:01] the Sonar Cube service. It's running, [811:03] man. Oh, inactive. Let's try to restart [811:07] that again. Sonar cube is active. [811:12] I want to check the status again. Code [811:15] this is fine. Sonar cube current this [811:18] sonar.sh this is fine. Let me [811:23] go to my GitHub documentation. Let me [811:26] see if this is loading anything here [811:30] where there could be some error from my [811:32] side only. [811:35] Hold on. [811:37] Let me try to [811:40] run this. [811:42] And [811:45] I want to run this one. [811:48] I ran this previously. Does it have it [811:51] in its history? [811:53] It may, but let's do it like this. And [811:57] here I will write sonar. [812:01] Oh, it doesn't see anything here. [812:06] H what I've done is I've also added [812:09] troubleshooting steps. So let's go to [812:11] the browser [812:13] and then here if I scroll down [812:19] further down further down [812:22] further down. Yeah. So let's check the [812:27] logs [812:35] and because all these things are [812:37] stopping in sequence. If I run free [812:42] age used free is there. Let me run that [812:47] kernel parameter tweaks again. Um I [812:50] think those did not run properly. [812:54] Do piston ctl restart [812:59] n cube. [813:24] If I [813:28] stopped again, [813:31] let me check other logs. I'll go here. [813:38] So now let me check the web logs. [813:49] connection to local host. Check the host [813:52] name and [813:54] TCIP connections. Hold on. I think there [813:58] is some problem with the sonar cube [814:03] configuration sonar.properties. [814:06] So let me quickly verify that this file. [814:11] Let me just see if we have entered [814:13] everything correctly. It's fine. Sonar [814:16] sonar and this is postress SQL. And if I [814:21] come here, we had to write 5432, [814:26] not 5431. [814:29] This was the problem, guys. I'll save [814:32] this and [814:35] more troubleshooting. Right. [814:38] sudo system [814:42] ctl [814:44] restart sonar cube [814:48] and this time it should work and let me [814:53] write status. So that's why reading logs [814:56] is important. So it gave us exactly what [815:00] we needed. It said this and then we [815:04] reached this conclusion. If I check [815:06] status [815:08] s a t us I have to spell it right. [815:11] Status is now running. If I go to [815:13] browser and then I hit refresh. [815:17] Perfect. It is working. Now the username [815:20] and password the default username and [815:23] password for sonar cube is admin. Admin. [815:28] So let us allow it some time. [815:32] Home. Yeah. admin admin so we'll log in [815:37] here [815:40] and then never old password is admin new [815:45] password I'll set a secure password here [815:50] press enter and this I will do never [815:53] later so now our sonar cube instance is [815:57] ready now we have all the three [816:00] components that we need ready so Let's [816:04] start with the demo. If I go to my [816:08] diagram, I will take you to this screen. [816:11] So first step is check out from private [816:15] git repository. Now for that we first [816:18] must have a repository. So I will go to [816:22] this screen. I will go to my browser. I [816:26] will go to this screen. Then let me go [816:31] to the top here. And then from here [816:33] cloud with VJosh. And then I can close [816:38] this one. This is not needed. I can [816:40] close the YouTube link as well. And here [816:44] I will go to repositories. Now let's [816:47] click on new. This repository I will [816:50] call cloud with vjos private repo. This [816:54] will be private. Here I will scroll down [816:56] and click on create repository. And now [816:59] we have a repository created which is [817:02] private. But I want you to clearly know [817:04] that when you create a private [817:06] repository, you need credentials to [817:08] access it. So now we will be creating [817:10] credentials that we will use to access [817:13] this repository. So I'll go to settings. [817:16] I'll scroll down go to developer [817:18] settings. And here I will go to personal [817:20] access tokens. And then I will go to [817:22] tokens classic. Here I will choose [817:25] generate new token. generate new token [817:29] classic and I will call this token [817:32] Jenkins GitHub so that I know what this [817:35] token is used for. I will choose repo [817:38] here and then I will scroll down and I [817:42] will click on generate token. Again do [817:46] not show these tokens to your folks or [817:48] colleagues. It is supposed to be [817:50] private. I will delete these credentials [817:53] before uploading this video. So I'll [817:56] copy this and then I will go to visual [817:59] studio code and here I will just create [818:03] a new file in the root itself and not [818:07] here I would rather create this file in [818:10] project files new files and I will call [818:13] this creds and in this creds file I will [818:17] just paste it you don't have to worry [818:20] about rest of the details that are here [818:22] we'll dissect them but for now just to [818:25] this. Copy this here. Let me explain the [818:29] code to you. So I will go to Java Maven. [818:33] This is where I'm holding my code. So I [818:36] have the src directory and src has main [818:40] and test. As we know main will have my [818:44] actual application code which is in the [818:47] file named app. Java. And I have added [818:51] some vulnerabilities so that I can demo [818:54] this to you at a very high level. This [818:58] code will just be displaying cloud with [819:00] varos simple dev seccops demo. And then [819:04] there are test frameworks that I have [819:07] defined. I have defined the JUnit test [819:10] framework and I have also defined a [819:13] couple of test cases so that we can also [819:17] see how code coverage behaves and then I [819:21] have my pom.xml. [819:24] I have a simple pom.xml. I won't be [819:27] explaining this pom.xml to you because [819:29] we have already covered this deeply in [819:32] our Maven lecture. So this is our code. [819:36] Now I want this to be production-l like [819:39] which is why we will be committing the [819:42] source code onto our GitHub repository [819:46] side by side. But one point I want you [819:50] to understand that I will be using this [819:53] token for my Jenkins to interact with [819:56] this and I am acting as a developer and [819:59] I am also using this GitHub token. But [820:03] in actual production you will have two [820:06] different tokens. One token will be used [820:09] by Jenkins and another token would be [820:12] used by developers. So with that [820:16] understanding let's start our work. So I [820:20] will go to the terminal first because I [820:22] want the code to be in my GitHub [820:25] repository. So we will start by creating [820:29] a new tab. And in this new tab what I [820:32] will do is I will cd into courses and [820:36] here jenkins and then this is common and [820:40] java dot maven. So java.m maven is the [820:44] directory that holds my src and pom.xml. [820:49] So if I press ls you see pom.xml and [820:53] src. So if I run tree you see I have [820:56] app.java Java in my main directory and I [821:00] have app test on Java in my test [821:03] directory which is the standard [821:05] structure that Maven would create for [821:07] you and I have the pom.xml in the root [821:10] itself. So first things first I will run [821:15] get init. So get init will initialize [821:19] this as a g repository and then I will [821:22] write get add dot so that all the files [821:27] that are currently untragged will be [821:30] tracked by git and then I will write get [821:34] commit- m and then I will call this [821:38] commit initial commit with code and [821:44] pom.xml. [821:46] So always have meaningful messages. I [821:49] will press enter. And then if I write [821:52] get branch-v [821:55] then you would see my current branch is [821:59] called master. But if I talk about [822:01] GitHub they would have named my branch [822:04] main. I like to keep things consistent [822:07] which is why I will be renaming this [822:10] from master to main. So I will write get [822:13] branch- [822:15] m and I will write just main here. So if [822:18] I run that command again, you would see [822:21] my branch is now renamed to main. If I [822:26] write get status here, you see nothing [822:29] to commit. We have a working tree clean. [822:32] So because we have already committed [822:35] this initially. Now what I want is I [822:39] want to add a private repository. So for [822:42] that what I will do is I will write get [822:45] remote add private repo and then the URL [822:50] of that repo. So I will have to spell [822:52] this correctly. I'm calling this private [822:54] repo. You would have seen a lot of folks [822:57] use origin here and which is fine but I [823:00] like to keep the name meaningful which [823:02] is why I have used private repo. Now I [823:06] will go to my browser. I will go here. [823:11] Here where I've created my private repo. [823:13] I will copy the address of my private [823:16] repo from here. And then I will just [823:19] simply paste my repo URL here. I will [823:23] press enter. Now once I have done this, [823:26] I have not provided any credentials yet. [823:29] Okay. If I try to push it, it will ask [823:32] me for the username. So for that what I [823:36] now want to do is I want to provide some [823:39] credentials. So that is what we are [823:41] going to do in this step. So what I will [823:44] do is I will copy this and then I will [823:47] come to my visual studio code. I will [823:50] paste the command here and then I will [823:53] copy this token and the placeholder I [823:57] will paste this token here and then I [824:01] will run this command in my terminal. [824:06] Now once I do this now I can perform my [824:10] pushes. I will write get push private [824:14] repo main. I'll press enter. And then if [824:18] I go to my browser now and if I hit on [824:23] code again or you can hit refresh you [824:26] see we have src and pom.xml and this is [824:30] what you will see in your repository. [824:33] Now as we progress we will update this. [824:36] Now that we have done this let's create [824:40] our first job. I'll go to I'm already in [824:45] my browser and here I will go to [824:48] dashboard new item and I want to call [824:52] this job cloud with varosh dev sec ops [824:57] demo and this will be a pipeline job. [824:59] I'll click okay. Now at this stage [825:03] itself I want you to know one thing very [825:06] clearly that I am using the main branch [825:10] but in production main branches are [825:13] often protected. See I cannot cover [825:17] everything that is used in production in [825:19] a single project. So what I have done is [825:22] in this project I have focused on [825:24] security and both DevOps and I've tried [825:27] to maintain that balance but I already [825:31] have a lecture in my channel that tells [825:34] you how to work with multi-branch [825:37] pipeline jobs. So you can always watch [825:41] that lecture if you want to see how the [825:44] branching strategy in production works. [825:47] And again like I told you before as well [825:51] if you want full understanding then you [825:54] should also have watched the SDLC life [825:57] cycle the branching strategies and if [826:00] you go to my Jenkins playlist you will [826:02] find first two lectures they are not [826:05] relevant to Jenkins only they are [826:08] relevant to CI/CD in general. So there [826:11] we have discussed how branching strategy [826:13] works in production, where you should [826:15] keep your code, what are feature [826:17] branches, what are release branches and [826:19] so on. So look at that. But for now we [826:23] are creating a pipeline job and we will [826:25] be working with the main branch itself. [826:28] Now I will scroll down and here I will [826:31] choose pipeline script from SCM. Why [826:36] pipeline script from SCM? because what [826:39] we will be doing in this demo is we will [826:41] be doing this production style. So our [826:43] Jenkins file will exist in our [826:47] repository itself and then in SCM I will [826:51] choose git and now we have all the [826:54] things floated. I'll just move this to [826:57] this space and this I will move here. [827:01] And now it is asking for repository URL. [827:03] Repository URL is here. I will copy [827:06] this. I will paste this here. And [827:10] credentials, I will choose Jenkins. But [827:14] Jenkins is not the credentials for this. [827:17] Jenkins [827:18] were the credentials for our Jenkins [827:22] agent, not for GitHub. Now for GitHub, [827:26] we have used the credentials in our CLI. [827:30] But now Jenkins will also need access to [827:34] our repository if it has to check out. [827:37] So if you look at the first stage of our [827:41] pipeline, it is to check out from [827:43] private repository. If Jenkins must [827:45] perform that check out, Jenkins must be [827:48] provided by the credentials. You have to [827:52] engrave this in your mind. Beat whatever [827:55] CI/CD tooling you are using, your CI/CD [827:58] tooling is just acting as an [828:01] orchestrator, you will still have to [828:03] integrate different tools in your [828:06] pipeline so that whatever you are trying [828:09] to do, you can do those things. So I [828:12] will go to my browser again and this [828:15] time I will click on add Jenkins kind of [828:19] will be just secret text. Now when I [828:22] talk about GitHub kind will be username [828:26] with password. So when I talk about [828:28] GitHub, GitHub expects the username [828:31] along with the token. Which is why we [828:33] are using username with password. Now [828:36] why I'm mentioning this distinction is [828:38] because if I talk about sonar cube there [828:41] we will just use secret text because [828:43] sonar cube is not expecting the [828:45] username. it is just expecting the token [828:48] and based on that token it will figure [828:50] out with which user that token is [828:53] associated but with GitHub you provide [828:56] both username with password so I'll [828:58] click here I will keep the scope global [829:02] so that my pipeline can also access [829:05] these credentials then I'll scroll down [829:08] username would be cloud with varosh that [829:10] is my GitHub username you will have to [829:12] choose yours and in the password section [829:16] section I will have to paste the GitHub [829:19] token. So I will copy this from here and [829:22] I will paste this token here. ID would [829:26] be GitHub pat that's a cool name GitHub [829:29] personal access token. I will click on [829:32] add. Now as soon as I choose those [829:35] credentials this red mark should go. If [829:38] the red mark still says that implies [829:42] your token has anomalies. So fix it and [829:46] provide the right token. And here is the [829:48] branch specifier. My branch is main that [829:52] is why I will write main here. Again if [829:56] you want to know about multibranch [829:58] pipelines then you should refer to the [830:01] lecture that is already there in my [830:04] Jenkins CI/CD course. I will click on [830:07] save here and then let's click on build [830:12] now. [830:16] Firstly, I want you to notice that it is [830:19] showing the current time. And why it is [830:22] doing that? Because we changed the time [830:24] for both the operating system and also [830:27] for JVM. So if I click here, I can go to [830:32] the console output and it will tell me [830:34] why it failed. Now it is saying unable [830:37] to find Jenkins file from kit and this [830:41] failure was obvious. Now whenever we are [830:45] running a pipeline job we supplied that [830:48] the Jenkins file you will find in the [830:51] root and that file was not there. So if [830:53] I go to my repository there is no [830:55] Jenkins file here which is why this job [830:58] failed. So if I go to configure section [831:01] and I scroll down, it was looking for a [831:05] Jenkins file. Now if you have not named [831:07] it Jenkins file, you can provide that [831:09] custom name and custom path here and it [831:12] will look for that file. But here we are [831:14] keeping the things at default and that [831:16] is what it is recommended you keep the [831:19] name Jenkins file itself because your [831:22] repository is private anyway. [831:25] So you have Jenkins file but we have not [831:27] supplied the Jenkins file. So let's [831:30] start the coolest part that is writing [831:34] Jenkins file. So I will go to my visual [831:37] studio code and always use an IDE which [831:41] understands Groovy JSON to write these [831:45] files so that you get color coding and I [831:48] will show you why this is so important [831:50] in a little while. So I'll right click [831:52] on Java Maven. I'll click on new file [831:54] here and I will call this new file [831:58] Jenkins file and I'll press enter and [832:02] let's start writing our Jenkins file. [832:05] Now the first things first I will write [832:10] pipeline. Now pipeline is from where [832:13] everything starts. So whenever you are [832:16] dealing with Jenkins file, you start [832:17] with pipeline and I start a curly brace. [832:21] And as soon as I add one curly brace, my [832:23] ID is intelligent enough to know that he [832:26] would also need a closing curly brace, [832:29] which is why it added that. I will press [832:33] enter. Now let's write the first thing [832:37] that we want to do. We want to tell our [832:40] pipeline that the agent that you should [832:43] be using to run this pipeline or to run [832:46] this job, what is the name of that [832:50] agent? I will not be supplying the name [832:53] of that agent. I will be supplying the [832:56] label associated with this agent. [832:59] Understand that for my entire pipeline [833:02] I'm just defining [833:05] a single label which implies the agent [833:08] that has this label will be running [833:12] everything in my pipeline all the stages [833:14] of my pipeline. You can have different [833:18] agents with different labels and then [833:22] these labels or these agents can be [833:24] called at different stages of your [833:26] pipeline. For example, you may have an [833:30] agent that only has cubectl. In that [833:34] scenario, you would be using that agent [833:37] to run cubectl specific commands. Maybe [833:40] there is an agent that has Maven, sonar, [833:43] cube and so on installed. So to run [833:45] those kind of jobs, you assign that [833:50] label, assign that agent. But in our [833:53] example, we will be using a single agent [833:56] which will be running our entire [833:58] pipeline which is why I'm defining this [834:01] at the pipeline level itself. You also [834:04] have the functionality to define the [834:06] agent section or the label section [834:09] within a stage as well. So with that [834:13] understanding I will write docker maven [834:17] and privy. understand that this name [834:21] must match the name that you defined [834:24] here. So if I open this one, I go to [834:27] this agent, then you would see this one [834:30] I called docker maven tree which is why [834:34] docker maven trivia. Understanding this [834:38] piece is critical my dear friends. And [834:41] then let's do one thing. I will press [834:43] enter. And now I want you to see the [834:48] important piece. I will write stages [834:51] here. stages is a plural which is why I [834:55] can have multiple stages. So I open a [834:58] curly brace. Press enter. Now inside [835:02] pipeline I have agent and stages. Agent [835:06] and stages are at the same level. But [835:10] now we want to add multiple stages. [835:14] Hence we must define different stages [835:17] within the stages block. For example, I [835:21] will start with my first stage. And the [835:25] first stage, what I'm calling my first [835:27] stage is privy file system scan. [835:32] Understand that within stages, we have [835:35] the stage block. But within the stage [835:38] block, we will have to do something, [835:40] right? We'll have to perform certain [835:43] steps within the stage block. And what [835:46] those steps are? we define within the [835:50] stage block. So in this stage I am [835:54] performing certain steps and in this [835:58] stage I will only be performing one step [836:02] which is sh is calling the shell. I will [836:08] write privy fs exit code one severity [836:14] high critical and a dot. What does this [836:17] mean varun? Firstly, we are telling our [836:22] pipeline job that the first thing that [836:25] you need to do is you have to assign [836:28] this job to an agent that has label [836:33] docker maven trivia. And once you have [836:36] found that agent, you run this stage [836:40] onto that agent. And that agent will be [836:45] running this in its shell. Understand? [836:48] We are defining in its shell. And what [836:51] is the command that it should be running [836:53] in its shell? It is trivy file system [836:57] exit code one severity high critical. [836:59] Now what we are saying with this line is [837:02] if there are any vulnerabilities that [837:05] have high and critical associated with [837:08] it then I want you to fail this [837:10] pipeline. fail this pipeline because any [837:13] exit code but zero implies failure. So [837:18] we are saying you fail the entire [837:21] pipeline if there are any high or [837:24] critical vulnerabilities. Now again [837:29] important thing is to understand how [837:32] this syntax is working which is why [837:35] Visual Studio Code is so powerful or or [837:38] any IDE that supports this color coding. [837:42] See steps has green curly brace right. [837:45] If I click on green curly brace it tells [837:48] me which one is the closing curly brace. [837:50] So that is the color coding that you [837:52] get. And you also see it is showing a [837:56] box outside this. And this box signifies [837:59] that this opening curly brace is [838:03] associated with this closing curly [838:05] brace. So if I click on this pink curly [838:09] brace, it shows me it highlights the [838:13] opening curly brace. Similarly, if I [838:15] check this closing curly brace, it tells [838:20] me its association of the opening curly [838:24] brace that it is associated with. So you [838:27] get two things. First is color coding [838:29] and then you get this identification [838:30] because as you progress you would see [838:33] that there could be multiple white curly [838:35] braces. Then this box helps you identify [838:40] which one are the pair of opening and [838:44] closing. So this is important to us. [838:46] This one you see pink is here as well [838:48] and pink is here as well. So you know [838:50] this opening curly brace is associated [838:54] with this closing curly brace. So keep [838:57] that thing in mind. Now let's get rid of [839:02] the syntax. Can we run this command? [839:05] Let's try actual production workflow [839:09] here. Jenkins file is in the root of our [839:13] repository. So if I go to my terminal [839:16] and I run ls, it will show me that right [839:19] now you are not tracking this file. Uh [839:22] why it is oh if I write get status I'm [839:25] sorry if I write get status it will tell [839:27] you that Jenkins file is not being [839:30] tracked right now. So let's do one [839:32] thing. get add dot and then let's add [839:38] get commit- m added tree stage. Let's [839:43] just write this for now. I'll press [839:45] enter and then I will write get push [839:49] private repo main. I'll press enter. Oh, [839:53] sorry I have to spell it right. Private [839:56] repo main. And this will push it. Now if [839:59] I go to my browser and I go to my [840:03] private repository, I hit refresh. I see [840:07] Jenkins file now exists and it was [840:09] pushed you know now I will now go here. [840:13] I'll go here and I will now click on [840:16] build now. Now now let's see what [840:18] happens. I will go to stages. And this [840:20] is a very good UI to see what is [840:23] happening. So if I go to this, it tells [840:25] me that my job has failed again. My [840:28] pipeline has failed. If I click here, [840:31] SCM checkout was automatically added [840:34] because now it found the Jenkins file [840:36] there. Previously, Jenkins file was not [840:38] there. Hence, we didn't see this. Since [840:40] the Jenkins file was there, it said your [840:43] checkout works seamlessly because you [840:45] provided the correct credentials. But [840:47] the first stage in your pipeline so to [840:50] speak, which is Trivy FS scan, did not [840:53] succeed. And why didn't it succeed? It [840:57] says preview not found. And this brings [841:00] me to that important point that I [841:02] mentioned at the beginning. Your Jenkins [841:05] is just an orchestrator. If your agent [841:08] who is performing this stage of the [841:10] build does not have Trivy installed, [841:13] then your pipeline would fail. So now [841:17] let's install Trivy in our Jenkins agent [841:20] because agent is the one that is doing [841:22] the leg work that is running our jobs. [841:26] So I will go here and then in this stage [841:31] privy fs scan I will go here and then I [841:37] will scroll down and this is how you [841:41] install Trivy. Now you should be logged [841:43] into your agent VM firstly. So let's [841:47] install trib and I am installing this in [841:51] the default path itself in your I'll [841:55] show this to you. I'll just copy this [841:56] right now all the commands and this is [841:59] the version of tree that we are [842:00] installing. If I go to my CLI I will [842:03] paste this in my agent Jenkins agent VM. [842:08] First I will exit this because I want to [842:11] run pseudo specific commands and my [842:14] Ubuntu user if I write id Ubuntu is part [842:18] of the pseudo group which is why it can [842:22] run sudo specific command. So I'll paste [842:25] all those commands and I'll press enter. [842:27] Now this will install trivia. Now you [842:29] would be thinking wun but this will gen [842:33] user will have access to it. it will [842:35] have access to it and why it will have [842:37] access to it because the location where [842:41] TV is installed all the users have [842:43] access to that location. So if I run [842:46] which trivy you would see that it is [842:48] installed in usr bin. So if I run ls-l [842:53] on this location then you would see that [842:57] everyone has permissions to execute this [842:59] binary. You see uh owner has full and [843:03] then execute permissions are with groups [843:06] and others as well. So our Jenkins user [843:10] can also run this. Now that we have [843:13] installed Trivy, let's go to our browser [843:16] and run this again. So I will rerun [843:19] this. And this time our Trivy file [843:23] system scan should succeed. And let's [843:27] see what it has for us. It says that you [843:31] know you see all the logs here for this [843:33] stage and this says it did not find any [843:36] vulnerabilities. But you would be [843:38] thinking Vun how is that possible? You [843:42] deliberately added vulnerabilities in [843:44] your code right? If I go here if I show [843:48] src to you and then main and then app [843:52] java you have problems in your code but [843:55] did not flag it. Why didn't it flag [843:58] these? I want you to clearly know that [844:01] privy we are using for SCA. We are in [844:05] this stage and we are using privy for [844:08] SCA. It will only look for [844:13] vulnerabilities in your dependencies [844:16] not the actual code. For your actual [844:20] code you have sashed. Understanding this [844:23] distinction is important. That is why TV [844:27] did not detect anything. Now I don't [844:30] want to keep you hanging. What I want to [844:32] do is I want to deliberately add a [844:35] problem so that TV can detect it. So I [844:38] will right click here. I will click on [844:40] new file and I'll call this file [844:42] secrets.ext. [844:44] Okay. And then what I will do is I will [844:47] add something that is in format of AWS [844:51] secret access key. It is not a genuine [844:54] key but it works for our demo. And in [844:58] pom.xml [844:59] I also want to add a dependency. I want [845:03] you to clearly know that when I talk [845:05] about trivi it can perform SCA plus more [845:09] to detect CVE linked vulnerabilities [845:14] plus misconfigurations and known secret [845:17] patterns that SAS cannot find. So for [845:20] example, if I talk about this, this is a [845:22] known pattern for secrets. But if I go [845:26] to my pom.xml, if I go to my Jenkins [845:29] code here, if I go here, then I have [845:34] hardcoded something. But this is not a [845:37] known pattern, which is why TV won't [845:40] detect it. And critical point TV is [845:44] mainly for the vulnerabilities that are [845:47] there in your third-party libraries. And [845:50] in pom.xml, [845:52] where do we define third-party libraries [845:55] on which our application depends? We [845:57] define them in the dependencies section, [846:01] which is what we are going to do. So [846:04] I'll copy I'll go to my browser and I [846:08] have noted all these points here. So [846:10] I'll go here. I'll scroll down and here [846:15] is the vulnerability that I want to add. [846:19] Hold on. Yeah, simulator problem. So I [846:23] will copy this and then let's go to our [846:26] pom.xml [846:28] and here. So this is dependencies is a [846:31] plural. This is the first dependency. [846:34] This is the opening tag. This is the [846:37] closing tag because it has this forward [846:39] slash. I'll come here. I will paste it [846:42] here. And then let me just, you know, [846:45] align it properly so that it looks neat. [846:50] And then this as well. I'll select this [846:54] block. I'll press command and closing [846:57] curly brace. This is fine. This is under [847:01] it. Yeah, this indentation looks good to [847:04] me. I'll save this. And now let's commit [847:07] this. Understand? We have to commit. A [847:10] lot of folks run it just like that. We [847:13] have to commit the changes that we have [847:15] done. So get add and then commit addit [847:19] review stage with vulnerabilities. [847:23] Enter. And then push private repo main. [847:27] And now I will come to my Jenkins screen [847:30] again. This time I want this to check [847:33] out the code again. So this rerun should [847:36] not be used. I will come here and then I [847:38] will click on build now. And then let me [847:42] go to stages. Let me go here. And this [847:45] time the TV file system scan failed. And [847:50] let's see why it failed. It says your [847:53] pom.xml has three vulnerabilities and [847:56] your secrets.ext has one secret that is [847:59] there. You fix that dear DevOps engineer [848:04] and then I will let you pass because I [848:07] have been configured to fail the [848:10] pipeline if there are any high or [848:15] critical level vulnerabilities and I see [848:19] your code has three which is why I am [848:23] failing. And then it also defines what [848:26] has the problem and what you need to [848:29] fix. And to fix that we simply will [848:33] delete our secrets.ext. [848:35] I'll delete this. And what I can do is I [848:38] can come to my code. I can come to my [848:41] pom.xml. [848:42] And this particular dependency I can [848:45] just comment using command and forward [848:48] slash in Mac. I'll save this. I will [848:51] again commit this commit with [848:55] vulnerabilities. I will just write fixed [848:58] here and then I will push this again. [849:02] And if I run this again, I'll come here. [849:06] I will run build now. If I go to stages [849:10] now, you would see this has now [849:13] succeeded. So if you come here, this [849:16] says everything is fine. Now that we [849:21] have done this, let's see what we want [849:24] to do next. Next is your build and test. [849:29] So this is the part where we will be [849:33] building our application and we will be [849:36] testing our application. So what I will [849:39] do is I will just copy paste the command [849:42] lit because it is a lengthy command and [849:46] then I will be explaining you step by [849:48] step what we are going to do. So let me [849:52] come to Visual Studio Code and I will [849:56] open my Jenkins file here. Stages we [850:00] have added the first stage. We identify [850:03] where is this stage ending. I will click [850:06] on this. It tells me that this is the [850:08] closing curly brace for this. I'll come [850:10] here, press enter so that I can add [850:14] another stage here and then I will just [850:17] copy that stage for you. So that let I [850:20] have to explain a few more things. So [850:22] let's do one thing. I will ex I will [850:25] write that stage first. stage and I am [850:28] calling this stage let's say build and [850:33] sonar because we are doing sonar [850:36] scanning here and then understand that [850:41] we want to access our sonar cube [850:46] installation from our Jenkins. So to do [850:49] that we must first provide credentials [850:53] to do the same. So now let's add those [850:57] credentials so that we can do the same. [851:01] So let's go to our [851:05] chrome browser and then here I will go [851:08] to sonar cube and here I will create a [851:12] local project and this local project I [851:16] will call cloud with varosh dev secops [851:20] demo and the project key will also be [851:23] the same. The branch will be main. I'll [851:24] click on next and follows the instance [851:27] default which is current default [851:29] previous version. So what we are saying [851:31] is new code is only the one that has [851:36] changed from the previous version. [851:38] Understand as DevOps engineers our [851:41] primary moto is to reduce the time that [851:44] the pipeline takes to run while [851:47] performing everything that has to be [851:49] performed. So if you have already [851:51] scanned 80% of the code, I don't want [851:55] you to rescan that 80% of the code. I [851:58] only want you to scan the new code. That [852:01] is what we are defining here. So I'll [852:03] create project. And this has created a [852:06] project for us. Now the project is [852:09] created. But what we also want to do is [852:13] we want to create a quality gate. Now by [852:15] default the project that we have created [852:18] will get associated with the default [852:21] quality gate that we have. But I also [852:24] want to show you how to create a quality [852:26] gate. So we will be creating a quality [852:29] gate as well. So if I click on create, [852:31] you would see that this UI appears and [852:35] it is asking us to name our quality [852:38] gate. So I will name it cloud with [852:40] varosh dev sec ops demo and I will add a [852:45] QP here so that I know this is a quality [852:48] gate QG rather so that I know I click on [852:52] create and you see it has copied all the [852:55] rules all the conditions that were there [852:58] in the sonar way built-in quality gate. [853:01] Now if you want deeper information I [853:03] would highly encourage you to watch the [853:05] sonar cube lecture but for now we will [853:10] add one more condition here and that [853:12] condition will be for the overall code [853:15] because we want to fail the pipeline if [853:18] the overall code does not adhere to what [853:21] we have defined. So let's say I want to [853:25] fail the pipeline if the overall code [853:28] coverage is less than 80%, I will add [853:32] this condition. Now this quality gate is [853:35] created but right now it does not have [853:37] any project associated with it. I told [853:40] you that a project when created gets [853:43] associated with the default quality gate [853:46] profile. But right now we want to [853:49] associate our project with the quality [853:52] gate that we created. So with without I [853:56] will click on this checkbox and then if [853:58] I come here again I will see my project [854:02] is associated with this quality gate. So [854:04] this work is done. We have created a [854:06] project. We have associated a quality [854:08] gate with that project. Now we want to [854:12] create a token. Now one way is you [854:15] create the token for administrator user [854:19] itself but we don't want to do that we [854:22] will create a new user and I will call [854:24] it Jenkins sonar and password I will [854:28] give it a secure password and we are not [854:31] going to use this password anyway but we [854:33] have to keep a secure password and name [854:36] let's call this Jenkins sonar itself and [854:40] then email is ABC at the xyz.com and [854:44] then we can create this. Now we have to [854:47] create a token for this user and that is [854:49] what is important. But before I do that [854:53] I will give this access to sonar [854:56] administrators. I'll press enter and [854:58] then here I will create a new token. I [855:02] will call this Jenkins sonar and then [855:05] expires in 30 days let's say and then [855:08] I'll click on generate. You only see [855:10] that once like GitHub token. So I'll [855:12] copy this. I'll go to Visual Studio Code [855:14] Creds and I will paste it here. Now I [855:17] will click on close. We have the token. [855:21] Now we have to add this credential into [855:26] our pipeline. How do I do that? See [855:30] there is a way via which you can [855:34] generate what you have to add in this [855:37] step. So what I will now do is I will go [855:40] to my Jenkins dashboard and here let's [855:45] go to Jenkins and then I'll go to my job [855:48] and I'll hit configure here and then if [855:51] I scroll down I see pipeline syntax [855:54] here. Okay, this can be used to generate [855:56] certain scripts. I'll click on pipeline [855:59] syntax and I will use a sample step [856:02] called with credentials. I will choose [856:05] with credentials bind credentials to [856:07] variables. So this creates a binding. [856:09] Now why this is important? Because any [856:12] credentials that you use like this will [856:14] not be exposed in the console or in the [856:18] you know if you check the logs of your [856:21] Jenkins. This is good because you want [856:23] your credentials to be redacted which is [856:25] why we always use binding. So I'll click [856:29] add here. And this time with GitHub we [856:32] used username and password. But like I [856:35] told you, Sonar Cube does not expect you [856:37] to specify a username which is why I [856:40] will choose secret text here. Now what [856:44] is the variable that you want to use [856:47] which you will use to call this secret? [856:49] Understand? I have stored the tokens but [856:52] I need a way via which I can call this [856:55] token and that way is variable. So you [856:58] define a variable and I will define a [857:00] very descriptive one sonar token so that [857:02] I can use this variable. Now what are [857:05] the credentials that you want to use for [857:07] this? I will click on add here. I'll go [857:10] to Jenkins and this time I will choose [857:14] secret text. Here it is asking what is [857:17] your secret text. I will go to my visual [857:19] studio code. I will go to creds. I will [857:22] copy this one. [857:24] I will paste it here. And the ID is [857:28] let's call this sonar cube token. And [857:32] then we can click on add. And then I [857:35] click on generate pipeline. So it will [857:37] generate the exact syntax that is needed [857:42] for us. So I will just copy this or I [857:45] can use this mark here as well. I'll go [857:48] to Visual Studio Code. I'll go to my [857:50] Jenkins file. I will come here within [857:53] this stage. And within this stage I will [857:57] have to you know open a curly brace. I [858:01] will open that curly brace and then I [858:05] will have to also add steps. What are [858:08] the steps I want to use for this stage [858:11] and within the steps block I will paste [858:15] what I have copied. I'll hold on. I'll [858:20] go here. Let me just copy this the [858:22] oldfashioned way. I'll come here and [858:26] this is this pink is for entire stages. [858:30] This is for the pipeline and this curly [858:33] brace within steps that is green is for [858:37] our step. So I will add this here. I'll [858:39] go to view and then I will also word [858:42] this. [858:44] So with credentials string credentials [858:46] ID is sonar cube token. Variable that we [858:48] want to use is sonar cube token. And [858:50] then here is where you can add the next [858:54] commands that you want to run. And this [858:56] is where I want to use copy paste. So I [859:00] will copy the block that I want to use [859:03] here. And then I will just paste it [859:07] here. [859:09] So here I am running a shell command. [859:12] I'm saying mvn clean verify. And if you [859:17] have not been through Maven lesson, this [859:19] will not make sense to you. But for now, [859:22] what I can tell you is I'll go to this [859:24] here. Maven clean will automatically run [859:27] the pre-clean for you and the clean as [859:29] well. So it runs the phase that you are [859:32] running along with the previous all [859:34] phases. In the next phase, I am saying [859:37] MVN verify. So first it will clean and [859:40] then it will run MVN verify. MVN verify [859:44] is this phase which will run all the [859:47] phases that are before MVN verify. So it [859:51] will also package which is why you will [859:54] get a jar file dot jar and it will also [859:58] perform unit tests. Now how it can [860:00] perform unit test? So if I go to my [860:03] visual studio code I have my pom.xml XML [860:06] here I have defined JUnit as a [860:10] dependency which is why Maven knows that [860:15] JUnit tests are there and secondly there [860:19] is also a surefire plug-in which is [860:23] being used and if you have watched the [860:26] sonar cube lecture all of this will make [860:29] sense to you so if I come here we are [860:32] essentially doing this we are performing [860:35] doing JUnit tests and why we can perform [860:38] this because we have defined a testing [860:40] framework which is JUnit and then I [860:43] showed you the code as well where we [860:45] have defined test cases and in our Maven [860:48] we have shorefire plug-in which is [860:51] performing these tests for us and then [860:54] further we have Jakokco Jakokco is Java [860:59] code coverage plug-in Jakoko records [861:02] code coverage for sonar cube and it [861:04] delivers that to our Sonar Cube server [861:08] and that is how Sonar Cube knows that [861:12] the code coverage is this. Understand [861:14] that Sonar Cube can do SAS but it cannot [861:18] do code coverage for code coverage [861:20] reports. It depends on JCO. So let's [861:24] move back and again watch the Sonar Cube [861:27] lecture if this is not making sense. And [861:29] here we have the shorefire plug-in which [861:31] will be running the tests and Jakokco [861:35] plug-in is here which is you know [861:38] delivering the reports to our sonar cube [861:42] and based on this sonar cube will be [861:44] passing or failing our pipeline. Again [861:49] in the quality gate that we have defined [861:51] we have said that if the coverage is [861:53] less than 80% then you mark this as [861:56] failed but there are other parameters as [862:00] well that you could have defined. So if [862:02] I look the default profile if I look at [862:04] this quality gate then you see there are [862:08] multiple conditions that you can add [862:10] based on which you can fail your [862:12] pipeline. Now let's go to our visual [862:16] studio code and see rest of our Jenkins [862:20] file. Then first I'm running clean then [862:23] verify then I'm running sonar sonar. So [862:25] sonar sonar implies I am invoking the [862:28] sonar plug-in and in that plug-in I'm [862:31] invoking the sonar goal. So then we are [862:36] specifying the project key that you want [862:40] to use is this one. And this is the [862:43] project key that we created in the UI [862:45] while we were defining a local project. [862:48] Now this is an important piece here. I'm [862:51] saying sonar host URL is this sonar IP. [862:56] But we have not defined this variable [862:59] anywhere. How do I define variables in a [863:04] Jenkins file? For that, let's come at [863:08] the very top of the pipeline. And here [863:12] alongside agent, I will create a line [863:16] environment [863:18] environment. And here I will use a [863:21] opening curly brace. And then within [863:24] this environment block, I will add sonar [863:28] IP. And the IP that I want to use for [863:31] sonar cube is I want to use the internal [863:35] IP of my sonar cube virtual machine [863:38] which is this one. I'll copy this. I [863:42] will paste this here. Always use private [863:44] IP addresses. However, you could have [863:47] also used public IP address. It would [863:49] work just fine. But I wanted you to see [863:52] how production implementations work. Now [863:56] sonar IP value we have defined sonar [863:59] token it will take the credentials from [864:02] here that also we have defined and [864:04] quality gate you have to wait for [864:06] quality gate that is true that is [864:08] selfexplanatory. [864:09] So we have done all these things. Let's [864:13] run this pipeline. And I want you to [864:15] think in your head, will this pipeline [864:17] fail or will this pipeline pass? And [864:20] here the step would be build and sonar. [864:24] Okay. And then let's push this. And then [864:28] let's come here. I will go here. Here. [864:33] And I will build it. Now I'll close [864:36] this. Close this. Let's It has failed. [864:40] So I go here. Let me go to stages [864:43] rather. So our this stage succeeded and [864:48] then build and sonar failed and it says [864:51] MVN clean this this this this Maven not [864:55] found. Remember what I told you your [864:59] agent must have whatever it is trying to [865:03] do here. We asked our agent to run MVN [865:09] specific commands but we do not have [865:11] Maven installed in our Jenkins agent. So [865:15] either you install Maven in your agent [865:19] or add Maven as a tool. Now we are going [865:24] to add Maven as a tool in our Jenkins so [865:28] that our agents can run Maven specific [865:31] command. So I'll go to my Jenkins. I'll [865:34] go to settings. I will go to tools here [865:37] and then if I scroll down and then here [865:42] I will add Maven and I am calling this [865:45] Maven 3. Now understand you could be [865:49] building different projects that may [865:51] require different versions of Maven [865:54] which is why this exists. What I can do [865:57] is I can call this Maven 3911 and know [866:01] that this name is associated with this [866:05] Maven version and other installation [866:08] that I add of Maven that I can call [866:11] Maven 3.9.1 [866:13] if I have certain installations that [866:18] depend on this Maven version. Understand [866:21] when you are doing this Jenkins will be [866:24] installing the required Maven on the [866:26] fly. So your agent VM must have internet [866:30] connectivity. If you do not have then [866:33] you can also add a zip. That [866:37] functionality you have. So I'm calling [866:39] this Maven 3 and the version I'm using [866:42] is 3.9.11. I will save this and then I [866:46] have defined a tool but I have not [866:50] called that tool in my pipeline yet. So [866:52] let me def do that. I will come here and [866:56] here let's write it here below the [867:00] agent. What I will do is I will add a [867:04] section called tools. And this is where [867:06] we define our tools. And the tools that [867:09] I want to define right now is just [867:11] Maven. But which Maven my friend? I want [867:15] to define Maven 3. And Maven 3 is the [867:19] name that we gave to our installation. [867:22] You see here Maven 3 was the name which [867:26] is why I'm calling this Maven 3. Try to [867:30] connect the dots. Now that we have done [867:32] this, I will go to my terminal. I will [867:36] do get add then get commit and then I [867:42] will push this. So if I go back to my [867:46] browser here I can run this job again. [867:51] So I'll go here this one and then I will [867:54] click on build now and I'll go to stages [867:57] and let's see what is happening. So TV [868:00] stage succeeded build and sonar. Let's [868:03] see if it will succeed or it will fail. [868:06] So it is running maven specific [868:08] commands. We know that because you see [868:10] this has started progressing. And now [868:14] let's see whether it will fail or it [868:17] will succeed. And I want you to clearly [868:20] know that in our pipeline we will have [868:24] three places that will decide whether [868:26] the pipeline will fail or not. that is [868:28] TV file system scan, quality gate scan [868:32] and TV image scan. So let's go to our [868:35] browser again and this has failed again [868:38] but this time the reason is different. [868:41] Maven specific commands did run [868:43] successfully but if I scroll down my [868:46] logs will tell me a different story. So [868:50] let's see what my logs are saying. It is [868:53] saying script return exit code one. [868:57] It is not giving me the complete logs. [869:01] If I scroll above. Okay, if I scroll [869:05] further up, let me just zoom out a [869:08] little so that I can see properly. [869:12] Has it redact redacted the logs? Hold [869:15] on. Maybe it has purged the logs. Let me [869:19] come at the very bottom. Oh yeah, now I [869:23] see the full logs. It says build failure [869:26] failed to execute goal this this this [869:29] this on this project status failed. What [869:33] is this? And it is asking you to check [869:36] here. It says the quality gate has [869:38] failed. And why has quality gate failed? [869:42] Let's see that. I'll go here. I'll go to [869:45] projects. What this is automatically [869:48] created for us. And I want you to check [869:50] a nuanced thing. They have named it this [869:54] we named it cloud with vjosh - dev sec [869:57] ops demo but from where did this name [870:00] come into picture so if I go to my [870:02] visual studio code and I go to pom.xml [870:06] then the name that I have defined is [870:09] this which is why it is using that name. [870:12] Now if you have not watched maven [870:14] lecture you are at loss here. So you see [870:17] here and if I go here then you see why [870:21] it has failed. It says the coverage is [870:24] just 3.2%. [870:26] The required coverage is this much. And [870:28] then there are other problems with the [870:30] code as well. If I go to issues, it will [870:33] mark the other problems. Now at this [870:36] stage, this is the point where you reach [870:39] out to your developers that we are [870:41] facing, we are getting these issues. [870:43] Please fix this. Please make the [870:45] coverage. All right? Only then the [870:48] pipeline will succeed. But since we are [870:50] doing this in our lab environment, I [870:53] think we can pass this using the means [870:57] that you should never be using in [870:59] production. So what I will do is I'll go [871:02] to my quality gate. I will go here and [871:05] then I will edit this rule and I will [871:08] mark the coverage just 2%. So anything [871:11] that is less than 2% you then fail. But [871:15] our code coverage is 3.2% so it will [871:17] succeed. But you should never do this in [871:20] production. The minimum code coverage [871:22] that I have seen in production is 65 [871:25] 70%, not below that. So I'll update this [871:28] condition and once this condition is [871:31] updated I will come back to my [871:35] this job and then what we can do is I [871:39] can restart from a stage and that stage [871:43] is build do sonar. Let's see how it [871:46] works. Now I'll go to stages and then [871:50] let's see what happens in this stage. [871:52] Now this time the build is success [871:55] because the quality gate allowed this to [871:59] pass. Now we have a good understanding [872:02] of how this works. So let's see what we [872:05] want to do next. So I'll go to my [872:07] diagrams and in my diagram we have done [872:10] both of these things. Now we have to [872:12] perform ECR login and image build. So [872:16] let's first perform ECR login and then [872:21] see what we have to do. So I'll come to [872:23] my Jenkins pipeline. But before we do [872:26] anything pipeline, I think it makes [872:29] sense to have a repository first. So [872:33] let's go to AWS. I'll go to my browser [872:36] and then here I will look for ECR. Yeah, [872:41] I want to use Amazon ECR. Here I am in [872:45] Mumbai region. I want to create a [872:47] repository and what I want to name this [872:50] repository is cloud with varosh dev [872:53] seccop ops- demo. And I'll just zoom in [872:56] a little so that you can see properly. [872:58] Image tags setting mutable or immutable. [873:02] In production you would always use [873:06] immutable. Now this defines whether [873:09] redundant tags can be pushed or not. So [873:13] always use immutable so that if 2.6 [873:17] version let's say exists then no one [873:19] else can repush 2.6 because that creates [873:24] chaos. You don't want that. Now it is [873:27] asking are there any tag exclusions and [873:31] I will have tag exclusions. I will have [873:33] a latest tag here. Now understand that [873:37] when you are downloading let's say an [873:40] image and you do not supply a tag you [873:43] docker typically adds latest as you know [873:48] the tag but for that to work an image [873:52] with latest tag must exist in your [873:56] repository if that does not exist then [873:59] latest won't work that is one thing [874:02] second what I'm saying is if someone has [874:06] pushed a latest image and then if [874:08] someone else pushes the latest image [874:11] with a tag latest then you should allow [874:13] that because we'll have to allow that so [874:16] that anyone who wants to use or work [874:19] with the latest version of the [874:21] application they can just pull the [874:23] latest tag. However, some level of [874:26] controls must be there in your [874:28] organization so that whenever you are [874:31] pushing a version that is the latest [874:34] indeed should have a latest tag as well. [874:38] And that is what we will be doing. We'll [874:40] be pushing image with both the tags. A [874:43] version tag which we'll see what version [874:47] tag we'll be using and then a latest [874:50] tag. So I've added latest here. I'll [874:52] click on add filter. Now if you are [874:54] using KMS you can use that you can have [874:57] your customer managed keys your own keys [874:59] or AWS manage but for this one I'll be [875:02] using AES 256 [875:05] here I will just click on create now as [875:09] soon as you click on create it will tell [875:12] you what all are the commands that you [875:14] can use to push this. The URI is also [875:18] available at your disposal. So if I [875:20] click on this I can view push commands [875:23] from here and this is what we will be [875:27] using. So let's go to our Jenkins file [875:31] and let's add another stage. So this [875:34] stage started with assan curly brace and [875:39] then I will add a new stage here. This [875:42] stage is let's call this ECR login. So [875:46] we are going to login into ECR and here [875:51] in this stage these are the steps and [875:55] I'll add steps here and then I'll add [875:58] another curly brace and then I will [876:00] mention the step here. Now see what I [876:04] can do. I can just quickly copy this and [876:07] I will paste this here. SH I want to run [876:11] this then this is fine. But see your [876:15] pipeline will not look good if you have [876:18] these lengthy URIs here. Okay. So let's [876:22] do one thing. Let's variableize this [876:24] part. And we already know how to do [876:27] that. So I'll copy this and then I will [876:30] come at the top and in the environment [876:32] section I will add another variable [876:35] here. Let's call this ECR registry. And [876:38] the URI is this. This is the registry. [876:41] So I have added this here. Now what I [876:44] want to do is this is the variable name. [876:47] So I will copy this and then instead of [876:51] this I will be using that. See if you [876:55] look at this command this is AWS [876:58] specific command. It is using AWS. Hence [877:01] we must have a binary that understands [877:04] AWS. What is AWS? It is essentially AWS [877:08] CLI. To run this command, our Jenkins [877:12] agent must have AWS CLI installed. [877:16] That's first. Second, we are also using [877:20] Docker login. So, Docker must also be [877:23] installed in our agent if we want this [877:27] command to run successfully. So, let's [877:30] do that. So, I will go to my browser and [877:34] here I will go to my GitHub notes. This [877:37] one I will come to the very top first [877:41] and then I will go to this stage. This [877:45] stage is ECR login. Install AWS CLI on [877:50] Jenkins agent. I'll go here. This is not [877:53] working. The click is not working. I'll [877:55] have to fix that. But anyway, here are [877:58] the steps to install AWS CLI. And you [878:01] have to do this in your Jenkins agent. [878:04] So, I will copy all of these. And again [878:06] this binary we are installing I think [878:10] this will be installed in usr bin for [878:13] which all the users have access. So that [878:15] will be available to our Jenkins user by [878:19] default. So I'll enter these commands [878:21] and this will install AWS CLI. And here [878:25] we if I run which AWS it's installed [878:29] here and all the users will have access [878:31] to this. And we can also verify this [878:34] using this command. So if I run this as [878:37] Jenkins user then I can check the AWS [878:41] version. So AWS CLI is indeed installed. [878:45] Now let's docker engine installation. So [878:49] all these steps are taken from the [878:52] official documentation. Here in the [878:55] first step we are removing if there are [878:57] any installations of Docker already in [878:59] your machine. This is not needed for our [879:02] use case because we have set up a new [879:03] VM. Then we are updating the index of [879:06] all the packages. Here are the things [879:08] that are required by Docker. We are [879:10] adding all the keys. Then we are adding [879:13] the repository. And finally we are [879:15] installing these things from that [879:17] repository. So let's let's run all of [879:20] these commands. I'll go to my terminal [879:24] and then I will paste them here. Unable [879:26] to locate is fine because the first [879:28] command won't run successfully as we do [879:30] not have Docker already installed. So [879:33] that's expected and once we are done [879:35] with this then what we will be doing is [879:38] this is an important piece we will be [879:41] adding our Jenkins user to the docker [879:44] group so that our Jenkins user can run [879:48] docker specific commands and then again [879:52] we are also adding the Ubuntu user if [879:54] you want wish to skip this you can [879:57] ideally only Jenkins should be allowed [880:00] but I added this so that you can quickly [880:03] test docker right from your Ubuntu user [880:05] itself. But in production you will only [880:08] use this not this one. For this one I [880:10] can use both of these. So I'll come [880:12] here. I will paste these and I have done [880:15] this and then let's see the docker [880:17] version. So first we are checking the [880:19] docker version as Ubuntu user and then [880:22] this is that output and then we are [880:24] checking the docker version as Jenkins [880:26] user. So both of these work. This is [880:29] good. Now I want you to understand one [880:33] more thing. Will this pipeline run? Will [880:35] this stage be successful? If not, can [880:38] you tell me why? So let's do one thing. [880:42] Let's first commit and then we'll come [880:44] to we'll see the logs and see how it [880:47] works. Commit. And this one is your ECR [880:53] login. And then I will write here push. [880:57] And then let's go to our browser and I [881:01] will come here here and I will build it [881:07] now. [881:08] Now you can go to stages and see how it [881:11] is working. So all of these succeeded. [881:13] We know they had to succeed. And here [881:17] let's see what the log says. AWS ECR get [881:21] login this docker login this unable to [881:24] locate credentials. You can configure [881:26] credentials by running AWS login. So you [881:29] are trying to access Amazon ECR service [881:34] from your Jenkins agent without [881:38] supplying any AWS credentials. How will [881:41] you be able to access Amazon ECR if you [881:45] have not provided the credentials to [881:48] access the same? Now if your Jenkins [881:51] agent was deployed outside of AWS then [881:54] you could be using AWS secret key or [881:57] there are other means as well but we are [882:01] in AWS itself. Both our controller and [882:04] agent are in AWS. Our agent wants to [882:08] access Amazon ECR service. How do I [882:12] grant access to my agent? It is simple. [882:17] I just attach appropriate AM role and my [882:23] agent will be assuming that role while [882:27] it tries to access Amazon ECR. So let's [882:32] associate an instance profile with our [882:37] Jenkins agent VM. But before doing that [882:41] we will have to create an AM role. So [882:45] I'll close this. [clears throat] Here I [882:47] will choose roam [882:50] ro. I'll come here and here I will say [882:54] I'll zoom in a little. Then I will say [882:57] create role for AWS service. And yes it [883:01] will be EC2. EC2 is fine here. I will [883:05] click next. And what is the policy that [883:09] you want to associate with this role? [883:11] Now I will use an existing policy. So [883:15] here are the policies for public [883:18] registries and for private as well. So [883:20] if I go here you see public and you see [883:24] private. So we are using the private [883:26] registry right now. That is what you [883:28] would see in your you know your [883:30] applications. So we will come here and [883:33] then I will choose EC2 container [883:36] registry power user. I will select this [883:40] policy and then I will click on next and [883:44] here I want to give the role name. So I [883:47] will call this Jenkins agent role. [883:50] Giving this name and then we will be [883:54] using this name later as well. So name [883:58] it in such a way that you remember it [884:01] and always follow nomenclature that is [884:04] understandable. So we are using this [884:06] name and then I will click on create [884:10] role. Now this will give our Jenkins [884:13] agent access to Amazon ECR. So let's [884:17] come back to the Jenkins job and then [884:21] let's rerun this from a specific stage [884:25] and that stage would be ECR login and [884:27] let's see if this works this time or [884:31] not. And this time it also fails. Let's [884:35] see why did it fail. Who will attach the [884:39] role to the instance? So let's attach [884:41] that role to the instance. We have just [884:43] created the role. Let's attach that. And [884:46] I didn't do it deliberately. Okay. I I [884:48] genuinely forgot. Security modify IM [884:51] role. And here I will choose the IM role [884:54] that I just created. This is called the [884:57] instance profile in this language. I'll [884:59] select this one and update IM ro. And [885:03] now if I run the pipeline, this should [885:06] succeed in my opinion. Let's use a rerun [885:09] this time and let it run. This GUI is so [885:13] intuitive to me. I, you know, like [885:16] seeing this everything green, green, [885:18] green. It's so satisfying when you have [885:21] a running pipeline. It has 10, 15 stages [885:23] and all of those are green. It's so [885:26] satisfactory, I tell you. And then you [885:28] see the login has succeeded. So we have [885:31] performed the ECR login but we have not [885:35] built our image yet. So let's start [885:39] building our container image. So I will [885:43] go to my Visual Studio Code. Firstly to [885:47] build any container image we must have a [885:51] docker file and docker file you can find [885:54] the docker file in the project files [885:57] itself in the repository. So we will be [886:00] upgrading this Java Maven with different [886:02] things. We'll now add docker file. So I [886:05] want to copy this for now and then I [886:08] will paste it in Java M. I'm pasting it [886:11] in the root itself. So if I see the [886:14] docker file, it's a simple docker file. [886:17] We are using Eclipse Turin 21 that is [886:20] Java 21. Working directory is / app. And [886:25] then we are copying whatever is [886:27] generated as app do.jar [886:31] inside /app directory. Now see I told [886:36] you whatever leg work is being done is [886:40] done by the Jenkins agent itself. And if [886:43] you remember while configuring agent we [886:46] also defined what would be the Jenkins [886:48] home in the agent. Where will the [886:51] workspace be saved? where what would be [886:53] the scratch space that the agent would [886:56] would be using and if you remember we [886:59] supplied forward slhomeward slashjenkins [887:03] there so I am in the Jenkins agent right [887:05] now so if I pseudo-en [887:08] u jenkins I log in as sudu jenkins bash [887:13] have to supply the command that I want [887:14] to run anyway so I'll go to cd home [887:18] jenkins and that's the home of our [887:20] Jenkins user and if I press ls is you [887:23] see workspace here and you also see [887:26] tools here. Now this tools will have [887:28] Maven. So if I run tree here do I have [887:30] tree? [clears throat] Anyway, forget it. [887:32] I'll cd into tools. And then if I do ls, [887:36] you see Maven installation is here. I [887:39] told you your agent will require [887:41] internet access if you're using that [887:43] tools section and you're not providing [887:45] the installer. Anyway, so I'll come back [887:47] and if I go to CD workspace, you would [887:50] see that whatever was there in our [887:51] repository, you see here and this is the [887:54] name that we gave to our pipeline job. [887:56] So if I go here [887:59] and do ls then you see Jenkins file is [888:02] here pom.xml is here src there is also [888:05] target and this was created because we [888:08] also ran mvn verify and that's why this [888:11] was created. So if I cd into target, if [888:14] I do ls, then you would see a file named [888:18] like this was created for us. And that [888:20] is exactly what we are doing with this [888:25] command. So what we are saying is copy [888:28] from target and you go to this file and [888:31] copy that as app.tchar. So you see this [888:34] is the file that we are copying as try [888:37] to you know think why we are doing [888:40] something and then we are exposing our [888:42] application on port 8080. This is just [888:44] for documentation. Your application [888:46] should also be listening on port 8080. [888:49] Our application is which is why I have [888:52] added that. Now another thing is entry [888:55] point. This is the command that I want [888:57] to run when my application is running. I [888:59] want to run this app.jar jar this file [889:03] and that is what will show you cloud [889:04] with varoosh welcome to cloud with [889:06] varosh and so on whatever we are doing [889:08] in the application so we have the docker [889:11] file in the root now if we commit [889:14] jenkins would know there the docker file [889:16] that I need to refer when I'm building [889:18] something but that build step we have [889:22] not yet added so let's start with that [889:26] step I will come here steps stage can [889:31] closing bracket that I'll come here. I [889:34] will add another stage and this stage [889:39] let's call this one image build image [889:42] build image I have to spell this right [889:45] so that it looks neat and then in the [889:48] stage here I'll add this curly braces [889:52] and then I will have steps I will again [889:55] have curly braces now I will copy the [890:00] command that I want to run just give me [890:03] a moment moment so that I can copy it. I [890:07] will paste it here. Perfect. Again, I'm [890:11] invoking a shell. Then I'm writing [890:13] export docker build kit zero. Now why I [890:19] am doing this? So this will essentially [890:21] disable docker build kit. So builds you [890:24] know run in classic and predictable mode [890:27] only. That's the reason for this and I [890:29] will tell you why I'm doing this in a [890:31] little while. Then our normal usual [890:35] docker build command. I'm saying docker [890:38] build- platform Linux AMD 64. Now this [890:43] is telling docker that the image that I [890:46] am building the application that will [890:48] result from this supports AMD 64 [890:53] architecture and which [clears throat] [890:55] is the reason the EKS nodes that we will [890:59] provision as part of this lecture. The [891:01] Amazon EKS cluster we provision the [891:04] worker nodes will also have AMD 64 [891:07] architecture. Now you know why this [891:10] matters. If you have developed your [891:12] application for ARM and then you are [891:16] trying to run that application onto a [891:18] worker node that has AMD 64 then it [891:23] won't work. And for further knowledge, [891:26] AMD is the one that introduced 64-bit [891:29] architecture. Then Intel no didn't want [891:32] to call it AMD 64. So Intel called it [891:35] x86_64. [891:38] But x86_64 [891:40] from Intel or AMD 64 they imply the same [891:44] thing. Why this matters is because the [891:46] application that we are building it [891:49] supports AMD 64 architecture and the [891:52] base image that I am using also supports [891:55] AMD 64 architecture. Hence the image [891:58] that I want to create is for the [892:01] platforms that have AMD 64 architecture. [892:04] That is what we are defining here. Now [892:07] why this becomes important is when you [892:09] are dealing with enterprisegrade [892:11] applications you want to be precise. You [892:13] want to tell everything that is needed [892:15] so that you do not get extra things [892:19] free. So that's why I'm defining this. [892:22] And when I was not defining this, my [892:26] Amazon ECR had a few more blanks pushed [892:31] to it. Which is why I added this Docker [892:34] build kit is equal to zero. Nothing [892:37] fancy. We have already used Docker build [892:40] so many times in this course. U in [892:43] Jenkins course or in my Kubernetes [892:45] course. If you're new here, Docker build [892:47] is just building the container image. If [892:50] you do not understand this, you are in [892:52] the wrong lecture because this is a [892:54] CI/CD pipeline lecture which deploys [892:57] onto Kubernetes. And to do that, [892:59] creating container image is the first [893:02] step. Anyway, I'm then defining tags. I [893:05] am pushing two tags. I'm saying the [893:08] first tag is image repo colon build [893:13] number but varun you have not defined [893:15] image repo yet let's define that now I [893:19] will go here and then I will say my [893:24] image repo this would be I'm adding [893:28] double quotes here okay because I will [893:30] be using a variable within a variable [893:33] and then I will call this this and I'll [893:36] tell you what I'm doing I'm saying my [893:39] image repo is ECR registry this [893:42] forward/cloud [893:43] with var dev seccopsy demo and this is [893:48] exactly what you saw here on the console [893:51] so if I come here exactly this I'm [893:55] defining this URI and then I will just [893:58] define the tag and what is the tag that [894:02] you are defining Vun I'm saying dollar [894:05] build number and then latest is I'm sure [894:09] you understand but what about build [894:13] number now there are certain environment [894:16] variables that are exposed by Jenkins [894:20] and we are using one of those [894:23] environment variables. Now if you want [894:25] to access those environment variables [894:28] what you can always do is you can first [894:30] copy this name. I'll copy it from here. [894:34] I will paste it here and then I will [894:37] write env.html [894:41] and I press enter. And these are all the [894:45] environment variables that you can use [894:47] that are available for your pipeline [894:50] jobs. And here what I'm using is I'm [894:53] using a build number and this is the [894:56] build number that I'm talking about. So [894:58] if I go to my pipeline, everything has a [895:02] build number associated with all the [895:04] runs they have a build number. Now I am [895:07] tagging my application with the build [895:10] number so that when I see my image I [895:13] exactly know which build resulted in [895:17] this build number. So what I will now do [895:20] is this is understood. Now you will also [895:24] have your applications versioning as [895:27] well like which version of application [895:29] you are using. I have not shown that [895:31] here but you must keep that in mind. So [895:35] what you could be seeing is here you [895:38] could be using you know like something [895:40] like this app version and then a build [895:43] number. So you are tagging your image [895:45] with application version as well and [895:47] then the build number from which that [895:49] image resulted. So that gives you full [895:52] control when you see the image itself. [895:55] Anyway, so we will just use this for now [895:58] and we have supplied that. Now that we [896:01] have done both of these things, we can [896:05] see whether these steps works for us or [896:08] not. So I will go to my terminal again [896:12] and here I will go here. I will first [896:16] add this and then I will commit this. I [896:20] will say container image build. Press [896:24] enter. And then I will push this. Once [896:27] this is pushed, I will come here. I will [896:31] run build now. And this time I will go [896:36] to stages. And you see build number 12 [896:39] has started. [896:41] and then [896:47] permission denied while trying to [896:50] connect to docker API. Now understand we [896:54] installed docker after connecting to the [896:58] agent. Correct? So we have to refresh [897:03] that so that our controller adds agent [897:08] with the refreshed permission. So what I [897:10] will do now is I will go to my Jenkins [897:14] screen. From here I will go to agent. I [897:18] will disconnect this. Yes. And then I [897:21] will again launch agent. I'll see the [897:23] logs. It says connection successful. Now [897:27] I will run this job again. I'll come [897:30] here and I will click on build now. And [897:34] let's see the stages. And if I go here, [897:39] this is running all the stages that we [897:41] have defined up till now. And at the [897:44] very last, we see the build image stage. [897:46] So let's see how it works now. [897:50] And we are in this stage. And this time [897:54] it has started building the things for [897:58] us. No more that error. So just [898:00] disconnect your agent and reconnect it [898:03] if you also see those problems. Now it [898:06] has completed. So it has completed [898:10] building the image. So if I go to my [898:15] terminal, I am already logged into my [898:18] agent and if let's say I run docker ps [898:23] here, docker images here, I'm sorry. [898:26] Then you see the images. This is the [898:29] base image that we are using. We don't [898:31] care about this one right now. You see [898:34] there are two images that are created [898:36] for us. One with the latest tag and one [898:40] with tag 13. And why is this 13? This is [898:45] 13 because this is build number 13. And [898:48] we added the build number in the command [898:52] that we were running in our Jenkins [898:54] file. So we said that you must use the [898:58] build number. Now that you have this [899:01] understanding, let's move to the next [899:04] step. What you would be thinking, Vun, [899:06] why haven't you pushed yet? My dear [899:08] friends, we'll have to perform an image [899:11] scan to ensure that this image, which is [899:15] our base image. When I talk about the [899:18] Docker file used by our application, is [899:21] vulnerability free. And that is what [899:24] privy image vulnerability scan will do [899:28] for you. It will scan your base image [899:32] and its OS packages to see whether there [899:36] are any vulnerabilities associated with [899:40] it or not. So let's just quickly do that [899:44] step. So I'll go to my visual studio [899:46] code and same this time I will check [899:50] this is the cyan curly brace and then [899:53] I'll come here I will add another stage [899:57] and this stage I will call privy image [900:01] scan and then we have opening closing [900:05] curly braces. I'll press enter and here [900:08] we'll have steps. I'll again have an [900:10] opening closing curly brace and inside [900:13] steps I will have the command and the [900:16] command is extremely simple. I'll paste [900:20] it here. Now I want you to see the [900:23] difference. In the previous command we [900:26] said that tree file system scan and I [900:29] want you to run this on everything that [900:31] is in the current directory. And I want [900:33] you to clearly know what is the current [900:35] directory. Current directory is the [900:38] workspace where it has cloned [900:40] everything. This is the current working [900:43] directory. So it is saying scan [900:44] everything here. But this time we are [900:47] saying TV image. FS has changed to TV [900:52] image because we are performing an image [900:55] scan this time and if you find any high [900:58] or critical vulnerabilities then you [901:00] fail the pipeline. So this is a quality [901:03] gate that we are applying that any high [901:06] or critical vulnerabilities you fail the [901:09] pipeline and vulnerabilities in the base [901:12] image. And here we are supplying the [901:15] image on which the scan must be [901:18] performed. Simple. And then I will come [901:22] to my terminal [901:24] and I will go here. And then I will [901:28] quickly add commit with I will write [901:32] here [901:34] image scan. And these messages are [901:37] important guys when you are you know [901:38] looking back at your history. And then [901:41] I'll press enter. I will come to my [901:44] browser. I will come here again build [901:46] now and then I will go to stages come [901:51] here and then let's see what our build [901:55] image stage results in. Build image is [901:59] already here from the last time. We want [902:02] to see the next one that is TV image [902:05] scan. TV image scan. It's so good, [902:08] right? It labels everything so properly. [902:11] So TV image scan has appeared here. If I [902:14] click here then it is performing the [902:16] scanning in our container image. And you [902:21] see our command has expanded. Whatever [902:25] we had variableized it now has 14 [902:28] because this is builder number 14. So [902:31] whatever variable is being defined that [902:36] is resulting. It is not scanning [902:38] anything stale. It is scanning what we [902:42] asked it to scan. Now this scan is [902:46] successful. Understand the image that we [902:49] are using is free of vulnerabilities [902:51] which is why it has not flagged [902:53] anything. So with this said we are good. [902:57] And if I expand this further yeah output [902:59] was you know redacted by this terminal. [903:02] Now you see the full output and here is [903:05] the detail for that. no vulnerabilities [903:09] and it is not just performing image [903:11] scan, it is also performing something [903:14] more because it also scanned app app.jar [903:19] as well. In this step, we wrote trivi [903:22] image. Now to know more about trivi you [903:24] will have to read about trivi. It is not [903:27] that this is a trivial class. Understand [903:31] that when you are integrating a new [903:33] tool, you must first understand what [903:35] that tool is using. We know as a [903:37] principle that Trivy is used for SCA and [903:41] is used for image scanning primarily. [903:44] Other things you read the documentation [903:47] and explore further and once you [903:50] understand one tool then other tools [903:52] become so easy to understand. So for [903:55] example if you know Jenkins, CI/CD then [903:57] be it GitHub actions or GitLab they seem [904:01] like a cakewalk. Now that we have done [904:04] this, let's see what we want to do next. [904:08] Next is we want to push to AWS ECR and [904:14] they call it Amazon ECR not AWS ECR. So [904:18] we'll push to Amazon ECR. Let's see what [904:22] are the steps for it. And again these [904:24] steps would be simple. Simple as in [904:26] whatever you do when you are using [904:29] manual methods. It is just Jenkins is [904:32] automating things for us. So I will [904:35] directly copy the entire block for this [904:38] because I don't want to waste time in [904:40] doing simple things with you. Here this [904:43] and this I'll come here. I will add [904:46] this. Now I have added that stage as [904:50] well. Now this is push to ECR. Steps is [904:54] docker push this the image tag and then [904:59] the second image tag. We want to push [905:00] both of them. So I'll go to my terminal [905:03] and here I will write get add this get [905:07] commit. I have to have a good commit [905:10] message. Anyway, ignore this one. And [905:12] here I will write get push private repo [905:16] main. And then I will go to my browser [905:21] and here again I would run the same [905:23] thing build now and then I will go to [905:26] stages. Let this complete and once this [905:29] is complete we should see that our [905:31] images our tags will be here. Understand [905:36] that we call this a repository right [905:39] like Ubuntu is a repository and then [905:42] that repository has multiple tags and [905:46] this repository cloud with VJO dev [905:48] secops demo has two images as in two [905:52] tags one with 15 and another is latest. [905:57] That is how it works. Now the push has [906:02] already happened. now is the last and [906:05] the most interesting stage that is [906:07] deploying onto an Amazon EKS cluster. So [906:11] if I go to Jenkins now you would see [906:14] this has pushed to ECR successfully and [906:17] then you can also see all the logs here. [906:21] Now is the interesting stage that is [906:25] deploy to Amazon EKS and this is also [906:30] where we are failing. In case the [906:32] deployment is not successful, we will be [906:35] rolling back. So let's start working on [906:39] this step. Now before we start working [906:41] on this, I quickly want to create a [906:43] cluster and I will just run the commands [906:46] to create a cluster and then I will walk [906:48] you through what I'm doing. Okay, I'll [906:50] just cd I'll come one step back and from [906:54] here I will cd into project files and [906:58] I'll do ls and I will run eksctl [907:02] create cluster and the file that I want [907:06] to use to create cluster is eks-configy. [907:13] Now what I'm saying is create a cluster [907:16] using this configuration file. Now to [907:18] run this command your thin client as in [907:21] your laptop or whatever you are using to [907:24] you know run this lab should have eksl [907:27] installed. You will find all these [907:29] details in the github repository. So do [907:32] not worry about them but you should have [907:35] that eksl installed AWS CLI installed [907:38] and we I will attach the steps on how to [907:40] do all those things in the GitHub notes. [907:42] You will press enter and cannot find EC2 [907:47] key. Hold on, hold on. Let me run AWS [907:54] configure [907:56] list profiles. [907:59] Yeah, this is the profile that I am [908:02] using. Okay, hold on. Let me make this [908:05] profile default. Okay, export AWS [908:10] default. I think uh I mean I closed the [908:13] session from where I was working in the [908:15] practice sessions which is why I have to [908:16] you know define the default profile [908:19] again and there are ways via which I can [908:22] make this profile as you know perpetual [908:25] default but I don't want to go there [908:26] right now. So default profile is CWVJ [908:31] - SJ AWS default profile. Now we'll [908:36] press enter. And now if I try to run [908:39] that command that should succeed because [908:41] yeah it has now succeeded. So that's [908:45] small roadblock. Now what we can do is [908:48] now let me explain you what we are going [908:50] to do because this creation takes some [908:53] time. I'll go to my diagrams and this is [908:56] what we are going to deploy. We are [908:59] going to create an EKS cluster and you [909:02] know in EKS cluster. There is a control [909:04] plane and then there is a data plane. [909:07] The control plane part is managed by [909:09] AWS. They will have redundant API [909:11] servers and your HCD nodes and they have [909:15] three AS where they will deploy your [909:17] database at CD and then your API server [909:21] along with other control plane [909:22] components is also redundant. So here [909:26] for this part you do not have to care [909:28] because it is a managed service. Here is [909:31] where I am using EKS managed worker [909:34] nodes wherein we will be creating two [909:38] worker nodes exactly like this in two [909:41] different availability zones. So if we [909:44] are creating two worker nodes and that [909:47] two in different availability zones, we [909:49] want to ensure that the application that [909:51] I am deploying is deployed in such a way [909:55] that the pods associated with that [909:57] application leverages both these [909:59] availability zones. Otherwise there is [910:01] no point for me to have two a2 [910:05] does not have any pods pertaining to my [910:08] application. If my application warrants [910:11] high availability of two A's then I must [910:15] ensure I have means via which I can [910:18] ensure that if I spin 10 pods five will [910:21] be in a 1 and five will be in a 2. That [910:25] is your responsibility as a DevOps [910:28] administrator. So for that to happen we [910:31] will have things in place. So this is [910:34] the cluster that we are creating. Now [910:36] let me walk you through the [910:38] configuration file that I have used to [910:41] create this cluster. Under project files [910:44] we used EKS config.yml. [910:48] Extremely simple. We are creating a [910:51] cluster. We are calling that cluster [910:53] cloud with various devs sec ops EKS. We [910:56] are creating that in the Mumbai region. [910:58] The version we are specifying is 1.32. [911:00] AWS is offering 1.33 as well now. So you [911:03] can use that. And then I am adding [911:07] certain tags to the resources that will [911:09] be created for this cluster. So these [911:12] are those tags owner tag environment [911:14] project purpose and then I'm specifying [911:17] I want to use two availability zones I [911:20] am with oidc is true and we discussed [911:23] this in if you want to you know look at [911:26] greater detail for this there is a there [911:29] is an ingress demo in my uh channel and [911:33] a stateful set demo in my channel very [911:35] detailed production like and there I [911:37] discuss all these things in greater [911:39] detail you Check that out. Then I'm [911:43] creating a managed node group. What I'm [911:45] saying is I'm calling this managed node [911:47] group cloud with vosh nng public and [911:50] public I'm calling this public because [911:53] the private networking is false here. So [911:56] my nodes worker nodes will be accessible [911:58] over the internet which you may not [912:01] need. I am doing this because the [912:04] application that I will be deploying [912:05] will be exposed using node port service. [912:08] But War Vun who uses note port in [912:10] production and I agree with you 100%. [912:15] But I wanted this demo to be within 3 4 [912:18] hours. I did not want it to stretch so [912:20] long which is why I have not created any [912:23] ingress or I have not deployed AWS's [912:25] load balancer controller. If you want to [912:28] see that scenario, there is a detailed [912:30] 2.5 hours video in my channel that walk [912:33] you step by step on how to do that and [912:36] how to expose your applications using [912:38] that. You are very welcome to integrate [912:42] that application with this CI/CD [912:44] pipeline. But for now, we will be [912:46] exposing the application that we are [912:48] deploying with note portut. Then I'm [912:52] saying allow SSH. We won't need SSH but [912:55] I'm just saying allow SSH and then [912:58] public key name that I'm using is the [913:00] same one that I use for instances. We [913:01] won't be logging in into our instances. [913:04] So this won't be used. Then I'm defining [913:07] few add-on policies here. I'm saying [913:09] that true true these things should be [913:11] allowed. Labels our nodes will have [913:15] these Kubernetes labels and then these [913:19] will be the tags associated with our [913:21] nodes. That is what I have defined. Now [913:24] let's see where is our installation. So [913:26] the installation is still going on till [913:29] that point. Let's do a few more things [913:33] that we want to do. So I'll come to [913:36] Visual Studio Code and then I'll come to [913:38] Jenkins file and then let's see the [913:42] Kubernetes manifests that we will be [913:44] dealing with. There is deploy svc.yamel [913:50] in project files that you can use. [913:52] deploy as in deployment plus the service [913:55] manifest exists here. So if I go here, [913:58] it is a generic deployment that you [914:00] would have used thousand of times. I'm [914:03] using two replicas and then this is the [914:07] important part. This is the part that [914:10] will ensure if I have 10 replicas, five [914:12] is in one a, five is in another a [914:16] because here the topology key that I'm [914:20] defining is zone. So if I have let's say [914:23] two zones, it will try to you know [914:26] deploy one pod in one zone and second [914:29] pod in another zone. When unsatisfiable [914:32] I'm saying do not schedule. So this is a [914:35] strict rule. You can be lenient with [914:38] this. So if you hover over this when [914:40] unsatisfiable [914:41] I think this will tell you. I mean [914:43] schedule anyway is another option I [914:45] think that you can have here. label [914:47] selector I'm saying this is applicable [914:50] for all the pods that have app is equal [914:53] to cloud with varos dev secc ops demo [914:55] and that is the label that we are adding [914:59] to the pods that will be part of this [915:02] deployment. So very simple max skew is [915:06] one that is you know uh default you [915:08] cannot have this zero. The max ske one [915:11] says the imbalance of one is applicable [915:15] can be there but not more than that. So [915:18] skew is the imbalance that you can have [915:21] here is the container section and you [915:23] see these lines because we have not [915:24] defined resources section as in limits [915:27] and requests which is why you are seeing [915:28] this container name is this image. Now [915:32] this is the important piece. I am saying [915:35] you have to use this image. Now I'm [915:37] saying latest here but do not worry. We [915:40] will have something in place which will [915:42] have a specific tag here and we'll see [915:45] how that works. For now just understand [915:49] this is the entire URI of your [915:51] repository. So if I go here to my [915:54] browser, you see entire this. If I go to [915:59] image repositories here, the entire this [916:03] is here. And then we are saying the tag [916:07] we are supplying the tag. Port says [916:09] container listens on port 8080 which is [916:12] on what my application listens on. Then [916:15] I am defining a node port service. It's [916:17] a simple node port service that listens [916:20] on port 31,000. [916:22] So we should be able to access this [916:25] application on any of the nodes public [916:28] IP on port 31,000 once the deployment is [916:32] successful. So we have understood this. [916:35] Now what I want to do is I want to add a [916:38] stage in my Jenkins file which will [916:41] ensure that this file you know firstly [916:46] I'll move this to my this because it [916:51] will be required for to run cubectl [916:54] commands and it will be required by the [916:56] next stage where we'll be editing this [916:59] file to change this latest to something [917:03] significant as in the build number. [917:05] Let's do that. I'll come to my Jenkins [917:08] file. I have this stage here in can. And [917:11] then I will add another stage here. And [917:14] this stage I will just call this stage [917:19] proper naming because in this stage we [917:22] will be updating the deployment with a [917:24] proper name. So I'll call this update [917:27] deployment. Okay. And then here I will [917:32] have opening curly brace, closing curly [917:34] brace. Then here I will have steps and [917:37] within steps I will only have one step [917:40] and that step is this one. Now here I am [917:45] using set. So set is a substitute [917:48] utility which will let you substitute [917:51] something with something. So here what [917:53] I'm saying is substitute my image you [917:56] know with this one and do that globally. [918:00] So globally will not matter because we [918:03] only have one place where we have to [918:05] perform the changes. I just added G for [918:08] completeness. So this what will this do [918:11] is it will go to image and it will find [918:13] this and then it will replace this with [918:16] this variable and this variable both of [918:20] these variables have already been [918:21] defined in our Jenkins pipeline. So [918:24] image repo is this and then the tag is [918:27] build number. So that is what this stage [918:30] will do. Now let's execute this and I'll [918:32] show you how this will work. Oh this [918:34] cluster creation is still ongoing. I [918:37] wanted to push. So let me do one thing. [918:39] I will have another terminal opened. I [918:42] will quickly cd into courses courses and [918:47] then here I will go to Jenkins and then [918:50] common and from common I want to go to [918:55] Java Maven right Java Maven is where I [918:58] want to go here. Let's just quickly [919:00] revise what we have done. It should have [919:03] deployment and service manifest which it [919:06] has. It should have Jenkins updated file [919:09] which is which it has. So let's run this [919:12] now. Let's commit this and then push it. [919:16] So I'll go to my terminal here. I will [919:19] write get status. It should show that [919:22] unttracked files. Yeah, modified is [919:24] Jenkins and this is unttracked. So I [919:27] will write get add dot and then get [919:31] commit- [919:32] m updated deployment image tag. Yeah. [919:38] And then I'll press enter. And then I [919:41] will write get push private repo main. [919:47] And here I'll check the spelling and [919:49] then I'll press enter. So now let's go [919:52] to our Jenkins UI and I will quickly run [919:56] this and I'll go to stages and by the [919:59] time this is running let's go to our [920:04] terminal and see what is the status of [920:06] our cluster. So the cluster is still [920:09] being created. Let's again go back to [920:11] our browser and build 16 has failed has [920:17] failed before even starting. Why is this [920:21] console output update deployment? Oh, [920:26] hold on. [920:28] This should be in single quotes. That is [920:32] what happens when you do things in a [920:35] hurry. Now the color coding has [920:38] returned. Right? There are two [920:40] parentheses after this. So this should [920:42] now work. Let me go here and then get [920:47] add get commit get push. [920:52] Now let's come back to Jenkins and here [920:55] I will build now and then I will go to [920:58] stages. Okay, this looks good to me now. [921:03] So now what I want to show you is how it [921:05] has modified. So let's go to our [921:08] terminal. And here I want to go to [921:12] agent. Yeah. And if I do ls, [921:16] you see deploy svc exists. Now [921:19] previously it did not because it is just [921:23] this stage that we added this. Now I'll [921:25] go to this one and it has updated the [921:28] deployment. So this has ran [921:31] successfully. But how do we verify that? [921:34] So if I come here, if I cat into this [921:37] deploy.yamel, [921:39] you see here this has changed. This was [921:43] latest in our source code. But the [921:46] checkout that Jenkins has performed the [921:48] file here in our temporary workspace has [921:52] 17. I hope this is making sense and you [921:55] are within yourself saying, "Ah, this is [921:58] how it all works." And now let's go to [922:01] our terminal. We are already there. [922:03] Let's see if cluster has been created or [922:06] not. So cluster is still not created. [922:08] Let's do one thing. I want to add that [922:10] cluster step as well. So here is the [922:13] cyan brace. I will add another stage. As [922:16] long as you follow these curly braces [922:18] and the color coding, you will never [922:20] face problems with your you know Jenkins [922:23] files and their syntax. And anyway, if [922:25] you face any problems, you can always [922:27] give it to you know AI tools to you know [922:30] format your files. It is all doable and [922:34] then I can have here [922:37] deploy to Kubernetes. Correct? So this [922:41] is fine. Now I want you to know one more [922:45] thing. We will be deploying these onto [922:47] our Kubernetes cluster. Right? So our [922:50] Jenkins agent will be using cubectl [922:53] client to run cubectl apply commands. [922:55] Correct? But our Jenkins agent does not [922:59] have cubectl installed. So let's do that [923:02] installation now. So I'll go to my [923:05] browser and I will close this and then [923:10] here is yes [923:12] stage four is done. We are on the last [923:16] stage right now and that stage is deploy [923:19] to AW. We have already discussed this [923:22] and then we have done this as well and [923:26] then it is deploy to Kubernetes. [923:29] So first I want to check the [923:32] installation steps. Yeah, install [923:34] cubectdl on Jenkins agent. So I'll copy [923:37] this. I will go to my terminal. I will [923:41] go to my agent and then I will no no [923:46] [clears throat] woron. I will exit. I [923:48] have to run this as Ubuntu user because [923:50] Jenkins is not part of the pseudo group. [923:54] And then I will run this. This will tell [923:58] me whether the second one will tell me [924:00] whether Jenkins can access and Jenkins [924:02] can indeed access Jenkins user can [924:06] indeed access the cubectl client. Once [924:09] we have done that, let's scroll up. I [924:13] want the cluster to be created. Okay, [924:16] the cluster has created. [924:17] Congratulations. So, let's go to the [924:20] CLI. And here I want to show you a few [924:23] things before we move forward. Elastic [924:26] Kubernetes service clusters. And if I [924:29] click here, if I go to [924:33] tags, you see the tags that we supplied [924:36] are automatically added here, which is [924:39] what we wanted. So if I go here, if I [924:43] now run cubectl get nodes, then I should [924:47] see two nodes because in my EKS config [924:51] file, I asked it to create two nodes. [924:56] Minimum nodes two and desired capacity [924:58] is also two of 20 GB each. So you see [925:01] two nodes are there for you and they are [925:04] running version the cubelet in my nodes [925:07] is running version 1.32.9. [925:10] So this is understandable. Now what I [925:13] want to do is I have created an EKS [925:17] cluster. I have installed cubectl [925:20] utility in my Jenkins agent. But but but [925:26] I want to run this command. So let's go [925:30] to our Visual Studio Code and I want to [925:35] explain this to you first here. So we [925:38] have this and then I will add steps here [925:43] and the steps that I want you to perform [925:46] dear is these. So let me paste these [925:50] steps. Okay, just give me a moment. If I [925:54] come here, these are the steps I want to [925:57] perform. So these are hefty steps. [925:59] Firstly, I am running multiple commands [926:03] in this shell, not a single command. [926:05] Previously I was running single [926:07] commands. Hence I was providing here I [926:09] provided shell two times. But here I am [926:12] running a chunk which is why I have [926:14] added all these under a single. So you [926:17] have sh here then I have three single [926:20] quotes. I have added them. When you are [926:22] running multiple command and you want to [926:23] invoke the shell only once, you can use [926:26] this. And this one is the preferred way [926:28] to do it. And then I'm saying AWS EKS [926:34] update cube config. Understand [926:37] my Jenkins agent will be running the [926:41] cubectl commands. Here [926:44] I was running the cubectl commands as [926:47] vun. Jooshi user. If I write AWS [926:51] STS get caller identity [926:56] then it is varun.jooshi [926:59] that was running the commands and [927:02] vun.jooshi has administrative level [927:04] privileges which is why he could see [927:07] what all are the nodes that are running. [927:09] Okay, using cubectl commands wun could [927:13] see what the cluster has. If I run [927:15] cubectl get pods, I can see this because [927:19] Wun is by default Wun. Jooshi is by [927:23] default [927:25] have administrative access to the [927:27] cluster. He's part of master's group and [927:30] we'll see what is that later. But for [927:32] now understand that user Vun. Jooshi is [927:36] able to do that. But our Jenkins agent [927:40] has one identity and that identity is [927:44] the role that we associated with [927:46] Jenkins. Right? In Jenkins agent, we [927:49] associated a role with our Jenkins [927:53] agent. So if I go here, if I go to [927:55] Jenkins agent, if I go to actions, [927:57] security, modify IM roles, then we added [928:02] Jenkins agent role to our Jenkins agent, [928:06] which is why Jenkins agent EC2 instance [928:09] was able to access Amazon ECR. Now we [928:14] want Jenkins agent role to access our [928:20] cluster. Now if you are familiar with [928:24] Kubernetes concepts, you know there is a [928:28] cube config file that is the source of [928:30] authentication that your cubectl client [928:33] uses to interact with your cluster. And [928:38] now what we are doing is in this step [928:41] what we are telling our Jenkins agent is [928:45] we are creating its cube config file. So [928:49] we are saying update the cube config [928:51] file and you update that cube config [928:54] file for a cluster that is in AP south 1 [928:58] that is in Mumbai region. The name of [929:00] that cluster is cloud with var dev [929:03] secops EKS because that is what we named [929:06] our cluster. So keep that in mind. And [929:09] then I am saying the cube config file [929:12] will be stored here. So for this to run [929:16] this command what I want is I want AWS [929:21] CLI to be installed and then I want to [929:24] update these details. In the second step [929:27] I'm saying cube cuttle create namespace [929:30] this and then I'm saying apply this [929:33] manifest file. Now to do these things my [929:37] Jenkins agent role must have access to [929:41] my Kubernetes cluster. so that it can [929:45] perform [929:46] namespace creation, it can perform [929:49] deployment creation and service [929:52] creation. But we have not allowed our [929:56] Jenkins role to do those things yet. [930:00] Hence, we first must do that. So, let's [930:06] start with it. What I have done is I [930:08] have added AWS cube config policy. Now [930:12] to update this information [930:15] your Jenkins agent role must have access [930:19] to your cluster. It has to perform [930:22] describe cluster action. For that we'll [930:26] have to create a policy and attach this [930:30] policy. So I'll copy this. I will go to [930:33] my browser and then in here I will go to [930:39] AM roles. Roles are here. Jenkins agent [930:41] role and here what I will do is I will [930:45] add permissions and this time let's [930:47] create an inline policy. Now inline [930:49] policy can only be associated with one [930:52] entity. That's what I'm doing here. I [930:54] will choose EC2 here and then I will say [930:59] I want to paste a JSON and then I will [931:02] paste that JSON here. Understand that to [931:06] run this command to run this command [931:11] your Jenkins agent must have access to [931:15] this cluster. You are running EKS [931:18] specific command. Your Jenkins agent [931:20] does not have access to Amazon EKS yet. [931:23] It just has access to Amazon ECR. You [931:27] can only take actions on services for [931:30] which you have been granted access. Now [931:33] in this step I'm granting that access to [931:36] this Jenkins agent role so that it can [931:39] run this command make a note that this [931:44] still cannot be performed and I'll show [931:46] you why and this time I'll write create [931:49] policy policy name is required temp EKS [931:54] node access okay so that I know that [931:56] this is a temporary policy I'll create [931:59] this and this is now added do this. Now [932:03] let's do one thing. I will come here and [932:06] then I will try to run this. Okay, I'll [932:09] come here first and then let's do add [932:13] and then let's commit and write here [932:17] deploy to EKS and then let's write get [932:22] push private repo main. And once we have [932:26] done this, we can come to our browser [932:31] and go to Jenkins screen. And here we [932:34] can run build now. And once you do that, [932:38] you will see it has started running. So [932:40] let's go to stages and let's see how [932:43] this pans out for us. If firstly the [932:46] deployment will fail. I'll am telling [932:48] you right away. But let's see the logs [932:50] to understand why it has failed. It has [932:53] failed. And if I come here, it's saying [932:56] added new context. And this is the [932:59] important part. You must be logged into [933:01] the server. You are unauthorized. It is [933:05] using the correct cube config file. It [933:07] is not saying that it failed in this [933:10] command. This command worked [933:13] successfully. But when it tried to run [933:16] this, it received an error that you are [933:19] unauthorized. And if you have watched my [933:22] Kubernetes authorization and [933:24] authentication lectures, this will make [933:28] so much sense to you. And we took the [933:30] example of Jenkins itself in those [933:33] lectures, especially in the service [933:34] account lecture. And now let's do one [933:37] thing. Let's come back here and fix the [933:40] permission problems. Now before I do [933:42] that, I want you to know one thing. When [933:44] I talk about AWS, it is the AWS O config [933:50] map that handles the authentication [933:54] part. I'm choosing my words very [933:57] carefully. AWS O is the config map that [934:01] handles the authentication part. So you [934:04] must first authenticate to the [934:06] Kubernetes cluster and then if you have [934:09] authorizations then you can perform [934:12] certain actions here. We are just [934:15] updating the cube config file Kubernetes [934:18] cluster. [934:20] We have not yet authenticated. We have [934:22] not yet authorized to perform any [934:26] actions in there. So we have to do two [934:28] things. First we have to fix the [934:29] authentication and then we have to work [934:32] on the authorization part. First we will [934:35] work on the authentication and AWS o is [934:38] the config map that takes care of the [934:43] authentication. So if I write cubectl [934:46] edit cm - n cube system and then aws o [934:54] is the name of that config map. You see [934:57] here we say map roles. Right now this [935:02] section that is added here this is [935:06] allowing your nodes to join this cluster [935:10] because this exists your nodes can join [935:14] the cluster. But Warun how come you are [935:17] able to access your cluster if you run [935:21] cubectl get pods you have not added [935:24] yourself anywhere and you must be logged [935:27] in into the server. Hold on. I think [935:30] just give me a moment. I used this one [935:33] to. So if I run cubectl get pods, I am [935:37] able to run this command. And here it [935:39] was not running because here my identity [935:41] is not that. If I run AWS STS get caller [935:47] identity. This failed because my [935:50] identity is not varun. It is varun. [935:52] Jooshi. But it is not for the account [935:54] that we are working in. So we are [935:57] working in the account which has at the [936:00] ending 648. Okay. But this CLI window [936:05] which I am in the terminal this is [936:08] working on an AWS account that has 470 [936:11] at the last. Although user is same but [936:13] it's a different account. But if I talk [936:15] about this, if I run AWS STS get caller [936:20] identity here, then this is using the [936:24] 648 account and I used this user to [936:27] create my cluster, which is why this [936:29] user has by default access to anything [936:33] in this cluster. But this piece is not [936:37] shown in the AWS O config map just for [936:41] your information. It is invisible. It's [936:44] like a golden key that the master has. [936:47] Now we want to create keys for other [936:50] users. And that is what we are going to [936:53] do. We are going to add entry so that [936:57] our Jenkins agent role can authenticate [937:01] to this cluster and let's do that. So I [937:04] will run that command again. cubectl [937:07] edit cm and then I will go into insert [937:10] mode and then I will come here and I [937:14] will go back here. I'll go back to [937:17] visual studio code now because I have to [937:19] copy the part that I have to add here [937:21] where I have added that it is AWS oedit. [937:25] So I will copy this. I'll explain this [937:27] to you in a while. It's very important [937:29] you know and then I will go to my [937:31] terminal and here I will just paste [937:34] this. I have to indent this properly [937:36] because you all know how critical YAML [937:40] is about indentations. So this has to be [937:43] just underneath this username can come [937:46] here group come here and we are indented [937:50] properly. So what I'm saying is the role [937:53] ARN is Jenkins agent role and you know [937:58] how you fetch your answer. You go here. [938:01] You go here and ARN is this. And I have [938:06] just added this ARN here. And I'm saying [938:10] Jenkins agent role. I'm saying username [938:14] is Jenkins agent role and groups is [938:17] system authenticated. If I would have [938:20] added masters then that would have given [938:22] you know all privileges. But I'm saying [938:25] this is an authenticated user for me. [938:28] Now once I have done this, this is the [938:31] authentication piece. I have cleared the [938:33] authentication piece. Now I have to walk [938:36] in through the authorization piece. And [938:39] for authorization, I have an arbback for [938:43] you. And here is cluster ro and cluster [938:47] ro bindings. Again go to my CKA playlist [938:51] and search for authorization. Learn [938:54] about these only then it will make sense [938:56] to you. Here I'm creating a cluster [938:58] role. That cluster role is called [939:00] Jenkins deployer role. And that Jenkins [939:05] deployer role will have access to [939:07] deployments plus replica set. It will be [939:10] able to perform all these actions. Then [939:12] it will have access to pods all these [939:14] actions access to services all these [939:17] actions. Access to name spaces all these [939:20] actions. And why I'm giving access to [939:22] namespaces because we know that in our [939:27] Jenkins file we are also creating a [939:30] namespace here. So and then finally we [939:33] are creating a cluster role binding. Now [939:35] cluster role binding binds user service [939:39] accounts or groups to cluster roles. Now [939:43] here this would be name spacewide and if [939:46] you want more information about this go [939:48] and watch my authorization lecture. So [939:51] here we are saying kind is user name of [939:54] the user is Jenkins agent role must [939:57] match the username in AWS O. So if I go [940:00] to my CLI again and I run a describe on [940:06] this config map then you would see that [940:09] the username is Jenkins agent role. The [940:12] Jenkins agent role is here and then the [940:15] role reference which is the role that [940:17] you want to associate with this user. [940:20] That role reference is Jenkins deployer [940:23] role. And where is Jenkins deployer [940:24] role? It is here. So this user will have [940:28] all these accesses. So what I will do is [940:31] I will apply this cluster role. So I'll [940:34] go to the terminal and then here I will [940:39] press ls and then I will run cubectl [940:42] apply - f I have to apply what I have to [940:47] apply cluster ro bindings.yamel YAML and [940:50] once I have done this that cluster role [940:53] and cluster cluster role binding would [940:56] have been created for me. Now I will [941:00] [snorts] come to my Jenkins pipeline [941:02] again to see what do I have to do next. [941:05] This will also work. This will also [941:08] work. Now we have edited the AWS O [941:11] config map. So I think we have done [941:13] whatever we wanted to do. Now let me [941:15] quickly just commit this. Okay, I will [941:18] run get add get commit- m deploy to EKS. [941:24] No. Oh, here we we have just done the [941:26] changes at cluster level. We have not [941:28] modified anything in our Jenkins file. [941:31] So, this should work. And just let me [941:33] quickly see whether this file is already [941:36] here. Docker file is already there. We [941:38] don't need any of these. So, let's apply [941:41] this. I will go to our browser and then [941:44] here I will go here and then let's run [941:50] this thing build now and then I will go [941:53] to stages and this should show me what [941:56] is happening. [942:07] deploy to Kubernetes deployment [942:10] successfully rolled out. So if I come [942:13] here, I go to this terminal here, if I [942:18] run cubectl get pods, I don't see [942:22] anything. But if I write the name space [942:25] name which was cloud with v jo dev sec [942:31] ops then you would see I have two pods [942:34] running and I want to show that these [942:37] pods are running on different nodes [942:39] because we have applied that setting no [942:43] topology spread constant. So if I write [942:45] hyphen o wide in here then you would see [942:48] that this is deployed onto this node and [942:51] this is deployed onto this node. So if I [942:54] go to my browser I go to instances I hit [942:59] refresh then you would see that two more [943:01] instances should appear here for my [943:03] cluster and these are the two the first [943:05] one and the last one. We are good here [943:08] and both of these are deployed into [943:10] different availability zones. So if I [943:12] check here 1 A and 1 B rest of our infra [943:17] resides in availability zone 1 A. So we [943:21] are good here. Now let's check the [943:23] accessibility of our application. So I [943:26] will copy this public IP address. I'll [943:28] go here. I will write http col// [943:33] I'll add that and then it is 31,000. [943:36] I'll press enter. [943:39] It is not working. Can someone tell me [943:41] why it is not working? It is not working [943:43] because we have not allowed port 31,000 [943:47] in our security group. So let's see what [943:50] is the security group that is associated [943:52] with these instances. So if I go to uh [943:56] security and then I see these are the [943:59] two security groups. Now I can edit [944:01] either of these but I will prefer this [944:03] one. I'll go here edit inbound rules. I [944:07] will add an inbound rule. I will keep it [944:10] custom and here I will add 31,000 and [944:14] then here from anywhere I can save this [944:17] rule and this should now work. Yeah, [944:21] cloud with varos simple dev sec op demo. [944:26] So we have deployed an end toend [944:29] pipeline. Now just to improve certain [944:32] things what we can do is we can have [944:34] certain post actions in our pipeline. So [944:37] that is exactly what we will do now. So [944:41] I will come here. I also want to show [944:44] you the logs for this one. So I'll go [944:46] here. Here. And then you see rolled out [944:50] was successful. And this worked because [944:53] in our pipeline here we said if we gave [944:57] 60 seconds in case this was unsuccessful [945:01] then we would roll it back. [945:03] [clears throat] [945:04] Now we have done this. This is this [945:07] stage. What I want to do is outside of [945:10] stage I want to add certain post [945:12] actions. Again you can have post actions [945:16] within stages but you can have them for [945:18] the entire pipeline. Now this action I [945:22] want to have it for the entire pipeline. [945:24] So stages were starting here but [945:28] pipeline was starting here. So I want to [945:31] add this within the pipeline block. So [945:35] this closing curly brace was for this [945:39] stage. This closing curly brace is for [945:43] the stages section. [945:46] You know you see this highlighted here. [945:48] And then after this I want to add our [945:52] post actions and this is the post [945:54] action. So if it is success pipeline is [945:57] successful we will print this and [946:00] failure we will print this. Always we [946:02] will print this no matter what. We'll [946:05] say finished. We'll save this. We will [946:08] go to iterm. We will go to this screen. [946:12] We will add this. Then we will commit [946:17] this and we'll add a meaningful message [946:20] post actions. [946:22] And then we will just simply push it [946:26] using this command. Now if I run this [946:30] pipeline, we will also see post actions [946:33] and that is what we want to see now. So [946:35] you can click on stages to see oh I have [946:40] a mistake. Let me go to the console [946:42] output workflow. [946:45] Hold on. I have a curly brace error. Oh [946:50] this there is no curly brace that is [946:53] closing this pipeline. Right. This is [946:55] this has turned red which is why. So, [946:58] oh, then I'll add this here. Okay, try [947:02] speaking getting recorded at the same [947:05] time. It's not easy as it seems. And [947:08] then I have this. This should work now. [947:11] So, I'll go to my terminal. And then I [947:15] will add and then I will commit. And [947:18] then I will push. And then I will go [947:21] back crying. And then I will click build [947:25] now. And this should now work. [947:27] I'm hopeful. Okay, it has started at [947:30] least. So, something would work, right? [947:33] I'll go here and this should work. And [947:35] by the time this completes, I want to [947:37] walk you through some more things that [947:40] you would be dealing in production. [947:42] Stage nine, post actions. Yeah, post [947:45] actions. Now, see why would you use post [947:48] actions? Now, I have given a very simple [947:50] example. You could be using post actions [947:53] to send emails. You can you know [947:55] regardless of what happened inside the [947:57] stage you can send deployment [947:59] notifications push artifacts tag builds [948:02] as stable there are so many things [948:04] create Jira ticket log failure metrics [948:06] in case of failures and so on. So you [948:09] could be using all these options and [948:11] maybe as I create another project or [948:14] another demo I will be using some of [948:16] these but I want you to keep these [948:19] things in mind. If it's a success you [948:21] would would want to send an email that [948:23] the deployment succeeded. You can access [948:26] the application here and so on. Then [948:30] what we have been doing up till now is [948:33] we have been clicking on build now for [948:36] our pipeline to run. But you can always [948:40] create web hook and when you create web [948:43] hook whenever you push onto your [948:46] repository a pipeline will trigger [948:49] automatically. I have jotted down all [948:52] the steps to do that. It's extremely [948:54] simple. I would highly highly encourage [948:57] you to do that. And then lastly I want [949:01] to talk about a few things. This [949:04] pipeline is production ready but there [949:08] are major chances of improvement. You [949:11] can improve this. First is deploying [949:14] Kubernetes inress. So I have detailed [949:18] lecture for that. You can refer that [949:20] multibranch pipelines. You will never [949:23] ever be pushing directly onto main. All [949:26] these main branch developer development [949:28] branch, production branch, whatever g [949:31] strategy you are using. It could be [949:33] trunk based or it could be git flow [949:35] based. But your main branches, your [949:38] development branches, your staging [949:41] branches, all these will be protected. [949:43] You will only be allowed to push onto [949:45] your feature branches and then a PR will [949:48] be raised. I have covered this in depth [949:51] in my multibranch pipeline lecture. So [949:54] do watch this and try to implement the [949:56] pipeline that we just implemented using [949:59] this. Before I move to the next one, [950:01] let's see what is Jenkins doing. So the [950:04] deployment is successful and as [950:07] suggested post actions are here. It says [950:10] build 21 finished, build 21 succeeded. [950:14] That is what we wanted to see. So let's [950:17] continue what we were discussing. So [950:19] here I have not introduced Argo CD [950:21] because I don't want to cover any topic [950:24] that I have not covered in this channel. [950:26] So I will soon be covering Argo CD and [950:28] once I do that I will be creating [950:30] projects that would be leveraging Argo [950:32] CD. We have used private IP addresses [950:36] somewhere but we should be using DNS [950:38] names. Use image digests. Now we have [950:42] provided tags but in production [950:44] implementations what you would see is [950:47] you use image digests instead of tags [950:51] and you see here our latest tag is being [950:54] replaced as we move up with the [950:56] pipeline. So we started creating our [950:59] images from build number 15 and we [951:02] completed this demo in build number 21. [951:05] So if you go here, this is our build 21, [951:08] which is why you see 21 is the latest [951:11] one. And here is the digest for this, [951:13] which is what I was saying. You would be [951:16] typically using digests in production [951:19] implementations, not these tags because [951:23] these are indeed immutable. You you use [951:27] digests in production. Then other [951:31] improvements that can be done signed [951:33] images plus policy enforcement. There [951:35] are things like Kyverno that can be [951:37] integrated centralized secrets manager. [951:39] So right now we have used credential [951:42] bindings. Bindings is one step in [951:45] securing your credentials but you can [951:48] also use a product that is meant for [951:50] secrets like Hashi Corp vault or AWS's [951:53] secrets manager. Then observability [951:56] integration. So we have not integrated [951:58] any cloudatch data dog or anything. We [952:00] are not doing any monitoring of sort. So [952:02] you could be improving this pipeline by [952:04] this policy as a code. We have not used [952:08] you know any security related policies [952:11] or any Helm or Terraform for deployment. [952:13] You could be using those things. Argo [952:16] rollouts I've already covered automated [952:19] cleanups and cost optimization. So we [952:21] have not done these things as well. You [952:23] could be doing these and there could be [952:26] plethora of other improvements that you [952:29] already are thinking. So we can [952:31] integrate all of these things but I [952:33] assure you a demo with this much depth [952:37] will clear a lot of your doubts. Now I [952:41] will go to my diagrams and then I will [952:43] wrap up this lecture. So we have created [952:46] this robust pipeline and with this on [952:50] screen I wanted to end this lecture. So [952:52] that wraps up dev secc Ops mega project [952:55] where we built a complete production [952:58] style CI/CD pipeline using Jenkins TV [953:02] sonar cube Amazon ECR and multi-AZ [953:05] Amazon EKS cluster. You saw how real [953:09] teams use security gates SAS image [953:12] scanning immutable artifacts am with [953:15] arbback rollouts checks and automatic [953:18] rollbacks to run safe deployments at [953:21] scale. If this session helped you, drop [953:24] a quick thanks or hi in the comments and [953:26] subscribe for more advanced EKS GitHops [953:30] Dev Sec Ops content that is coming soon. [953:33] I will see you in the next session. [953:36] Thank you very much. Hi, I am Vun Jooshi [953:39] and welcome to the final lecture of our [953:42] Jenkins basics to production course. [953:45] Today we cover one of the most important [953:48] Jenkins features, shared libraries. [953:52] Instead of repeating the same build, [953:55] test, deploy steps in every Jenkins [953:58] files, shared libraries let you [954:02] centralize common pipeline logic and [954:06] reuse it across all your projects, which [954:10] is super cool in my opinion. We will [954:14] understand why these exist, how the [954:18] directory structure works, how to [954:21] register a private library in Jenkins, [954:24] and then wire two private apps to the [954:28] same shared steps. We will finish by [954:32] changing a couple of lines in the [954:34] library and watching how both [954:37] applications update instantly. a perfect [954:41] way to close this series. So let's get [954:45] started. Why Jenkins shared library? Now [954:50] remove the name Jenkins shared library [954:53] from your memory for some time and let's [954:56] understand what we have been doing thus [954:59] far. Now let's say you are dealing with [955:02] 20 applications and these 20 [955:05] applications each of these will have a [955:09] repository and then all of these [955:12] repositories will also hold a Jenkins [955:15] file and there could be multiple steps [955:20] that are repeated in all of these [955:23] Jenkins file. But Vun why would I repeat [955:27] my steps? I want you to know that when [955:31] you are dealing with let's say a project [955:33] and that project is supporting let's say [955:36] 20 applications [955:38] you won't be creating different image [955:42] registries for all these 20 apps. You [955:45] will essentially have a common image [955:48] registry and in that image registry you [955:52] will have all the container images be it [955:55] for development workloads, staging [955:57] workloads or production workloads for [956:01] all those 20 applications and not just [956:04] image registry. Think of it like that. [956:07] the Sonar Cube server, you will have a [956:10] common Sonar Cube server across all [956:13] these 20 apps. Privy. Now when you are [956:17] performing a scan, let's say you are [956:20] flagging high and critical level [956:23] vulnerabilities in your container images [956:26] and for that the command that you run is [956:30] common. Hence that command is repeated [956:35] in all the 20 Jenkins file. Let's say [956:39] for example now you are flagging your [956:43] high and critical level vulnerabilities [956:45] in your images using a specific tree [956:48] command that we saw in our mega project [956:52] which is the previous video. And now [956:54] your manager comes to you and tells you, [956:58] "Okay, Abishek, for now we were only [957:02] flagging high and critical level [957:04] vulnerabilities, but now I also want you [957:08] to include medium level vulnerabilities. [957:12] If Trivy finds any medium level [957:15] vulnerabilities, [957:17] you fail the pipeline." Now in that case [957:20] if your command is repeated 20 times in [957:24] those 20 Jenkins file then you would [957:27] have to make modifications at 20 [957:30] different places and that is brutal. Now [957:34] someone is thinking why can't I [957:36] automate? Now developing an automation [957:38] for just 20 changes might feel [957:41] overwhelming. So let's say you decided [957:43] to do all those changes manually. What [957:47] if there are errors in copy paste while [957:50] you are doing this manually? This [957:53] increases the risk. Hence, what if you [957:58] centralized that command at a central [958:01] place and then in all of those 20 [958:05] Jenkins file, you were calling that [958:07] central place. Wouldn't that be awesome? [958:10] And if you have to make any changes, you [958:13] just make the change in that central [958:15] place. And since all of those Jenkins [958:18] file for your 20 apps is already [958:21] referring to that shared location to [958:23] that central location, [958:25] your work is done. You just make the [958:28] change at one place and you are done. [958:31] And I just gave you example of image [958:34] registry, sonar cube server, tree [958:38] commands. But imagine docker push [958:40] command, docker build commands. All of [958:43] these are common in most of the [958:46] pipelines. You do not want to repeat [958:50] things again and again when you can have [958:54] them centralized at a single place. And [958:57] if you are from a programming [958:58] background, I want you to immediately [959:01] contrast this with functions [959:04] for things that you have to do again and [959:07] again. You don't write code for that [959:11] same thing again and again. You just [959:13] create a function and then you call that [959:16] function whenever you want to do that [959:18] specific thing. And Jenkins shared [959:23] library [959:24] helps you do the same thing. It allows [959:28] you to extract all the commonalities and [959:32] dump those commonalities into a central [959:35] shared place from which you can call [959:39] those things. As simple as that. I hope [959:45] this has given you a highlevel view of [959:49] why we need something at a central [959:51] place. Since I already have a slide for [959:54] it, let's read what we are saying here. [959:57] Too many pipelines repeat identical [960:00] CI/CD steps everywhere. Changing a [960:03] standard step requires editing every [960:07] project. So you have to edit 20 Jenkins [960:11] file in our example. And here I [snorts] [960:14] was just saying 20 apps. But let's say [960:16] each application has four microservices. [960:20] Then 4 into 20 you are dealing with 80 [960:24] Jenkins files and imagine editing those [960:28] files. It's cumbersome. You don't want [960:32] to do that. You want a way via which you [960:35] can make the changes at just one place. [960:38] And all those 80 microservices, all [960:41] those 80 Jenkins file can just refer [960:46] that central location and it will find [960:48] out whatever is needed and how all of [960:52] these things are wired. We will see that [960:55] diagrammatically. But at this point of [960:57] time, I just want you to understand why [961:01] something like that was needed. Because [961:04] we DevOps folks do not just handle 20 30 [961:09] microservices. You will be you will be [961:12] part of projects where you are dealing [961:14] with hundreds thousands of microservices [961:18] and in those scenarios you do not want [961:22] to deal with single Jenkins file. You [961:24] will have a central location which will [961:27] be holding all the commonalities. [961:30] Inconsistencies appear when teams copy [961:33] paste. Like we discussed, updating tool [961:36] URLs or credentials become slow and [961:39] risky. Now imagine you have sonar cube [961:43] at a specific IP address. And for this [961:45] example, let's say you have configured [961:48] your Jenkins file to reach out to an IP [961:50] address which you should never be doing. [961:53] You should be using DNS names always. [961:56] But let's say that IP address changes. [961:59] then you will have to make changes in 20 [962:02] different places or 80 different places. [962:04] You don't want that. And tool URLs can [962:09] change. So keep these things in mind [962:12] whenever you are architecting a CI/CD [962:14] pipeline or whenever you are [962:16] architecting a solution. All these [962:18] things scalable. Is the solution I am [962:21] proposing scalable? The client today [962:24] wants 20 microservices. But shall I ask [962:27] a question to client? How about 6 months [962:30] down the line? Do you intend to deploy [962:32] more microservices? Do you intend to [962:35] rearchitect the application that are [962:38] traditional VMs into microervices at a [962:42] later stage? So all these plannings, all [962:45] these questions should come to your mind [962:48] even when the client says I just have 40 [962:50] microservices or 20 microservices. [962:53] Always ensure the solution that you are [962:57] proposing can be scaled. And this is not [962:59] just applicable for Jenkins shared [963:01] library here. It is applicable for cloud [963:04] environments, for traditional virtual [963:06] machines, for on premises, for CI/CD, [963:09] for Kubernetes or any IT solution you [963:12] are dealing with or even your personal [963:16] business. Is that scalable? How do I [963:19] scale that? And if I want to scale, what [963:23] are the other things that I must ensure [963:26] must be in line when scaling becomes [963:30] necessary. [963:31] Last one is [963:33] central teams struggle to enforce [963:36] organizationwide [963:38] pipeline standards. Now if you have a [963:40] central place, you can always enforce [963:44] the usage of shared libraries. So that [963:48] once you have defined that this is the [963:50] sonar cube server that everyone must be [963:53] using then they just call that shared [963:55] library and they can use that sonar cube [963:58] server. Now this is important for your [964:00] security teams because security teams [964:02] would want that anyone that is running [964:06] any pipelines their pipelines must be [964:09] scanned for SAS for code quality code [964:13] coverage and so on. For that you must [964:17] enforce these things and all these [964:20] results must be dumped onto a single [964:23] server or single sonar cube server for [964:26] example and then shared library is [964:29] something that will help you achieve [964:32] that. Now all these points cover why do [964:36] we need Jenkins shared library. Now [964:39] let's see what is a Jenkins shared [964:43] library. A Jenkins shared library is a [964:46] central like we said version controlled [964:50] repository. I want you to know right now [964:53] itself that the shared library that you [964:57] are creating will be updated as your [965:01] application progresses. Maybe you added [965:03] a sonar cube step. So you will be adding [965:05] that sonar cube step in your shared [965:08] library. Maybe now you want to scan [965:11] medium level vulnerabilities as well. So [965:13] you will be editing your shared library. [965:15] Now since it is version controlled, you [965:19] as an administrator have a choice [965:22] whether to use the latest version of the [965:24] shared library or a version that was [965:28] pushed on let's say 21st of July. So it [965:31] gives you that level of freedom. And [965:33] then while configuring Jenkins shared [965:36] library, we will see what all other [965:38] options are available. You can even [965:41] supply a commit hash. So I want my [965:44] pipeline to run with the Jenkins shared [965:47] library status as it was on this commit [965:51] hash. So this gives you a lot of [965:53] flexibility. Maybe for application one [965:57] who are using the same shared library a [966:00] specific version or a specific tag does [966:03] not perform medium level scans and that [966:06] is exactly what is needed by app one. [966:08] But app 2 requires even medium level [966:13] vulnerabilities to be scanned. So that [966:15] is possible because these shared [966:18] libraries are version controlled and [966:20] Jenkins gives you the freedom to choose [966:24] the version of the shared library that [966:27] you want to deal with. So let's read [966:29] this again. Is a central version [966:32] controlled repository for reusable and [966:35] standardized pipeline logic. A central [966:38] repo to store reusable pipeline logic. [966:41] Standardizes CI/CD steps across many [966:44] projects. Version control library [966:46] pipelines can be imported by Jenkins [966:50] files. Now this is interesting. How does [966:55] my pipeline job knows which library, [967:01] which shared library to use? because I [967:05] can have multiple shared libraries but I [967:08] also want app one to use that shared [967:11] library but app 2 to use that shared [967:14] library. How do I do that? You define [967:17] that in the Jenkins file itself. You [967:20] define that this Jenkins file should be [967:24] using this shared library. That Jenkins [967:27] file should be using shared library too [967:30] and so on. Next is single source of [967:33] truths for build, test, deploy, scan and [967:38] image ops. Now there could be multiple [967:40] stages in your CI/CD pipelines. The [967:44] steps or stages are not limited to what [967:46] you see here. Maybe you are doing [967:49] something else as well. If there are [967:51] commonalities across multiple [967:53] applications or multiple microservices, [967:56] you can always take the common thing out [968:00] into a shared library and that's the [968:03] reason it is called shared library [968:06] because it is shared between multiple [968:09] Jenkins files and those Jenkins files [968:12] could be pertaining to your standalone [968:15] applications or it could be for your [968:17] three- tier architecture or it could be [968:20] for your microservices. Now we have the [968:22] understanding of why and what of Jenkins [968:26] shared library. Now let's see how [968:30] Jenkins connects your apps through a [968:33] shared library. Now let's understand [968:36] what is happening in this diagram. We'll [968:39] be dissecting this step by step. [968:41] Firstly, we have two applications and [968:44] these two applications have certain [968:47] commonalities and we now want to use [968:51] Jenkins shared library here. Let's see [968:55] what all are the moving parts that are [968:57] there and how to wire these moving parts [969:01] together. So we have app one private [969:04] repository. As you know your application [969:07] code would be residing in private repos. [969:10] So the example that I'm giving here is [969:13] close to production. So app one private [969:16] repo is here. App 2 private repo is [969:18] here. And then we also have a repository [969:22] for the shared library. As I told you [969:25] shared library is version controlled. So [969:28] if it is version controlled, how it is [969:30] version controlled? Because it has its [969:33] own repository. In this example, we are [969:36] saying we have three repositories. And [969:39] in this architecture, all three [969:41] repositories are created in GitHub. All [969:44] three repos live separately in GitHub. [969:48] But Jenkins stitches them together [969:51] through the shared library. Now at the [969:53] very top you see Jenkins controller, [969:56] checks out these repos and loads the [970:00] shared library. Why I'm saying loads the [970:03] shared library? because it is the [970:06] Jenkins file in which you define [970:10] which shared library to use. Now I will [970:14] quickly give you a glimpse of what is in [970:17] app one private repo. It has the app [970:19] code plus it has the docker file but [970:22] plus it has a Jenkins file. Now docker [970:24] file may be or may not be there. If you [970:27] are dealing with non-containerized [970:29] applications and you are using your [970:31] CI/CD to deploy those then docker file [970:34] of course will not be there. But I am [970:36] taking an example which we will see in [970:39] the demo. Hence I also have the docker [970:42] file. Now let's see what all things you [970:47] will change in Jenkins file when you are [970:50] dealing with shared library. So first is [970:53] import the shared library and for [970:56] reference I have marked the step here [970:59] itself. Now understand if my Jenkins [971:02] file should be using and when I say [971:04] Jenkins file understand that your [971:07] application or your microservices will [971:09] have Jenkins file for them. So I am [971:13] talking about that Jenkins file that [971:15] would essentially be placed with your [971:18] application code in the repository where [971:21] your code is committed or pushed. So you [971:25] have Jenkins file here. Now in the [971:27] Jenkins file at the very top I mentioned [971:31] that hey Jenkins file this is the shared [971:35] library I wish to use. So here I'm [971:39] saying use my first shared library at [971:43] the rate main is you have to use the [971:46] main branch and we will see why this [971:48] matters once we move to the demo [971:51] section. So I have two demos for you. [971:53] This first demo is a simple one so that [971:56] you we can stitch the story together and [971:58] then second demo is a little advanced [972:01] one which you would see in production. [972:04] It's like production. Then second is [972:08] call the shared library function. So [972:11] first is import the shared library and [972:14] how we are importing this shared library [972:16] by defining this step at the very top of [972:19] our Jenkins file. Then call the shared [972:21] library function. I will come to this in [972:24] a little while. Now when you are [972:27] creating your shared library here is the [972:31] shared library. Since I told you that [972:34] our shared library is also private, it [972:37] is obvious that when you are configuring [972:39] your shared library in your Jenkins, you [972:41] will be supplying the repository URL [972:45] along with the credentials that Jenkins [972:48] must be using to access your shared [972:51] library private repo. So that is what [972:53] you will be doing here. Also, you will [972:55] be selecting the branch or tag and I [972:59] have missed one thing or the commit [973:02] hash. You can supply these things and [973:04] we'll see what these things are later. [973:07] For now, understand that the Jenkins [973:10] file has at the very top the library it [973:13] wishes to use. Now let's see what is [973:17] happening here. Shared library has a [973:21] defined structure. So for example here I [973:25] have shared library private. This has [973:27] three directories src, v and resources. [973:32] Now, right off the bat, v is the most [973:35] critical one for us devops folks. Now, v [973:40] is the directory that contains simple [973:43] top level groovy scripts that act like [973:46] functions. Now, this is a mouthful, I [973:48] know, but we will be dissecting it step [973:51] by step. In this lecture, we will be [973:54] only discussing the v one because this [973:56] is the one which is the most important [973:58] one. SRC and resources are for advanced [974:02] use cases. For example, you would use [974:04] src that will hold your groovy and java [974:07] classes for complex object-oriented [974:10] logic and resources. Anything other than [974:14] Groovy or Groovy files you will find in [974:17] resources section. So for example, if [974:19] you are using YAML, JSON, templates or [974:22] scripts, all of those will reside in [974:24] resources. Now I will not be covering [974:27] SRC and resources because this lecture [974:30] is meant to give you a view of shared [974:33] library and the things that are most [974:35] commonly used. If I would have covered [974:37] SRC and resources, this would become a [974:40] programming class and it will be too [974:41] complex. So I do not want to do that. [974:43] But I assure you once you have [974:46] understanding of what I'm teaching here [974:48] then you can go forward and conquer any [974:51] of these you know shared libraries and [974:53] you will understand what is happening. [974:55] So with that said let's see what is [974:58] there in V. Anything placed inside V is [975:04] treated as a global variable and becomes [975:08] directly [975:10] callable from your Jenkins file. Now [975:13] what do I mean? For example, if you see [975:16] this snip here, what we are doing is [975:19] this is my Jenkins file and I have [975:21] written build with Maven here. Now why I [975:25] have written build with Maven here? [975:27] Because inside the V directory I have [975:31] created a file that is named build with [975:34] Maven.groovy. [975:36] I'm not adding groovy here because [975:38] extensions you are not supposed to add [975:41] but you call this thing using the file [975:46] name without the extension. So if you [975:48] want to call this function that you see [975:51] here that is doing maven clean install [975:55] then you would essentially call that by [975:58] using the file name. This is a critical [976:01] piece to understand. You do not have to [976:04] specify anything that is inside. What [976:07] you do is you just call the file name [976:11] without the extension. And what will [976:14] that do? It will go to that file and [976:17] invoke the function. And what is the [976:21] function? The main function in anything. [976:25] Groovy inside. V is the call function. [976:29] And within the call function, we are [976:32] running a shell command that says mvn [976:35] clean install. Now I want you to imagine [976:38] with me right now. Let's say you have 20 [976:42] applications. All those 20 applications [976:45] are running MVN clean install. Your [976:50] manager walks to you and tells you I [976:53] don't want to run MVN clean install now. [976:56] I just want to run MVN clean package. [977:00] Now if you have been through Maven, you [977:02] will understand what I'm saying. But if [977:04] you do not have understanding of that, [977:06] go and watch that lecture. I have that [977:08] in this channel or just understand we [977:11] are changing certain command. So here [977:14] our manager says change this install to [977:17] package. I don't want you to run the [977:18] install just run package. So you will [977:21] just be changing it here and your [977:24] Jenkins file is already calling that [977:26] function from here already calling that [977:29] file from here. So once you change it [977:32] here all those 20 Jenkins file that are [977:36] using this shared library will be [977:38] running MVN clean package instead of [977:42] install. I hope this makes sense. Now if [977:45] you have understood up till here that is [977:48] all there is to Jenkins shared library. [977:52] Having said that let's see what we want [977:55] to discuss next. Each file must use the [977:58] Groovy extension. That is understood we [978:00] are using it. The file name becomes the [978:03] function name as I told you. So here I'm [978:06] calling it build with Maven because this [978:08] file is named build with Maven. We do [978:11] not include the extension when we are [978:13] calling it from the Jenkins file. [978:15] Recommended naming style is camel case. [978:18] So that's why you see the first letter [978:22] is lower case and then each word after [978:25] that starts with uppercase letter that [978:30] is camel case for you. Jenkins looks for [978:33] a call function inside each file. Like I [978:37] told you the main function that is [978:40] called is call. So you would see that [978:44] call is always there. This is the main [978:46] function that gets called whenever you [978:48] are calling a file. And then that main [978:51] function can have multiple sub [978:52] functions. But the main function that [978:55] gets called is the call function. So we [978:59] have this understanding. Now let's do [979:02] first a simple demo and then we will [979:05] move to a complex demo. So let's go to [979:10] our browser first and then I am creating [979:14] a Jenkins server along with you a [979:16] Jenkins controller. Now if you already [979:19] know the steps involved in Jenkins [979:21] installation you are good to go. You can [979:24] skip this section if you already have a [979:26] Jenkins up and running. If you don't you [979:28] can follow along with me. So I'll write [979:30] Jenkins controller because that is what [979:32] I am creating. I want to use Ubuntu. [979:36] Ubuntu 24.04 LTS is fine by me. And then [979:40] here I am choosing this instance C7i [979:44] flex large because this will make things [979:47] faster and I want that 2 vCPUs 4 GIB. [979:50] these instances which have free tier [979:52] eligible. I'm using these because I can [979:56] use these with free AWS credits. I'll [979:59] choose this one. Then here the key pair [980:02] that I wish to use is cloud with VJOS SJ [980:05] Mumbai. And then I will scroll further [980:07] down. I already have a security group [980:10] created and the security group that I [980:13] wish to use is Jenkins controller SG. [980:16] And if you have followed the previous [980:19] mega demo then you know Jenkins [980:21] controller SG only allows two things. It [980:24] allows port 22 from everywhere which you [980:27] will not be doing in production and it [980:29] allows 8080 that is the port on which [980:32] Jenkins listen and that also from [980:34] everywhere. So for now only two ports [980:38] are opened for the outside world. I'll [980:41] scroll down. I'll change this to 20 just [980:43] to be on the safe side. Although we [980:45] wouldn't be using this much storage. [980:47] I'll scroll further down and we are good [980:50] here. I'll click on launch instance. By [980:52] the time it launches this instance, I'll [980:54] go to my GitHub repository. I'll go to [980:57] Jenkins basics to production. That is [980:59] the repository for this lecture. And [981:02] then I will go to shared library. Now [981:05] whatever we'll be doing in any of these [981:07] demos or whatever theory I have [981:09] discussed is thoroughly covered in this [981:12] repository. So I would encourage you to [981:15] go through these and if you find this [981:18] helpful I would encourage you to start [981:21] the repository as well. So here I'll go [981:23] to lab install Jenkins controller. So [981:27] these things we have already done. I [981:29] will scroll down and then what I want to [981:32] do is I want to SSH into the machine and [981:34] then I want to give this machine a [981:37] logical host name so that we know what [981:40] we are dealing with. So I'll come here. [981:42] I'll go to instances. I will click on [981:44] this. I'll click on connect. Then I will [981:47] copy the command that can be used to [981:50] initiate the SSH connection. I'll go to [981:52] my terminal. I am in my home directory. [981:55] I'll cd into SSH. I already have the key [981:59] that I wish to use here. It is cloud [982:02] with varosh sjmumbai.pm and I already [982:05] have set the correct permissions and you [982:07] can follow the documentation that I have [982:10] shared in my GitHub notes if you want to [982:13] set the permissions as well. So I'll [982:14] press enter and I will click on yes this [982:17] is trust on first use. I will clear my [982:20] screen so that I have more real estate [982:23] and then I will copy this command to set [982:27] the host name. Now the host name is set [982:30] to Jenkins controller. After that I will [982:33] also set the time zone so that we can [982:37] study the logs properly. Now this will [982:40] set the time zone for this virtual [982:42] machine for this operating system. But [982:44] JVM also requires the time to be set [982:47] correctly. And JVM will be there once we [982:49] install Java 21. And we are installing [982:53] JDK. JDK is Java development kit. I keep [982:57] on mentioning this because I want [982:59] everyone to understand that there is a [983:01] difference between JRE and JDK. You [983:04] install JRE when you want to run Java [983:07] applications. But when you want to build [983:09] the Java applications, you also want [983:11] compiler to be there. and JDK is the one [983:14] that comes with compiler and other [983:17] essential tools that are required for [983:19] building a Java application or [983:21] troubleshooting a Java application. I'll [983:23] copy this and then I will paste this [983:26] here. So this will install Java for me. [983:29] Then I will scroll further down. Now we [983:32] will begin installing Jenkins. Now [983:36] installing Java is a prerec for Jenkins [983:39] which is why we installed Java. Now we [983:42] will be first adding the repositories [983:44] for Jenkins and then we'll be installing [983:48] Jenkins. So I'll copy all of these. I'll [983:50] go to my terminal. I'll let this [983:52] complete first and once it is I'll paste [983:55] the commands. Okay, I'll paste the [983:57] command. Now you can see that we have [983:59] all the things. Java compiler version is [984:02] 21.0.9 [984:04] and open JDK version is 21.0.9 which is [984:08] fine. I'll click on enter and then I [984:11] will move to my repository again. Here [984:13] is the part where I am setting the JVM [984:17] time zone. So I will be using this [984:19] non-interactive command to do this. You [984:21] can use either of these. Both of these [984:23] are doing the same thing. This will just [984:25] do whatever I want in just one click. [984:27] I'll go here. I'll paste it here. I'll [984:29] press enter. And then I'll come here. [984:33] Damon reload will reload the unit file. [984:36] This will enable Jenkins so that even [984:38] after a reboot Jenkins starts [984:40] automatically. Then we are checking the [984:43] status in the final command. So I'll [984:44] paste it here. I'll press enter. So I [984:48] see the Jenkins service is running. I'll [984:50] quit this. I'll go to my browser. I will [984:53] come here. I will go to instances. I [984:56] will tell him to remind me later. And [984:58] then I will copy this public IP address. [985:01] http. It's not HTTPS. Remember folks, [985:05] I'll paste it here. It is 8080. I'll [985:08] press enter. And here it is saying you [985:11] can copy the password from this [985:12] location. I will go here. P sudo cat. [985:15] I'll paste that here. I'll copy the [985:18] password. And then I will paste the [985:21] password here. I will click on continue. [985:24] Never. And then here install suggested [985:27] plugins. That is what I want you to [985:29] choose. Now I want you to know that [985:31] [snorts] [985:32] once this will take a minute or so we'll [985:35] go back to our repository. So we have [985:38] checked the accessibility we are able to [985:40] access Jenkins. Now we also must install [985:43] Docker engine onto our Jenkins [985:46] controller. But Warun why now understand [985:48] that Jenkins is just an orchestrator. [985:51] You would want the tools that are [985:53] required for your pipeline to be [985:55] available in Jenkins. So if you have [985:57] watched my previous lecture, if we are [985:58] doing TV scans, we want TV to be [986:02] installed into Jenkins. If we are [986:04] logging into AWS ECR or running AWS [986:07] specific commands via AWS CLI, then we [986:10] would want AWS CLI to be installed onto [986:13] our Jenkins controller or to our Jenkins [986:17] agent. If Jenkins agent is the one that [986:20] is actually running the jobs and as we [986:22] have seen in this series thus far in [986:26] production environments, you will have [986:28] the controller that will just be used [986:30] for creating jobs and so on and the [986:34] actual leg work of running the jobs will [986:37] be performed by your agent and we saw [986:39] that in our mega demo. But in this [986:42] lecture, our controller is acting as [986:46] both controller and it it is also doing [986:48] the leg work. Leg work as in it is the [986:50] one that is running the jobs for us. [986:52] Since our jobs that we'll be creating in [986:55] this lecture, we'll be using docker [986:57] commands because we'll be building a [986:59] container image and then we'll be [987:01] deploying a container using that image. [987:04] For that, we need Docker Engine [987:06] installed, which is why these are the [987:09] steps that we'll be using to install [987:12] Docker Engine onto our Jenkins [987:15] controller. So, I'll copy all these [987:17] commands. I will go to the terminal here [987:20] and I'll clear my screen. I'll paste all [987:23] of these commands here and I'll press [987:25] enter. So, this will install everything. [987:27] I have omitted the first one because [987:29] this is a fresh VM. I'm pretty sure it [987:32] does not have Docker engine already [987:34] installed. Once this is done, in the [987:37] next step, what we'll be doing is we'll [987:40] be adding our Jenkins user in the Docker [987:44] group because all the Jenkins related [987:47] jobs will be running with the user [987:50] Jenkins. So, we want that user Jenkins [987:52] to be able to create Docker containers [987:55] for us, which is why we are making [987:57] Jenkins user part of the Docker group. [988:00] when in the next step we are making [988:02] Ubuntu user also part of the docker [988:05] group because the user that I am using [988:08] to perform tasks is Ubuntu and I would [988:10] want this user to also able to run [988:13] docker specific commands because I'll be [988:15] using this user for verification. So [988:19] I'll quickly copy these I'll paste them [988:22] here and before I can run before I run [988:25] the restart command let's see that if [988:28] this is okay. So here I will create a [988:31] user which will be named cloud with [988:33] varosh. I will give it a secure [988:37] password. Here again I'll confirm the [988:40] password. Full name is Vun Jooshi. Email [988:45] address is abcthexyz.com. [988:48] I'll save and continue here. And here I [988:52] will leave this as is. Save and finish. [988:54] Start using Jenkins. Now finally I will [988:57] run this command that I that will [989:00] restart Jenkins. I'll go to my CLI. I [989:04] will paste that here and this will [989:08] restart. And then finally what I'll do [989:10] is I will verify that the user Ubuntu [989:13] can run docker - version and then I will [989:17] verify that user Jenkins can also run [989:19] dockery version. I'll come here. I will [989:23] paste these commands here and I can see [989:26] both of these can run this command. So [989:29] docker installation is done and now [989:32] Jenkins and Ubuntu user can run docker [989:35] specific command. So we are good here. [989:39] Now let's move to the demo section. I'll [989:43] remain in my browser. So what I will do [989:45] is I will go at the very top and then I [989:50] will open this thing and then I will go [989:53] to repositories and now we want to do a [989:57] simple demo. So what I will do now I [990:00] will create a public repository first [990:02] and then we'll move to the private [990:03] repository. I'll click on new and I will [990:06] call this shared [990:09] library demo. I'll scroll down. This is [990:13] already public which is fine. I will [990:15] click on create repository. Now when you [990:18] are dealing with this you won't see an [990:20] option to create a directory. Remember [990:22] what I told you. If you see here you [990:25] will see three folders [990:28] src v and resources. We will not be [990:31] discussing src or resources. So we'll [990:33] just be talking about v. We'll go here. [990:36] We have to create a directory named v. [990:39] So I'll click on create a new file. Here [990:42] I will write V but I will also add a [990:45] forward slash which will make V a [990:47] directory. Then here what I want to name [990:51] this file. Remember what I told you [990:54] whatever name here you give you can call [990:58] that in your Jenkins file without that [991:01] extension. So and we have to use camel [991:05] case because that is what is [991:07] recommended. So I will come here and I [991:10] will just call this simple echo because [991:14] in this demo we'll just be echoing [991:16] something and then I will also mention [991:19] dot groovy. This is fine what I want to [991:23] enter in this file. So I will quickly [991:27] copy the contents and I'll paste it [991:29] here. Now let me explain this to you. I [991:32] already told you that we will be calling [991:36] a function named call. So call is the [991:39] main function which will get called [991:42] whenever you use simple echo in your [991:45] Jenkins file. And then what we are doing [991:48] we have this uh parenthesis here. We [991:51] open and close a parenthesis. Then we [991:53] have a curly brace. And within the curly [991:56] brace we are saying echo Jenkins shared [991:59] library project with cloud with valor. [992:02] And then we are closing the curly brace. [992:04] That is all what we are doing here. I [992:07] will click on commit changes. Copilot [992:09] will be you know give me a commit [992:12] message. Add simple echo function to [992:14] echo a message. I'll click on commit [992:17] changes. So we have one file created [992:21] here. Now if I go to the diagram we have [992:26] this part sorted. Now we want to create [992:29] a repo. Here private repo is shown but [992:32] for now we'll be creating a public repo. [992:34] And then we have to create a shared [992:38] library in Jenkins. So let's go to our [992:42] Jenkins UI. And here first let me change [992:48] this to dark. And then I will zoom in a [992:52] little. And here I will go here. I will [992:56] hold on cloud with where Josh was my [993:00] user. Then I'll enter my secure [993:02] password. keep me signed in. Sign in. [993:05] Okay. Why this has changed the [993:07] appearance again? I'll move it to dark [993:09] again. System and in system I will [993:13] search for global and then I move to [993:17] global trusted pipeline libraries. I'll [993:20] click on add. It is asking me to name [993:24] the shared library and I will choose a [993:28] cliched name. My first shared library. [993:33] Yeah, spells right. Default version. Now [993:37] remember what I told you. Your shared [993:41] libraries are version controlled. So you [993:44] have the flexibility of choosing the [993:48] version that you want to choose. So if I [993:50] click on this question mark, you can [993:52] either have a branch name, tag name or a [993:57] commit hash. But here in our example we [994:02] will be using main. So what we are [994:05] saying is whatever is there in the main [994:07] branch you just pick that up that is the [994:11] version of shared library we wish to use [994:14] and there are other things as well that [994:16] you can use as you can see etc here [994:19] according to the SCM. So I am choosing [994:22] main here. But I want you to clearly [994:25] know that why were we calling our [994:30] library with at the rate main? Because [994:33] we wanted our library to use [994:38] main. And now you'll understand this as [994:41] I progress further. I promise you. Then [994:43] here is load implicitly. I do not want [994:46] to load this shared library implicitly. [994:49] I only want to load this when someone is [994:52] calling this shared library. So what I'm [994:54] saying is when someone writes in their [994:56] Jenkins file that this is the shared [994:58] library I wish to use only then that [995:02] Jenkins file can use this shared [995:04] library. Next is allow default version [995:08] to be overridden. So this is the [995:10] important piece. So for example if I [995:12] have defined main here then I don't have [995:15] to type at the rate main here. I can [995:18] even omit that because the default [995:20] version is already main. But I'm saying [995:25] allow default version to be overridden. [995:28] But if I want to let's say mention a [995:30] commit hash for application two but I [995:33] want the main for application one I can [995:36] do that. Remember I told you maybe for [995:39] application one I want to scan medium [995:42] level vulnerabilities as well which was [995:45] part of a specific commit for that you [995:49] can use a commit hash here and here you [995:52] can continue to use main because main [995:55] has what you wanted. So here I am [995:59] allowing default version to be [996:02] overridden. Next is include at the rate [996:06] library changes in job recent changes. [996:10] What does this mean varun? So remember [996:14] when we were discussing different jobs [996:17] there was a feature when you pushed [996:20] something in your repository that will [996:22] trigger your pipelines automatically. [996:26] Right? And this point says you whenever [996:31] there is a change in your shared library [996:36] and any of the pipelines any of the jobs [996:39] that are using that shared library if [996:41] they are configured to run automatically [996:45] then they will also run whenever there [996:49] is a change in the shared library. So [996:52] that is what it is saying include at the [996:54] rate library changes in the job recent [996:58] changes. But understand this will only [997:01] work if you have configured your [997:05] pipeline to get triggered whenever let's [997:08] say there is some activity in your [997:11] repository. So it will only work then [997:14] next one is cache fetched versions on [997:18] controller for quick retrieval. Now [997:21] understand in our example we just have a [997:23] couple of Jenkins files. We'll be [997:25] dealing with app one and app two as we [997:27] progress to demo two. In demo one we'll [997:29] just have one application and that to [997:32] test application. But remember what I [997:34] told you you as DevOps engineers will be [997:38] aiming to reduce the runtime of your [997:41] pipeline to minimum while covering all [997:46] the compliance or all the requirements [997:48] for your application. We said that [997:50] before. So here if you are dealing with [997:54] 20 30 40 applications you don't want all [997:57] of those or you don't want your [997:59] controller to fetch the shared library [998:01] again. You want it to cache it so that [998:05] your builds or whatever you are doing [998:07] with your pipelines can be finished in a [998:10] breeze. So that is what this is saying [998:12] cache fetged versions. Retrieval method [998:15] modern SCM is fine. source code [998:18] management is get project repository. So [998:21] this is the one that is stitching [998:23] everything. So you in your job will be [998:28] defining that this is the shared library [998:30] you wish to use and within the shared [998:33] library configuration you are defining [998:36] the repository that is linked with this [998:39] shared library. So understand create [998:42] that mental picture in your head folks [998:44] and then I will enter the repository [998:47] URL. So this is what we created just [998:49] now. I will go here and then I will [998:52] click here. I will copy this and then I [998:55] will paste it here simply. Credentials [998:58] none because this is a public [999:00] repository. So I will click on save [999:03] here. This is saved. Now I'll click on [999:06] new item. I want to create a job and [999:10] this job I will call shared library job [999:14] as expected. This will be a pipeline [999:16] job. I'll click okay. But here I will be [999:21] choosing pipeline script not pipeline [999:24] script from SCM because in the next demo [999:27] we'll be doing this. So for now because [999:29] we are understanding how shared library [999:32] work we are just choosing pipeline [999:35] script here. Now what is the pipeline [999:38] script that you want to run. So I will [999:41] quickly copy that and then I will paste [999:43] it here. Now see how simple this is. I [999:46] am saying this pipeline job you should [999:49] be using my first shared library at the [999:54] rate main. And this is part of the [999:55] syntax. You have a space here then you [999:57] have an underscore here and then the [1000:00] pipeline job like you have the [1000:03] difference is here now how we are [1000:07] calling what we created. So we created [1000:10] this here in the v directory we have [1000:14] simple echo.groovy. [1000:16] So what we are saying is whenever [1000:19] someone writes simple echo you invoke [1000:24] the main function. So whenever I am [1000:26] calling this here I'm saying in and in [1000:28] the steps we are saying simple echo. So [1000:31] that will trigger this and what will [1000:34] happen? What will happen is it will type [1000:37] Jenkins shared library project with [1000:40] cloud with varos. Now why this is so [1000:44] powerful right now you would be saying [1000:46] wun why can't I just write you know echo [1000:50] here why do I have to go to so much [1000:53] trouble by creating that shared library [1000:55] and so on. This is just one Jenkins job. [1000:59] Imagine you had 100 Jenkins job which [1001:02] were displaying this for you and your TL [1001:06] walks to you and tells you that I want [1001:10] CWVJ to be expanded. It should say cloud [1001:13] with VJOS complete. You won't be making [1001:15] changes into 100 of your jobs. You will [1001:18] just make the change here and it will be [1001:22] reflected in all your pipelines when you [1001:25] run your pipeline. So here I am calling [1001:28] that function. I will click on apply [1001:32] save and when I click on build now this [1001:36] will do what I have asked it to do. So [1001:40] let's check the console output from [1001:43] here. So as expected you will see both [1001:47] here. So see it is referencing shared [1001:50] library demo as well because this is a [1001:52] public repository. We didn't supply any [1001:54] credentials but we supplied this in the [1001:58] shared library we created within Jenkins [1002:01] and then you can see the job ran and it [1002:05] said Jenkins shared library project with [1002:08] cloud with varos as simple as this I [1002:14] hope this makes complete sense to you [1002:17] now that we have this understanding [1002:20] let's move to demo two where we'll be [1002:23] dealing with a production like scenario. [1002:27] So let me give you a brief of demo 2. [1002:31] Demo two will be exactly like this. I [1002:34] would say we will be creating three [1002:36] private repository. One for app one, [1002:38] another for app 2 and the last one for [1002:41] shared library. So let's start doing [1002:44] this. So I'll go to my GitHub. I will go [1002:49] here and cloud with vos. I will go to [1002:54] repositories. Here I'll click on new and [1002:57] here I will call it app one private [1003:01] which is here conveniently. I will click [1003:04] on private and then I will click on [1003:07] create a repository. [1003:09] Once this is done I will again go here. [1003:13] I will click on repository. new [1003:16] repository. I will be creating a [1003:19] repository for app 2 as well and app 2 [1003:24] private and this will also be private [1003:27] and then I will click on create a [1003:29] repository and then I will also create a [1003:32] repository for shared library. So I'll [1003:34] go to repositories again. This is [1003:37] private. This is private. App one [1003:39] private app 2 private. I will create a [1003:42] repository for shared library private as [1003:46] well. So this I will change to private [1003:49] and then I will click on create [1003:50] repository. Now this part is fine. So we [1003:54] have three private repositories. Now [1003:56] when you have a private repository you [1003:58] need credentials to log into that [1004:01] private repository. So let's do that. So [1004:04] what I will do is I will click here. I [1004:07] will go to settings and then I will [1004:11] scroll down developer settings. Here I [1004:15] will go [snorts] to personal access [1004:16] tokens tokens classic. So we will be [1004:19] using a personal access token classic to [1004:22] access our private repositories. [1004:24] Understand Jenkins will only be able to [1004:27] check out or to see whatever is there in [1004:29] your repositories only when you provide [1004:32] credentials for the same. So in this [1004:33] step we will be generating the that [1004:36] token and then in the next step we will [1004:38] be configuring those credentials within [1004:40] our Jenkins. So I'll click here classic [1004:44] and then let's call this Jenkins GitHub [1004:47] and expiration could be 7 days. I want [1004:50] repo level access. I will scroll down. I [1004:53] will click on generate token. So you [1004:55] will only see this once. So copy this [1004:57] and do not worry. I will be removing [1005:00] these credentials before I upload this [1005:03] video. So I'll paste this here for now [1005:05] and then I will go to my Jenkins console [1005:10] here. I will go to Jenkins. I will go to [1005:12] settings and here I will go to [1005:15] credentials [1005:17] system global credentials and here I [1005:21] will click on add credentials. Now I [1005:24] want to add credentials which has [1005:26] username with password. I want to I want [1005:29] those to be global because I want my job [1005:32] to be accessing those credentials. [1005:34] Username is cloud with varosh. I'll [1005:38] enter that here. And then password that [1005:41] I wish to use is I will copy that token. [1005:45] Ctrl C and then I will paste it here. [1005:48] Now we'll be using this token for two [1005:51] things. first is for Jenkins accessing [1005:55] these three repositories and then we'll [1005:57] also use these this token for our thin [1006:02] client that is my laptop to push code [1006:05] onto our private repository. Now often [1006:07] these would be different not often [1006:09] always these would be different. Your [1006:11] Jenkins will have a separate pad [1006:14] personal access token and your thin [1006:18] client your laptop or your you know [1006:21] bastion host jump host or whatever means [1006:25] you are using to access your [1006:26] repositories will have different [1006:28] personal access token. So ID I will call [1006:32] this Jenkins ID I will give it GitHub [1006:37] pat. Now this is important because this [1006:40] is how you call the credentials. So [1006:43] GitHub pat is here. I will click on [1006:45] create. So this will create the [1006:47] credentials for us. Now these [1006:49] credentials are created. What we now [1006:52] want to do is I want to show you what we [1006:56] are building. So let's go to our Visual [1007:01] Studio code. So I already have a project [1007:05] files directory which has app one and [1007:08] app 2 in it. App one has the logic for [1007:13] app 1. So here is the code for app 1. [1007:16] App one listens on port 5000. App one [1007:19] will just say I am app 1. Welcome to [1007:22] cloud with VJOS. App one your friendly [1007:24] demo micros service built with flask [1007:27] shipped by Jenkins running in docker [1007:30] then we have a requirements.ext [1007:33] now with Python you will see that [1007:35] whatever packages that you want to you [1007:39] know install are there defined [1007:42] separately in requirements.ext and we [1007:44] want flask as flask is the framework [1007:48] which we'll be using for our [1007:50] application. Then here is the docker [1007:53] file. So docker file is simple. We are [1007:55] saying we want to use python 3.11 slim [1007:58] workdir is forward/ app. We are copying [1008:02] requirement.ext [1008:03] and then we are installing the packages [1008:06] that we need referring to [1008:09] requirements.ext [1008:10] so that our image size is small. We are [1008:13] saying do not cache anything. Then we [1008:17] are copying app 1.py pi into forward [1008:21] slash app. Finally, we are running our [1008:25] application. We are running our [1008:27] application with a username app user [1008:30] because you should not be running your [1008:31] applications as user root. If you want [1008:34] to understand this further, go to my [1008:37] Kubernetes playlist and search for [1008:40] security or pod security. You will find [1008:43] what all measures you can take to ensure [1008:45] your pods do not run as root. Then the [1008:50] user that I wish to use is app user. [1008:52] Expose is just for documentation here. [1008:54] So that my DevOps admins know that this [1008:58] application listens on port 5,000. And [1009:00] finally I'm running that application [1009:02] using Python app 1.py. A simple thing [1009:06] that we are doing. Then is the Jenkins [1009:09] file. Now we have already done so many [1009:13] demos. We have written these files from [1009:16] the very scratch. So I did not want to [1009:18] waste anyone's time in that. Hence I [1009:21] have already written a very small [1009:24] Jenkins file. It just has two stages. It [1009:27] has a build stage and a deploy stage. We [1009:31] have defined three variables. Repo app [1009:35] one flask tag is we are tagging this [1009:38] with the build number. This is available [1009:41] by Jenkins. So we are using the existing [1009:44] variables that are available for our [1009:47] pipeline jobs. Then port I am defining [1009:49] as 5,000 because 5,000 is the port on [1009:53] which my application listens and the [1009:55] host port for this will also be 5,000. [1010:00] Then there are stages. Build stage we [1010:02] are docker build- t repo is app one [1010:06] flask and tag is this one. Extremely [1010:09] simple, right? And then we have stage. [1010:12] We have a deploy stage. Steps is we are [1010:14] doing two steps. First I am saying [1010:17] remove any container with the name app [1010:21] one flask because we could be running [1010:23] this job multiple times. So for the very [1010:25] first time the job will succeed but next [1010:27] time if I don't have this step the job [1010:30] will fail saying that there is a [1010:32] container with the same name already [1010:34] there. So we don't want that to happen [1010:36] which is why I am first running this [1010:38] command. But Warun if this command fails [1010:42] for the very first time then pipeline [1010:44] will also fail because when you run this [1010:46] job for the very first time [1010:48] there will be no container with this [1010:51] name. So that's why I am writing [1010:53] doublepipe true which says that if this [1010:57] fails then run this command which will [1010:59] throw the exit code zero. So it will [1011:01] proceed. So this command will only run [1011:04] if this one fails. That is what [1011:06] doublepipe will do for you. Then in the [1011:10] next one we are saying docker run run [1011:12] run in the background hyphen d and then [1011:15] or in detached mode name of the [1011:18] container will be dollar repo that is [1011:21] app one flask will be the name of the [1011:23] container hyphen port. The port [1011:24] forwarding is whenever someone hits port [1011:27] 5,000 on the host they should be [1011:29] directed to port 5,000 on the container. [1011:33] And then the image that should be used [1011:36] while creating this container is this [1011:38] one. And this is the image we created in [1011:40] the build step. Very simple. Now in [1011:44] application 2, everything is same. [1011:47] Instead of app one, you have app 2. So [1011:49] if I go here, so app 2.py has everything [1011:53] app 2. So app 2, app 2 and then docker [1011:57] file as well app 2 because we are [1012:01] copying app 2.py. Jenkins file as well. [1012:04] Here you see app to flask and this [1012:07] application is listening on port 5050. [1012:13] Now we have done two main things. We [1012:16] have created our personal access token. [1012:20] We have done a walk through of both the [1012:23] applications that we are about to [1012:24] deploy. Now what we want to do is we [1012:28] want to push the code that is here onto [1012:32] our private repositories. Now what I [1012:35] want to do right now is first I want to [1012:39] show you how this looks without shared [1012:42] library. We will just do a smoke test. [1012:45] We will run the jobs and see everything [1012:48] is working as expected. So let's go to [1012:51] the command line. I will go to my [1012:53] terminal and this is the controller. I [1012:55] don't want to deal with here. Now I'm [1012:57] working with my laptop. So I'll go to [1013:00] this terminal. I'm already in project [1013:02] files. If I run ls, you see I already [1013:06] have app one and app 2. So I'll cd into [1013:09] app one. So right now if I do get [1013:13] status, I see this is showing the [1013:16] results of my you know another [1013:19] repository another uh you know [1013:21] directory. But for this directory in F1, [1013:25] this is not get initialized yet. So I'll [1013:28] clear my screen. I will run get init [1013:31] first. Now this will initialize this. [1013:34] Now if I run get status now, you would [1013:37] see that it is saying all these files [1013:40] are currently unttracked. So if I want [1013:42] to track these, I will write get add [1013:46] dot. Now this will start tracking all [1013:48] the files. Dot represents all these [1013:50] files. I will write get commit- m I will [1013:56] write initial comet with code and [1014:01] Jenkins file we'll ignore if there are [1014:05] any spelling errors I'll press enter and [1014:08] this part is understood but understand [1014:11] up till now if I go to my repository in [1014:15] the browser I go here here rather and [1014:19] then if I go to here. I want to go to [1014:25] home. Then uh I should have gone via [1014:30] show more. And then if I go to app one [1014:33] private, what I see here is they call [1014:37] this main branch. But if I go here get [1014:42] branch v, then I'll come to know that [1014:46] this is calling it master. So what we [1014:48] want to do is we want to change the name [1014:51] of our branch. So what I will do is I'll [1014:53] zoom in as well a little so that you can [1014:56] see properly. I will write get and [1014:59] branch [1015:01] and what I want to do is I want to name [1015:03] my branch main. So if I now run get [1015:06] branch v you will see it is now changed [1015:10] to main. Now that this is done what I [1015:13] will do is I will run a command. Let me [1015:15] copy that command. What I'm doing is I [1015:19] am adding a remote and this remote I am [1015:22] naming it private repo. You would have [1015:24] often seen that people name it origin. [1015:26] So either is fine. I prefer private repo [1015:29] here. So I have written private repo. I [1015:32] have added a remote and this one you [1015:35] have to replace with your username. Just [1015:37] give me a moment. I will write here [1015:41] cloud with varosh. Cloud with [1015:45] varosh. I will just quickly see my [1015:48] username there. I don't think these are [1015:50] case sensitive but I don't want to have [1015:53] any problem. So this is written in you [1015:56] know the case that it prefers. So I will [1015:59] write press enter remote private already [1016:03] exists. So I'll have to remove that [1016:05] private first. get remote remove private [1016:11] repo and then I will run this again. So [1016:16] this has now added. Now if I what I want [1016:20] to do is I want to provide credentials [1016:22] because I will not be able to push until [1016:25] I provide credentials. So what I will do [1016:27] is I'll copy this and then let me come [1016:31] to Visual Studio Code. So I have [1016:32] somewhere where I can you know uh I'll [1016:36] let me create a new file. I'll just [1016:39] create it directly here. I will call [1016:42] this creds and then I will paste it here [1016:48] and then I will go to my browser. I will [1016:50] go to this tab. I will copy this token. [1016:54] I will paste this token here. And then I [1017:00] will copy this entire thing and then [1017:04] I'll come to my terminal. I'll paste it [1017:06] here. And then I will run simple get [1017:10] push private repo. I would have done [1017:12] this beforehand but I was thinking that [1017:15] maybe there are some students who would [1017:16] like to see the entire workflow which is [1017:18] why I'm doing this again. If you have [1017:20] already seen this a million times in [1017:22] this series, please pardon me. Then I [1017:24] will write main here because what I'm [1017:26] doing is I am pushing the main branch to [1017:30] this remote. I'll press enter. [1017:33] Now this will do the push. If I go to my [1017:36] browser now and I go to app one, if I [1017:39] hit refresh, I would see that whatever [1017:42] files were there are available in my app [1017:45] one private. Now I want to do this thing [1017:48] for app 2 private as well. So what I [1017:50] will do is I'll quickly copy paste this [1017:53] time. Okay, I'll copy and then I'll come [1017:56] to Visual Studio Code. I will come here. [1018:00] I'll paste these commands here. I will [1018:03] copy this username [1018:06] and I'll paste it here. [1018:09] Okay. And then I will copy this entire [1018:12] command and I will paste this here. I'll [1018:16] have to change this to app to private. [1018:20] And I'll have to also change this to app [1018:22] to private. And I think rest everything [1018:27] looks okay. I will copy this. And then I [1018:31] will paste it here. Let me in the last [1018:34] command. I just want to do a force. I'll [1018:39] copy this. I'll paste it here. [1018:42] I'll do a force push. And this should [1018:45] work. I will go to my browser. I will [1018:49] refresh it. More learning for you. This [1018:52] is brute force, okay? Do not do this in [1018:55] production. Do not force push it until [1018:58] and unless you know what you're doing. I [1019:00] know what I'm doing right now, which is [1019:02] why it is fine to run these flags. Use [1019:06] these flags with caution. RM- RF orce. [1019:11] These can cause destruction, folks. I'm [1019:14] telling you people. Anyway, so let's [1019:17] let's move ahead. So we have done this. [1019:19] This is good. Now we have the code in [1019:23] our repository. What we want to do is we [1019:26] want to create the jobs that will be [1019:29] using these repositories. Right? So I [1019:32] will go to my Jenkins console. I will go [1019:35] here. I will create a new item. I will [1019:39] call this simple app one job. I'll call [1019:42] this a pipeline. And then I will scroll [1019:45] down and this time I will choose [1019:48] pipeline script from SCM because I have [1019:51] my Jenkins file committed in my private [1019:54] repository. Understand these things my [1019:57] friends and then I'll choose get here [1020:00] and then repository URL. As soon as I [1020:04] will paste that URL it will give me a [1020:07] red flag that my dear friend I want [1020:10] credentials. This is a private repo [1020:13] and as expected and here I will say use [1020:16] these credentials and be happy and it is [1020:19] happy. Now I'll write main here I will [1020:23] because main is the branch where [1020:25] everything is placed and then I will [1020:27] click on save. [1020:29] Let's click on build now. Now this [1020:32] should succeed. Why this should succeed? [1020:35] Because right now we are saying run this [1020:38] job normally. No shared library nothing [1020:41] here. Now this should run. I'll click on [1020:44] build now. Until the time this runs [1020:47] let's create another job. Let's call [1020:49] this app 2- job. And this will also be a [1020:53] pipeline job. Okay. And here I scroll [1020:56] down. I choose pipeline script from SCM. [1020:59] I go to app to private. I copy [1021:03] the URL. I go here. I choose get as SCM. [1021:10] I paste that here. It won't be happy for [1021:13] some time. Let's make it happy by [1021:15] providing the credentials. This I want [1021:18] to be main. I will scroll down. I will [1021:21] click on save. I will build this now. I [1021:24] will go to Jenkins and app one has run [1021:28] successfully. And this is deploying in [1021:30] our Jenkins controller itself. So if I [1021:33] go to my item, I go to this controller, [1021:36] I clear my screen for more real estate. [1021:39] I run docker ps. [1021:42] Permission denied. Why would you deny me [1021:45] permission, my friend? Let me just [1021:48] quickly I'll go to my repository [1021:52] here. Install Jenkins controller. [1021:56] We verified this, right? sudo hyphen new [1021:59] docker [1022:01] version [1022:04] Jenkins can run it but Ubuntu cannot why [1022:07] is that sudo docker ps we were able to [1022:12] run it when I last checked [1022:15] why is this [1022:17] idbuntu [1022:20] is part of the docker group [1022:24] what was that command new gp let Let me [1022:28] try to run docker ps. That's strange. [1022:33] Let me add p sudo. Anyway, we'll [1022:35] troubleshoot that later. But app one [1022:38] flask is created. And if I go to my [1022:41] browser, why is app 2 flask not created [1022:45] yet. I will click on refresh. This job [1022:48] has failed. So let's see why this has [1022:50] failed. I'll click here. Console output. [1022:54] If I scroll down, scroll down. Scroll [1022:57] down. Invalid IP address 5050. Oh, I [1023:02] think my Docker. My Jenkins file has a [1023:05] problem. So, let me quickly rectify this [1023:07] with you. More troubleshooting. [1023:10] Better Jenkins file. I'm saying hyphen p [1023:14] port but port port 2 * 5050. That's [1023:18] unacceptable. [1023:19] Let's correct this. App one has this [1023:22] correct. This one I have corrected right [1023:25] now. these things happen and then I'll [1023:28] go to iterm remember we have to commit [1023:32] this we have to push this so I will go [1023:35] to get add dot if I run get status first [1023:39] you will see that there have been [1023:41] changes made to Jenkins file I'll write [1023:44] get add dot get commit- m fixed port [1023:51] variable [1023:53] because we want our messages to be [1023:57] explanatory. I'll press enter. Get push [1024:02] private repo main. I'll press enter and [1024:07] then I will go to my browser. I will go [1024:11] to app 2 job build now. And if this [1024:16] succeeds, I will be running [1024:19] app one job again because I want to see [1024:21] symmetry. Build one, build two. It has [1024:24] succeeded now. So let me quickly run app [1024:27] one job again. So that VC symmetry. So [1024:31] if I go to my item and go to my [1024:35] controller, if I run sudo [1024:39] docker ps, you see I have both of these [1024:44] containers running. App one flask, app 2 [1024:47] flask. And if I go to my browser, if I [1024:52] copy the public IP address of my [1024:56] instance and I write http [1025:00] I forward slash forward slash I paste [1025:02] the IP address and it is listening on [1025:05] port 5000. This won't work. And someone [1025:08] is saying wun you have not entered the [1025:12] port number in the security group. How [1025:14] would it work? So I have to open two [1025:17] ports. 5,000 and 50/50. So, let's do [1025:20] that. 50/50. Interesting, right? I'll go [1025:23] here. Go to the security group. Edit [1025:26] inbound rules. Add rules. [1025:29] 5,000. [1025:32] I'll open it for everyone. Do not do [1025:34] this in production. [1025:36] 5050. [1025:38] And then here for everyone. You can [1025:42] write descriptions as well. And this [1025:44] should now work here. http [1025:48] this is not working [1025:57] 5,000 5,000 ports this [1026:03] yeah took some time I'll copy this I'll [1026:07] paste this here and then let's also [1026:10] check 50/50 and this should also work so [1026:13] this is showing app 2 that is what we [1026:15] expected this is showing app one which [1026:19] is what we expected. So things are [1026:22] working as expected. Now let's do one [1026:24] thing. Now let's create our shared [1026:28] library. So let's now create our shared [1026:31] library. So I'll go to my dashboard [1026:32] here. I will go to settings. I will go [1026:35] to system. We have been here before. [1026:38] Remember previously we were here when we [1026:40] were creating a shared library that was [1026:43] pointing to our public shared library [1026:45] that we created in our previous demo. [1026:48] But this time our shared library repo is [1026:51] private. So we have to supply certain [1026:54] level of credentials there don't we? [1026:56] I'll keep the name as is. I will keep [1026:59] all these settings as is. What I will do [1027:02] this time is I will just be adding our [1027:06] private repo here. So if I open this in [1027:10] a new tab, what is this new thing coming [1027:12] here? Open link in split view. Oh, this [1027:15] is interesting. So I can have one tab [1027:18] here, one tab here. We'll use it in some [1027:21] course or in this lecture if need be. I [1027:23] will go to repositories here and shared [1027:27] library private. I will copy the name [1027:31] from here. We don't have anything in [1027:33] this repository right now and we will [1027:35] soon have something. Don't worry about [1027:37] it. I'll go here. I will paste this here [1027:41] and then credentials. We have the same [1027:44] credentials that we'll be using and save [1027:47] this. That's all we had to do. We edited [1027:50] our existing shared library. And now our [1027:54] existing shared library that is my first [1027:56] shared library is pointing to this [1028:01] shared library private repo. [1028:04] What we want is we want to create [1028:07] certain files in this repository which [1028:12] will help us externalize the [1028:16] commonality. So let's do this step by [1028:19] step. So I'll go to Visual Studio Code. [1028:22] Let me close this. Let close this. And [1028:25] let me close this. So that we can see [1028:27] step by step what we are doing here. [1028:29] Okay. I will come here. I will first [1028:32] create a new folder here. I will call [1028:36] this shared [1028:38] library. I will go to shared library and [1028:41] I will create a new folder and I will [1028:44] call this v. I told you that we won't be [1028:46] creating src or resources but we will be [1028:49] creating v.s. Now what I'm doing is I'm [1028:51] creating a structure here and then we'll [1028:54] push it like we did for app one and app [1028:56] 2. So we have v created. Now I am [1028:59] creating a new file here. Now watch this [1029:02] very carefully. I will call this build [1029:06] image and then I will give a name dot [1029:10] groovy. Simple, right? Extremely simple. [1029:14] Nothing major we have done. I will open [1029:17] Jenkins file for one of our apps app one [1029:21] and what I will do is I'll do this split [1029:24] right so that we can see these things [1029:27] side by side we have our Jenkins file [1029:30] here and this I will also view and I [1029:33] will word wrap this now I have this here [1029:37] I what I want to do is I want to take [1029:40] these steps out because I will tell you [1029:44] later I why I want these out but for now [1029:48] I don't want these steps to be here. I [1029:51] want a certain function which will call [1029:54] my you know library here and my function [1029:57] will run this. So for now what I will do [1030:00] is watch very carefully. I will just cut [1030:03] it and I'll paste it here. Since this is [1030:06] build image.groovy I am only concerned [1030:10] about build right now. I don't care [1030:12] about anything else like we saw before. [1030:15] Here I will have the same function that [1030:18] we will always have and then I will have [1030:21] a opening curly brace. I'll come here [1030:24] and then I will come down and I will [1030:27] have a closing curly brace. We have done [1030:32] this. Now in this Jenkins file I will [1030:36] just have one thing. What I will tell [1030:39] this is I will write build image here [1030:44] and then I will just close this. What [1030:47] I'm saying is in this step my dear [1030:51] friend you just call this function and [1030:54] this function will know what to do. And [1030:57] what is that we are doing? We are doing [1031:01] this command. We are building an image [1031:05] here but we have externalized the common [1031:09] thing. Both of these Jenkins file had [1031:13] this common step which we have now [1031:16] externalized. We have centralized that [1031:19] step. That is what we have done. Now [1031:22] what I want to do is I want to do [1031:24] something similar for my deploy step. [1031:28] Now what I will now do is remember we [1031:31] are creating different files for [1031:34] different stages for this stage and [1031:38] these steps I will create a different [1031:41] file and that file will be called and [1031:44] once that file is called the function in [1031:47] that file that will be under call will [1031:50] be invoked. So we have done this for [1031:53] build image this stage. Now I want to do [1031:57] it for deploy. So I'll right click here [1031:59] again I'll write a new file and this [1032:02] file I will call deploy app dot groovy [1032:08] simple right and then I will take this [1032:11] here so that you can see side by side [1032:14] and I will do something similar I will [1032:16] copy this entire thing I will paste it [1032:19] here of course I'll have to remove this [1032:22] step from here and I will just copy [1032:25] these steps rather cut these steps and [1032:28] paste these steps here. I will go here [1032:32] view and I will word wrap this so that [1032:34] you can see more. It's already word [1032:36] wrapped. It's hung or what view. Okay. [1032:42] Okay. Anyway, so this let me close this. [1032:47] Now we have this here, right? This is [1032:50] exactly what we want to do. But here [1032:53] what I want to do is I want to call the [1032:58] function. And how do I call this [1033:00] function? I just write deploy app and [1033:03] that will call this function. And this [1033:06] will be run. So I will write here deploy [1033:10] app. That's it. This will go here and [1033:14] run this. Now this is exactly what I [1033:17] will do in the Jenkins file of app 2. [1033:20] But I will just have certain changes. So [1033:23] I'll go here to Jenkins file for app 2. [1033:27] Here everything else is fine. Here I [1033:29] will just write the name of that file [1033:32] that is build image. And then here I [1033:37] will write deploy. This will call this. [1033:40] I will save these things. I will close [1033:43] this. I will close this. I will close [1033:46] this. Let me just quickly review. This [1033:48] looks good. We are building the image [1033:51] here. Here we are deploying a container [1033:56] using that image. This looks okay to me. [1033:59] And this Jenkins file also looks okay to [1034:04] me. We are calling build image. There is [1034:08] correct spelling here as well. Deploy [1034:10] app. If I go here to this Jenkins file, [1034:15] this is build image and this is deploy [1034:18] app. The spelling is correct in both of [1034:21] these. I will save these. I will go to [1034:23] my terminal now. I will go to this tab. [1034:28] I will we are in app 2 right now. So get [1034:31] add dot get because we have made changes [1034:34] in the Jenkins file. So we have to [1034:36] commit uh both app one and app 2. The [1034:38] Jenkins file has changed. So I will [1034:40] write get commit Jenkins file changed to [1034:45] have shared library and I will have to [1034:51] also write hyphen m here and then press [1034:55] enter get push private repo main. [1035:01] This is done. I will come one step back. [1035:04] I will cd into app one and this time I [1035:08] will write get add dot and then [1035:12] get commit jenkins file to shared [1035:15] library. This is fine and then I will [1035:18] just push this to main and we also have [1035:23] to do it for our shared library. So I'll [1035:26] come one step back and then I will go to [1035:29] shared library directory. [1035:32] We run ls here and I run get init first [1035:37] because I don't think this was [1035:38] initialized. Yes, this was not [1035:40] initialized. I will run get add dot I [1035:44] will run get commit- m then I will write [1035:50] here first commit in shared library and [1035:57] uh we'll have to supply the credentials [1035:58] as well here. So if I run get branch- v, [1036:04] this is the master branch. So I will run [1036:07] get branch- [1036:09] m main. And then let's go there to our [1036:14] visual studio code. I have creds here. [1036:17] Then we want to run these things. So let [1036:19] me quickly copy that. I will go to my [1036:23] browser and this is the this is the [1036:27] directory and here I want to just copy [1036:30] this name from here. I will paste this [1036:34] here and I will paste this [1036:38] here. I hope there's no space right. [1036:41] It's correct. And then I will just copy [1036:45] these three things. And then I will come [1036:48] here. I will paste it here. and I'll [1036:50] press enter. [1036:52] So this is done. I will go to my [1036:54] repository and then I will just refresh [1036:57] this to see I can see all those things [1036:59] here. I can see v here first commit and [1037:02] shared library and then I see both my [1037:05] groovy files here. So I will now go to [1037:07] Jenkins and then let's rerun these jobs. [1037:11] So we have done the shared library part. [1037:14] We have defined the private repository. [1037:16] We have pushed to our private [1037:19] repositories. We have the fresh code [1037:21] everywhere. Now finally let's run this. [1037:25] I'll click on this. I'll click on this. [1037:28] Let's go to one of these jobs and see [1037:30] everything is working. So this is this [1037:32] has failed. Beginner's luck is not with [1037:36] us today. So I will go to console output [1037:38] and see why this has failed. Error error [1037:42] error. [1037:43] Build image found among. This is [1037:48] interesting. [1037:51] Okay, this is for app one. Let's see if [1037:54] app 2 is showing the same thing. We copy [1037:56] pasteed. So that's expected. I will go [1037:59] here. Console output [1038:03] no walk Jenkins change Jenkins file [1038:06] change to have shared library. That's [1038:08] fine. That's our commit message. End of [1038:11] pipeline stage deploy skip due to [1038:13] earlier failure. So this failed [1038:15] somewhere here. itself. [1038:27] Okay, let me see the code again. [1038:40] Wow. [1038:42] Just wow. [1038:45] Can anyone tell me what is wrong? [1038:49] We don't have that library block at the [1038:52] top. How would this work without that? [1038:55] So, let me quickly add that folks. [1038:59] I'll come here. I will add that here. I [1039:03] will add that in Jenkins file [1039:07] of app one as well. These things happen. [1039:11] Okay. And I did not plan this. This I [1039:14] actually forgot. So I'll go to my [1039:16] terminal. I will come one step back. I [1039:21] will cd into app one. I will [1039:25] get [1039:27] add dot [1039:29] get commit- m [1039:33] fixed. Let's write that. [1039:36] and get push [1039:39] private repo main. This is fine. Now [1039:43] let's first run app one. Okay, then [1039:45] we'll come to app two. Let's see some [1039:48] success this time. App one job. I'll run [1039:52] this. And this is running. I can go to [1039:56] the console output from here itself. And [1039:59] I think this time it succeeded. Okay, [1040:01] it's a success folks. [1040:04] Some success. I'll go to my [1040:08] terminal and let me first see I did it [1040:11] in both the files right I added this [1040:13] here as well right let me just go one [1040:16] step back I will go to cd app 2 and here [1040:21] I will run get add dot [1040:24] get commit- m fixed in production you [1040:28] will write you'll expand this message [1040:30] not just fixed I will write get push [1040:35] private repo main. So I'm pushing main [1040:38] to this private repo. This is done. [1040:40] Everything looks good. I'll go to Chrome [1040:42] now. I will go to Jenkins. I will now [1040:45] run app 2. So this was build number four [1040:49] for app one and build number three for [1040:52] app two. [1040:54] I will quickly refresh this. Now both [1040:57] have completed successfully. I will go [1040:59] to my [1041:02] terminal. Now I'll go to this controller [1041:05] and let me run sudo docker ps and I see [1041:11] both of these containers now running app [1041:14] one flask and app 2 flask both are [1041:17] running and I also want to show you the [1041:22] logs for one of these jobs. So I'll come [1041:24] here and then I can go to stages and [1041:29] this will tell me what happened in each [1041:32] of these stages. So our containers are [1041:36] running just fine. And if I go to build [1041:38] I can see what happened in the build [1041:40] stage. Both the shared library and our [1041:45] private repos along with the shared [1041:47] library repo were used to build our [1041:51] application. So if you come here and I [1041:54] go to console output you can see all the [1041:56] details here. So shared library was used [1041:59] and then I would encourage you to check [1042:01] all these logs to understand what was [1042:05] unfolding underneath. How was Jenkins [1042:08] orchestrating everything. So I would [1042:10] encourage you to do that. [1042:12] But for now you would be thinking Vun [1042:15] you have done these things pat on your [1042:17] back. It is good. But how this is useful [1042:21] for me? Now I want you to imagine your [1042:23] team lead comes to you and tells you [1042:28] Abhishek yesterday someone yesterday we [1042:34] patched our docker host and upon the [1042:38] reboot of that host [1042:40] our containers didn't come up. First is [1042:44] this. I want you to fix this [1042:46] immediately. [1042:48] Second is there are no labels in your [1042:53] container images. I want you to label [1042:55] the container image with the maintainers [1042:59] email address. You Abishek are already [1043:03] using shared library. You know and you [1043:06] nod to your TL I will get this done as [1043:09] soon as possible. You raise the change [1043:12] request and then the request is tracked [1043:15] there. So what you will do to fix this? [1043:19] Now if you were not using shared [1043:22] library, what would you have done? You [1043:25] would have come to each Jenkins file and [1043:28] you would have changed the command to [1043:31] add certain flags. Now what those flags [1043:34] are, we'll see later. But if you had 20 [1043:37] applications or 200 microservices, you [1043:40] would have made changes in all of these. [1043:42] But you are smart. you are using Jenkins [1043:45] shared library. So what you would do is [1043:48] you would come here first. Let's [1043:50] replicate that problem. So if I go to my [1043:53] iterm right now both of my containers [1043:56] are running. So if I run sudo systemctl [1044:01] restart docker and I press enter. Now [1044:05] this will restart docker for me. We all [1044:08] know that. But after docker has [1044:11] restarted, let's see what is the status [1044:14] of our container. So if I do that, you [1044:17] see both my containers are in stopped [1044:20] state. So if I run docker ps now and I [1044:23] have to add p sudo here and I have to [1044:26] add hyphen a flag to see the stopped [1044:28] containers. You see that both of these [1044:30] containers have exited. So the docker [1044:32] service was restarted or when the host [1044:35] will be restarted, we would see that our [1044:37] containers will stop and we don't want [1044:40] that. We want our containers to be [1044:42] running 24 + 7 because these containers [1044:44] are running a web application which is [1044:47] accessed let's say 24 + 7. So we would [1044:50] want these containers to automatically [1044:53] restart start when the docker host is [1044:57] rebooted. So what I will do is I will [1045:00] come to my visual studio code. I will go [1045:04] to my shared library code. I won't be [1045:06] touching my Jenkins file. Okay. I'll go [1045:08] to this deploy. And here I will add a [1045:13] flag. Now the flag that I will add is I [1045:16] will call this restart and then I will [1045:20] say unless stop. Now if I have manually [1045:23] stopped this container then you don't [1045:25] restart it. But if there is a docker you [1045:29] know service restart or the host is [1045:31] rebooted you automatically restart this [1045:35] container. So I have added this step. [1045:37] Now this was requirement one. [1045:40] Requirement two was I want a maintainer [1045:43] label. So what I will do is I will come [1045:46] to this build image and here I will add [1045:50] another flag. I will add label and here [1045:53] I will write maintainer is equal to [1045:57] cloud with vjosh at the rategmail.com [1046:02] and by the way this is a valid email [1046:04] address. If you want to interact with me [1046:06] if you have certain feedback feel free [1046:08] to reach me on this email address. Now [1046:11] we have added both of these points. [1046:13] We'll have to first commit these and [1046:15] once we have committed we can rerun our [1046:19] pipeline jobs to see whether our changes [1046:23] have taken effect or not. So I'll come [1046:25] to my terminal and here I will go to [1046:29] this tab. I will go one step back. I [1046:33] will cd into shared library. Here I will [1046:36] run get status and this will show me [1046:38] that you have made changes to two of [1046:40] your files. Now let's run get add dot [1046:44] get commit- m. I will just write [1046:47] automated a few things and then I'll [1046:51] press enter and finally I will run get [1046:54] push private main. I'll press enter [1046:57] again. I will go to my browser and I [1047:01] have a habit of always refreshing my [1047:04] repo when I have pushed something new. I [1047:06] don't have a clue why because if that h [1047:09] terminal has shown success it is there [1047:12] right but I do this refresh all the time [1047:15] and then once this is done I will come [1047:18] to my Jenkins screen I will click here [1047:21] and I will click on build now and once I [1047:24] go here this is running and I want to do [1047:27] the same thing for app 2 job as well. So [1047:31] I wanted to do certain changes. I did [1047:33] those changes in my central shared [1047:36] library repo. I did not modify my [1047:40] Jenkins files that were placed in app [1047:43] one and app 2. And just to let you know [1047:45] all these files I will be committing [1047:48] onto the GitHub repositories and I will [1047:51] also include the old Jenkins files so [1047:53] that you can follow along. So what I [1047:55] will now do is I'll go to my terminal [1047:58] here. I will yeah I'll go to the [1048:00] terminal I'll go to this controller and [1048:03] now I will run docker ps and I see both [1048:07] of my containers are now running. So if [1048:10] I first run docker images, this will [1048:12] show me all the images that are there in [1048:15] this docker host. And I see for app 2, [1048:18] this is build five. For app one, build [1048:22] five is the latest one. And if I go to [1048:25] my browser, what you will see is if I [1048:28] refresh this, build five is the latest [1048:30] one. And both of these builds have [1048:32] succeeded. So if I run docker inspect, I [1048:37] will write this one. Let's check it for [1048:40] app 2. I'll run this and I'll press [1048:42] enter and I'll have to add pseudo. I'll [1048:45] do this. And if I scroll up, you will [1048:48] see there is maintainer cloud with [1048:50] varjosh at the rategmail.com. So first [1048:53] requirement is done. Second requirement [1048:56] was our containers should be [1048:59] automatically started when there is a [1049:01] host reboot or the docker service is [1049:04] rebooted. So let's do that as well. I [1049:06] will run the command p sudo systemct ctl [1049:09] restart docker and then we'll be [1049:12] checking the status of our container. So [1049:16] let it do its magic and then let's run [1049:20] docker ps and you see our containers [1049:24] have been up for 5 seconds because that [1049:26] is when the docker service came up. So [1049:29] our containers both of these are up and [1049:33] running. Now this is something very [1049:35] simple that I have shown you but I want [1049:37] you to understand that right now I have [1049:41] not moved certain parts of my pipeline [1049:44] into my shared library. For example, if [1049:47] I let's say move all these things also [1049:50] there then changing the repo or changing [1049:54] the tag port all of these things becomes [1049:57] so simple. So what I want to now do is I [1050:00] want to quickly take you to my GitHub [1050:02] repository and I want to show you what [1050:06] all are the possibilities when you are [1050:09] using shared library. So I will move at [1050:12] the very top and then I will scroll down [1050:16] and here I will choose why centralizing [1050:19] config matters. And here you will see a [1050:23] lot of things. I'll suggest you give it [1050:25] a read. So this really starts to shine [1050:28] in real CI setups where multiple apps [1050:31] rely on common tools like SAS scanners, [1050:35] dashed tools, image scanners, code [1050:37] quality platforms. So all of these [1050:40] things are common in most of the [1050:43] applications. So what you can do is you [1050:45] can move these things out. Maven [1050:48] commands, you are running the same Maven [1050:50] command for most of your apps. you can [1050:53] move that to a shared library. So I hope [1050:57] all of these things make sense to you. [1051:00] What I will also do is all these files [1051:03] that we have used you will find in the [1051:05] repository. I will also move the old [1051:08] Jenkins file that we were using [1051:10] initially to the repository of this [1051:13] lecture. With that said, it's a wrap for [1051:17] this lecture. This was our final lecture [1051:20] in the Jenkins basics to production [1051:23] course. You saw how shared libraries [1051:26] remove duplication, simplify [1051:28] maintenance, and let you update many [1051:32] pipelines by changing one file. Both [1051:36] apps picked up a new build and deploy [1051:39] logic automatically, proving why real [1051:42] teams rely on this approach. [1051:45] If you have any questions, feel free to [1051:48] ask them in the comment section and I [1051:50] will reply. [1051:52] If this helped, do consider liking, [1051:56] sharing, and subscribing. It will mean a [1051:59] lot to me. Thank you very much. If you [1052:02] have made it this far, well done. [1052:05] Completing a long hands-on CI/CD master [1052:08] class takes focus and discipline, and [1052:11] that effort matters. By now you should [1052:15] have a clear understanding of how CI/CD [1052:18] works in production. Not just Jenkins [1052:20] pipelines, but how code flows from [1052:24] commit to build, testing, security, and [1052:27] deployment in real teams. If you [1052:30] practiced along with the demos, you [1052:32] should also feel more confident [1052:34] reasoning about CI/CD systems, [1052:37] identifying gaps, and understanding why [1052:40] pipelines are designed the way they are. [1052:42] I also want to say this clearly. Your [1052:46] feedback matters to me. If this course [1052:49] helped you in production or interviews, [1052:51] I would genuinely love to hear that. And [1052:54] if something felt unclear or incomplete, [1052:57] tell me that too. I read the comments [1053:00] and use that feedback to improve future [1053:04] content. All the notes, YAML, and [1053:07] Jenkins files used in this course are [1053:10] available in the GitHub repository [1053:12] linked below. Treat it as your long-term [1053:15] reference when working on real [1053:17] pipelines. If this course added value, [1053:20] the simplest way to support the channel [1053:22] is by liking the video, subscribing to [1053:24] the channel, and leaving a comment. It [1053:27] really helps this content reach more [1053:30] learners. And if you would like to [1053:32] support the channel further, consider [1053:35] joining as a channel member. I don't [1053:38] lock content behind PayWall's membership [1053:41] is simply a way to support the free [1053:44] in-depth learning I try to put out [1053:46] consistently. Finally, keep building, [1053:50] experiment, break things, fix them, and [1053:53] keep asking why systems are designed the [1053:56] way they are. That curiosity is what [1053:59] will make you strong DevOps engineer in [1054:02] the long run. I will see you in the next [1054:05] video. Thank you very much.