Introduction to DevOps SDLC (CI/CD)

In this day and age having a functioning and secure Software Development Life Cycle (SDLC) process in place is becoming a key component of a successful organization. And one methodology that is becoming increasingly popular is DevOps. Mainly, because the methodology itself is designed to produce fast and robust software development. In this article, we will focus on how we can incorporate security into CI/CD and turn DevOps into DevSecOps easily and with automation in mind.

It’s quite a long article, so in case you are already familiar with some of the terms, feel free to skip to whatever part pleases your curiosity :)

CI/CD (an explanation of Continuous Integration, Development, and Delivery)

Within the methodology, there are two processes that have been particularly standing out lately, especially as buzzwords in the industry — Continuous Integration (CI) and Continuous Delivery (CD). In fact, both processes usually have the same direction in the SDLC flow, but they just finish at different points (that’s why they usually are phrased together as CI/CD). Put shortly (and oversimplified) CI is a software development practice where performed code changes and integrations are tested automatically for errors and issues (these are often functional and/or unit tests). Additionally, a crucial part of CI is that developers integrate their work frequently and on a continuous basis. An integral part of a CI strategy is also a culture that cultivates communication, collaboration, and learning in order to merge small code changes fast. In fact, CI works best with small and frequent code commits.

CD, additionally, extends CI and automates code releases within the SDLC flow. With Continuous Delivery, builds (that were tested under CI) are automatically pushed to a testing environment where more in-depth testing is done. This way, the build is finally ready for production. Some organizations prefer to do some manual testing and review at this stage, and others prefer automation. In a case where this release process is automated, the procedure goes under the name of Continuous Deployment.

Well, why is CI/CD so hot these days? The two main reasons are simple, money and money. Of course, syntactically, the two reasons seem pretty much the same, however substantially one reason is making money, and the other is saving money.

In today’s world where almost every company is software driven, and where the SaaS business atmosphere is so dynamic and competitive, innovation and speed are key to keeping up with the market. Businesses get an edge over their competition when they integrate key business decisions to a fast and innovative software development process. By automating the software release process in a safe and functional manner, the CI/CD strategy helps businesses deliver custom features, services and innovations to end-users quickly and effectively. Something that organizations sticking to traditional SDLC methodologies can’t keep up with.

Software defects can be quite costly, especially if they are left undetected or unfixed for long periods of time. In fact, the sooner an issue (whether functional or security) is found, the less costly it is to fix it. According to a recent report by Rice Consulting (STBC: The Economics of Testing), defects are 100 times more costly to fix when detected in production, and 10 times more costly to fix in testing. And the numbers don’t even include costs that might come from the potential damage of a bug, or a security issue in production. Either way, developers have an easier time finding and fixing an issue that is relatively new, compared to an issue that’s been there for weeks or even months. This is where the automated build and tests used by the CI/CD methodology shine and save organizations tons of time and money. With the automated build and testing throughout the whole SDLC (from Code to Release), companies make sure that their software is free of any major functional and security bugs early in the process. So even though these bugs are bound to appear in the dynamic workflow that CI/CD allows, they are detected and fixed early and quickly.

Application Security Testing in CI/CD

Now that we know what CI/CD is, we can see how adopting DevOps practices and implementing CI/CD allows organizations to meet customer and business demands faster, or even release their product in a timely and effective manner. Ideally, CI/CD would allow for safe and perfectly functional products to go into the market quickly. However, security and smooth experience are often compromised for speed and cost. Since companies often try to meet business demand and beat competitors to the finish line, speed and efficiency are usually the keys here, and unfortunately, security and smooth functionality aren’t. So, how can we make sure that security is taken into account in such a fast-paced atmosphere? Well, we must not sacrifice automation and speed for it.

“We need to turn DevOps into DevSecOps without sacrificing automation”

To be able to understand how to do that, we need to get acquainted with the typical Application Security Testing techniques: DAST (Dynamic Application Security Testing) and SAST (Static Application Security Testing). After DAST and SAST are explained and compared, I will try to provide you with some ‘recipes’ of including them into your CI/CD and automating security easily.

DAST and SAST

Obviously, both DAST and SAST have the same purpose — to test the security of your application. However, they take two quite different approaches.

DAST tests web applications and services by using mock attacks via HTTP/HTTPS, similar to what a hacker would do. This method is called black-box testing. Simply, a DAST scanner doesn’t need to have access to the code and the inner workings of an application. In fact, it automates what a hacker would do in a live application. Thus, when a vulnerability is found it usually means that the vulnerability could be exploited. Often, a good DAST scanner will show evidence that the vulnerability is real by safely exploiting it, and will show you how you can fix the issue. This allows pentesters and developers to save time. However, DAST’s biggest asset comes at a price: DAST scanners can only test a functioning application, meaning, you can only scan with a DAST scanner once an application was successfully built.

SAST scanners, on the other hand, don’t require a running, functioning application. Instead of scanning the application from the outside, they scan the code directly. A Static Analysis tool (SAST) would try to find out how data flows through the application to a database. So, let’s say from an input source (like a form field) all the way down to a security-sensitive sync point (like a database call). By analyzing this flow the tool checks if these security-sensitive syncs are safe and sanitized, and that’s how a SAST tool finds vulnerabilities. So, instead of attacking the application from the outside, as a DAST would, a SAST figures out the internal workings of your code base and tries to find vulnerabilities from within. Because of this, SAST doesn’t require the application to be functional and running.

To understand the different approaches better, here’s a simple real-life scenario. Imagine you are building a bike. Well, there are two ways to test the security/quality of the bike you are building. One way is to build the bike fully, and take it on a test ride. However, you try to change the gears, the chain won’t switch. Now, you know you have a problem with the gear system, and you focus on that specific issue to fix it. That’s the DAST approach. Using a SAST methodology, you would dismantle the bike before putting it all together, and check the quality of each individual part, and the quality of each individual link between the parts. Using a DAST, you could be a bit confused about which part of the gear system causes this problem, whereas when using a SAST you will know the exact reason why and where the gear change problem happened, but you would spend a lot more time and effort checking individual parts where no problem is apparent. Even though SAST is a lot more thorough, it’s usually a lot more time-consuming than DAST (after all, you are still checking each part of the system individually). Here, it’s important not to confuse the point I am making when I say “time-consuming”. Actually, the run-time of a SAST scan is usually shorter than that of a DAST, however, what is time-consuming with SAST is figuring out which finding is risky/worthy enough to pay attention to.

Oftentimes, DAST and SAST are pitted against each other, as if they are on opposing teams, but both of them help you secure your application. So, the best case scenario is to use both methods. However, in reality, most companies (especially small and mid-sized ones) don’t have the budget and the time to do so.

So… which one is better?

Unfortunately, you won’t get a definitive answer. In fact, which one is better, might be the wrong question to ask. The important question is: Which method suits me and my team best?

The case for SAST

As we saw previously, SAST scans your code and finds vulnerabilities. Conceptually it sounds great — you scan the security of a code in the build, find vulnerabilities, and fix them. In fact, on the surface, it seems to be the most applicable security solution in a DevOps environment. For one, you can scan for security issues really early in the process. It doesn’t need a running, functioning app as DAST does, and often, it doesn’t even need the code to compile. Another advantage of SAST is that it provides us with a deep and comprehensive idea of the inner workings of an app.

However, SAST also came across a few obstacles when it came to adapting to DevOps practices like CI/CD. One problem it had was that scans often took an hour or more. And in the fast-paced spirit of DevOps, everything that takes more than 10 minutes in the build is a sin.
Fortunately, the industry adapted, and came up with a solution:\

• Run long (full) scans over the weekend\
• Run increment (delta) scans in the build — these ones, of course, are usually limited and miss some vulnerabilities, but it’s quite a viable solution as long as you are running the full scans (and fixing vulnerabilities) regularly too.

However, the biggest problems that SAST faces today are still to be dealt with: false positives and providing developers with overwhelming amounts of information. Due to the method that SAST scanners use, they are very likely to report either false positives or irrelevant vulnerabilities that bare no risk in the real world. Thus, you might spend a lot of time either fixing irrelevant vulnerabilities or figuring out what the scan did right and wrong. This can be quite frustrating, especially when trying to save time and speed up to deploy. In many cases, developers don’t buy into SAST findings, which leads to some vulnerabilities being found, but not fixed.

So, if you want to start the application security testing process early, and if you feel like you will be able to maneuver through the overwhelming amount of findings and successfully map them to actual risk, then SAST might be right for you. A good way to implement SAST into CI/CD is to follow the solution above: run long scans over the weekends, and small incremental scans with every build. You can find some awesome tips on integrating SAST into your CI/CD in this article written by Meera Rao from Synopsys.

The case for DAST

DAST has some challenges too. Compared to SAST, in theory, DAST seems to be less adequate for a DevOps ‘continuous everything’ paradigm. DAST scans can take longer to complete — the scanner first crawls through your whole application and tries to attack it, looking for issues (so you can imagine why). Other than that, DAST’s most important obstacle is the fact that it requires a running, functional app to test against. This is DAST’s biggest gift and a curse. Due to the fact that DAST needs a running app, in order to scan you will need a dedicated testing environment for that. Ideally, everyone should have this (and if you don’t, I highly suggest you do), but oftentimes, organizations lack this environment and consequently, deters them from using DAST.
But, if we see the bigger picture, DAST has great potential for CI/CD integration, and here are the reasons why:

#1 DAST findings relate to Risk

The fact that a DAST spiders through the application and tries to attack it as a hacker would, allows findings to be mapped directly to risk. In the security space, it is widely accepted that Risk=Impact x Probability. DAST findings will often have a higher Probability compared to SAST findings since we can be sure that whatever vulnerability was found by DAST is exposed through a web interface that is (or would be) out there on the internet accessible to everyone. However, there’s another dimension to Probability, and that’s the probability of a finding being exploited. For example, if a vulnerability requires the attacker to perform an action on a victim (like in some cases of XSS) then the Probability (=Risk) would be lower when compared to a vulnerability that can be exploited directly by a hacker. The Impact of a finding depends on the finding’s severity and some business-driven parameters regarding the application (for example if the app is a revenue generating).

In Probely, our own DAST vulnerability scanner, we rank vulnerabilities by priority. We map the risk of a vulnerability by taking into account numerous variables. For example, we rank SQL Injections as high priority or high-risk vulnerabilities. Why? Well, the probability to be exploited is quite high, and the impact can be detrimental, allowing the attacker to access your database, or delete and alter parts of your database.

This allows developers to use DAST to tackle vulnerabilities with the highest risk first and focus on what matters. Unlike DAST, even though most SAST tools have risk indicators, SAST has difficulties adapting the risk to the context of the vulnerability. We can see this in how SAST would treat an XSS that can only be exploited if the victim is using IE6. Even though the risk is quite low, a SAST won’t be able to pick that up and would still report the vulnerability as either Medium or High risk. In fact, a majority of DAST tools do the same, however, more modern tools like Probely are advanced when it comes to applying more context to the vulnerability.

#2 DAST relates to the real world

In today’s world applications are usually built using multiple components using multiple languages, by multiple teams and companies. Often, SAST tools are dependent on a certain programming language, and extensive on-boarding and adaptation. However, since DAST uses the web interface of an app to run security tests, it tests the end result delivered by all these different components and teams. In addition to that, DAST scanners, like Probely, are usually technology agnostic (it doesn’t matter which language the application is written in). DAST scanners are at an advantage here since they test how different components interact within the app and find vulnerabilities without much insight into the inner workings of the individual components. SAST scanners usually don’t look at the interactions between the different components but test each component individually.

#3 Developers love DAST!

The fact that DAST scanners work like hackers, and can directly demonstrate the problem and present evidence of the exploit, makes DAST scanners appreciated by developers. This way, the developers can easily see the importance of a finding and the context in which the vulnerability is found. Furthermore, modern DAST tools like Probely also provide developers with some information on how to fix the issues found. Besides showing evidence that the vulnerability is real, Probely detects the language or the technology used to build the app, and based on that, provides users with tailored instructions on how to fix the vulnerabilities.

Compared to SAST, DAST scanners report minimal amounts (if not none) of false positives — Saving developers tons of time and effort and complementing the fast pace of a DevOps methodology.

Recipes for using DAST in CI/CD

Now that we know the drawbacks and the benefits of DAST in CI/CD, we can consider it as a serious option for including security into your CI/CD, and a way to turn DevOps into DevSecOps. One thing to note before going forward is that there’s no one clear answer on how to use DAST in CI/CD. Here are some ways that we’ve seen DAST being used efficiently in CI/CD. However, you should note that the best way to use DAST is the one that works best for your company and your existing environment.

General best practices

Whatever strategy or recipe you use to incorporate DAST into your CI/CD pipeline, here are some general practices that are applicable for all.

Customize your tests

The good thing about modern DAST scanners is that they can be greatly customized to your needs. At Probely, for example, we specifically pay attention to this aspect. We offer a full API so that users can really integrate security in their workflow on their terms. A couple of things that you should pay attention to when customizing: what kind of scans do you run at which point of your pipeline, which results are allowed to block the pipeline (or block a deploy), and when should you require a DAST scan.

What kind of scans should you run at which point?

In Probely we offer 4 different types of scans (we call them scanning profiles) — lightning, safe, normal, and full. Each profile should be used in different environments and at different points of the pipeline. For example, lightning scans are very fast scans (usually take less than 60 seconds), however, they only check for vulnerabilities related to SSL/TLS, HTTP headers, and cookies attributes. Since they are quite fast you can run them together with your unit and functional test and they won’t stall the flow. Think of the lightning scans as fly fishing for only small fish. However, these ‘small fish vulnerabilities’ still provide a lot of value since they are important for your security posture, and it’s through them that hackers often put their foot in the door and get onto exploiting greater things. Due to the types of vulnerabilities, it checks for, it’s important to run the lighting scan in an environment where you have the same (or similar) server settings as in production.

The ‘safe scans’ cover all vulnerabilities we support (over 1000), but with a limited set of payloads and methods. Since the scanner won’t make POST, PUT, DELETE or UPDATE requests, these scans are best suited to be run against your production environment, triggered by a deploy.

Lastly, you can run the ‘full scan’ in your Staging (testing) environment since it’s the most complete scan with the largest amount of payloads. Even though some of the payloads might be risky, you don’t run the risk of breaking something in production (doesn’t really happen often, but it’s good to be safe).

Which results are allowed to block the pipeline (or block a deployment)?

The fact that most modern DAST scanners can accurately map vulnerabilities to risk allows users to customize how each result (based on severity) affects the pipeline or the deployment process. In Probely we have 3 different risk levels (High, Medium, and Low). Having that in mind, you can use our API and set up a process where the pipeline or deployment is blocked only if there is a medium or high vulnerability. For example, in order to push a change to production, there should be a process in place that checks if there’s a medium or high vulnerability in the last DAST scan in staging (and if there’s no scan currently running). If that test passes, then the change is secure enough to be pushed in production. This is highly customizable (and I recommend you to take advantage of it). For example, you can block a deployment only for certain vulnerabilities, like SQL injections (because of the impact).

Take time into account

Application security testing (whether it’s SAST or DAST) takes time, and that’s the reality we all have to accept. There are a couple of ways to go about this issue (in fact, a large chunk of this article is about how to spend as little time possible on AST), but none of them is straightforward, it all depends on your case and time management in your company. As a starting point it’s important to be aware that DAST (and SAST) scans can take hours, so planning a time buffer in your workflow is crucial.

Integrate DAST either in Staging or Production

We already know that DAST requires a running, functional app. And often times your QA or testing environment is under constant change from new pushes and tests already running, thus it’s quite probable that there’s going to be some break or a functional issue in there. Having that in mind, scanning in Staging and Production is your best bet when it comes to DAST. Of course, if that’s not your case, feel free to go with whatever feels right (just keep in mind that DAST requires a running application).

Make scans a requirement before pushing to Production

This one is pretty straightforward and obvious. You don’t want to have a vulnerable application exposed in production.

If you have the chance to use SAST in earlier stages too

We will discuss this a bit later in the article, but having a SAST in place during the Commit and Build stage is highly recommendable.

Blueprint approaches for integrating DAST in CI/CD

These are some blueprint ideas that we’ve seen out there being used effectively. You can either use these and get inspiration when integrating DAST in your CI/CD pipeline, or you can customize the approaches and take whatever works best for you. We’d love to see how you include security tests in your CI/CD so feel free to send us your own approach.

#1 Automated tests in Staging before a Deploy to Production

One way to automate DAST in CD is to trigger scans on every push to staging. So, once integration and unit tests are complete (even SAST, if you have one in place), you will want to automatically trigger a DAST scan. Now, as I said, you can customize your flow. For example, you can have an automatic push to production if only low vulnerabilities were found. If there’s a medium or high vulnerability, then you can block the push and require manual approval. Alternatively, you can always have manual pushes to production, once the scan is done. How you go about it really depends on your company culture, your ability to take risks, and the nature of your application.

One thing is for sure, you need to make DAST scans a requirement before any deployment to production. Once in production, you can additionally run our ‘Safe’ scans.

#2 Scheduled Tests

Another way to use a DAST scanner, like Probely, is to schedule periodic tests. Based on your needs you can schedule weekly, bi-weekly, or even nightly scans. Since DAST scans can take a while, I’d recommend you to scan after a day of work, at night. You can make these scans a requirement for pushing to production. So, if you configure bi-weekly scans you would have two pushes to production per week. Thus, this method works well in a weekly/bi-weekly production schedule. To successfully run scheduled tests and not block the pipeline, you can create a ‘separate’ pipeline rather than integrating them into your current staging one.

You can schedule scans either through Probely’s API or using the web interface. You can get some help on our API documentation page, or if you prefer the web interface, check out this article. If you use Jenkins or CircleCI, we have plugins that you can use to easily integrate Probely in your CI/CD. You can learn how to integrate Probely with Jenkins here.

#3 Best Case Scenario

The best case scenario for Application Security in CI/CD is mixing variations of both approaches (SAST and DAST) with other methodologies at different points of your pipeline.

Ideally, every activity is a checkpoint in itself. For instance, real-time SAST scans are there to find vulnerabilities early while a developer is coding, once the Pre-commit checks are done, the pipeline is ready to go. And from there, an incremental SAST scan is run (we talked about this before). When incremental scans are done, the pipeline should move to a complete and thorough SAST scan. Note that when performing an incremental SAST scan you should be able to break the build in a case of a critical finding (this is an issue for another day though). After the functional and unit tests and the SAST scan pass successfully, you have a certain degree of confidence that you have a fully functional app. Now, the running app is ready for DAST scans and hardening checks to ensure the robustness of the application. Once issue-free, the app is pushed to production. In production, you can run Safe DAST Scans using Probely. For a complete security testing framework, you should also do yearly Pen-testing; Monitor your live application; and when the security of your app is more mature, open a Bug Bounty program. To learn more about the differences between Automatic Security Testing, Pentesting, and Bug Bounties you can check out this article.

In a perfect world, you want to have this in place. However, in reality, we understand that it is a work in process and this pipeline can’t (and shouldn’t) be built overnight.

What to expect for the future and final words

Even though DAST scanners are not necessarily blocking CI/CD pipelines, there is still a long way to go for a ‘painless’ adoption. We, at Probely, are working really hard to make that happen, however, there are a lot of specific use cases that we need to cover. For now, I’d advise you to sit tight (let us work our magic) and run tests frequently parallel with other CD Pipelines in your workflow.

That’s it for now folks!

If you have any questions or if you need any help with implementing DAST or Probely in your CI/CD Pipeline please don’t hesitate to contact us. We are always happy to help!

In case you are tired of reading, you can Start a Free 14-day Trial with Probely and see all of this in action!