DevSecOps: Best Practices for CI/CD Pipeline Security

CI/CD pipeline refers to a series of sequential practices comprising Continuous Integration (CI) and Continuous Deployment and/or Continuous Deployment (CD). Commonly utilized by DevOps teams, the CI/CD pipeline is one of the most efficient methods to build, test and deploy code, largely via automation tools.

Leveraging CI/CD pipeline has been reliably provided to facilitate the construction and deployment of more updates (and better ones) to software. This is largely because successful CI/CD pipelines require consistent collaboration and honest implementation of Agile and DevOps principles.

How To Integrate Security Into the DevOps Toolchain

Traditional Security Conundrum in DevOps

DevOps tactics and tools are significantly transforming the way businesses innovate. However, amidst this transformation, IT decision-makers are cognizing that traditional ‘siloed’ security approaches are hampering organizations from realizing the full potential of DevOps. In fact, the conventional security methods and controls are perceived as inhibitors to speed, agility, and scalability offered by DevOps.

Baking Security into DevOps

In response, forward-thinking and fortune 500 companies have started integrating security practices and controls into each phase of the DevOps software development lifecycle, a methodology popularly known as DevSecOps. It integrates security practices and procedures into DevOps tools and underlying policies, making security an integral part of software development. As DevSecOps gathers steam, IT firms are more likely to blend vulnerability assessment, risk modeling, and security automation into DevOps processes and toolchains. As a result, it improves security and compliance maturity levels of the DevOps pipeline and toolchain, while enhancing product quality and delivery. How? DevSecOps enables seamless flow of application changes through DevOps pipelines, bestowing on the developers the authority and autonomy, without axing security or increasing risk.

The Evolution of DevSecOps

I wrote The Future of DevSecOps in June 2019 after gathering insights from professionals who foresaw:
  1. greater adoption,
  2. security ingrained in development, and,
  3. AI/ML-driven automation.
For this article, I wanted to go back and see how the adoption of DevSecOps has proceeded over the past two years. In a subsequent article, I‘ll share what these IT professionals now see as the future for DevSecOps.

I received input from more than 40 IT professionals. Based on their feedback, the most significant evolution of DevSecOps over the past couple of years has been:

  1. the expansion and adoption of tools,
  2. businesses realizing the necessity of DevSecOps, and,
  3. software delivery automation.

Tools

Joseph Feiman, Chief Strategy Officer at WhiteHat Security:

A critical step toward DevSecOps has been taken by DevOps itself, which started offering its own application security technologies. Application security vendors, as well as open-source security communities, have started addressing this emerged opportunity as well. They have begun integrating their existing technologies in the unified DevOps, thus serving it with intermediate solutions (intermediate – because those solutions have not been designed for new paradigms). At the same time, those security vendors/communities have been/will be rapidly developing native solutions for the emerged DevOps.

Opportunities for DevSecOps in 2021

I wrote The Future of DevSecOps in June 2019 after gathering insights from IT professionals who foresaw:
  1. greater adoption,
  2. security being ingrained in development, and,
  3. AI/ML-driven automation.
For this article, I’m sharing what IT professionals now see as the potential for DevSecOps. I previously shared how these IT professionals have seen the recent evolution of DevSecOps, as well.

I received input from more than 40 IT professionals. Based on their feedback, the greatest opportunities for DevSecOps are:
  1. Alignment of Organizations,
  2. Security of the Software Pipeline,
  3. Automation, and,
  4. AI/ML.

Alignment

Gregg Ostrowski, Regional CTO at Cisco AppDynamics:
The biggest opportunity comes with the addition of “biz.” In a recent article for DZone, I described how BizDevSecOps is the evolution of DevSecOps, and in many ways, this reality is already here. When developing an application, user experience needs to be a top priority as end users are among the most important stakeholders. This is especially true now when the primary way for a customer to interact with a business is through their digital services. Business teams now have user experience top-of-mind because it drives customer satisfaction and that is a key contributor to revenue. By breaking down silos and incorporating their input into overall DevSecOps, teams can create better-performing and more seamless and secure applications.

Rick Vanover, Senior Director Product Strategy, Veeam:
There is an incredible opportunity for traditional IT organizations to align to the DevSecOps practices today as well as work to modernize legacy platforms. This is important in a post-COVID world as organizations scale and change as the world sets itself on the next normal mode of behavior. Having obsolete platforms and applications does not align to the agility requirements of today, much less tomorrow.

Gary Duan, CTO at NeuVector:
Better integration of purpose-built toolings for development, monitoring, threat visibility, and protection throughout the entire pipeline and at runtime. Security automation and real-time protection are the key criteria for ensuring the success of the DevSecOps movement.
 
Saif Gunja, Director of Product Marketing, DevOps, Jack Marsal, Director of Product Marketing, Cybersecurity, and Ajay Gandhi, VP Product Marketing of Dynatrace:
There is a real opportunity for a BizDevSecOps approach to application security to form a new focus for digital transformation. Traditional app security models are buckling under the pressure of dynamic cloud-native environments and applications like Kubernetes, mobile, and serverless. The monitoring tools most organizations deploy to catch vulnerabilities create blind spots and bottlenecks that are only growing. This problem is made worse by siloed teams, manual processes, and outdated approaches that leave vulnerabilities missed in preproduction and production environments. In fact, 93% of CIOs say IT’s ability to maximize value for the business is hindered by challenges like siloed IT and business teams. However, when developers collaborate with ops, or ops with business teams, or the business with developers, everyone can quickly get on the same page, drawing data from a single source of truth.

Dan Hubbard, Chief Product Officer at Lacework:
Alignment of both organizations and architectures. Organizationally security is aligning with the most technical outcomes with developers and more business-driven outcomes with CISO’s. DevSecOps sits in the middle and plays a big role in bridging the gap. 

I believe the biggest opportunity now is being able to actually tie all these DevSecOps requirements, risks, and opportunities into a broader workflow within the organization. Microservices architecture introduces a lot of moving parts. Today, most of these parts are managed as isolated requirements or items. At scale, that makes it really hard to manage, monitor, and secure. I expect to see a tighter workflow between DevOps, DevSecOps, and the overall infrastructure team as part of the continued evolution.
 

Security

Anders Wallgren, VP of Technology Strategy at CloudBees:
There will be an increased focus on the security of the software pipeline itself, as it is a core part of the software supply chain. You may be doing all the right things to your software, but you also need to make sure all the right things are happening in your software delivery pipelines, and that you have control over the security of those pipelines.
 
Jeff Williams, CTO and Co-Founder of Contrast Security:
Most organizations are just getting started with DevSecOps, so there are a ton of opportunities. Focusing on fast and highly accurate Appsec tools for security testing and open source library analysis is a good place to start.  Maturing and expanding threat modeling, standard defenses, Appsec training, and champions program are also strong moves.  One key opportunity is the “SecOps” piece of DevSecOps. Most organizations don’t have visibility into who is attacking them, what attacks they’re using, and which systems they are targeting. This is critical threat intelligence that can both help operations protect the application layer and feedback into the development team. This feedback loop is a great way to build the culture of security innovation and learning that’s at the core of DevSecOps. Supply chain security has also become critical for every organization. DevSecOps must expand its scope to cover these challenges.  There are three parts of the software supply chain to secure.
  1. Your custom code whether developed by staff, consultants, or outsourced. We are pretty bad at this as 20 years of Appsec haven’t moved the needle. Look at IAST and RASP to enhance traditional SAST/DAST/WAF. Note that ordinary Appsec typically only looks for inadvertent mistakes, not malicious code.
  2. Your third-party code, whether OSS or commercial components.  We are also very weak here because with current SCA tools we can’t even stop using libraries with *known* vulns much less deliberately malicious code. RASP can help prevent zero-day library vulns from being exploited.
  3. All the software you use in your software factory: IDEs, build tools, test tools, etc. Little emphasis here currently by defenders. Developer environments are often wide open. An attack here can do anything a malicious developer could do.
All 3 kinds of code in the supply chain are potentially a SolarWinds type debacle. Attackers, who have historically focused on (1) have started probing (2) and (3) in recent years.  We have a lot of work to do to ensure the integrity of the software supply chain. Other industries (electronics, aviation, pharma, etc...) are decades ahead.
 
Deepak Kumar, CEO and founder of Adaptiva:
The greatest opportunities lie in improving the CI/CD (continuous integration/continuous development) pipeline with improved security and tools to help validate third-party code as well as the natively developed code is. Increasing scrutiny on this so-called “shadow code” necessarily improves security, and these additional processes, if implemented properly, can help prevent similar supply-chain attacks in the future.
 
 

Automation

Zeev Avidan, Chief Product Officer at OpenLegacy:
More than ever, teams can develop and deploy confidently in the knowledge that they're meeting corporate security standards. As we see it, the next frontier is the extension of this principle – unlocking team innovation through automation – to the realm of integrations, particularly monolithic core systems, which tend to be the last bastion of centralized IT control.
 
Peter Oggel, Chief Technology Officer at Irdeto:
Automation enables DevSecOps to monitor an attack surface that is increasingly widespread, and almost impossible to monitor without automation technology. This will help prevent organizations from succumbing to cyberattacks with financial and reputational repercussions, while also reducing the risk of non-compliance within regulated industries.

Buck Flannigan, VP Global Partners at Fluree:
Hyper Automation will continue apace with AIOps, but there is a crucial need to “Trust, but (cryptographically) Verify” the data sets being ingested as part of an overall ML Governance strategy.  Privacy and regulatory compliance will increasingly be automated, and attestations require the ability to reproduce the state of code and data going back in increasingly lengthier timeframes.
 
 

AI/ML

Saumitra Das, CTO and Co-founder of Blue Hexagon:
DevSecOps needs to integrate Artificial Intelligence engines for deeper scanning for malicious code into either the build or ship (registry scanning) phase. Just looking for CVEs is a commodity and does not protect against the biggest issue: all the big attacks of 2021 went after unknown CVEs. It is critical to look for malicious code in addition to CVEs.

New approaches to detecting malicious code with very high efficacy and speed are now commercially available. These can be integrated into either the build phase or the ship phase to scan all code that is being put into production. Deep Learning can provide fast verdicts in milliseconds (similar to how self-driving cars make decisions in milliseconds to drive) at scale so that thousands of containers can be deep scanned per day for supply chain and other attack vectors. 


Thanks also, to the following for sharing their insights for this article:

2021: Autonomous Cloud Operations, Agile App Security, Customer-First Approach Will Drive Second Wave in Digital Transformation

The pandemic did not create the need for digital transformation, but it has accelerated it dramatically. Some organizations’ capacity for digital services was, to put it simply, unprepared for the volume of new users. Other organizations were positioned better to adapt quickly because they invested in capabilities such as AI-assistance and continuous automation beforehand.

Last year’s increased demand for digital services and the urgency to adapt quickly to users’ needs will precipitate a rise this year in user experience-driven digital transformations, a renewed need to streamline operations by integrating application security into DevSecOps, and accelerated adoption of autonomous cloud operations.

Can Your Software Development Processes Withstand a Software Supply Chain Attack?

Enterprise software development has graduated from the “waterfall” framework of development and operations - and has become less linear, more complex and, in several ways, more difficult to secure. While contemporary software supply chain practices allow developers to manage that complexity and deliver software efficiently at scale, unaddressed gaps and vulnerabilities within the process continue to be exploited by threat actors.

That’s why security measures within every step of software development and supply chain must take top priority as attacks continue to be directed to the application layer — and often succeed in penetrating the network and executing malicious instructions.

DevSecOps, SecDevOps, or RainbowMonkeyUnicornPony? [Interview with DJ Schleen]

While DevOps is forging boldly into the future, security is still trailing those advances in many organizations. So it’s important that we understand how to apply notions of (traditionally static) security into environments that are built to foster continuous development. I, for one, would like to raise the torch to the fledgling category of DevSecOps and learn how it is successfully implemented by industry leaders. In the first of a series of interviews with DevSecOps community leaders, I chat with DJ Schleen, DevSecOps Advocate at Sonatype.

Helen: I think that the market is light on shared DevSecOps reference architectures to help the community learn and grow. Do you agree and what can we do about it?

DJ: There are a lot of missing pieces out there and I think it's because nobody really knows where to go with it. If you do a search for DevSecOps reference architectures, you're going to see that infinity logo with a bunch of locks around it which doesn't really tell you much. I’ve created this one, but the community does need to share. I think it's because people don't really know which community they're part of; are they part of Secure DevOps, SecDevOps, OpsSecDev? I think there's confusion. So you might see some security reference architectures, but I don't know if they're really taking into consideration flow across the whole technology value stream.

The Edge of Test Automation: DevTestOps and DevSecOps

Don't push me 'cause I'm close to the edge.


The state of software development is evolving rapidly as software developers and testers have started moving to DevOps. In the beginning, organizations used to have a simple process for development and operations, but now users demand a more enhanced and improved experience with every product, which requires proper testing, security, data, AI, and more.

DevSecOps and the Problem of Machine-Scale Data

Machine-scale data might overwhelm your DevSecOps implementation.

“Shifting Left” From DevOps to DevSecOps

When development teams using waterfall approaches couldn’t keep up with customer requirements, they adopted DevOps and Agile SDLCs. While these flexible approaches attempt to meet customer demands, security processes get left behind. You either skip security, or you aren’t really Agile. Either way, you’re losing the benefits of adapting rapidly to customer needs.

Now that new regulations and consumer awareness have made privacy and security a priority, the industry’s recognized that they need to be built into the SDLC. “Shift left” means integrating processes and testing that have traditionally happened at the end into the development process itself, and you often hear that term used to describe a transition from DevOps to DevSecOps.

Top 5 Challenges of DevSecOps and How to Overcome Them

DevSecOps emphasizes the need for better collaboration between development, operations, and security. It is the constant integration of efforts of all teams at every step of the process. The ultimate goal is to move into a world that is automated and synced, making most of the manual tasks obsolete.

But to get there, there are changes to be made not just to the process but to the behavior as well. However, according to a survey by Threat Stack, 68% of companies state that their CEO demands security and DevOps teams not do anything that slows down the business. This is one of the biggest challenges of DevSecOps and why many quit the transition halfway.