Many businesses are adopting containers as a foundational technology used to manage and run their applications. If you’ve worked much with containers, it’s easy to see why: they enable entirely new levels of portability and scalability. But the adoption of containers, like any other new technology, also means new ways to exploit applications.
Depending on the container’s configuration, an exploited application can eventually lead to the compromise of the host that the container is running on. There are also other implications to consider, such as potential secrets stored as environment variables in the container and what they have access to. If you want to know more about Docker containers security best practices specifically, GitGuardian proposes a useful cheat sheet.
With the exponential increase in container adoption, it's more critical than ever for teams to ensure that proper security and threat management infrastructure and practices are in place. This Refcard presents a comprehensive examination of threat detection for containerized environments, spanning several focus areas such as common cloud security architectures and Kubernetes hardening guidelines. And central to this Refcard are the fundamentals of container threat detection, including concepts like resource limits, static image vulnerability scanning, configuration validation, and much more.
Introduction to log4j2 Mitigation
The log4j2 vulnerability like the OpenSSL Heartbleed and Apache Struts vulnerabilities that came before it are poignant reminders to digital businesses that it’s not just enough to respond to a vulnerability by redeploying applications once a patch is available, you also have to be able to discover instances of the vulnerability being exploited in real time in your production platform and stop them. In this tutorial, we’ll show you how to use Deepfence ThreatMapper and ThreatStryker to help you do just that.
Deepfence ThreatMapper is an open-source security observability platform that hunts for vulnerabilities – including log4j2 – in applications in production across containers, Kubernetes, clouds, serverless environments, VMs, and bare metal, and then ranks them based on their risk of exploit. ThreatMapper eliminates the noise and false positives generated by scanning tools by further calculating the risk of exploit for each of these vulnerabilities, so that you can target the issues that present the greatest risk to the security of your applications.
In my previous two articles, we discussed Kubernetes security and created a guideline for enhancing K8s. If you haven't read them yet, here are the links:
Hardening Your Kubernetes Cluster - Threat Model (Pt. 1)
Covering the essentials of security in Kubernetes environments, this Refcard addresses the three primary areas of attack within a Kubernetes cluster. Security concepts range from the software supply chain — images, build systems, and container registry security — to Kubernetes infrastructure, as well as deploy-time and runtime security. Key examples like threat vectors, security measures, and vulnerability and violation types within each section will help you continue strengthening your Kubernetes environment security as you automate and scale the deployment and management of your cloud-native applications.
Container security is the process of implementing security tools and policies to protect the container, its application, and performance, including infrastructure, software supply chain, system tools, system libraries, and runtime against security threats.
Runtime security is a critical piece in a cloud-native security story. Access control and policy enforcement are important prevention techniques, but runtime security is needed to detect threats that evade preventions.
Container registries serve as libraries to store and access third-party container images required during the build phase of the SDLC and the images produced for deployment to test, staging, and production environments. While public container registries are accessible and convenient, private registries can better integrate into existing CI/CD workflows, offer greater control over access and security, as well as help ensure build repeatability and reliability. This Refcard covers key container concepts and terminology; common use cases; and guidelines for container registry configuration, operation, security, and storage.
So you have your Docker Containers deployed, which in turn are hosting critical applications of your organization? Great! So far, so good!
For the interest of the organization, it remains extremely crucial to keep not only the Containers but also the hosted applications protected from security threats. By default, a deployed Docker originally remains secured through an auto-generated profile docker-default for its containers. This profile, however, provides moderate security on the application level, and thus it remains highly recommended to implement a security profile through AppArmor which works at the process/program level of an application.
Introduction
Securing a container platform is a multi-step process spanning from development to production. The process of securing containers is continuous. It should be integrated into your development process, automated to remove the number of manual touchpoints, and extended into the maintenance and operation of the underlying infrastructure. Container security thus is the process of implementing security tools and policies to ensure that your container is running as intended.
There are a lot of Docker security practices and Kubernetes security practices. But there are gaps: there could be Image vulnerabilities or container abnormalities still. Some of these can be captured and addressed using various static container image scanning tools. But there is a need to analyze the container behavior at run-time to detect any bogus configurations, which are intentional or not, leading to data loss, security intrusions, and eventually leading to different vulnerabilities. If a container is not running as expected, then it could attack that exploited an existing vulnerability. So the run-time container scanning is essential to detect any action that deviates from the norm.
Significant changes were made to the Istio service mesh in its version 1.5 release earlier this year, including notable modifications to the control plane architecture and the creation of a single model for extending Istio and its Envoy proxies using WebAssembly. Istio’s latest quarterly release, version 1.6, at first glance may seem to carry less weight in comparison, however, this update includes several important enhancements that continue to improve its operability.
Installation and Configuration Management
Reducing Upgrade Risks
Istio 1.6 introduces canary support for upgrading versions of the Istio control plane, enabling users to deploy numerous releases of Istiod within the same cluster and migrate pods to a newer version. This will significantly reduce any risks that arise when carrying out upgrades in a production cluster. When installing new control plane versions, the istioctl the command-line tool now supports assigning names to versions that can be utilized when assigning workloads to each specific Istiod version running in the service mesh.
Despite the maturity of the platform, security is still a big challenge for Kubernetes users. While Kubernetes offers maximum flexibility, modularity, and ease of use in other areas, the complex nature of Kubernetes-based environments means securing the cloud environment completely is a complex task to complete.
There are a lot of tools and services that focus on improving security for Kubernetes. Aqua Security, however, is the most comprehensive one on the market. (If you haven’t heard of Aqua Security yet, read my previous blog article here.) Using tried and tested technology, Aqua Security is capable of securing the entire Kubernetes environment with a holistic approach. It is an all-in-one Kubernetes security tool.
In this article, readers will learn about each best practice in Open Source Kubernetes as well as Oracle’s Kubernetes managed service (OKE) running on Oracle Cloud Infrastructure (OCI).
Kubernetes has gained rapid traction over the last three years and is being deployed in production by many companies. While in general, Kubernetes does follow the core software security principles, some ownership of security falls on the shoulders of the end users. Just like a shared security responsibility model exists between all cloud providers and the customers, there is a shared security responsibility for managed Kubernetes services being offered by cloud providers. Managed Kubernetes Services Cloud providers like Oracle Cloud Infrastructure Container Engine for Kubernetes (also known as Oracle Kubernetes Engine or OKE), Azure Kubernetes Service (AKS), and others are typically responsible for managing and securing the control plane (API Server, scheduler, etcd, controllers) of the Kubernetes cluster and customers of the managed service are responsible for the securing the data plane (node pools, ingress, networking, service mesh etc).
Ideally, highly-threaded I/O intensive Linux containers running on Kubernetes would have all the CPU time they need. But just how compatible is that goal with reality? To find the answer – and optimize Linux containers – application developers and DevOps teams must understand how Linux schedules tasks and allocates them CPU time.
The goal behind “real-time” containers is enabling your most important containers – those with mission-critical requirements around time-sensitive performance and reliability – to share the same hardware as non-real-time containers. But before committing to this strategy, it’s important to first determine how practical and achievable this is. In a strategy reliant on real-time containers, under-resourcing those containers could produce application performance shortfalls and hamper security efforts.
Docker as Root
Docker runs its containers as root. But does your workload really needs root permissions? The answer is rarely. Still, your containers, by default, continue to run as a root-user. This could have serious security concerns. A process that runs inside the container as root is in fact a process running as root on the host itself. This provides an opportunity for a malicious attempt to gain unrestricted access to the host itself.
You can check it by yourself, just use the following command on any image that you commonly use:
The use of containers continues to rise in popularity in enterprise environments, increasing the need for a means to manage and orchestrate them. There’s no dispute that Kubernetes (K8s) has emerged as the market leader in container orchestration for cloud-native environments.
Since Kubernetes plays a critical role in managing who and what could be done with containerized workloads, security should be well-understood and managed. It is therefore essential to use the right deployment architecture and security best practices for all deployments.
Security has long been a sticking point for many DevOps teams (including my own, at a Canadian insurance and financial services co-operative). While available tools have enabled automation across plenty of other parts of our CI/CD pipeline — and made automated deployment of our container-based applications the norm — security automation has largely lagged behind.
Like most DevOps teams, we put automated vulnerability scanning into place, but the manual effort of building security policies to safeguard production application workloads remained a pain point.
To understand the current and future state of Kubernetes (K8s) in the enterprise, we gathered insights from IT executives at 22 companies. We asked, "How does K8s help you secure containers?" Here’s what we learned.
Microservices, Security, and Kubernetes (K8s)
RBAC
- K8s helps with authorization and authentication via workload identity. Role-based access control (RBAC) is provided out of the box. Service mesh ensures communication between microservices.
- K8s solves more problems than it creates. RBAC enforces relationships between resources like pod security policies to control the level of access pods have to each other, how many resources pods have access to. It’s too complicated but K8s provides tools to build a more stable infrastructure.
- RBAC mechanisms. Good security profile and posture. K8s provides access and mechanisms to use other things to secure containers.
- K8s provides a pluggable infrastructure so you can customize it to the security demands of your organization. The API was purpose-built for extensibility to ensure, for example, that you can scan workloads before they go into production clusters. You can apply RBAC rules for who can access your environments, and you can use webhooks for all kinds of sophisticated desired-state security and compliance policies that mitigate operational, security, and compliance risk.
- The advantage of K8s is in its open-source ecosystem, which offers several security tools like CIS Benchmarks, Network Policy, Istio, Grafeas, Clair, etc. that can help you enforce security. K8s also has comprehensive support for RBAC on System and Custom resources. Container firewalls help enforce network security to K8s. Due to the increased autonomy of microservices deployed as pods in K8s, having a thorough vulnerability assessment on each service, change control enforcement on the security architecture, and strict security enforcement is critical to fending off security threats. Things like automated monitoring-auditing-alerting, OS hardening, and continuous system patching must be done.
- The great part about the industry adopting K8s as the de facto standard for orchestration means that many talented people have collaboratively focused on building secure best practices into the core system, such as RBAC, namespaces, and admission controllers. We take the security of our own architecture and that of our customers very seriously, and the project and other open-source supporting projects releasing patches and new versions quickly in the event of common vulnerabilities and exposures (CVE) makes it possible for us to always ensure we are keeping systems fully up to date. We have built-in support for automated K8s upgrades. We are always rolling out new versions and patches and providing our users with the notifications and tooling to keep their systems up to date and secure.
Reduced Exposure
- You have services deployed individually in separated containers. Even if someone gets into a container, they’re only getting into one microservice, rather than being able to get into an entire server infrastructure. Integration with Istio provides virtualization and security on the access side.
- Since the beginning, containers have been perceived as a potential security threat because you are running these entities on the same machine with low isolation. There's a perceived risk of data leakage, moving from one container to another. I see the security benefits of containers and K8s outweigh the risks because containers tend to be much smaller than a VM running NGINX will run a full OS with many processes and servers. Containers have far less exposure and attack surface. You can keep containers clean and small for minimal attack surface. K8s has a lot of security functionality embedded in the platform. Security is turned on by default with K8s. When you turn it on, there are a lot of security features in the platform. The microservices and container model is based on immutable infrastructure. You offer limited access to the container running the application. You're able to lock down containers in a better way and be in charge of what our application does. We are now using ML to understand what service or set of containers are doing and we can lock down the service.
- You need to look at security differently with containers to be more secure. Containers allow you to reduce the attack surface with fewer privileges. Limit access to the production environment. K8s has a number of security mechanisms built-in like authentication and authorization to control access to resources. You need to learn to configure. Be careful about setting the right privileges.
K8s Security Policies
- My team helps with full and automatic application discovery spread across multiple data centers and cloud, and creating clean infrastructure policies and runtime discovery. Dynamic policies help lock down containers and apps built on top of containers.
- You’re only as secure as you design yourself to be in the first place. By automating concepts and constructs of where things go using rules and stabilizing the environment, it eliminates a lot of human errors that occur in a manual configuration process. K8s standardizes the way we deploy containers.
- Namespaces, pod security policies, and network layer firewalling where we just define a policy using Calico and then kernel-level security that’s easy for us since we’re running on top of K8s. Kaneko runs Docker builds inside of Docker. Kaneko came out of Google.
- 1) Helps us with features like namespaces to segregate networks, from a team perspective. 2) Second is network policies. This pod cannot talk to that database and helps prevent mal-use of software. 3) Theres benefits from K8s protecting individual containers. This mitigates problems escaping from containers and helps you stay isolated.
Other
- It’s not built-in per se, that’s why there are a number of security tools coming up. Things like scanning Docker images. as K8s does not scan. A number of security companies are coming out with continuous scanning of Docker images before they are deployed, looking for security vulnerabilities during the SDLC. DevSecOps moves security checking and scanning to occur earlier in the development process. There are tools that are popping up to do that.
- If you enable the security capabilities provided, it’s an advantage. There are capabilities in K8s that control whether you have the ability to pull up a container. It has to be set up correctly. You need to learn to use the capabilities. You need to think about the security of every container.
- Security is a very important topic. One area is open source and the level of involvement and the number of developers involved can help drive greater security in the environment. Cloud-native security with the code and the cluster. For customers to leverage K8s in the cloud it changes the investment you have to make because you are inheriting the security capabilities of the cloud provider and dramatically lowering costs. K8s has API-level automation built-in.
- Our container images are created using the Linux package security update mechanism to ensure the images include the latest security patches. Further, our container image is published to the Red Hat Container Catalog which requires these security measures to be applied as part of the publishing process. In addition, domain and database administrative commands are authenticated using TLS secure certificate authentication and LDAP, as well, domain meta-data, application SQL commands, and user data communications are all protected using the AES-256-CTR encryption cipher.
- K8s provides only minimal structures for security, and it is largely the responsibility of implementers to provide security. You can build quite a lot on top of the Secrets API, such as implementing TLS in your applications or using it to store password objects.
- K8s-orchestrated containerized environments and microservices present a large attack surface. The highly-dynamic container-to-container communications internal to these environments offer an opportune space for attacks to grow and escalate if they aren’t detected and thwarted. At the same time, K8s itself is drawing attackers’ attention: just last year a critical vulnerability exposing the K8s API server presented the first major known weakness in K8s security, but certainly not the last. To secure K8s environments, enterprises must introduce effective container network security and host security, which must include the visibility to closely monitor container traffic. Enterprise environments must be protected along each of the many vectors through which attackers may attempt entry. To do so, developers should implement security strategies featuring layer-7 inspection (capable of identifying any possible application layer issues). At the same time, the rise of production container environments that handle personally identifiable information (PII) has made data loss prevention a key security concern. This is especially true considering industry and governmental regulatory compliance requirements dictating how sensitive data must be handled and protected.
Here’s who shared their insights:
Do you have the keys to unlock DevOps security?
The rapid adoption of container technology, DevOps practices and principals, microservices application architectures, and the rise of Kubernetes as the de facto standard for container orchestration are the key drivers of modern digital transformation. Whether an application is built in the cloud, on-premises, or in hybrid environments using container technologies, or it's being ported to a containerized infrastructure, containerization has clear advantages in terms of scalability, portability, and continuous development and improvement.
In a medium article, Tinder's Engineering Team recently announced their move to Kubernetes to solve scale and stability challenges. Twitter is another company that has announced their own migration from Mesos to Kubernetes. New York Times, Reddit, Airbnb, and Pintrest are just a few more examples.
