Accelerate DevOps By Offering a Certificate Service for CI/CD Pipelines

Application development teams need to move fast. Yet they often need to reinvent the wheel when it comes to machine identities such as SSL/TLS certificates. They frequently create their own security infrastructure, using a combination of Open SSL, secrets management tools, DevOps platforms, and scripts. Then, as environments and tools change, apps are migrated and regulatory frameworks change, those same developers need to spend time re-coding applications, updating scripts. or learning new certificate authority APIs.

Why Do Developers Reinvent the Wheel?

Developers prefer to stay within their existing toolchain and often view Information Security has a barrier rather than an enabler. Often, security processes for SSL/TLS certificates are antiquated and require manual steps such as submitting a ticket, which are incompatible with the dynamic, ephemeral DevOps environments. As a result, developers take on the burden of creating their own security infrastructure, even though they are not PKI experts. This diverts resources away from their core responsibilities, ultimately slowing them down.

How to Install Payara 5 With NGINX and Let’s Encrypt Over Oracle Linux 7.x

From field experience, I must affirm that one of the greatest and stable combinations I've seen is Java Application Servers + Reverse Proxies. Although some of the functionality is a clear overlap, I tend to put reverse proxies in front of application servers for the following reasons (please see this NGINX page for more details):

  • Load balancing: The reverse proxy acts as a traffic cop and could be used as an API gateway for clustered instances/backing services
  • Web acceleration: Most of our modern applications use SPA frameworks, hence it is worth caching all the JS/CSS/HTML files and freeing the application server from that responsibility
  • Security: Most HTTP requests could be intercepted by the reverse proxy before any attempt against the application server, increasing the opportunity to define rules
  • SSL Management: It is easier to install/manage/deploy OpenSSL certificates in Apache/NGINX compared to Java KeyStores. Besides this, Let's Encrypt officially supports NGINX with plugins.

Requirements

To demonstrate this functionality, this tutorial combines the following stack in a classic (non-Docker) way, though most of the concepts could be useful for Docker deployments:

When Machine Identities Go Bad

Managing machine identities, such as SSL/TLS certificates is boring, right? It’s not inspiring work and it’s easily overlooked or forgotten in the day to day onslaught of changes and incidents in a typical enterprise technology department. And they seem like such little things… but when certificates go bad, well, life can turn pretty dark. Here are some real-life nightmares that happened as the result of mismanagement of machine identities.

1. Expired Certificates Delayed Breach Detection

The notorious breach at Equifax — talk about reputational damage, right? Nearly 150 million customer records stolen including date of birth and social security numbers. That’s a lot of people having sleepless nights about ID fraud thanks to an error somewhere in Equifax’s approach to cybersecurity. While the initial attack was performed via a Struts vulnerability (a common one I still frequently see during application scanning), the detection of the breach took 76 days. The reason it took 76 days to detect: misconfiguration of the device inspecting encrypted traffic on the network. The reason for the misconfiguration of the device: a digital certificate that had expired ten months previously.

How to Take the Burden of Machine Identity Management Off the Backs of DevOps

When I moved into an apartment, I didn’t build scaffolding around the building to support a rope and pulley system to lift boxes of my furniture and belongings to the 19th floor. My stuff was put into an elevator with a dedicated shaft, supported by specifically designed mechanical infrastructure and a simple computer system. The latter way is much safer, more effective, and automated.

In my last post, I wrote about how many DevOps practitioners are still manually generating and managing their machine identities, especially TLS certificates. Think about all of the load balancers, servers, containers, virtual machines, and other network entities that are constantly launched and killed within a DevOps environment. They all need machine identities, yet some of those entities have lifespans of only a few hours.