Sam Aydlette

Cloud service providers entering regulated markets should consider what NYC restaurateurs know. You won't get far without a skilled maître d'. By focusing on an "upscale clientele," the maître d' preserves the exclusivity of hot nightlife locations. Similarly, having a CI/CD pipeline that enforces the use of hardened components is like having a skilled maître d' for your deployments - very selective of who gets in and unimpressed by your excuses about why your sketchy package should be on the list. Cloud service providers can elevate their security, lower overall operating costs, and unlock opportunities in multiple regulated markets by ensuring that what is deployed into production is already secure.

Why Deploying Hardened Components via DevSecOps is Necessary: The Big Picture

General purpose images typically include unnecessary services, unused libraries and default credentials. Using general purpose images is especially risky because supply chain attacks are increasing. Sophisticated threat actors aren't just targeting your perimeter anymore, they're targeting your software vendors to create a master key to every door before you even install them. Ensuring adequate provenance and integrity of your components is no longer optional. Simply put, organizations that aren't hardening will get breached, if they haven't been already. The 2024 Sophos State of ransomware report indicated that organizations with unpatched vulnerabilities were 2.8 times more likely to experience ransomware.

Due to this heightened threat landscape, maintaining a continuous inventory of hardened components has become the foundation that all compliance frameworks are rapidly moving towards these days. You can't patch once a quarter and expect to squeak through an audit for much longer. Ignore this at your peril, unless you enjoy explaining audit findings to executives who suddenly develop an intense interest in the technical details of your "SSH into EC2 and then run bash scripts" approach to code deployments.

By using hardened cloud components, organizations can eliminate vulnerabilities before deployment rather than frantically patching systems on live production hosts while praying that nothing breaks too badly. Integrating security into CI/CD also provides organizations with the flexibility to more easily modularize their cloud model to enter different regulated markets using the same hardened tech stack across multiple compliance requirements and sovereign geographic locations.

How to Implement Hardening

Organizations seeking to implement hardened components in their cloud environments have two options: building an internal hardening process (the "we own the pain" approach) or leveraging pre-hardened images from specialized vendors (the "please take my money and make this problem go away" approach).

Option 1: DIY Hardening

Hardening objectives are often based on established frameworks like Security Technical Implementation Guides (STIGs) developed by the Defense Information Systems Agency (DISA). STIGs are meant to apply to your entire stack and many options exist for operating systems, databases, applications and network devices.

Applying STIGs typically involves a four-step process that combines bureaucracy, technical challenges, and existential dread:

1. Baseline creation: Developing hardened base images for each component you are deploying.
2. Automated configuration management: Using configuration management and validation tools to maintain consistency.
3. Deviation management: Creating exceptions where you either accept or lower the risk of not applying a specific hardening rule.
4. Continuous monitoring: Constantly verifying that nobody has quietly disabled all your security controls when you weren't looking.

While effective in theory, this approach comes with its share of challenges.

• Resource intensive: Every time new vendor patches are released or STIGs are updated, a new build is required. This ongoing process of building images to strip unneeded functionality and remove CVEs is resource intensive.
• Expertise requirements: Organizations need specialized knowledge across multiple technologies, and these unicorn employees are hard to find and even harder to retain.
• Management overhead: Hardened components must be thoroughly tested to ensure they don't break application functionality. The more complex the tech stack becomes, the more dependencies become increasingly unwieldy.

Generally speaking, only the largest or most advanced organizations have the expertise and the economy of scale necessary to make "in house" component hardening worth it. Even then, it's hard to pull off. For everyone else, it's probably better to pay someone to do it for you.

Option 2: Buying Pre-Hardened Images from Specialized Vendors

The alternative approach is leveraging pre-hardened images from specialized vendors, who provide components specifically designed with security as a primary focus. This is the "I have money and would prefer to keep my sanity" solution.

Cost Considerations

While likely less expensive than doing everything yourself, buying pre-hardened images from a vendor is not cheap, and there are still additional costs. These include:

• Licensing costs: Subscription fees based on deployment scale
• Integration expenses: Resources required to incorporate hardened images into existing pipelines
• Training requirements: Staff may need training on new tooling
• Support costs: Enterprise support agreements for when things inevitably break at 3 AM on a holiday weekend

Vendor Reliance

Relying on a 3rd party vendor for hardened images also means that you are adding risk to your supply chain. A vendor may go out of business suddenly or get acquired and decide to stop supporting the images you use. If this happens, you may have to quickly migrate off of the purchased images en masse, which is risky.

Point being, there are substantial costs either way. If I figure out how to make money without investing any, I'll probably stop writing these articles. However, buying pre-hardened images from a quality vendor provides a number of advantages:

• Broad ecosystem support: Vendors typically provide hardened versions of popular components (NGINX, Redis, PostgreSQL, etc.) faster than your team could possibly harden them all
• Cloud provider integration: Pre-built solutions for major cloud platforms that work (almost) perfectly out of the box
• Customization options: Ability to tailor hardened images to specific requirements
• Deployment flexibility: Support for various environmental needs, from VMs to containers to whatever new deployment paradigm emerges next week

The Compounding Benefits of Deploying Hardened Components

Regardless of implementation approach, hardened components deliver significant security advantages for cloud environments.

Minimal CVEs and Faster Detection and Response

Hardened components dramatically reduce the number of potential vulnerabilities by:

• Eliminating unnecessary packages: Removing software that isn't required but is just there, such as development tools, documentation files, or unused language runtimes
• Configuring secure defaults: Because default settings are typically about as secure as using "password" as your password
• Regular updates: Maintained images receive security patches promptly, instead of sitting in your "I'll get to it later" queue for six months

This approach improves security metrics across the board:

• Faster Mean Time to Detection (MTTD): With fewer components to monitor, security teams can identify issues faster
• Faster Mean Time to Remediation (MTTR): Standardized environments make patching and updates more straightforward
• Overall risk posture improvement: The attack surface is reduced so significantly you might actually get to sleep through the night occasionally

Multi-Cloud Inventory of Hardened Components

Package URL (PURL) has emerged as a crucial standard for component identification across diverse ecosystems. As a precise, decentralized identification method, PURL offers significant advantages, bringing order to chaos like a particularly effective librarian. The best part of using PURL is that it works across any tech stack or cloud infrastructure - you can deploy hardened images and identify them with PURL regardless of whether they are running in AWS, Azure, GCP or on-premise:

• Unambiguous identification: Each component is uniquely identifiable regardless of source or repository, eliminating the "which version is this again?" problem
• Cross-platform consistency: Works the same across languages, package managers, and deployment models
• Automation support: Enables automated inventory tracking and vulnerability correlation
• Supply chain transparency: Facilitates clear tracking of component origins and dependencies

Organizations leveraging PURL can maintain accurate inventories of all deployed components across complex tech stacks, enabling rapid identification of affected systems when new vulnerabilities emerge. Below is an example of a PURL for a specific PostgreSQL image:

Supply Chain Security with Build-Time SBOMs

Software Bills of Materials (SBOMs) are another essential artifact for demonstrating security, the "list of ingredients" that lets you proudly show your customers what exactly they are consuming on an ongoing basis. Generating built-time SBOMs from hardened components facilitates:

• Comprehensive dependency visibility: Clear documentation of all included software, including the dependencies of dependencies of dependencies
• Provenance verification: Evidence of component origins and build processes
• Automated policy enforcement: SBOMs enable automated checks against security policies, catching violations before they make it into production
• Compliance documentation: Ready evidence for regulatory requirements, saving you from panic-generating documentation the night before an audit

Build-time SBOM generation ensures that security is integrated from the earliest stages of deployment rather than being assessed retroactively.

Build Integrity with SLSA Framework

Supply chain Levels for Software Artifacts (SLSA, pronounced "salsa") is a framework that provides a “chain of custody” demonstrating that your deployment hasn't been tampered with. SLSA offers a security framework with progressive levels of maturity. Organizations implementing SLSA alongside hardened components gain significant advantages: - **Build provenance**: Cryptographically verifiable records of how, when, and where your components were built - **Chain of custody**: Complete visibility into every handoff in your pipeline, ensuring nothing suspicious happened between source code and deployment - **Automated verification**: Integration into CI/CD to automatically reject components that don't meet your required SLSA level - **Progressive security maturity**: The ability to start simply and gradually increase your security posture in measurable increments Here's an example of SLSA provenance data for the hardened component that we identified with PURL earlier:

Bonus: Achieving Economies of Scale

Many cloud service providers leverage the same or similar sets of components. For example, many vendors likely use Alpine Linux container images, NGINX web servers, and MongoDB databases. These and similar sets of hardened components, or alternatives like Chainguard's Wolfi that offer similar functionality, may be offered as a consolidated product that takes advantage of maximized efficiency to lower overall cost to different groups within a broader organization and the cloud industry as a whole.

SLSA doesn't just tell you that your components were hardened; it provides cryptographic proof of exactly how they were hardened and that they haven't been compromised since.

Putting It Together: Enforcing Security Outcomes with Infrastructure as Code

Infrastructure as Code (IaC) serves as the mechanism that integrates hardened components, PURL tracking, SBOM documentation, and SLSA validation into a cohesive security posture. IaC enables automated policy enforcement throughout your software development lifecycle, ensuring that all components meet specific security outcomes before they reach production. Your CI/CD pipelines become security checkpoints, automatically rejecting builds that incorporate vulnerable components or lack proper provenance and attestation. When hardened components are managed through IaC, your build environments maintain their integrity through immutable infrastructure principles. Each deployment uses pre-vetted, minimal-CVE images that remain consistent across environments, eliminating configuration drift and unauthorized modifications. The PURL identification system seamlessly integrates with this approach, creating a universal addressing scheme that uniquely identifies each component regardless of its source ecosystem. This standardized identification enables precise tracking across disparate environments and simplifies vulnerability correlation. SLSA is also applied directly in your deployment pipeline code. Each build generates cryptographically signed attestations that verify the integrity of your software supply chain, creating an unbroken chain of custody from source to deployment. This approach is complemented by automated SBOM generation, which documents all dependencies for every deployment artifact. The result is a comprehensive security framework where images must come from approved registries, contain verifiable documentation, include signed attestations, and pass vulnerability scans before deployment. This integrated approach creates a multiplicative security effect. Your CI/CD pipeline becomes an end-to-end trust model for your deployments, the ultimate maître d' for your production environment.​​​​​​​​​​​​​​​​