7 min read

Container Image Security Part 3: Image Integrity and Azure Policy

Container Image Security Part 3: Image Integrity and Azure Policy

We have discussed how we can ensure the security of our container images in part one of this series, in part two we have configured container image scanning in our pipeline. Now it's time to validate our container images at run time. After all, in a process with so many steps from building to deployment and runtime there are many things that can go wrong, and many steps were malicious actors can compromise the integrity of our containers.

Container Image Signing

The first question: Why would we sign our container images? Well, it is all about trust. Is the container image really who it says it is? An image can be stored in our container registry, but that doesn't mean we know who put it there or that we have a way to verify it. Look at it differently: Look at it differently: Why do you have a passport or an ID card? To prove who you are and provide authorities with the means to verify.

This article is part one of a series on container image security. In the previous posts we have discussed container image scanning and vulnerability detection when container images are stored, in the second article we looked at vulnerability scanning as part of our build and deploy pipelines.

Upon writing this I decided we need a part 4 as well. As there are two approaches we can leverage here:

  • We can use Image Integrity (which runs using Ratify) and leverage Azure Policy to detect unsigned images.
  • We can use full blown Ratify to prevent unsigned images from even running on our cluster.

Both require different approaches. And one can be more complex than the other. For this post we are going to focus on Image Integrity. We are saving a full blown Ratify configuration and deployment for the next post!

A couple of technologies we are going to use here:

  • Azure Container Registry to store the container images.
  • Azure Kubernetes Service to run the containers.
  • Azure Policy addon for AKS.
  • Azure Key Vault to store the certificates.
  • Notation to sign our image.
  • Ratify to detect if our container images are signed.

Configuration and Deployment

We are going to deploy the appropriate configuration, based on the documentation by Microsoft: here.

We will somewhat deviate from the documentation as we have more articles to write and will prepare the steps to come!

Before we start we need to register the features: ImageIntegrityPreview and AzurePolicyExternalData.

As you can see, it clearly states "Preview". Even though we are in the process of building a production ready scenario, this feature should not be used in production just yet. Regardless, I am very excited about it.

az feature register --namespace "Microsoft.ContainerService" --name "EnableImageIntegrityPreview"
az feature register --namespace "Microsoft.ContainerService" --name "AKS-AzurePolicyExternalData"
az provider register --namespace Microsoft.ContainerService

While we wait for the features to be registered, some good-to-knows:

We need the ImageIntegrityPreview feature as this will deploy Ratify on our cluster once we apply the appropriate Azure Policy. The AKS-AzurePolicyExternalData feature we need for a number of reasons. Azure Policy deploys an implementation of Gatekeeper on the AKS Cluster. Gatekeeper comes with many features but until this feature was released, we were missing a very specific one called External Data. This allows Gatekeeper to interact with external sources such as container registries and Azure Key Vaults. Sounds like something we need if we wish to verify the integrity of our images with an external source. Before the availability of this feature to deploy and manage our own version of Gatekeeper, essentially breaking the out of the box integration with Azure Policy.

Right.. Enough chit chats. Once the features are registered we need to add additionally capabilities to our cluster. Specifically, we are going to enable OIDC and workload identity.

az aks update -g rg-aks-wesh02 -n aks-wesh02 --enable-oidc-issuer --enable-workload-identity 
az aks show -n aks-wesh02 -g rg-aks-wesh02 --query "oidcIssuerProfile.issuerUrl" -otsv

Please note, we don't actually need the workload identity for the Image Integrity checks, but we will in the next article as workload identity is the technology, we will use to authenticate to our external sources from our cluster with Ratify. Better to be prepared.

Now it's time to configure Azure Policy and start monitoring our running containers. We will do this as follows:

$subscriptionID = "SubscriptionID"
$resourceGroup = "rg-aks-wesh02"
$location = "west europe"
$scope = "/subscriptions/$subscriptionID/resourceGroups/$resourceGroup"

az policy assignment create --name 'deploy-trustedimages' --policy-set-definition 'af28bf8b-c669-4dd3-9137-1e68fdc61bd6' --display-name 'Audit deployment with unsigned container images' --scope $scope --mi-system-assigned --role Contributor --identity-scope $scope --location $location

This will deploy an Azure Policy Initiative containing three policies:

  • [Image Integrity] Kubernetes clusters should only use images signed by Notation.
  • [Preview]: Deploy Image Integrity on Azure Kubernetes Service.
  • Deploy Azure Policy Add-on to Azure Kubernetes Service clusters.

The first policy uses the Audit effect and ensures Azure Policy will report back when a container image is not signed using notation. Please note that currently only image signed by Notation are supported and only the Audit effect is supported.

The second policy will deploy a Ratify pod on the AKS cluster and the third policy ensures the Azure Policy Add-on is configured for AKS. The latter you obviously don't need as you always enable Azure Policy for AKS! 😄

After a little while, we should be able to see a Ratify deployment running within the gatekeeper-system namespace.

We just need one more thing. We need to configure our Certificate Store and Verifier for Ratify. As we are using the demo images provided by Deislabs, we are also going to use the examples provided by them and Microsoft as documented here.

apiVersion: config.ratify.deislabs.io/v1beta1
kind: CertificateStore
  name: certstore-inline
  provider: inline
    value: |
      -----BEGIN CERTIFICATE-----
      -----END CERTIFICATE-----
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Store
  name: store-oras
  name: oras
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
  name: verifier-notary-inline
  name: notation
  artifactTypes: application/vnd.cncf.notary.signature
    verificationCertStores:  # certificates for validating signatures
      certs: # name of the trustStore
        - certstore-inline # name of the certificate store CRD to include in this trustStore
    trustPolicyDoc: # policy language that indicates which identities are trusted to produce artifacts
      version: "1.0"
        - name: default
            - "*"
            level: strict
            - ca:certs
            - "*"

We save this as a file and then simply kubectl apply -f .\filename.yaml.

This deploys the store we need and the verifier configuration so that we can verify images that are being used for our running containers.

Please note, this is a very basic configuration, but Ratify allows you to do so much more (can't wait for the next post!). I found this part in particular very, very cool:

SBOM Validation | Ratify
This document outlines how Ratify can be used to verify SBOM (Software bill of material). The sbom verifier is added as a plugin to the Ratify verification framework. Currently the SBOM verifier is in 2.0.0-alpha release, and it supports the following SBOM validation:

But... Like I said, more on advanced usage of Ratify in the next blog post. We will now deploy two containers from images hosted by Deislabs. One signed image and one unsigned image.

Now we wait for Azure Policy to report back. Fingers crossed.

As we deployed two pods one signed and one unsigned Azure Policy should report back that we have one container that is non-compliant. And.. it does!

Hold up! What about all the other containers on my cluster that are not signed by Notation? Should they not also report back as non-compliant?

That is correct. None of the containers running as part of our AKS deployment in the namespaces kube-system, gatekeeper-system, etc. are signed by Notation. The default deployment of the policy excludes these namespaces like so:

If it wouldn't we would have a lot more non-compliant containers. However, having the ability to modify these parameters allows us to fine tune the auditing to our liking.

Wrapping up

Container image signing allows to build a trust relationship and verify the integrity of our images. Ratify plays an important part in this and looks very promising for the future. Especially with the development Microsoft is doing in extending the capabilities of the Azure Policy gatekeeper implementation and even deploying Ratify as part of the specific Azure Policy we used in this post.

But there's more. We are just scratching the surface when it comes to container image signing. In this post we just verified a signed image. The next step is to use Notation to sign our own images, leverage Azure Key Vault and then use Ratify to validate our images before a Pod is started. Preventing these uncompliant images from even running on our cluster. Stay tuned!