Container Image Security Part 3: Image Integrity and Azure Policy

We have discussed how we can ensure the security of our container images in part one of this series, in part two we have configured container image scanning in our pipeline. Now it's time to validate our container images at run time. After all, in a process with so many steps from building to deployment and runtime there are many things that can go wrong, and many steps were malicious actors can compromise the integrity of our containers.

Container Image Signing

The first question: Why would we sign our container images? Well, it is all about trust. Is the container image really who it says it is? An image can be stored in our container registry, but that doesn't mean we know who put it there or that we have a way to verify it. Look at it differently: Look at it differently: Why do you have a passport or an ID card? To prove who you are and provide authorities with the means to verify.

Upon writing this I decided we need a part 4 as well. As there are two approaches we can leverage here:

• We can use Image Integrity (which runs using Ratify) and leverage Azure Policy to detect unsigned images.
• We can use full blown Ratify to prevent unsigned images from even running on our cluster.

Both require different approaches. And one can be more complex than the other. For this post we are going to focus on Image Integrity. We are saving a full blown Ratify configuration and deployment for the next post!

A couple of technologies we are going to use here:

• Azure Container Registry to store the container images.
• Azure Kubernetes Service to run the containers.
• Azure Policy addon for AKS.
• Azure Key Vault to store the certificates.
• Notation to sign our image.
• Ratify to detect if our container images are signed.

Configuration and Deployment

We are going to deploy the appropriate configuration, based on the documentation by Microsoft: here.

We will somewhat deviate from the documentation as we have more articles to write and will prepare the steps to come!

Before we start we need to register the features: ImageIntegrityPreview and AzurePolicyExternalData.

As you can see, it clearly states "Preview". Even though we are in the process of building a production ready scenario, this feature should not be used in production just yet. Regardless, I am very excited about it.

az feature register --namespace "Microsoft.ContainerService" --name "EnableImageIntegrityPreview"
az feature register --namespace "Microsoft.ContainerService" --name "AKS-AzurePolicyExternalData"
az provider register --namespace Microsoft.ContainerService

While we wait for the features to be registered, some good-to-knows:

We need the ImageIntegrityPreview feature as this will deploy Ratify on our cluster once we apply the appropriate Azure Policy. The AKS-AzurePolicyExternalData feature we need for a number of reasons. Azure Policy deploys an implementation of Gatekeeper on the AKS Cluster. Gatekeeper comes with many features but until this feature was released, we were missing a very specific one called External Data. This allows Gatekeeper to interact with external sources such as container registries and Azure Key Vaults. Sounds like something we need if we wish to verify the integrity of our images with an external source. Before the availability of this feature to deploy and manage our own version of Gatekeeper, essentially breaking the out of the box integration with Azure Policy.

Right.. Enough chit chats. Once the features are registered we need to add additionally capabilities to our cluster. Specifically, we are going to enable OIDC and workload identity.

az aks update -g rg-aks-wesh02 -n aks-wesh02 --enable-oidc-issuer --enable-workload-identity 
az aks show -n aks-wesh02 -g rg-aks-wesh02 --query "oidcIssuerProfile.issuerUrl" -otsv

Please note, we don't actually need the workload identity for the Image Integrity checks, but we will in the next article as workload identity is the technology, we will use to authenticate to our external sources from our cluster with Ratify. Better to be prepared.

Now it's time to configure Azure Policy and start monitoring our running containers. We will do this as follows:

$subscriptionID = "SubscriptionID"
$resourceGroup = "rg-aks-wesh02"
$location = "west europe"
$scope = "/subscriptions/$subscriptionID/resourceGroups/$resourceGroup"

az policy assignment create --name 'deploy-trustedimages' --policy-set-definition 'af28bf8b-c669-4dd3-9137-1e68fdc61bd6' --display-name 'Audit deployment with unsigned container images' --scope $scope --mi-system-assigned --role Contributor --identity-scope $scope --location $location

This will deploy an Azure Policy Initiative containing three policies:

• [Image Integrity] Kubernetes clusters should only use images signed by Notation.
• [Preview]: Deploy Image Integrity on Azure Kubernetes Service.
• Deploy Azure Policy Add-on to Azure Kubernetes Service clusters.

The first policy uses the Audit effect and ensures Azure Policy will report back when a container image is not signed using notation. Please note that currently only image signed by Notation are supported and only the Audit effect is supported.

The second policy will deploy a Ratify pod on the AKS cluster and the third policy ensures the Azure Policy Add-on is configured for AKS. The latter you obviously don't need as you always enable Azure Policy for AKS! 😄

After a little while, we should be able to see a Ratify deployment running within the gatekeeper-system namespace.

We just need one more thing. We need to configure our Certificate Store and Verifier for Ratify. As we are using the demo images provided by Deislabs, we are also going to use the examples provided by them and Microsoft as documented here.

apiVersion: config.ratify.deislabs.io/v1beta1
kind: CertificateStore
metadata:
  name: certstore-inline
spec:
  provider: inline
  parameters:
    value: |
      -----BEGIN CERTIFICATE-----
      MIIDQzCCAiugAwIBAgIUDxHQ9JxxmnrLWTA5rAtIZCzY8mMwDQYJKoZIhvcNAQEL
      BQAwKTEPMA0GA1UECgwGUmF0aWZ5MRYwFAYDVQQDDA1SYXRpZnkgU2FtcGxlMB4X
      DTIzMDYyOTA1MjgzMloXDTMzMDYyNjA1MjgzMlowKTEPMA0GA1UECgwGUmF0aWZ5
      MRYwFAYDVQQDDA1SYXRpZnkgU2FtcGxlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
      MIIBCgKCAQEAshmsL2VM9ojhgTVUUuEsZro9jfI27VKZJ4naWSHJihmOki7IoZS8
      3/3ATpkE1lGbduJ77M9UxQbEW1PnESB0bWtMQtjIbser3mFCn15yz4nBXiTIu/K4
      FYv6HVdc6/cds3jgfEFNw/8RVMBUGNUiSEWa1lV1zDM2v/8GekUr6SNvMyqtY8oo
      ItwxfUvlhgMNlLgd96mVnnPVLmPkCmXFN9iBMhSce6sn6P9oDIB+pr1ZpE4F5bwa
      gRBg2tWN3Tz9H/z2a51Xbn7hCT5OLBRlkorHJl2HKKRoXz1hBgR8xOL+zRySH9Qo
      3yx6WvluYDNfVbCREzKJf9fFiQeVe0EJOwIDAQABo2MwYTAdBgNVHQ4EFgQUKzci
      EKCDwPBn4I1YZ+sDdnxEir4wHwYDVR0jBBgwFoAUKzciEKCDwPBn4I1YZ+sDdnxE
      ir4wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAgQwDQYJKoZIhvcNAQEL
      BQADggEBAGh6duwc1MvV+PUYvIkDfgj158KtYX+bv4PmcV/aemQUoArqM1ECYFjt
      BlBVmTRJA0lijU5I0oZje80zW7P8M8pra0BM6x3cPnh/oZGrsuMizd4h5b5TnwuJ
      hRvKFFUVeHn9kORbyQwRQ5SpL8cRGyYp+T6ncEmo0jdIOM5dgfdhwHgb+i3TejcF
      90sUs65zovUjv1wa11SqOdu12cCj/MYp+H8j2lpaLL2t0cbFJlBY6DNJgxr5qync
      cz8gbXrZmNbzC7W5QK5J7fcx6tlffOpt5cm427f9NiK2tira50HU7gC3HJkbiSTp
      Xw10iXXMZzSbQ0/Hj2BF4B40WfAkgRg=
      -----END CERTIFICATE-----
---
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Store
metadata:
  name: store-oras
spec:
  name: oras
---
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-notary-inline
spec:
  name: notation
  artifactTypes: application/vnd.cncf.notary.signature
  parameters:
    verificationCertStores:  # certificates for validating signatures
      certs: # name of the trustStore
        - certstore-inline # name of the certificate store CRD to include in this trustStore
    trustPolicyDoc: # policy language that indicates which identities are trusted to produce artifacts
      version: "1.0"
      trustPolicies:
        - name: default
          registryScopes:
            - "*"
          signatureVerification:
            level: strict
          trustStores:
            - ca:certs
          trustedIdentities:
            - "*"

We save this as a file and then simply kubectl apply -f .\filename.yaml.

This deploys the store we need and the verifier configuration so that we can verify images that are being used for our running containers.

Please note, this is a very basic configuration, but Ratify allows you to do so much more (can't wait for the next post!). I found this part in particular very, very cool:

SBOM Validation | Ratify

This document outlines how Ratify can be used to verify SBOM (Software bill of material). The sbom verifier is added as a plugin to the Ratify verification framework. Currently the SBOM verifier is in 2.0.0-alpha release, and it supports the following SBOM validation:

Ratify

But... Like I said, more on advanced usage of Ratify in the next blog post. We will now deploy two containers from images hosted by Deislabs. One signed image and one unsigned image.

Now we wait for Azure Policy to report back. Fingers crossed.

As we deployed two pods one signed and one unsigned Azure Policy should report back that we have one container that is non-compliant. And.. it does!

That is correct. None of the containers running as part of our AKS deployment in the namespaces kube-system, gatekeeper-system, etc. are signed by Notation. The default deployment of the policy excludes these namespaces like so:

If it wouldn't we would have a lot more non-compliant containers. However, having the ability to modify these parameters allows us to fine tune the auditing to our liking.

Wrapping up

Container image signing allows to build a trust relationship and verify the integrity of our images. Ratify plays an important part in this and looks very promising for the future. Especially with the development Microsoft is doing in extending the capabilities of the Azure Policy gatekeeper implementation and even deploying Ratify as part of the specific Azure Policy we used in this post.

But there's more. We are just scratching the surface when it comes to container image signing. In this post we just verified a signed image. The next step is to use Notation to sign our own images, leverage Azure Key Vault and then use Ratify to validate our images before a Pod is started. Preventing these uncompliant images from even running on our cluster. Stay tuned!