Container Image Security Part 1: Azure Container Registry
When managing virtual machines, we have a practice of pushing security updates as soon as possible. After all, the longer a machine (Operating system or packages) is vulnerable, the bigger the risk. Unfortunately, when it comes to container images not everyone is as proactive. We need to treat container images as what they are, an operating system (even though very limited) that requires maintenance.
As an example in this post, I am using a container image built from: https://github.com/OWASP/vulnerable-container-hub which contains several vulnerabilities and even allows you to host CTF events. Perfect!
The story
Container images take quite the journey. From being built as part of a development cycle to being pushed to a registry and they will eventually end up running on a platform such as Kubernetes.
Each step in that journey can either positively or negatively impact your container image security and each step requires different techniques to mitigate potential threats. There is a proactive and reactive aspect of this. For known issues and CVE's, we can proactively scan images once we've built them, or even inspect the Dockerfile for potential vulnerable base images. But, what if a CVE is discovered for a package that is already part of your running container image and is already stored in your container registry? For this post we are going to focus on the registry, that happy place where all container images end up. As we are using Azure, we will be using Azure Container Registry.
Configuration and Deployment
Before we can test and play with this, we need a couple of things:
- Azure Container Registry with the vulnerable image (Example here)
- Azure Kubernetes Service cluster integrated with Azure Container Registry (Example here)
- Vulnerability Assessment for Azure with Microsoft Defender Vulnerability Management (https://learn.microsoft.com/en-us/azure/defender-for-cloud/agentless-vulnerability-assessment-azure)
The vulnerability assessment solution is the new way going forward to detect vulnerabilities inside our container images and during runtime! Please note that we do need to explicitly enable this for our Microsoft Defender for Cloud Plan. This feature is called "Agentless container vulnerability management" and can be found in your Microsoft Defender for Cloud environment settings under [Settings -> Defender Plans]
Once we have all that deployed and enabled, we can push our image to the Azure Container Registry. If you are unsure how to do that, the Microsoft Learn page will help you but, too long and don't want to read? Here it goes:
# Pull the code and Docker file from https://github.com/OWASP/vulnerable-container-hub
git clone https://github.com/OWASP/vulnerable-container-hub
cd .\vulnerable-container-hub\
git submodule init
git submodule update
cd .\ctf\CTF-web-to-system-01\
# Build the container image locally
docker build . -t vulnerableimage:0.0.1
# Login to Azure
az login
az acr login --name acrwesdemo001 # Make sure you use your own registry name here!
# Tag and push
docker tag vulnerableimage:0.0.1 acrwesdemo001.azurecr.io/vulnerableimage:0.0.1
docker push acrwesdemo001.azurecr.io/vulnerableimage:0.0.1
# Run the pod on Azure Kubernetes service
kubectl run vulnerablepod6531 --image=acrwesdemo001.azurecr.io/vulnerableimage:0.0.1
As a result, we should end up with a container image hosted in our Azure Container Registry and a pod running on our Azure Kubernetes Service cluster.
We can quickly inspect whether the pod is running using kubectl get pods
What's happening, how does this work?
Alright, so what's next? Well next we wait! Luckily, we don't have to wait for long. Once Microsoft Defender for Cloud is set up correctly it will only take a couple of minutes for Azure Container Registry to detect our vulnerable image.
On push of the image to the registry, Azure Container Registry triggers a scan of the image that will look for vulnerable OS packages and language specific packages and will report back through Microsoft Defender for Cloud.
Remember how we said new CVEs can be released in the future? No worries, the vulnerability assessment solution will rescan your images once every 24 hours.
The vulnerability assessment configuration leverages two specific policies/recommendations:
- Azure registry container images should have vulnerabilities resolved (powered by Microsoft Defender Vulnerability Management)
- Azure running container images should have vulnerabilities resolved (powered by Microsoft Defender Vulnerability Management)
And what do you know! We are getting some alerts!
Some investigation shows us that we have a container image with 662 vulnerabilities! Whoopsie. And, to make things worse, this image is also running on our cluster!
We determined that our detection works. So what is next? There is only one option: Remediate. But how do we go about that? This is all very manual. The answer, as in most cases is automation, automation, automation. What kind of automation is really up to you, do you want to automatically purge or quarantine images? Maybe, but that does means part of your solution might break; no image means no container running.
Let's be clear though: if you have an image with 662 vulnerabilities that is currently running in your cluster the answer is always: shut it down. However there are alternative options to be discovered. For example:
- Automate using different features that Microsoft Defender for Cloud provides such as the continuous export of alerts to Azure Event Hub and take it from there (https://learn.microsoft.com/en-us/azure/defender-for-cloud/alerts-overview).
- Stream alerts to Azure Sentinel
- Use Azure Resource Graph to query for vulnerabilities.
And then you decide what to do with the alerts. Perhaps you want to automatically create an issue or work item in your GitHub or Azure DevOps environment. Perhaps create a ticket in your ITSM system? Really, anything goes as long as you focus on having that feedback loop. That is the crucial part of success here: the feedback loop.
Bonus material
In this post we just focused on two recommendations/policies specifically for container images. If you are running Azure Kubernetes Service, there are more options, building on top of the same Microsoft Defender for Cloud concepts we discussed here. If you are interested, check out this blog post by Richard Hooper:
Pixel Robots.
Wrapping up
Azure provides very nice features for detecting vulnerabilities in your container images. However, the level of security depends on how you leverage those features. In this post we focused on the container image inside the registry, but good container image security requires action on different levels. A vulnerable image is not "fixed" inside Azure Container Registry, in fact we need to take that feedback and make sure we build a new container image that does not have these vulnerabilities. That may require interaction with your development team (what happens if you upgrade packages? Does everything still work?). And, maybe we can even perform some detection before we even push the image to the registry? Well that's for part two of this series!