Container Image Security part 5: Copa, Trivy and Azure Container Registry

Last year Microsoft released Copa (Project Copacetic), an Open Source Image Vulnerability Patching Tool and has recently been taken on as a sandbox project within CNCF. Copa is a tool that allows you patch vulnerable container images. That sounds like something we need in our pipelines. In this series we have looked at different strategies ranging from preventing Dockerfiles which resulted in vulnerable images from being merged to only allowing trusted/signed images on our cluster.

Truth is, there are many different strategies for developing, deploying and maintaining your container images. Depending on where you are in your journey or what your requirements are, Copa might just be for you!

Long story short: Copa can take the scanning results from vulnerability assessment solutions like Trivy and patch the existing image to mitigate existing vulnerabilities. How cool is that?!

If you want to follow along with the series, here you go!

Container Image Security Part 1: Azure Container Registry
Container Image Security Part 2: Building & The pipeline
Container Image Security Part 3: Image Integrity and Azure Policy
Container Image Security part 4: Azure Policy, Ratify and Notation

To be fair, when playing with this I really needed to get my head in the game to understand what's happening. The out of the box examples needed some modifications for this all to work. However, once up and running it proved to be very stable. Yes, this still is a young project but both Copa and the Copa Github Action are actively maintained and support for different strategies are increasing rapidly. A very promising future.

Okay, let's get to it.

What's about to happen

Credit where credit is due: a good initial workflow can be found here in the Copacetic GitHub Action repository. Additionally, Josh Duffney has written a good series on this which pointed out some ideas to be. However this did not work out of the box for me. I had issues with buildkit not being configured properly on my runners and some permission issues. Eventually I resorted to just running a custom script to grab the Copa version of my liking, this was a tip I found somewhere in a GitHub issue that for the life of me I cannot find anymore. Once I do, this blog post will be "patched" (see what I did there? 😄).

Alright. So we are incorporating different technologies here. The scenario is as follows:

• We have an Azure Container Registry
• In that registry we have an image that has not be scanned and potentially contains vulnerabilities
• We will use Trivy to scan the image
• We will use Copa to patch the image
• Image will then be pushed with a new version number

Alternatively we can build, scan and patch the image in one go, through a workflow but that does require containerd image store support. There are some pull requests in the making to make that a little easier in the copa-github-action repository. However, if you are up for some custom scripting and customizing the docker installation on your runner, it is already possible.

Before we look at the workflow we need to ensure our actions have the appropriate permissions to Azure Container Registry.

We need Pull access for Trivy and Copa and we need Push access for our Docker Push command. If you are following this series, you should already have most of this set up but if you haven't, here is how: https://learn.microsoft.com/en-us/azure/container-instances/container-instances-github-action?tabs=userlevel#configure-github-workflow

Just make sure we have these permissions set up and have our secrets configured in GitHub and we are good to go.

az role assignment create --assignee <ClientId> --scope $registryId --role AcrPush
az role assignment create --assignee <ClientId> --scope $registryId --role AcrPull

The GitHub Workflow

Let's take a look at the GitHub Workflow we will be using.

name: build-scan-fix-push

on:
  push:
    branches: [ "main" ]

permissions:
  contents: read

jobs:
  patchypatch:
    permissions:
      contents: read # for actions/checkout to fetch code
      packages: write
      security-events: write # for github/codeql-action/upload-sarif to upload SARIF results
      actions: read # only required for a private repository by hub/codeql-action/upload-sarif to get the Action run status
    name: patchypatch
    runs-on: ubuntu-latest

    strategy:
      fail-fast: false
      matrix:
        images: ['acrwesdemo001.azurecr.io/backend:v1.0.1']

    steps:
      - name: 'Docker Login'
        uses: azure/docker-login@v1
        with:
          login-server: ${{ secrets.REGISTRY_LOGIN_SERVER }}
          username: ${{ secrets.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}   

      - name: Login to Azure
        id: login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ matrix.images }}
          format: 'json'
          output: 'trivy-results.json'
          ignore-unfixed: true
          vuln-type: 'os'
          severity: 'CRITICAL,HIGH'
        env:
          TRIVY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
          TRIVY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}

      - name: Check vulnerability count
        id: vuln_count
        run: |
          report_file="trivy-results.json"
          vuln_count=$(jq 'if .Results then [.Results[] | select(.Class=="os-pkgs" and .Vulnerabilities!=null) | .Vulnerabilities[]] | length else 0 end' "$report_file")
          echo "vuln_count=$vuln_count" >> $GITHUB_OUTPUT

      - name: Copa patch
        if: steps.vuln_count.outputs.vuln_count > 0
        id: copa_patch
        run: |
          wget https://github.com/project-copacetic/copacetic/releases/download/v${COPA_VERSION}/copa_${COPA_VERSION}_linux_amd64.tar.gz
          tar -xzf copa_${COPA_VERSION}_linux_amd64.tar.gz
          chmod +x copa

          imageName=$(echo ${{ matrix.images }} | cut -d ':' -f1)
          current_tag=$(echo ${{ matrix.images }} | cut -d ':' -f2)

          major=$(echo $current_tag | awk -F. '{print $1}')
          minor=$(echo $current_tag | awk -F. '{print $2}')
          patch=$(echo $current_tag | awk -F. '{print $3}')
          ((patch++))

          new_tag="$major.$minor.$patch"
          patched_tag=$new_tag

          echo "patched_tag=$new_tag" >> $GITHUB_OUTPUT
          echo "imageName=$imageName" >> $GITHUB_OUTPUT

          docker buildx create --use --name builder
          ./copa patch -i ${{ matrix.images }} -r trivy-results.json -t $patched_tag
          echo "copa_patch=true" >> $GITHUB_OUTPUT
        env:
          COPA_VERSION: 0.6.0

      - name: Push patched image
        if: steps.vuln_count.outputs.vuln_count != '0'
        run: |
          docker push ${{ steps.copa_patch.outputs.imageName }}:${{ steps.copa_patch.outputs.patched_tag }}

Okay.. pretty big workflow. We will take a look at the interesting parts.

First of all, we're using a matrix strategy, this allows us to use multiple images. Alternative we can look at parameters for passing the latest image version upon runtime. That all depends on your strategy.

Next we're using Trivy. A more elaborate explanation on Trivy can be found in part 2 of this series: Container image security part 2: building the pipeline. We are making some small adjustments as we are going to authenticate to Azure Container Registry and provide a report to Copa.

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ matrix.images }}
          format: 'json'
          output: 'trivy-results.json'
          ignore-unfixed: true
          vuln-type: 'os'
          severity: 'CRITICAL,HIGH'
        env:
          TRIVY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
          TRIVY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}

For authentication we are using the TRIVY_USERNAME and TRIVY_PASSWORD environment variables and populate them with the secrets for the service principal we have stored in our GitHub Repository. Additionally we changed the output format to 'json' and are generating a file called 'trivy-results.json'. This is the file Copa will look at to determine if there are vulnerabilities that require patching.

After the count has been checked during the step "Check vulnerability count", it is time to run Copa.

     - name: Copa patch
        if: steps.vuln_count.outputs.vuln_count > 0
        id: copa_patch
        run: |
          wget https://github.com/project-copacetic/copacetic/releases/download/v${COPA_VERSION}/copa_${COPA_VERSION}_linux_amd64.tar.gz
          tar -xzf copa_${COPA_VERSION}_linux_amd64.tar.gz
          chmod +x copa

          imageName=$(echo ${{ matrix.images }} | cut -d ':' -f1)
          current_tag=$(echo ${{ matrix.images }} | cut -d ':' -f2)

          major=$(echo $current_tag | awk -F. '{print $1}')
          minor=$(echo $current_tag | awk -F. '{print $2}')
          patch=$(echo $current_tag | awk -F. '{print $3}')
          ((patch++))

          new_tag="$major.$minor.$patch"
          patched_tag=$new_tag

          echo "patched_tag=$new_tag" >> $GITHUB_OUTPUT
          echo "imageName=$imageName" >> $GITHUB_OUTPUT

          docker buildx create --use --name builder
          ./copa patch -i ${{ matrix.images }} -r trivy-results.json -t $patched_tag
          echo "copa_patch=true" >> $GITHUB_OUTPUT
        env:
          COPA_VERSION: 0.6.0

This step will run if the previous step (the counting of the vulnerabilities) returns a value bigger than zero. We are then pulling the desired version of Copa (as defined by COPA_VERSION in the environment variable) and install it.

Now the next step is optional and completely different for everyone. It all depends on your versioning strategy and how you want to go about container image patching. The image I am using has a tag of "v1.0.1" and my goal is to update the patch version once the image is patched. For that we are using awk to strip down the current tag and up the number by one. The result should then be "backend:v1.0.2".

And finally, we push the image to Azure Container Registry.

The result

We started with an image that contained vulnerabilities. Which was also detected By Microsoft Defender Vulnerability Management.

We then run our GitHub workflow and determine everything succeeded and the Copa was executed, which indicates that Trivy found vulnerabilities.

After taking a closer look we can also see that Copa indeed did some patching!

And now for the big moment!

We have an image with the correct tag (v1.0.2) and no vulnerabilities! Personally, I can run this over and over again but I am really loving how this works.

Wrapping up

Again, a million ways to implement this but the fact that we can patch existing container images, so quickly and with relative great ease is amazing. This is just one of the technologies we want in our build and deploy pipelines. Always implement multiple technologies such as Microsoft Defender for Cloud, Ratify, notation and Azure Policy. All of these technologies take a different approach in improving your container image security.