Container registries – Security considerations

Introduction

Container registries play a crucial role in developing and deploying cloud-native software applications. They act as repositories to store and distribute container images, which are executable software bundles that include all dependencies needed to run an application - the code, a runtime, and libraries. Container registries enable developers to pull and push images, facilitating easy version control, replication, and software distribution across various environments and systems. Understanding the security considerations of using container registries helps maintain your applications' integrity and protect them from potential vulnerabilities.

In container registries, a repository is a collection of related Docker images, often versions of the same application, grouped together. In simpler terms, if we compare a container registry to a library, the repository is like a book series, while each image is a specific release in that series.

This blog post will review security considerations when designing a container registry setup.

Public or Private

The first consideration is to choose between a public or private IP for the container registry. Some considerations are:

1. Accessibility: Public registries are reachable to anyone online, which means the possibility of unauthorized access is higher. Private registries, on the other hand, are usually only accessible within a specific network, reducing the risk of unauthorized access. Unless the container registry is intended to be open for anyone, the registry needs some form of Authentication and Authorization.
2. Defense in depth: We have one additional layer of security with private registries. Should there be a zero-day breach or bug in the container registry implementation from a cloud provider, you reduce the consequences by keeping it offline on the public Internet.
3. Vulnerabilities and Attacks: Public registries are more exposed to potential vulnerabilities and attacks, such as DDoS attacks, because they are accessible to anyone on the internet. Private registries, accessible to fewer people, can be easier to protect.

While public registries may offer more accessibility and ease of use, private registries offer stronger security controls and reduce the implication in zero-day bug scenarios.

Topology Complexity

The security evaluation of a setup can not be limited to only one setting, e.g., public or private container registries.

A complex setup with many components might be worse than a more straightforward setup because:

• It is more difficult to get an overview of the solution, which might lead to wrong configurations
• Potentially larger attack area
• Increase the time to recovery or bugfixes due to complicated release processes
• Reluctance to make setup changes due to complexity

This especially holds in cloud scenarios where the components are often fully managed and have a low chance of bugs. I have seen elaborate setups where there were private build agents, network pairing, routing, and configuration only to keep the container registries offline.

Number of registries

The number of registries depends on the desired level of isolation. Some options are:

• One container registry for all teams and environments
• One container registry per team but shared across the environments
• One container registry/team/environment

With one container registry for all teams and environments, you get reduced cost and efficient resource utilization. It is also easier to pull in the same image in the test environment that was tested in dev. However, container registries have limited functionality to scope certain repository names to individual teams. As a consequence, the teams might race to claim certain image names. Azure Container Registry uses the concept of *Tokens* and Scope Maps to access subsets of repositories. This could have been a viable solution to restrict teams and what repository names they can use. However, you need to manually update the Scope Maps whenever there are new repositories, which quickly becomes a management overhead.

With one container registry/team/environment, there is a complexity overhead of copying images from one registry to another as an application version progresses from development to test to production. However, this becomes a more viable solution in trunk-based scenarios with only one environment.

A balance is to use one container registry per team but shared across that team's environments. This setup avoids naming collisions and makes it easier to reduce the blast radius in case of bugs or accidental container registry deletions. The individual teams can also configure their individual container registries as they see fit.

Image encryption

Some container registries automatically encrypt their images at rest, e.g., Azure Container Registry. In the case of Azure Container registry, it is also possible to provide your own key. It is good practice not to store any secrets in the images, and encryption at rest with a provided key is primarily used for regulatory reasons, in my experience. Bringing your own key increases the topology complexity.

Image versioning

Do not use the latest tag when versioning your container images. It is a bad idea for the following reasons:

• latest does not necessarily indicate the latest version
• New pods might not get new versions without the imagePullPolicy: Always setting

Instead, create new versions with new tags.

Azure Kubernetes Service (AKS) does not have built-in capabilities to clean up old Azure Container Registry (ACR) images. However, ACR provides built-in capabilities to automatically clean up old images using ACR Tasks. ACR Tasks can be configured with custom policies to delete images older than a certain number of days or keep only a certain number of images. This allows you to manage your storage use within ACR and prevent it from becoming cluttered with outdated or unused images.

Image signing

Consider signing the images during creation and verify the signature before running the containers. This can be valuable to a well-crafted, secure software delivery flow. Another variant that can achieve the same output is to use image SHA´s as tags.

Container Registry access

Another attack vector is pushing new images to a container registry access. When designing who/what can push images to a container registry, it is important to consider the following:

• Avoid creating principals/service accounts that can push to all registries but scope it down to a team or system. In one registry/team, we can create a dedicated principal for that registry, for example.
• Create a dedicated principal/service account per application for even more granular access. This clarifies access logs as the principal is for single application use.
• Limit the registry access to principals/service accounts from deployment pipelines. Ideally, a user should only be able to read the container registry and use pipelines/automation for anything else.
• Carefully consider the pipeline principal/service account access to avoid leaking credentials and privilege escalation, and consider any consequences of pipeline changes.

Conclusion

In conclusion, the security considerations for container registries are multi-faceted and require careful planning and execution. Key factors such as the choice between public or private registries, the complexity of topology, the number of registries, and the implementation of image encryption and versioning can significantly impact the overall security of your applications. Furthermore, managing access to the container registry is crucial to maintain control and limit potential vulnerabilities. By understanding these aspects and making informed decisions, you can optimize the security of your container registries and ensure the safe and efficient operation of your cloud-native applications.