K8s environments seperation

Introduction

While working on my current project, I noticed that namespaces separated K8s environments. This blog post looks at some pros and cons of using namespaces instead of individual clusters for K8s environments.

Kubernetes(K8s) namespace isolation

Kubernetes namespaces provide a mechanism for isolating groups of resources within a single cluster. Here are some key namespace isolation capabilities in Kubernetes:

• Resource Isolation: Namespaces allow you to divide cluster resources among multiple users or teams, providing a scope for names and helping to prevent naming conflicts.
• Access Control: Role-Based Access Control (RBAC) can be applied at the namespace level, allowing fine-grained control over who can access and modify resources within each namespace.
• Resource Quotas: Administrators can set resource quotas for each namespace, limiting the total amount of CPU, memory, and storage consumed by the resources in that namespace.
• Network Policies: These can control traffic flow between pods in different namespaces, providing network-level isolation.
• Service Discovery: Services are typically only discoverable within their namespace, providing a level of isolation for inter-service communication.

GitOps

With techniques like GitOps and Kustomize, we can reduce the chances of deploying resources using the wrong namespace. One example is to set up repo syncing into specific namespaces and use the Kustomize namespace directive to make sure all YAML files only use that spesific namespace:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: my-namespace  # Set the namespace here

resources:
  - deployment.yaml
  - service.yaml

Reducing the chances of mistakes

While namespaces offer a degree of isolation, it's important to note that they do not initially provide the same level of security and resource separation as using separate clusters. Namespaces share the same underlying node resources and cluster-wide components, which can lead to potential security and resource contention issues in multi-tenant environments.

It is possible to use K8s techniques like node tain/tolerations or labels to isolate specific nodes to certain namespaces. Azure AKS also enabled us to create separate subnets, dedicating them to a node pool. With these separate subnets, you can further use Networking policies and Security Groups to segregate teams.

Using separate environment clusters introduces another layer of security against mistakes. We can still use many of the same mechanisms for namespace separation, but those are now limited to the different systems running in the clusters, not the environments. Additionally and most importantly, we keep the YAML for our applications precisely the same between the environments. Whereas with namespace-separated environments, we would need namespaces such as:

• prod-systemA
• test-systemA

This would make it very easy to make mistakes by missing prod for test when referencing K8s services in different namespaces should we need that.

Additionally, we want to promote applications from one environment to another without changing any of the YAML.

Cost

Typically, the managed K8s offerings run and manage the control plane for free. Thus, you only pay for the nodes/VMs, which we could argue would be somewhat the same whether they were added to the same cluster or a new one. With individual environment clusters, there would be an additional resource overhead of installing cross-team tools multiple times, e.g., the Prometheus operator.

Cluster upgrades

Using separate clusters for different environments provides significant advantages when it comes to cluster upgrades:

• Staged upgrades: You can upgrade non-production clusters first, allowing thorough testing before applying changes to production. If issues arise during an upgrade, you can keep the production cluster on a stable version without affecting other environments.
• Independent upgrade paths: If needed, different environments can be on different Kubernetes versions, allowing for more gradual adoption of new features across your infrastructure.

In Azure AKS, it is not possible to downgrade the control plane.

Conclusion

In conclusion, while Kubernetes namespaces offer some level of isolation and resource management within a single cluster, separating environments using individual clusters provides several significant advantages. These include enhanced security, easier environment-specific customization, independent scalability, and more straightforward upgrade processes. Although namespaces can be fortified with additional Kubernetes features like RBAC, resource quotas, and network policies, they share the same underlying infrastructure and control plane.

Separate clusters offer a clearer separation of concerns, reducing the risk of cross-environment issues and allowing for more robust testing and staging of upgrades. They also simplify the promotion of applications between environments without requiring changes to YAML configurations. While managing multiple clusters may be some additional complexity, the benefits in terms of security, flexibility, and reliability often outweigh these challenges.

Ultimately, the choice between using namespaces or separate clusters for environment separation depends on the organization's specific needs, including factors such as security requirements, scalability needs, and operational complexity. However, for many production scenarios, the added isolation and control provided by separate clusters make them a compelling choice for Kubernetes environment separation.