Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.It was first developed by Google and contributed to Open Source in 2014, and is now maintained by the Cloud Native Computing Foundation.
By default Kubernetes consists of components which is the control plane called Cluster. This cluster of Kubernetes consists of the worker machines called as Nodes. Nodes run containerized application. Components of the application workload are called as Pods which is hosted by worker nodes.
Kubernetes Control Plane has the following components
- kube-api-server – This is the frontend of the Kubernetes Control Plane which exposes the Kubernetes API.
- etcd – Key value store used as Kubernetes backing store for all cluster data.
- kube-scheduler – Watches for newly created Pods with no assigned nodes.
- kube-controller-manager – Runs different controller processes such as Node controller, Replication controller, Endpoints controller, Service Account and Token Controller.
- cloud-controller-manager – Embeds cloud specific control logic. This lets you link your cluster with the cloud providers’ API and separates the components that interact with your local cluster and cloud platform
Below listed are Node components which maintain running pods and provide Kubernetes Runtime Environment
- kubelet – It makes sure that containers are running in a Pod. An agent that runs on each node in the cluster.
- kube-proxy – Network proxy that run on each node in the cluster. It maintains network rules on nodes.
- Container runtime – This is responsible for running containers. Several container runtimes are supported by Kubernetes.
Kubernetes built-in security features
Kubernetes offers several security features which are inbuilt to help secure the above mentioned components.
RBAC
RBAC known as Role Based Access Control lets you specify what actions can be performed by any user using the Kubernetes API. This is achieved by defining Roles and Cluster Roles.
Pod Security Policies
Pod security policies are kind of opposite to that of RBAC. RBAC restricts users based on role whereas Pod security policy restricts the actions that pods can perform. For example we can create a policy which restricts containers from running as root.
Network Policies
Kubernetes network segmentation policy is a crucial key security control that helps you to control the flow of network between different pods and endpoints within the cluster. This helps you to lock down or block the networks within the cluster. In short we can say that this policy restrict communication between the Pods.
Secrets Management
Kubernetes has a built-in secrets management framework which helps you in keeping the secrets such as passwords, SSH Keys etc. secure. This can be done using kubectl command.
Network Encryption
As a default security configuration Kubernetes uses TLS to encrypt the network traffic.
Kubernetes native security provides the following advantages
- Increased Protection
- Greater Operational Efficiency
- Reduced Operational Risk
With the rise of technology, attack vectors also increases rapidly. The more we try to defend; the more hackers find ways to break the security. Therefore conducting regular pentest on the development/production environment is a good practice. In this tutorial we are going to write about Kubernetes Penetration Testing. Kuberenetes too has a wide range of security issues just like other platforms such as
- Unsecured Kubelet API
- Unrestricted Resource Access
- Unprotected Helm Tiller service
- Insecure Default Configuration
- Sensitive cloud metadata unrestricted
- Secrets not protected adequately
- Lack of Network Policy
- Internal services unprotected without Ingress authentication
- Unauthenticated etcd access
- Privileged/root containers
- Excessive service account privileges
Even though Kubernetes attack vectors are different from targets such as Windows or Linux, Azure Security Centre has mapped the attack techniques and has created the first Kubernetes attack matrix using ATT&CK matrix.
Kubernetes Security Measures
As we can see, there are many attack vectors for Kubernetes and we need to take the needed security measures to stay ahead of the attackers. On a small scale deployment or when we use experimental clusters it is easy to talk about the security as it is easy to manage. But when we have large deployment which comprises of tens or hundreds of pods, then we need to have a broad strategy. In this tutorial we will list out the security measures that can be followed to keep the Kubernetes environment secure.
- Use Namespace: Work with namespaces for Authentication, Authorization and Access Controls. Namespace provide reference for network policies, access control and other security controls it is a key isolation boundary for Kubernetes.
- Harden Pod Security and Network Security Policies: Pod security and Network security policies can help to enforce security restrictions as stated above. Even though Kubernetes has these security policies, by default these are not enabled. Even if they are enabled, you need to scrutinize the policies according to your needs. Level of hardening depends upon how far you are using restrictive policy such as preventing write access and limiting the non-essential network traffic.
- Reusable and Broad Policies: Write RBAC policies or Pod Security Policies that can be reused which can be applied to the entire cluster whenever the cluster changes or you redeploy the application. Therefore to be efficient write policies in a broader view such that the policies can be applied to the entire cluster rather than applying it for a single resource. Even though writing broad policies are not applicable all the time, Always try to write policies in a broader perspective as possible as it can be.
- Host Security: Kubernetes will not monitor or harden the host operating system and therefore it is the sole responsibility of the Admin who deploys Kubernetes to harden the base operating system which powers its nodes. Single node with an out-dated OS can lead to compromising all the machines. It is advised to choose Linux as the base OS and specifically enable SELinux, AppArmor or any other related security framework on the host system which is the best practice. These tools add extra layer of protection for the host system and protect it from certain exploits.
- Secure Communication across various services: Kubernetes communicate using API. Since it is API drive we should restrict user access or specify who can access the clusters as a security measure. This can be done by using Transport Layer Security (TLS) and certificate management between api-server, etcd, kubelet etc.
- Image Provenance and Scanning: Never deploy code from unknown sources. For Kubernetes, using images from known registries which are on allow list. Deploy phase based scan results and enforce policies for image scanning. Enforce validation of Admission Controls which is a feature of Kubernetes which will reject the deployment of the image if it was specified without scanning results or if the image build is more than 90 days old. Clair is one of the open source image scanner which can be used for this purpose.
- Allow List: The process of leveraging allow list will help to identify unexpected process running. This is done by initially identifying all processes that are executed normally and then using this list you can create allow list against the behaviour of future application.
- Secrets Management: Secrets are nothing but sensitive data Keys and Passwords required by the application. Using late-binding is the best practice for managing secrets and defer secrets from loading from secrets-store to the application run-time which is the initialization phase of the pod. Hashicorp Vault and the open source Nirmata Vault Client can be used for this purpose.
- Identity Management: By default Kubernetes does not provide any construct to manage user identities even though we have RBAC which provides control over access of entities. Using Central Identity Provider such as Active Directory or other directories to manage user identities is the best practice. While managing multiple clusters across different infrastructure, it is good for enterprises to use Single Sign-On (SSO).
- Compare and Analyse Runtime Activity: Replicas of the Containerized Applications that were created should behave identical. If the replicas are found to have significant deviations, then it needs to be looked upon for further investigation. Therefore it is advised to integrate your Kubernetes security tools with external systems such as SIEM, mail, slack, Google Cloud Security Centre etc. to raise an alert the responsible team to notify them about a potential threat that is detected.
- Logging and Auditing: Logging helps to get insights into security breaches and helps for investigating the past events. Even though Kubernetes generates log data, there is no feature provided for auditing or interpreting that data. Therefore you need to use third party application such as Sumo Logic for leveraging the log data as a basic security operation.
Cyberarch Consulting helps Organizations/Enterprises to secure their Docker/Kubernetes environment by conducting extensive Penetration Testing and providing security recommendations. Cyberarch Consulting has many years of experienced penetration testers who holds some of the high level certification in the industry.
Author : Meshach.M – Senior Security Consultant at Cyberarch Consulting
