Enhancing Native Kubernetes Security
August 7, 2019
By 2022, 75% of enterprises (a sharp rise from 30% in 2019) are expected to run containerized applications globally. The visible benefits of automating container orchestration with Kubernetes is a key force behind this rapid adoption. Unfortunately, this also increases risks as a compromise in Kubernetes can potentially impair the entire containerized environment.
The hyper-dynamic nature of the containerized environment makes Kubernetes security all the more challenging. The challenges would only worsen with rapid adoption as more deployments would uncover new gaps and vulnerability those went unnoticed so far (an example is CVE-2018-1002105, a privilege escalation vulnerability uncovered in 2018). Widespread deployment would also make Kubernetes an attractive target for the bad actors.
Old-school security with predefined policies and signatures was never designed to keep pace with the constant flux of containerized environments. Kubernetes security needs a newer approach.
Dealing with Kubernetes Security Challenges
Kubernetes automates the deployment, monitoring, and maintenance (updates, etc.) of containerized applications. Availability of multiple Kubernetes distributions makes its deployment fairly easy. However, keeping Kubernetes secure is where things get trickier, as it presents a new set of challenges:
New attack vectors
Each Kubernetes pod running one or more containers has a unique IP address for connectivity, which can become a doorway to launch attacks either from external networks or internally (example: by exploiting victims of a phishing attack). A misconfigured container is another attack vector that lets attackers explore weaknesses across the runtime environment. They can connect to other pods running in the same or different hosts to spread the attack. Attackers can also launch a reverse shell in a pod connected to a command/control server to steal data over network tunnels. IP-whitelisting is inadequate to counter these attacks, especially when trusted IP addresses are used.
Containers are spun up and tore down based on capacity demands. This ephemeral nature of containers renders perimeter and IP address-based security controls less effective. Moreover, overlay networks and IP address reuse obfuscate traceability. Logs and other evidence for forensic investigations may be lost when containers are reset in response to a security incident.
Containers do not operate in isolation, and they are not a true sandbox like a hypervisor is for virtual machines. Unlike virtual machines, containers share the host OS that they run on. Unless the orchestration layer can secure its management services an attack compromising one container or the host OS can quickly spread into other containers sharing the same OS.
Kubernetes doesn’t control communication between pods natively. A compromise in one pod can easily spread to other pods. When a pod is infected, there need to be ways to detect the infection and lockdown inter-pod communication.
Increased attack surface
Kubernetes can deploy containers across multiple physical machines and cloud domains which spikes lateral (east-west) traffic across the pods and the corresponding attack surface. Container runtime like Docker, vulnerable container images and the Kubernetes platform expand the attack surface of container infrastructure.
Attackers keep increasing their sophistry to evade detection by conventional security tools. In a recent attack, where an open Kubernetes management console was compromised to run cryptomining, the attackers could hide their server behind a reputable CDN service and made the spike in CPU usage unnoticeable due to throttling. When public services like Dropbox and Google Drive are used to upload stolen data, IP and domain blacklisting are ineffective.
Native security features are limited
Native security features in Kubernetes are not designed to address these challenges. Even when correctly configured, Kubernetes native security is limited to role-based access control to restrict containers’ access to server resources. It doesn’t have capabilities to audit and analyze container contents, detect behavioral abnormalities (privilege escalation, malicious files, etc.) that could signal a threat. Thus, native Kubernetes is not sufficient to protect containerized environments throughout the application lifecycle.
Kubernetes and Docker while automating application deployment, also increase risks since a compromised platform provides inroads to rest of the container resources. This further amplifies the need to enhance Kubernetes security.
Strengthening Kubernetes security
On top of native security, there are some deployment best practices that can provide the first line of defense, such as: use namespaces, enable SELinux, use least-privilege access control, regularly update system patches, verify configurations against CIS benchmarks, etc. However, without proper detection of threats, organizations could unwittingly be granting unauthorized access to Kubernetes clusters, applications, and customer data.
To evaluate the level of security of your Kubernetes deployment, here are some considerations:
- Is there adequate monitoring to flag unauthorized access to Kubernetes cluster by processes and users?
- Is there adequate visibility into communication across containers and pods to detect suspicious traffic?
- Can you ensure file integrity at the service and application layers to defend against malicious software?
- Is there process-level visibility in containers and pods to identify a suspicious chain of events?
In containerized environments, traditional tools are not as effective as tools designed with containers in mind. For example, due to lack of context-awareness traditional host-based security agents cannot enforce container-specific policies in the same host.
An optimal Kubernetes security solution needs to cater to these aspects:
Cyber-threats target organizations at multiple layers. You need a solution to identify the risks and threats for hosts, containers, applications, pods, and the Kubernetes-deployed infrastructure (includes publicly exposed and unsecured API servers, kubelets, and management consoles). File integrity monitoring helps to identify malicious tampering of container configuration and management.
Deep and Persistent Visibility
It is virtually impossible for traditional network and endpoint controls to secure ephemeral containers. You need a mechanism to quickly trace back to the origin of the attack. Attackers usually delete their trail. That’s why process-level traces are crucial. Once the event is logged at a process-level, it cannot be wiped out. This fine-grain, persistent visibility is necessary to prevent threat escapes in a containerized environment.
Automated intrusion detection
Machine learning-based behavioral baselining and automated threat defense can scale to thousands of containers in cloud and datacenter environments. A robust threat detection system monitors and logs all inter-process activities, including those within the same file, process hierarchy, communications between processes, machines, and clusters, changes in user privileges, internal and external data transfers, and so forth.
Audit and compliance
CIS Benchmark for Kubernetes Security is now available to run auditing and compliance checks. Automated auditing tools can continually monitor for Kubernetes misconfigurations and ensure compliance to thwart attacks.
Kubernetes for cloud workloads
Kubernetes allows new workloads to be deployed in minutes. However, it also implies that the attack surface can change more quickly than you can change the security. Traditional cloud security tools were never designed for these new dynamics. You need a solution that can provide continuous and deep visibility into processes at various layers of the container environment to accurately detect and alert potential threats, all across the application lifecycle.