Director of Research, Lacework Labs
Reports on in-the-wild attacks on Kubernetes clusters are somewhat sparse. This coupled with multiple attack vectors prompted us to deploy Kubernetes honeypots with very loose security controls to catch real-world attacks.
Our hypothesis was that an attack would happen quickly through the insecure API and that the attacker would abuse the cluster to deploy coinminers. We were partially correct. In this blog, we detail how our honeypot was compromised, how long it took, and connections we discovered to other cryptojacking campaigns.
Setting up a K8s Honeypot
For this honeypot we wanted an easy to deploy Kubernetes cluster with a small footprint. We also wanted something that would leave certain security controls open, primarily the insecure API. By default, the Kubernetes API serves on two ports: the secure port and the insecure port. The secure port uses TLS and API calls go through the configured authentication and authorization modules. The insecure port uses HTTP, bypassing authentication and authorization modules. Thus, you can see how exposing the insecure API port to the internet would result in a compromise.
Based on the above requirements and some reporting we found on various blogs, we decided microk8s would be a good tool for this experiment. MicroK8s is a single node K8s cluster in a Linux snap package. It’s designed for things like offline development, prototyping, testing, and IoT applications. It comes with all the same features you would expect from a full-fledged Kubernetes cluster. Enabling standard services such as DNS, Dashboard, and others is a breeze and can be accomplished with a single command.
Given these features, we simply spun up a honeypot on an EC2 instance and made it accessible to the internet. This includes the Dashboard and insecure API port.
Attack Details & Analysis
Based on Suppoie botnet reporting it seemed that a wide-open Kubernetes honeypot would get scanned and exploited within a couple of days. This was not the case. In fact, every day the insecure API on TCP port 8080 was scanned frequently, with the HTTP response data indicating it was open for the taking. However, it appears that these scans weren’t necessarily looking for open Kubernetes API servers.
Figure 1. Wireshark screenshot of requests to the insecure API.
On Day 31, we saw follow-up requests to the various API endpoints following an initial scan. This time instead of a wide variety of generic User-Agent strings we saw a User-Agent belonging to kubectl.
Figure 2. Wireshark screenshot of requests from kubectl to the insecure API.
After querying all the API various endpoints, the remote IP (220.127.116.11) made a POST request to add a ReplicationController. This ReplicationController provides five replicas of a CentOS image that runs a series of curl commands to download XMRig, its configuration file, and then executes it.
Figure 3. Wireshark screenshot of a POST request to create a ReplicationController containing pods for Monero mining.
One would think an attacker would probably pull an image that already has cryptojacking tools setup, however, this method allows the attacker to pull a clean image and also gives them the flexibility to change and update the commands (such as the C2 IP).
Figure 4. A pod description for a Monero mining pod.
The app name for the pods used in this attack “myresd02” is interesting as we can use it to track compromises on other Kubernetes clusters. More details on this in future posts. The attacker uses the following config file for XMRig.
Figure 5. XMRig configuration file.
In this config we note the user (Monero address) 4AB31XZu3bKeUWtwGQ43ZadTKCfCzq3wra6yNbKdsucpRfgofJP3YwqDiTutrufk8D17D7xw1zPGyMspv8Lqwwg36V5chYg and mining pools 18.104.22.168:3333, 22.214.171.124:3333, and 126.96.36.199:3333. In the following sections, we show details behind the attack IP, mining pools, Monero address, and C2 IP (used for download).
In May 2018 Fortinet observed this IP (188.8.131.52) hosting a shell script used to pull and run docker images that utilize XMRig for mining Monero. In February 2018, a security blogger noted the IP being used to serve a miner that was used in attacks against Redis. IBM X-Force also put out an advisory about the IP, mentioning its use in hosting cryptocurrency miners and being attributed to the 8220 Gang reported by Cisco Talos. The 360 Threat Intelligence Center also blogged about this IP being used as a C2 in attacks against Hadoop Yarn and associates the activity with the “8220 Mining Group.” In June 2018, Kromtech reported on 17 malicious Docker images they discovered on Docker Hub. This IP address was used as one of the C2s found in a compromised image. It’s also worth mentioning that Kromtech describes Kubernetes related compromises via internet accessible Kubelet ports.
Monero Address & Private Mining Pools
Taking a look at the Monero address 4AB31XZu3bKeUWtwGQ43ZadTKCfCzq3wra6yNbKdsucpRfgofJP3YwqDiTutrufk8D17D7xw1zPGyMspv8Lqwwg36V5chYg and private mining pools 184.108.40.206:3333, 220.127.116.11:3333, and 18.104.22.168:3333 we see a number of reports about cryptojacking activity.
During June 2018 there was a Medium blog post about a Hadoop YARN attack. In September 2018, Security Boulevard reported on Sustes malware targeting IoT devices and Linux servers. Also in September 2018, there was another report of a Kubernetes compromise.There is a GitHub page from December 2018 labeled “microk8s crypto mining exploit.” Also in December 2018, the address is mentioned in the comprehensive and previously mentioned write-up from Cisco Talos on tracking active cryptojacking campaigns.
This is just to name a few, there are multiple hits on samples in VirusTotal and HybridAnalysis as well.
Command and Control IP
There wasn’t much prior reporting on the C2 IP (22.214.171.124) that hosted the XMRig binary and config file. There were a couple of comments on blogs in mid-March 2019 that found similar curl commands as those seen on the attack on our Kubernetes cluster.
Based on prior reporting on cryptojacking campaigns and infrastructure used in this attack it appears this activity is likely a fairly new attack vector among a suite of attack vectors for an on-going campaign. Actors using overlapping infrastructure have used many methods of compromising hosts from malicious docker images, to attacks against Hadoop YARN, to multiple attacks against Kubernetes.
In the future, we will be sharing research on Kubernetes clusters with exposed insecure API ports globally along with findings on attacks against them. Our honeypot took about a month to be compromised, but it is safe to say that the vast majority of Kubernetes clusters exposed in the same manner are currently compromised by this cryptojacking campaign or similar ones to it.
If you would like to learn more about how Lacework provides cloud workload security solutions that detect attacks like this, follow this link to kick off a Free Cloud Risk & Threat Assessment to learn more.
Private Monero pool
Private Monero pool
Private Monero pool
XMRig config file