Your etcd is Showing: Thousands of Clusters Open to the Internet
By submitting this form, you agree to our privacy policy.
James Condon
Director of Research, Lacework Labs
Usage of the distributed key-value store etcd is at an all-time high. The fastest-growing open source project Kubernetes uses etcd to store data critical to the operation of its clusters. Like many open-source, easy to use data stores, the simplicity of setup is a double-edged sword. Admins running the datastore must handle implementing security configuration and controls. In this blog post we look at unsecured etcd clusters exposed to the internet and discuss security considerations for deploying etcd.
Background
etcd is an open source, distributed key-value store that provides a reliable way to store data across a cluster of machines. It has been around since 2013 and currently Kubernetes uses etcd as its backend for storing cluster state and configurations, in addition to providing service discovery. It is fast, benchmarked at 10,000 writes/sec, and also provides simple to use REST and gRPC APIs. It listens on TCP port 2379 for client communications and TCP port 2380 for server-to-server communications. Similar to other popular distributed key-value stores like Redis, it’s easy to use and set up, but security implementation is up to the user.
Prior to etcd 2.1 authentication did not exist. Currently, etcd supports TLS and client certificate authentication. These features aren’t enabled during a default install and must be configured by the user. If authentication isn’t properly setup, anyone with access to the API can access all the keys in etcd.
Shodan & etcd
In early in 2018, Giovanni Collazo blogged about the default authentication issue and reported that Shodan showed 2,284 open to the internet, many of which contained highly sensitive information. It’s been 3.5 years since etcd 2.1 was released and close to a year since Collazo reported his Shodan findings on etcd, surely the number of servers has declined a bit?
Not so much according to recent Shodan search there are 2,498 accessible etcd clusters.
Figure 1. Shodan research results for etcd.
Interestingly, the most common cloud providers were Alibaba, Amazon, and Tencent.
Figure 2. Breakdown of Top Organizations.
We didn’t go as far as to interrogate any of the identified servers, however, Shodan provides interesting banner information about the exposed etcd clusters. Taking a closer look at the data available from Shodan we saw a number of interesting etcd names related to Kubernetes such as:
- “kube-master-1”
- “kubemaster_etcd”
- “kubeadm-master2”
- “k8s_node2”
- “K8s-cluster-etcd”
Some of the cluster names identify the potential organizations of the cluster owners. The earliest etcd version we saw was 2.1.1 with 3.2.22 being the most common. Interestingly, there were a handful of etcd clusters with reported uptimes since 2014-2016.
Honeypot
Having an etcd cluster with no authentication open to the internet is dangerous. Anyone with access could dump all the keys by issuing the following curl command:
curl http://<ip address>:2379/v2/keys/?recursive=true
But how many malicious actors are looking to do this? Is it as frequent as attacks on similar products like Redis? It’s hard to say, but we did place an etcd honeypot out to see what would happen. To do this we simply spun up an Ubuntu 18.04 server. We installed etcd using:
$ apt install etcd
This install has etcd bound to localhost, so we need to edit the config file in /etc/default/etcd. Specifically we need to add the server’s IP address to the variables
ETCD_LISTEN_CLIENT_URLS
ETCD_ADVERTISE_CLIENT_URLS
Following this we start the service by running the commands:
$ systemctl enable etcd
$ systemctl start etcd
Lastly we verified that we could access the REST API from an external IP address.
After letting the server sit for a few days we only observed two unique hosts hitting the API. The first host attempted to query the etcd version but was missing a host header.
Figure 3. Malformed etcd HTTP request.
The second host repeatedly made request for the etcd keys, however the HTTP method was HEAD as opposed to GET, so the keys were not returned. It’s not clear whether this was intentional (research perhaps) or an unintentional mistake.
Figure 4. HTTP HEAD request for etcd keys.
We will continue to monitor for an interesting interactions.
Security Considerations
The risk of having even read-only access to etcd can escalate quickly. In a Kubernetes deployment, this could lead to the compromise of the entire cluster. If you are using etcd be sure to follow a number of best practices
- Restrict network API access to only hosts that need it. For example in a Kubernetes deployment only allow the Kubernetes API server network access to the etcd cluster.
- Enable authentication, using a least privilege approach for users and roles.
- Use etcd TLS capabilities for peer and client communications as well as client certificate authentication.
- Encrypt etcd data at rest.
- Enable audit logging.
- Use a security product like Lacework to monitor misconfigurations and unauthorized network access.
Monitoring & Detecting in Lacework
Lacework baselines the behavior of workloads, applications, and network communications in your cloud environment. Misconfigurations, such as exposed etcd APIs, are highlighted by anomalous network connections.
Figure 5. Event triage card for anomalous connections to etcd.
Final Thoughts
Misconfigurations are very common in today’s cloud deployments. This coupled with open source tools rely heavily on proper deployment by its users. To mitigate these risks it’s important to continually monitor your API endpoints for exposure to the internet.
Photo by Matt Artz on Unsplash
