Why visibility and permissions are key to Kubernetes security

Kubernetes (k8s) is a critical infrastructure platform that can help streamline your development and operations. As with any technology layer this complex, there are areas that need specific focus.

Security is definitely one of those and a topic I dove into recently with Dave Trader, Field CISO at Presidio.

This conversation was live streamed on the Lacework LinkedIn page as part of our “Life’s A Breach” series. Be sure to follow Lacework on LinkedIn to find about the next stream.

Why Kubernetes?

A recent survey from Canonical found that 64% of the teams adopting Kubernetes are trying to improve “maintenance, monitoring, and automation.”

That same report quotes Kelsey Hightower, Principle Engineer, Google Cloud on the drive for streamlining the developer and operations experience. One particular line in that quote stood out to me, “A lot of people like the fact that many decisions are just built into K8s, such as logging, monitoring and load balancing.”

The value those built in decisions and infrastructure consolidation deliver to your team is massive. Kubernetes allows you to modernize your development process and focus your operational efforts.

Kubernetes Biggest Challenge

Using that initial stat as a frame, I asked Dave what he sees as the biggest challenge with Kubernetes is in customer environments. Immediately, Dave called out the lack of visibility. I was surprised given the survey and modern approach k8s delivers when it comes to operations. He responded, “…you need to have that visibility as a baseline. And once you have that, then you can have elevated conversations around orchestration.” Therein lies the key.

To get the most out of Kubernetes, you need clear visibility and observability of what’s happening in your environment.

The Basics Of Securing Kubernetes

Before we dive into the more advanced topics and strategies around k8s, make sure you have a strong foundation.

That starts with the excellent project documentation. Specifically, with the “Securing a Cluster” section of the administration tasks guide. This section of the docs walks you through the basics of authentication, encrypting network traffic, restricting workloads and users, and more.

After you’ve researched and implemented the basics, you should work through the available security tutorials.

Next up, you should definitely watch this combination of videos on Kubernetes security from KubeCon 2021 in Europe and North America. Basics, then Advanced.

Both feature Ellen Körbes and Tabitha Sable, two of the top experts in all things k8s.

Visibility

Dave was right. Visibility is critical to the success of your Kubernetes environment. That starts with the logs of the actions taken in and by your clusters. Monitoring and analyzing these logs for abnormal and undesired behavior can direct your security response and fine tune your security practice.

One of the items to look for specifically in the logs is use of various permissions as users and systems with too many permissions are extremely common in any environment.

In an environment as nuanced as k8s, implementing the principle of least privilege is a constant challenge.

Too Many Permissions

In our discussion, Dave made a great point about elevated permissions, “You should have break glass detection…”

What exactly is “break glass detection?”

A break glass account is one that grants elevated privileges for a short amount of time. They are usually used in an emergency situation when the standard processes or people aren’t available to the team.

Detecting when this happens is extremely important. It should be during an incident response but if it isn’t, the team is either about to start working an incident response or kick one off.

This is not an everyday occurrence.

The use of a break glass account is a bit of an edge case. Far more common are teams of developers with far too many permissions in your clusters. The application of permissions can be complicated and frequent mistakes often lead to the “just give me full permissions” solution.

That’s bad.

Knowing what permissions are in use and who is using them will help you get those authorization grants under control.

Automation

So far this post has highlighted some areas of concern for maintenance and monitoring. These areas need to be addressed before you can really start to take advantage of automation…the third reason that people move to Kubernetes.

There are a lot of early wins with basic automation to be had dealing with the visibility and permissions challenges. But to move your security practice forward, you’ll want to leverage more advanced automations.

When you get to that point, the areas that could be of interested early on are:

• Validating a container meets your standards and best practices before if gets deployed to production
• Automating a response to specific actions of interest, like storing secrets
• Making sure that all logging is enabled (including with your CSP if you’re using a managed service) and centralized

What’s Next?

Dave said it best in our conversation, “If we would apply the best in [the] fundamentals that we’ve learned from coming up through the industry. We can secure these environments.”

Start with visibility– it’s the key to security success.

From there, move on to permissions. Kubernetes is a dynamic environment and permissions can be complicated. That’s a challenging combination so you need a strategy to continuously address it.

Finally, look for opportunities to automate everyday tasks and more advanced security ones. Kubernetes is extremely flexible and automating your workflows and environment can help unlock the platform’s power.

Remember to follow Lacework on LinkedIn where we regularly stream conversations like this one with Dave Trader, Field CISO at Presidio.