Traditional SIEM in Kubernetes environments

Question

How do companies manage SIEM for Kubernetes environments? I am specifically interested in running CIS benchmarks and auditing OS events on the nodes.

I already have a Wazuh cluster and agents rolled out to VMs (EL, Debian, and Windows), and the approach for Kubernetes clusters (ex.: EKS, on-prem, GKE) is quite cloudy to me. I think that the mindset has to shift because for example with VMs, it is normal for users to log in via SSH and change things, but not for Kubernetes worker, master, and etcd nodes where a lot of things work via the API.

I am looking for a strategic approach to dealing with security monitoring in Kubernetes. I know there are a lot of projects in the CNCF ecosystem tagged as "security" and I was considering whether or not a SIEM tool is still relevant.

Answer 1

There are two relevant levels of SIEM monitoring here:

1. Monitoring of the cluster itself
2. Monitoring the Apps in the cluster

Monitoring the Cluster

Wazuh itself describes a way to log Kubernetes on their blog

It creates a webhook listener that then receives the forwarded audit logs from the Cluster. Following this, then you can create rules around the most relevant actions, like creation of new roles, new Service Accounts etc.

Find more about Auditing Kubernetes in the official documentation

From a more strategic perspective, your options vary a lot on the specific implementation of your Cluster, e.g. in GCP you can use the native GCP tools to audit the cluster.

If you run it in your datacenter on some hosts, Wazuh or any other normal SIEM needs to monitor this hosts as well.

Monitoring the apps in the cluster

Since containers are more cattle than pets we need to find ways to automate the collection of logs from them. Kubernetes is application centric, so any form of logging or auditing must be built into the app instead into the cluster.

For this, you could use a syslog container as a sidecar that sends all application logs from a pod to your wazuh endpoints and must be deployed which every application.

Or you rebuild the app in a way that it accepts environment variables to stream the logs to the wazuh cluster if possible.
Then you would also catch all Replicas of a given deployment.

Or you add the syslog streamer to the base image in your CI/CD pipelines and provide it via the central registry.

Answer 2

Just to supplement the answer from @stefan-lorenz and address your point about whether a SIEM is still relevant in a Kubernetes environment.

I'd say that it's critical, perhaps even more so than in traditional environments, for a couple of reasons.

• Analyzing Kubernetes audit logs is the only way to easily track a lot of activity in a Kubernetes cluster. For example when a user creates a new authentication token (using the Token Request API) there is no permanent record in the cluster of that token, the only indication of its creation is an audit log entry (assuming your policy tracks that).

• Kubernetes environments use ephemeral containers where workloads will move from one node to another. As such centralized logging and security monitoring, using a solution which is k8s aware is vital to make sure you're able to provide a picture of what's happening in the cluster.