Kubernetes fingerprinting with Prometheus.pdf

How Attackers Use Exposed
Prometheus Server to Exploit
Kubernetes Clusters
Miguel Hernández & David de Torres, Sysdig

Miguel Hernandez
Security Researcher
Sysdig
@MiguelHzBz
Kubernetes fingerprinting with
Prometheus
Photo
David de Torres
Manager of Engineering
Sysdig
@maellyssa
https://xkcd.com/2347/
Exposed
metrics

Title
Assume you are a target, but not for free
● Follow the Kubernetes security best practices.
● Use Prometheus to monitor everything.
● But don’t let the door open.
https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/0/CTR_Kubernetes_Hardening_Guidance_1.1_20220315.PDF
We are not going to break and break into
Kubernetes Cluster or Prometheus.

Title
Why Kubernetes fingerprinting?
The first step in any pentesting, ethical hacking or cybercriminal groups, is to
gather as much information as you can about the target you want to breach.
Why? Simple, to know what technique to use or the appropriate tools to achieve
intrusion and evasion of defense systems.
Information on versions inside the cluster can map to CVE and vulnerabilities that
can be exploited.
Information on applications, tools and architectures can be used for competitors.
https://www.eccouncil.org/ethical-hacking/

Title
Kubernetes in the wild
https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

Kubernetes fingerprinting with Prometheus.pdf

Title
But Prometheus is only metrics…
https://www.cncf.io/online-programs/a-look-at-how-hackers-exploit-prometheus-grafana-fluentd-jaeger-more/
https://jfrog.com/blog/dont-let-prometheus-steal-your-fire/
https://github.com/juice-shop/juice-shop/issues/1275

Title
Prometheus in the wild
Prometheus collects and stores its metrics as time series data, i.e. metrics information is
stored with the timestamp at which it was recorded, alongside optional key-value pairs
called labels.
Prometheus allows (and recommends) using basic authentication, but not enabled by
default: https://prometheus.io/docs/operating/security/
Exposing open Prometheus endpoints to the Internet is a bad idea… and as every bad
idea, it's highly adopted:

Title
More Prometheus in the wild
Shodan -> favicons (https://github.com/sansatart/scrapts/blob/master/shodan-favicon-hashes.csv)
Censys (https://search.censys.io/)
Fofa (https://fofa.info/)
61854
161.274

Title
What will we us to fingerprint Kubernetes?
Two of the most widely used exporters offer most of the
information that we need:
Node Exporter
● Physical infrastructure
● Network interfaces
Kube State Metrics
● Host OS & kernel
● Kubernetes components
● Hostnames and network topology
● Logical hierarchy
● Secrets location
● Applications (and versions) deployed

Website
API
…
https://example.com
Scenario - NotARealCompanyForSure ©

Title
Fingerprinting Physical Infrastructure
Node Exporter:
node_dmi_info bios_vendor:
● SeaBIOS
● Amazon EC2
bios_version:
● seabios-1.9.1-qemu-project.org
● 8f19b21
● 1.0
bios_release:
● 1.0
bios_date:
● 10/16/2017
● 04/01/2014
chassis_asset_tag:
● Amazon EC2
chassis_vendor:
● Amazon EC2
● Alibaba Cloud
system_vendor:
● Tencent Cloud
● Amazon EC2
● Alibaba Cloud
product_name:
● m5.xlarge
● Alibaba Cloud ECS
product_version:
● pc-i440fx-2.1
board_vendor:
● Amazon EC2
board_asset_tag:
● i-00280f617XXXXX
board_vendor:
● Smdbmds
● Amazon EC2

Cloud provider
● AWS Keys
● Tencent Keys
● Alibaba Keys
https://www.example.com
Load
Balancer

Website
API
…
https://example.com
Cloud provider
Credentials:
● AWS Keys
● Tencent Keys
● Alibaba Keys

Title
Fingerprinting network interfaces
Node Exporter:
node_network_info{device=~'eth.+'}
{
address="06:d5:XX:XX:XX:XX",
broadcast="ff:ff:ff:ff:ff:ff",
device="eth0",
instance="172.31.XX.XX:9100",
instance_az="us-west-2a",
instance_id="i-XXXXX",
instance_name="XXX-XXX",
instance_type="c5.xlarge",
instance_vpc="vpc-XXXXXXX",
job="ec2_instances",
operstate="up"
}

Title
Fingerprinting network topology
KSM:
kube_node_info
kube_service_info * on (service) group_left group by
(service,type)(kube_service_spec_type{type="LoadBalancer"})
kube_ingress_info
Node hostname
Services in the cluster (specially load-balancers)
● namespace
● cluster IP
● node
● (application behind the service can be guessed
by name of service/namespace)
Ingresses in the cluster

Cloud provider
● AWS Keys
● Tencent Keys
● Alibaba Keys
Load
Balancer
AZ
VPC-ID
K8s Cluster
Node
namespace
Ingress
Pod

Website
API
…
https://example.com
Cloud provider
Credentials:
● AWS Keys
● Tencent Keys
● Alibaba Keys
Networking
● Load Balancer
● Region & AZ
● VPC
● Instance IP & ID
K8s Cluster
Topology
● Cluster IP
● Namespaces
● Nodes
● Ingress

Title
Fingerprinting Kubernetes hierarchy
KSM:
kube_namespace_status_phase
kube_deployment_spec_replicas
kube_daemonset_status_desired_number_scheduled
kube_statefulset_replicas
kube_replicaset_spec_replicas
kube_pod_info
kube_pod_container_info
kube_cronjob_info
kube_job_info
Namespaces
Workloads
Pods
Containers
CronJobs
Jobs
Nodes
Pods

Title
Fingerprinting Kubernetes Control Plane
Kubernetes:
kubernetes_build_info
Component
● API-server
● controller-manager
● kube-proxy…
Major, minor version
git version
git commit
build_date
go_version

Cloud provider
● AWS Keys
● Tencent Keys
● Alibaba Keys
Load
Balancer
AZ
VPC-ID
K8s Cluster
Master
Node
namespace
Pod
Pod
Container
Container
Ingress

Website
API
…
https://example.com
Cloud provider
Credentials:
● AWS Keys
● Tencent Keys
● Alibaba Keys
Networking
● Load Balancer
● Region & AZ
● VPC
K8s Cluster
Components:
● Kube-proxy
● Kube-admin
● Kubelet
Topology
● Cluster IP
● Namespaces
● Nodes
● Ingress
Known Vulnerabilities:
● CVE-2020-8554
● CVE-2020-8558
● CVE-2020-8559
● CVE-2021-25735
● CVE-2021-25737
● CVE-2021-25741

Title
Fingerprinting OS & Kernel
KSM Exporter:
kube_node_info
os_image:
● Ubuntu 18.04.4 LTS
● Ubuntu 20.04.3 LTS
● CentOS Linux 7 (Core)
● Tencent Linux 2.4
kernel_version:
● 5.11.0-1027-aws
● 4.15.0-142-generic
● 4.14.105-19-0020.1
● 3.10.0-1160.59.1.el7.x86_64

Website
API
…
https://example.com
Cloud provider
Credentials:
● AWS Keys
● Tencent Keys
● Alibaba Keys
Networking
● Load Balancer
● Region & AZ
● VPC
K8s Cluster
Components:
● Kube-proxy
● Kube-admin
● Kubelet
Topology
● Cluster IP
● Namespaces
● Nodes
Node
● Kernel
● OS
● Go version
● Git version
● CVE-2022-0847 -
dirty pipe (Kernel
Linux)
● CVE-2022-0185
● USN-3833-1: Linux
kernel (AWS)
vulnerabilities
○ CVE-2018-
18955
● CVE-2021-3156
● CVE-2020-8554
● CVE-2020-8558
● CVE-2020-8559
● CVE-2021-25735
● CVE-2021-25737
● CVE-2021-25741

Title
Applications versions
KSM:
kube_pod_container_info
Custom:
prometheus_build_info
pod (app name)
image name + tag + sha256
● docker.io/library/cassandra:3.11.6
● sha256:5aa8400b4b3b794b5eba85f79b75a9ed9326e41428a
e3a9d6b91cd731f2cf768
Prometheus version

Service
● Service-example
○ Website
○ API
○ …
○ https://example.com
Cloud provider
Credentials:
● AWS Keys
● Tencent Keys
● Alibaba Keys
Networking
● Load Balancer
● Region & AZ
● VPC
K8s Cluster
Components:
● Kube-proxy
● Kube-admin
● Kubelet
Topology
● Cluster IP
● Namespaces
● Nodes
Node
● Kernel
● OS
● Go version
● Git version
● Docker
Pod / Container
Registry:
● docker.io
Image:
● Image-id
● CVE-2020-8554
● CVE-2020-8558
● CVE-2020-8559
● CVE-2021-25735
● CVE-2021-25737
● CVE-2021-25741
● CVE-2022-0847 -
dirty pipe (Kernel
Linux)
● CVE-2022-0185
kernel (AWS)
vulnerabilities
○ CVE-2018-
18955
● CVE-2021-3156
● CVE-2021-44521 - Cassandra
● https://mariadb.com/kb/en/security/ - RCE
● CVE-2020-28035
● Wordpress
● CVE-2018-16850 - PostgreSQL
● CVE-2019-11043 - PHP
● CVE-2021-44228 - Log4j
● CVE-2022-22963 - Spring Cloud
● CVE-2020-13942 - Apache unomi

Title
Locating Kubernetes secrets
KSM:
kube_secret_info
kube_secret_type
kube_secret_annotations
Namespace
Secret name
Type
● Opaque
● service-account-token…
Kubectl last applied info (leak)
Application (application that uses the secret can be usually
guessed by the name of secret/namespace)
kube_secret_annotations{kubectl_kubernetes_io_last_applied_configuration != ""}

Cloud provider
● AWS Keys
● Tencent Keys
● Alibaba Keys
Load
Balancer
AZ
VPC-ID
K8s Cluster
Node Master
namespace
Pod
Pod
Container
Container
Namespace
secrets
Ingress

Service
● Service-example
○ Website
○ API
○ …
Cloud provider
Credentials:
● AWS Keys
● Tencent Keys
● Alibaba Keys
Networking
● Load Balancer
● Region & AZ
● VPC
K8s Cluster
Components:
● Kube-proxy
● Kube-admin
● Kubelet
Topology
● Cluster IP
● Namespaces
● Nodes
Node
● Kernel
● OS
● Go version
● Git version
● Docker
Pod / Container
Registry:
● docker.io
Image:
● Image-id
Kubernetes Secrets
● Service auth
tokens
● CVE-2020-8554
● CVE-2020-8558
● CVE-2020-8559
● CVE-2021-25735
● CVE-2021-25737
● CVE-2021-25741
● CVE-2022-0847 -
dirty pipe (Kernel
Linux)
● CVE-2022-0185
kernel (AWS)
vulnerabilities
○ CVE-2018-
18955
● CVE-2021-3156
● CVE-2020-28035
● Wordpress
● CVE-2019-11043 - PHP
● CVE-2021-44228 - Log4j

Title
Logging queries in Prometheus
Prometheus allows query logging… but it's not enabled by
default.
You can check if loggin is enabled by querying this metric:
prometheus_engine_query_log_enabled
https://prometheus.io/docs/guides/query-log/

Title
Real History
Now, the attacker prepares the journey and the intrusion target.
In this fictitious examples, the attacker might want to access the data leak,
use your machines for cryptomining or encrypt the victim's data (ransomware).
With this knowledge of Prometheus exposed, the attacker uses the specific
technique for each case.
https://cd.blokt.com/wp-content/uploads/2018/02/crypto-mining-e1518714481556.jpg
https://miro.medium.com/max/750/1*TSX7fu85EwGEdnhA-Sv4cA.jpeg

Title
Leak data scenario - Attacker Path
Service
● Service-example
○ Website
○ API
○ …
Cloud provider
Credentials:
● AWS Keys
● Tencent Keys
● Alibaba Keys
Networking
● Load Balancer
● Region & AZ
● VPC
K8s Cluster
Components:
● Kube-proxy
● Kube-admin
● Kubelet
Topology
● Cluster IP
● Namespaces
● Nodes
Node
● Kernel
● OS
● Go version
● Git version
● Docker
Pod / Container
Registry:
● docker.io
Image:
● Image-id
Kubernetes Secrets
● Service auth
tokens
● CVE-2020-8554
● CVE-2020-8558
● CVE-2020-8559
● CVE-2021-25735
● CVE-2021-25737
● CVE-2021-25741
● CVE-2022-0847 -
dirty pipe (Kernel
Linux)
● CVE-2022-0185
kernel (AWS)
vulnerabilities
○ CVE-2018-
18955
● CVE-2021-3156
● CVE-2020-28035
● Wordpress
● CVE-2019-11043 - PHP
● CVE-2021-44228 - Log4j

Title
Cryptomining scenario - Attacker Path
Service
● Service-example
○ Website
○ API
○ …
Cloud provider
Credentials:
● AWS Keys
● Tencent Keys
● Alibaba Keys
Networking
● Load Balancer
● Region & AZ
● VPC
K8s Cluster
Components:
● Kube-proxy
● Kube-admin
● Kubelet
Topology
● Cluster IP
● Namespaces
● Nodes
Node
● Kernel
● OS
● Go version
● Git version
● Docker
Pod / Container
Registry:
● docker.io
Image:
● Image-id
Kubernetes Secrets
● Service auth
tokens
● CVE-2020-8554
● CVE-2020-8558
● CVE-2020-8559
● CVE-2021-25735
● CVE-2021-25737
● CVE-2021-25741
● CVE-2022-0847 -
dirty pipe (Kernel
Linux)
● CVE-2022-0185
kernel (AWS)
vulnerabilities
○ CVE-2018-
18955
● CVE-2021-3156
● CVE-2020-28035
● Wordpress
● CVE-2019-11043 - PHP
● CVE-2021-44228 - Log4j
2
1
3
https://github.com/kozmer/log4j-shell-poc
https://dirtypipe.cm4all.com/
https://github.com/arget13/DDexec

Title
Ransomware scenario - Attacker Path
Service
● Service-example
○ Website
○ API
○ …
Cloud provider
Credentials:
● AWS Keys
● Tencent Keys
● Alibaba Keys
Networking
● Load Balancer
● Region & AZ
● VPC
K8s Cluster
Components:
● Kube-proxy
● Kube-admin
● Kubelet
Topology
● Cluster IP
● Namespaces
● Nodes
Node
● Kernel
● OS
● Go version
● Git version
● Docker
Pod / Container
Registry:
● docker.io
Image:
● Image-id
Kubernetes Secrets
● Service auth
tokens
● CVE-2020-8554
● CVE-2020-8558
● CVE-2020-8559
● CVE-2021-25735
● CVE-2021-25737
● CVE-2021-25741
● CVE-2022-0847 -
dirty pipe (Kernel
Linux)
● CVE-2022-0185
kernel (AWS)
vulnerabilities
○ CVE-2018-
18955
● CVE-2021-3156
● CVE-2020-28035
● Wordpress
● CVE-2019-11043 - PHP
● CVE-2021-44228 - Log4j
● …
1
2
3
https://github.com/hktalent/spring-spel-0day-poc

Title
Summary
We could think that metrics are not important in a security perspective,
but we show that’s not true.
It’s also important to mention that the proper services Kubernetes or
Prometheus advise of the problems to expose their data to the world

Title
Recommendations
Today, if we follow security best practices in every part of our chain,
we are safe from most security incidents.
We will have to continue to fight with new vulnerabilities that impact
our services and also, not least, a plan against insiders. But let's at
least make things difficult.
● Secure your Cloud provider with Principle of least privilege.
○ https://www.cisa.gov/uscert/ncas/current-activity/2020/01/24/nsa-releases-guidance-m
itigating-cloud-vulnerabilities
● Secure your Cluster Kubernetes
○ https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/0/CTR_Kubernetes_Hardening_Gu
idance_1.1_20220315.PDF
● Secure the Host / OS
○ https://nvlpubs.nist.gov/nistpubs/legacy/sp/nistspecialpublication800-123.pdf
● Secure the containers
○ https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-190.pdf
● Secure your code
○ https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-95.pdf
● Secure your Prometheus Metrics!
○ https://prometheus.io/docs/operating/security/#prometheus

Kubernetes fingerprinting with Prometheus.pdf

Related slideshows

More Related Content

Kubernetes fingerprinting with Prometheus.pdf