They are just a few clicks in the UI or a single API call, but how do security groups work at KVM hypervisor level? How do they filter traffic and what else do they do in addition to firewalling? What Anti-Spoofing policies are implemented by the security groups?
In this talk, Wido dives into the specifics of the security groups on the KVM hypervisor for both IPv4 and IPv6.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
Report
Share
Report
Share
1 of 19
Download to read offline
More Related Content
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
2. Hi!
Wido den Hollander
Netherlands
• 1986
• Married
• Father of two sons (2020 & 2022)
Active in the hosting industry since 2003
• Started my own hosting company in 2004
• Joined TWS (now Your.Online) in 2018
• Currently CTO of Your.Online
• Open Source, CloudStack, Ceph and IPv6 enthousiast
Your Ground to Build Online Possibilities
• In the online business
• mainly webhosting
• holding, private
• Started in the Netherlands, now also active in France and
Spain
• EUR ~200m revenue, 1 million customers, 600 employees
• Some of our brands: Gandi, Savvii, Yourhosting, Versio,
Axarnet, Nexylan, o2switch, ShockMedia and Realtime
Register
• Our brands use a lot of Open Source software
• Large CloudStack deployments
• Large number of Ceph clusters
3. KVM security groups
• Hypervisor level firewall
• Ingress
• Egress
• Multiple Instances can use the same Security Group
• API driven
• Updates are applied live to running Instances
5. But how?
• When the Virtual Machine is started the Management Server sends
the security group information to the CloudStack KVM Agent
• The Agent executes ‘security_group.py’ as part of the boot process of
the VM
7. But what?
• default_network_rules
• Creates ipset for IPv4 and IPv6
• Creates ebtables rules for MAC
• Creates ip(6)tables chains
• Basic security rules
• Anti spoofing
• No answering to DHCPv4 requests
• No Router Advertisement packets allowed to go out
• add_network_rules
• Sets ingress and egress rules
• TCP, UDP, ICMP and other protocols
8. iptables: IPv4 (stripped)
-A i-46-4318-def -m state --state RELATED,ESTABLISHED -j ACCEPT
-A i-46-4318-def -p udp -m physdev --physdev-in vnet4 --physdev-is-bridged -m udp --sport 68 --dport 67 -j
ACCEPT
-A i-46-4318-def -p udp -m physdev --physdev-out vnet4 --physdev-is-bridged -m udp --sport 67 --dport 68 -j
ACCEPT
-A i-46-4318-def -p udp -m physdev --physdev-in vnet4 --physdev-is-bridged -m udp --sport 67 -j DROP
-A i-46-4318-def -m physdev --physdev-in vnet4 --physdev-is-bridged -m set ! --match-set i-46-4318-VM src -j
DROP
-A i-46-4318-def -m physdev --physdev-out vnet4 --physdev-is-bridged -m set ! --match-set i-46-4318-VM dst -j
DROP
-A i-46-4318-def -p udp -m physdev --physdev-in vnet4 --physdev-is-bridged -m set --match-set i-46-4318-VM src -
m udp --dport 53 -j RETURN
-A i-46-4318-def -p tcp -m physdev --physdev-in vnet4 --physdev-is-bridged -m set --match-set i-46-4318-VM src -
m tcp --dport 53 -j RETURN
-A i-46-4318-def -m physdev --physdev-in vnet4 --physdev-is-bridged -m set --match-set i-46-4318-VM src -j i-46-
4318-VM-eg
-A i-46-4318-def -m physdev --physdev-out vnet4 --physdev-is-bridged -j i-46-4318-VM
VMs are not allowed to answer to DHCP requests no matter what they define in their Ingress/Egress rules
9. ebtables: No MAC and ARP spoofing!
-A i-46-4318-VM-in -j i-46-4318-VM-in-src
-A i-46-4318-VM-in -p ARP -j i-46-4318-VM-in-ips
-A i-46-4318-VM-in -p ARP --arp-op Request -j ACCEPT
-A i-46-4318-VM-in -p ARP --arp-op Reply -j ACCEPT
-A i-46-4318-VM-in -p ARP -j DROP
-A i-46-4318-VM-out -p ARP --arp-op Reply -j i-46-4318-VM-out-dst
-A i-46-4318-VM-out -p ARP -j i-46-4318-VM-out-ips
-A i-46-4318-VM-out -p ARP --arp-op Request -j ACCEPT
-A i-46-4318-VM-out -p ARP --arp-op Reply -j ACCEPT
-A i-46-4318-VM-out -p ARP -j DROP
-A i-46-4318-VM-in-ips -p ARP -s 1e:00:7e:00:3c:b7 --arp-ip-src 31.24.44.108 --arp-mac-src 1e:00:7e:00:3c:b7 -j
RETURN
-A i-46-4318-VM-in-ips -j DROP
-A i-46-4318-VM-out-ips -p ARP --arp-ip-dst 31.24.44.108 -j RETURN
-A i-46-4318-VM-out-ips -j DROP
-A i-46-4318-VM-in-src -s 1e:00:7e:00:3c:b7 -j RETURN
-A i-46-4318-VM-in-src -j DROP
-A i-46-4318-VM-out-dst -p ARP --arp-op Reply --arp-mac-dst 1e:00:7e:00:3c:b7 -j RETURN
-A i-46-4318-VM-out-dst -p ARP --arp-op Reply -j DROP
10. ip6tables: IPv6 (stripped)
-A i-46-4318-def -m state --state RELATED,ESTABLISHED -j ACCEPT
-A i-46-4318-def -s fe80::/64 -d ff02::1/128 -p ipv6-icmp -m physdev --physdev-out vnet4 --physdev-is-bridged -m
icmp6 --icmpv6-type 134 -m hl --hl-eq 255 -j ACCEPT
-A i-46-4318-def -d ff02::2/128 -p ipv6-icmp -m physdev --physdev-in vnet4 --physdev-is-bridged -m icmp6 --
icmpv6-type 133 -m hl --hl-eq 255 -j RETURN
-A i-46-4318-def -p ipv6-icmp -m physdev --physdev-in vnet4 --physdev-is-bridged -m icmp6 --icmpv6-type 134 -j
DROP
-A i-46-4318-def -p ipv6-icmp -m physdev --physdev-out vnet4 --physdev-is-bridged -m icmp6 --icmpv6-type 135 -m
hl --hl-eq 255 -j ACCEPT
-A i-46-4318-def -d ff02::16/128 -p ipv6-icmp -m physdev --physdev-in vnet4 --physdev-is-bridged -j RETURN
-A i-46-4318-def -s fe80::1c00:7eff:fe00:3cb7/128 -d ff02::1:2/128 -p udp -m physdev --physdev-in vnet4 --
physdev-is-bridged -m udp --sport 546 -j RETURN
-A i-46-4318-def -s fe80::/64 -d fe80::1c00:7eff:fe00:3cb7/128 -p udp -m physdev --physdev-out vnet4 --physdev-
is-bridged -m udp --dport 546 -j ACCEPT
-A i-46-4318-def ! -d fe80::/64 -p udp -m physdev --physdev-in vnet4 --physdev-is-bridged -m udp --sport 547 -j
DROP
-A i-46-4318-def -p udp -m physdev --physdev-in vnet4 --physdev-is-bridged -m udp --dport 53 -m set --match-set
i-46-4318-VM-6 src -j RETURN
-A i-46-4318-def -p tcp -m physdev --physdev-in vnet4 --physdev-is-bridged -m tcp --dport 53 -m set --match-set
i-46-4318-VM-6 src -j RETURN
-A i-46-4318-def -m physdev --physdev-in vnet4 --physdev-is-bridged -m set ! --match-set i-46-4318-VM-6 src -j
DROP
-A i-46-4318-def -m physdev --physdev-in vnet4 --physdev-is-bridged -m set --match-set i-46-4318-VM-6 src -j i-
46-4318-VM-eg
-A i-46-4318-def -m physdev --physdev-out vnet4 --physdev-is-bridged -j i-46-4318-VM
Router Advertisements should never originate from Inside a VM and thus be blocked
11. ipset
root@hypervisor:~# ipset list i-46-4318-VM
Name: i-46-4318-VM
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 240
References: 5
Number of entries: 1
Members:
31.24.44.108
root@hypervisor:~# ipset list i-46-4318-VM-6
Name: i-46-4318-VM-6
Type: hash:net
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536
Size in memory: 1384
References: 9
Number of entries: 2
Members:
fe80::1c00:7eff:fe00:3cb7
2a01:4b80:1:1:1c00:7eff:fe00:3cb7
12. BCP38: Prevent IP-spoofing
"BCP38 is RFC2827:
Network Ingress Filtering:
Defeating Denial of Service
Attacks which employ IP
Source Address Spoofing.”
Source: bcp38.info
Routers forward packets by
looking at the destination. Not
at the source.
13. Changes
• authorizeSecurityGroupIngress
API command is executed
• Command is sent to KVM
Agents
• The ‘security_group.py’ script is
executed
• On hypervisors where the
affected VMs run
• iptables/ip6tables rules are added
or removed
15. Performance
• Uses connection tracking on the host
• nf_conntrack
• Think about the limit!
• Might need to disable security groups on high performance
applications
• Requires a separate network in CloudStack
• Security groups can be enabled/disabled per network
16. nf_conntrack: table full, dropping packet
• Conntrack table can fill up during a DDoS attack
• The hypervisor needs to track every connection to and from a VM
• Statefull packet inspection is a problem with DDoS attacks
• Make sure you have DDoS protection higher in the network
• Do we need to track every packet?
• Maybe not, improvements can be made
-A i-46-4318-def -m state --state RELATED,ESTABLISHED -j ACCEPT
17. Libvirt Network Filters
“The goal of the network filtering XML is to enable administrators of a virtualized system to configure and enforce network
traffic filtering rules on virtual machines and manage the parameters of network traffic that virtual machines are allowed to
send or receive. The network traffic filtering rules are applied on the host when a virtual machine is started. Since the
filtering rules cannot be circumvented from within the virtual machine, it makes them mandatory from the point of view of a
virtual machine user. “
Network filtering support is available since 0.8.1 (QEMU, KVM)
<devices>
<interface type='bridge'>
<mac address='00:16:3e:5d:c7:9e'/>
<filterref filter='clean-traffic'>
<parameter name='IP' value='10.0.0.1’/>
<parameter name='IP' value=‘2001:db8::100/>
</filterref>
</interface>
</devices>
<rule action='accept' direction='out' priority='500'>
<icmp/>
</rule>
<rule action='accept' direction='out' priority='500'>
<udp dstportstart='53'/>
</rule>
18. Libvirt Network Filters
Maybe a better solution than our own Python scripts? Opinions? Ideas?
https://libvirt.org/formatnwfilter.html
Supported protocols
• MAC (Ethernet)
• VLAN (802.1Q) (Since 0.9.8)
• STP (Spanning Tree Protocol) (Since 0.9.8)
• ARP/RARP
• IPv4
• IPv6
• TCP/UDP/SCTP
• ICMP
• IGMP, ESP, AH, UDPLITE, 'ALL'
• TCP/UDP/SCTP over IPV6
• ICMPv6
• ESP, AH, UDPLITE, 'ALL' over IPv6