SlideShare a Scribd company logo

Under the Hood: Open vSwitch & OpenFlow in XCP & XenServer

•
•
The Linux Foundation
The Linux Foundation

As more businesses explore the benefits of cloud computing, network managers will have to increasingly meet the challenges of redesigning their networks for the cloud. This talk will introduce the basic concepts of Open vSwitch & Openflow and show how these technologies can help satisfy these needs. We will also explain how Open vSwitch fits into XCP and XenServer and routing of dataflows. First part of talk discussing the networking challenges that cloud implementers face. - Networking challeges - Data isolation Introducing Open vSwitch - What it is, and its features - Why it is important in a virtualized environment Openflow - Basics of Openflow - How flows are routed in XenServer & XCP

Related slideshows

Kernel Recipes 2019 - XDP closer integration with network stack
Kernel Recipes 2019 - XDP closer integration with network stackKernel Recipes 2019 - XDP closer integration with network stack
Kernel Recipes 2019 - XDP closer integration with network stack
 
Filesystem Comparison: NFS vs GFS2 vs OCFS2
Filesystem Comparison: NFS vs GFS2 vs OCFS2Filesystem Comparison: NFS vs GFS2 vs OCFS2
Filesystem Comparison: NFS vs GFS2 vs OCFS2
 
The Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitchThe Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitch
 
1 of 24
Download to read offline
CloudStack Collaboration Conference 2012 Under the Hood: Open vSwitch & OpenFlow in XCP & XenServer Dominic Curran Citrix XenServer Software Engineer Nov 2012 Slides available under CC BY-SA 3.0
2 Contents ● Linux Bridge ● What does Open vSwitch Provide ? ● Open vSwitch Project ● How Open vSwitch works ● How Openflow works Slides available under CC BY-SA 3.0
3 What's wrong with Linux Bridge ? ● Linux Bridge: ● Open vSwitch: – Nothing ! – Full L2-L4 matching capability – L2 switch. Fast & reliable – Decision in userspace – Simple forwarding – ACLs, QoS & Bonding – Mobility of state Userspace Module Userspace Module VM VM VM VM Bridge Kernel Module Bridge Kernel Module OVS Kernel Module OVS Kernel Module VM VM VM VM packet packet Slides available under CC BY-SA 3.0
4 What does Open vSwitch Provide ? ● Provide greater visibility into the XenServer networking layer ● Provide distributed fine grained networking configuration and control policies ● Allows XenServer to be part of switching fabric ● Makes Network Admins happy. ● A Little History: – Research project from Standford University – Bay Area startup – Nicira – Partnered with Nicira 2009 Slides available under CC BY-SA 3.0
5 Open vSwitch Project ● http://openvswitch.org ● Developed under GPL & Apache 2 License ● Mailing Lists – Announcements: announce@openvswitch.org – User-level discussion: discuss@openvswitch.org – Dev (code review, etc): dev@openvswitch.org – Mail Archives available ● As of Linux 3.3 patches started to be accepted for Open vSwitch. Slides available under CC BY-SA 3.0
master git clone git://openvswitch.org/openvswitch branch-1.8 branch-1.7 branch-1.6 branch-1.5 branch-1.4.3 XS 6.1 branch-1.4.2 LTS branch-1.4.1 Open vSwitch Git Tree Source repository: branch-1.4 branch-1.3 Slides available under CC BY-SA 3.0 branch-1.2 XS 6.0.2 branch-1.1 XS 6.0 LTS $ vlan-maint ● (Open vSwitch 1.0) 6
7 Open vSwitch OpenSource Contributors $ git log | grep Author | cut -d “:” -f 2 | sort | uniq -c | sort | -nr In no particular order: ● Redhat ● IBM ● NEC ● HP ● Nicira ● Fujitsu ● Google ● Broadcom ● Cisco ● Intel ● Citrix Slides available under CC BY-SA 3.0
8 Main Components ● Open vSwitch – Open source: www.openvswitch.org – Kernel mode module for packet switching (openvswitch_mod.ko) – User space daemon for rules evaluation (ovsdb-server & ovs-switchd) – Configured via OpenFlow protocol: www.openflow.org XenServer XenServer XenServer XenServer ● OpenFlow Controller Host Host Host vSwitch HostvSwitch vSwitch vSwitch – Open vSwitch has a number of nice standalone features – BUT remote management is probably its most powerful – Nearly every aspect can be controlled remotely OpenFlow OpenFlow – Typically a deployment uses a centralized Controller Controller Controller – DVSC (Distributed Virtual Switch Controller) ● One DVS Controller can manage multiple XenServer pools XenServer XenServer ● Can be hosted in the same pool as it is managing Host Host vSwitch ● Provides Flexible way to group hosts for network management vSwitch Slides available under CC BY-SA 3.0
9 Architecture of Open vSwitch ● Each host runs a vSwitch userspace daemon plus kernel module – ovs-vswitchd & ovsdb-server – Drop-in replacement for the Linux bridge based stack. – Daemon evaluates flow rules to determine destination of each frame. – Pushes rules down into the kernel as necessary. ● In isolation the vSwitch acts much as the Linux bridge does: – Layer-2 MAC learning on physical and virtual interfaces. – From the XenServers toolstack level there is very little observable difference. – Under the hood the management tools, configuration of the stack etc are very different. ● Optionally may connect to central Controller for distributed coordination and control Slides available under CC BY-SA 3.0
1 0 Main Components on one XenServer Host JSON/RPC ovsdb-server ovs-vswitchd Packet Classifier FlowTable FlowTable FlowTable User Kernel Netlink openvswitch_mod.ko Management Protocol (Unix Socket) Hash lookup Table Slides available under CC BY-SA 3.0
1 1 ovsdb-server & ovs-vswitchd ● ovsdb-server – Database that holds switch-level configuration – Custom database with properties of: ● Value contraints ● Weak references ● Garbage collection – Log-based (helpful from debugging) – Speaks Management protocol to ovs-vswitchd & Controller ● ovs-vswitchd – Core component in the system – Supports multiple independent datapaths (bridges) – Communicates ● With controller using OpenFlow ● With ovsdb-server using management protocol ● With kernel module using netlink ● Packet classifier supports flow lookup with wildcard rules Slides available under CC BY-SA 3.0
1 2 openvswitch_mod.ko ● Fast path ● Kernel module handles packet switching ● Exact-match cache of flows – Lookup & forwarding ● Designed to be fast and simple – Knows nothing of Openflow ● Implements Tunnel encapsulation/decapsulation Slides available under CC BY-SA 3.0
1 3 How the Controller fits with the other Components Controller ssl Off-box # ovs-vsctl … xenbr1 xenbr1 xenbr2 xenbr2 JSON/RPC # ovs-appctl ... # ovs-ofctl ... ovsdb-server ovs-vswitchd Packet Classifier FlowTable FlowTable User Kernel Netlink openvswitch_mod.ko Management Protocol (6632/TCP) Hash lookup OpenFlow Channel (6633/TCP) Table Command Line (ovs-vsctl, ovs-appctl, ..) Slides available under CC BY-SA 3.0
1 4 XenServer Packet Flow – Packet Miss DVS Controller Accessible via Browser OFPT_PACKET_IN DomU ssl Dom0 xenbr1 xenbr1 xenbr2 xenbr2 JSON/RPC XAPI ovsdb-server ovs-vswitchd MISS #2 Toolstack Packet Classifier FlowTable FlowTable User Kernel Incoming openvswitch_mod.ko Packet MISS #1 eth1 Hash lookup Table Slides available under CC BY-SA 3.0
XenServer Host VM- Domain2 Flow Table Entries 10.80.228.31 1 5 xenbr0 xenbr0 (Charlie) ● VM Charlie sends a ping to Bob. Port 1 Port 2 # ovs-dpctl show xenbr0 system@xapi0: PIF lookups: hit:103033 missed:77944 lost:0 eth0 vif2.0 flows: 30 port 0: xenbr0 (internal) port 1: eth0 port 2: vif2.0 ● Then dump flows: 10.80.2.144 (Bob) # ovs-dpctl dump-flows xenbr0 | grep "10.80.2.144" in_port(2),eth(src=72:41:36:a2:8c:d9,dst=00:21:1b:f3:63:45),eth_type(0x0800),i Flow 1 pv4(src=10.80.228.31,dst=10.80.2.144,proto=1,tos=0,ttl=64,frag=no),icmp(type=8 ,code=0), packets:5013, bytes:491274, used:0.760s, actions:1 in_port(1),eth(src=00:21:1b:f3:63:45,dst=72:41:36:a2:8c:d9),eth_type(0x0800),i Flow 2 pv4(src=10.80.2.144,dst=10.80.228.31,proto=1,tos=0,ttl=62,frag=no),icmp(type=0 ,code=0), packets:5013, bytes:491274, used:0.760s, actions:2 ● L2-L4 Exact Match ● Total Number of packet matches of this type ● Total Number of bytes for this flow match ● Time flow was last updated ● Actions. In this case switch packet to Port 2. Slides available under CC BY-SA 3.0
1 6 What about SYN attacks ? ● SYN attack form of Denial-of-Service attacks Client SYN Server Attacker SYN Server SYN-ACK Wait.. SYN-ACK Wait.. And wait... And wait... ACK OK Timeout! TCP – Three Way Handshake Legitimate Client SYN ● For Open vSwitch this is particular problem ! ?? – Each SYN is a new flow, thus a user-kernel interaction ● Recent Open vSwitch introduced a 'Flow Governor' – Detects large quantities of short-lived flows – Stops flow setup in kernel module, thus missing a user to kernel interaction Slides available under CC BY-SA 3.0
1 7 Open vSwitch Features ● Supports OpenFlow 1.0 ● Bonding ● 802.1Q VLAN ● Tunneling – Supports GRE mode ● Ingress policing ● NetFlow v5 ● Port mirroring (SPAN and RSPAN) ● ACLs (through OpenFlow) ● Bridge compatibility ● Documented in ovs-vswitchd.conf man page: – http://openvswitch.org/cgi-bin/ovsman.cgi?page=vswitchd%2Fovs-vswitchd.conf.5 Slides available under CC BY-SA 3.0
1 8 What is OpenFlow ? ● OpenFlow allows the path of network packets to be setup by software. ● It is the enabler of 'Software Defined Networking'. ● Standard way to control flow-tables in commercial switches and routers. ● Gives Controller high level access to the datapath routing decisions of a switch. ● Currently, support is limited to 1.0 plus a number of extensions ● Now managed by Open Networking Foundation (ONF) Slides available under CC BY-SA 3.0
1 9 How does the Protocol work ? ● It uses a binary protocol for faster processing ● Header: 8-bit 8-bit 16-bit 32-bit version Type Length Transaction id ● Simple protocol for running on embedded CPUs in hardware switches ● vSwitch connects over SSL to port 6633 on the Controller. Slides available under CC BY-SA 3.0
2 0 OpenFlow Initial Setup Protocol Message Types vSwitch Controller ● Controller-to-Switch OFPT_HELLO - Version ● Asynchronous OFPT_HELLO of vSwitch - Version of ● Symmetric Controller OFPT_ECHO_REQUES T EPLY OFPT_ECHO_R Connection Established S OFPT_FEATURE - Datapath id - Capabilities OFPT_FEATURES_RE PLY - Actions - Ports - Name - HW Address - Port number Slides available under CC BY-SA 3.0
2 1 OpenFlow Protocol vSwitch Controller OFPFC_ADD Change entry OFPFC_MODIFY in FlowTable E OFPFC_DELET Reason to send: OFPT_PACKET_IN - Packet miss - In Port OR - Reason - Explicit rule - Buffer id - Data EQUEST REPLY with: OFPT_STATS_R - Switch description OFPT_STATS_REPLY - Flow statistics - Table statistics - Queue statistics - Port statistics _REQUEST OFPT_BARRIER Synchronize OFPT_BARRIER_REP LY Slides available under CC BY-SA 3.0
2 2 What happens if Controller connections is lost ? vSwitch OFPT_ECHO_REQUES T Controller ? ECHO_REQUEST timeout ??? ● Immediately enter one of two modes: – Fail secure mode – Fail standalone mode Slides available under CC BY-SA 3.0
2 3 WireShark Plugin for OpenFlow Protocol ● http://www.openflow.org/wk/index.php/OpenFlow_Wireshark_Dissector ● Command Line Alternative: # ovs-ofctl snoop <bridge> Slides available under CC BY-SA 3.0
2 4 Questions ? ● My Email: dominic.curran@citrix.com This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

More Related Content

Under the Hood: Open vSwitch & OpenFlow in XCP & XenServer

  • 1. CloudStack Collaboration Conference 2012 Under the Hood: Open vSwitch & OpenFlow in XCP & XenServer Dominic Curran Citrix XenServer Software Engineer Nov 2012 Slides available under CC BY-SA 3.0
  • 2. 2 Contents ● Linux Bridge ● What does Open vSwitch Provide ? ● Open vSwitch Project ● How Open vSwitch works ● How Openflow works Slides available under CC BY-SA 3.0
  • 3. 3 What's wrong with Linux Bridge ? ● Linux Bridge: ● Open vSwitch: – Nothing ! – Full L2-L4 matching capability – L2 switch. Fast & reliable – Decision in userspace – Simple forwarding – ACLs, QoS & Bonding – Mobility of state Userspace Module Userspace Module VM VM VM VM Bridge Kernel Module Bridge Kernel Module OVS Kernel Module OVS Kernel Module VM VM VM VM packet packet Slides available under CC BY-SA 3.0
  • 4. 4 What does Open vSwitch Provide ? ● Provide greater visibility into the XenServer networking layer ● Provide distributed fine grained networking configuration and control policies ● Allows XenServer to be part of switching fabric ● Makes Network Admins happy. ● A Little History: – Research project from Standford University – Bay Area startup – Nicira – Partnered with Nicira 2009 Slides available under CC BY-SA 3.0
  • 5. 5 Open vSwitch Project ● http://openvswitch.org ● Developed under GPL & Apache 2 License ● Mailing Lists – Announcements: announce@openvswitch.org – User-level discussion: discuss@openvswitch.org – Dev (code review, etc): dev@openvswitch.org – Mail Archives available ● As of Linux 3.3 patches started to be accepted for Open vSwitch. Slides available under CC BY-SA 3.0
  • 6. master git clone git://openvswitch.org/openvswitch branch-1.8 branch-1.7 branch-1.6 branch-1.5 branch-1.4.3 XS 6.1 branch-1.4.2 LTS branch-1.4.1 Open vSwitch Git Tree Source repository: branch-1.4 branch-1.3 Slides available under CC BY-SA 3.0 branch-1.2 XS 6.0.2 branch-1.1 XS 6.0 LTS $ vlan-maint ● (Open vSwitch 1.0) 6
  • 7. 7 Open vSwitch OpenSource Contributors $ git log | grep Author | cut -d “:” -f 2 | sort | uniq -c | sort | -nr In no particular order: ● Redhat ● IBM ● NEC ● HP ● Nicira ● Fujitsu ● Google ● Broadcom ● Cisco ● Intel ● Citrix Slides available under CC BY-SA 3.0
  • 8. 8 Main Components ● Open vSwitch – Open source: www.openvswitch.org – Kernel mode module for packet switching (openvswitch_mod.ko) – User space daemon for rules evaluation (ovsdb-server & ovs-switchd) – Configured via OpenFlow protocol: www.openflow.org XenServer XenServer XenServer XenServer ● OpenFlow Controller Host Host Host vSwitch HostvSwitch vSwitch vSwitch – Open vSwitch has a number of nice standalone features – BUT remote management is probably its most powerful – Nearly every aspect can be controlled remotely OpenFlow OpenFlow – Typically a deployment uses a centralized Controller Controller Controller – DVSC (Distributed Virtual Switch Controller) ● One DVS Controller can manage multiple XenServer pools XenServer XenServer ● Can be hosted in the same pool as it is managing Host Host vSwitch ● Provides Flexible way to group hosts for network management vSwitch Slides available under CC BY-SA 3.0
  • 9. 9 Architecture of Open vSwitch ● Each host runs a vSwitch userspace daemon plus kernel module – ovs-vswitchd & ovsdb-server – Drop-in replacement for the Linux bridge based stack. – Daemon evaluates flow rules to determine destination of each frame. – Pushes rules down into the kernel as necessary. ● In isolation the vSwitch acts much as the Linux bridge does: – Layer-2 MAC learning on physical and virtual interfaces. – From the XenServers toolstack level there is very little observable difference. – Under the hood the management tools, configuration of the stack etc are very different. ● Optionally may connect to central Controller for distributed coordination and control Slides available under CC BY-SA 3.0
  • 10. 1 0 Main Components on one XenServer Host JSON/RPC ovsdb-server ovs-vswitchd Packet Classifier FlowTable FlowTable FlowTable User Kernel Netlink openvswitch_mod.ko Management Protocol (Unix Socket) Hash lookup Table Slides available under CC BY-SA 3.0
  • 11. 1 1 ovsdb-server & ovs-vswitchd ● ovsdb-server – Database that holds switch-level configuration – Custom database with properties of: ● Value contraints ● Weak references ● Garbage collection – Log-based (helpful from debugging) – Speaks Management protocol to ovs-vswitchd & Controller ● ovs-vswitchd – Core component in the system – Supports multiple independent datapaths (bridges) – Communicates ● With controller using OpenFlow ● With ovsdb-server using management protocol ● With kernel module using netlink ● Packet classifier supports flow lookup with wildcard rules Slides available under CC BY-SA 3.0
  • 12. 1 2 openvswitch_mod.ko ● Fast path ● Kernel module handles packet switching ● Exact-match cache of flows – Lookup & forwarding ● Designed to be fast and simple – Knows nothing of Openflow ● Implements Tunnel encapsulation/decapsulation Slides available under CC BY-SA 3.0
  • 13. 1 3 How the Controller fits with the other Components Controller ssl Off-box # ovs-vsctl … xenbr1 xenbr1 xenbr2 xenbr2 JSON/RPC # ovs-appctl ... # ovs-ofctl ... ovsdb-server ovs-vswitchd Packet Classifier FlowTable FlowTable User Kernel Netlink openvswitch_mod.ko Management Protocol (6632/TCP) Hash lookup OpenFlow Channel (6633/TCP) Table Command Line (ovs-vsctl, ovs-appctl, ..) Slides available under CC BY-SA 3.0
  • 14. 1 4 XenServer Packet Flow – Packet Miss DVS Controller Accessible via Browser OFPT_PACKET_IN DomU ssl Dom0 xenbr1 xenbr1 xenbr2 xenbr2 JSON/RPC XAPI ovsdb-server ovs-vswitchd MISS #2 Toolstack Packet Classifier FlowTable FlowTable User Kernel Incoming openvswitch_mod.ko Packet MISS #1 eth1 Hash lookup Table Slides available under CC BY-SA 3.0
  • 15. XenServer Host VM- Domain2 Flow Table Entries 10.80.228.31 1 5 xenbr0 xenbr0 (Charlie) ● VM Charlie sends a ping to Bob. Port 1 Port 2 # ovs-dpctl show xenbr0 system@xapi0: PIF lookups: hit:103033 missed:77944 lost:0 eth0 vif2.0 flows: 30 port 0: xenbr0 (internal) port 1: eth0 port 2: vif2.0 ● Then dump flows: 10.80.2.144 (Bob) # ovs-dpctl dump-flows xenbr0 | grep "10.80.2.144" in_port(2),eth(src=72:41:36:a2:8c:d9,dst=00:21:1b:f3:63:45),eth_type(0x0800),i Flow 1 pv4(src=10.80.228.31,dst=10.80.2.144,proto=1,tos=0,ttl=64,frag=no),icmp(type=8 ,code=0), packets:5013, bytes:491274, used:0.760s, actions:1 in_port(1),eth(src=00:21:1b:f3:63:45,dst=72:41:36:a2:8c:d9),eth_type(0x0800),i Flow 2 pv4(src=10.80.2.144,dst=10.80.228.31,proto=1,tos=0,ttl=62,frag=no),icmp(type=0 ,code=0), packets:5013, bytes:491274, used:0.760s, actions:2 ● L2-L4 Exact Match ● Total Number of packet matches of this type ● Total Number of bytes for this flow match ● Time flow was last updated ● Actions. In this case switch packet to Port 2. Slides available under CC BY-SA 3.0
  • 16. 1 6 What about SYN attacks ? ● SYN attack form of Denial-of-Service attacks Client SYN Server Attacker SYN Server SYN-ACK Wait.. SYN-ACK Wait.. And wait... And wait... ACK OK Timeout! TCP – Three Way Handshake Legitimate Client SYN ● For Open vSwitch this is particular problem ! ?? – Each SYN is a new flow, thus a user-kernel interaction ● Recent Open vSwitch introduced a 'Flow Governor' – Detects large quantities of short-lived flows – Stops flow setup in kernel module, thus missing a user to kernel interaction Slides available under CC BY-SA 3.0
  • 17. 1 7 Open vSwitch Features ● Supports OpenFlow 1.0 ● Bonding ● 802.1Q VLAN ● Tunneling – Supports GRE mode ● Ingress policing ● NetFlow v5 ● Port mirroring (SPAN and RSPAN) ● ACLs (through OpenFlow) ● Bridge compatibility ● Documented in ovs-vswitchd.conf man page: – http://openvswitch.org/cgi-bin/ovsman.cgi?page=vswitchd%2Fovs-vswitchd.conf.5 Slides available under CC BY-SA 3.0
  • 18. 1 8 What is OpenFlow ? ● OpenFlow allows the path of network packets to be setup by software. ● It is the enabler of 'Software Defined Networking'. ● Standard way to control flow-tables in commercial switches and routers. ● Gives Controller high level access to the datapath routing decisions of a switch. ● Currently, support is limited to 1.0 plus a number of extensions ● Now managed by Open Networking Foundation (ONF) Slides available under CC BY-SA 3.0
  • 19. 1 9 How does the Protocol work ? ● It uses a binary protocol for faster processing ● Header: 8-bit 8-bit 16-bit 32-bit version Type Length Transaction id ● Simple protocol for running on embedded CPUs in hardware switches ● vSwitch connects over SSL to port 6633 on the Controller. Slides available under CC BY-SA 3.0
  • 20. 2 0 OpenFlow Initial Setup Protocol Message Types vSwitch Controller ● Controller-to-Switch OFPT_HELLO - Version ● Asynchronous OFPT_HELLO of vSwitch - Version of ● Symmetric Controller OFPT_ECHO_REQUES T EPLY OFPT_ECHO_R Connection Established S OFPT_FEATURE - Datapath id - Capabilities OFPT_FEATURES_RE PLY - Actions - Ports - Name - HW Address - Port number Slides available under CC BY-SA 3.0
  • 21. 2 1 OpenFlow Protocol vSwitch Controller OFPFC_ADD Change entry OFPFC_MODIFY in FlowTable E OFPFC_DELET Reason to send: OFPT_PACKET_IN - Packet miss - In Port OR - Reason - Explicit rule - Buffer id - Data EQUEST REPLY with: OFPT_STATS_R - Switch description OFPT_STATS_REPLY - Flow statistics - Table statistics - Queue statistics - Port statistics _REQUEST OFPT_BARRIER Synchronize OFPT_BARRIER_REP LY Slides available under CC BY-SA 3.0
  • 22. 2 2 What happens if Controller connections is lost ? vSwitch OFPT_ECHO_REQUES T Controller ? ECHO_REQUEST timeout ??? ● Immediately enter one of two modes: – Fail secure mode – Fail standalone mode Slides available under CC BY-SA 3.0
  • 23. 2 3 WireShark Plugin for OpenFlow Protocol ● http://www.openflow.org/wk/index.php/OpenFlow_Wireshark_Dissector ● Command Line Alternative: # ovs-ofctl snoop <bridge> Slides available under CC BY-SA 3.0
  • 24. 2 4 Questions ? ● My Email: dominic.curran@citrix.com This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.