OpenStack Neutron's Distributed Virtual Router
- 1. Distributed Virtual Router Introduced in the Juno Release of OpenStack Neutron Carl Baldwin DVR Illustrations courtesy of Jack McCann © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 2. DVR Architecture © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 3. Neutron deployment architecture without DVR neutron-server API auth ML2 plug-in database Network Service Node(s) © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 3 to change without notice. Compute hosts DHCP agent L3 agent message queue Metadata agent ovs agent Nova metadata ovs agent Open vSwitch Open vSwitch
- 4. Neutron deployment architecture with DVR neutron-server API auth ML2 plug-in database Metadata agent L3 agent message queue Network Service Node(s) Compute hosts DHCP agent Metadata agent Nova metadata L3 agent agent_mode = dvr ovs agent w/dvr agent Nova metadata ovs agent Open vSwitch Open vSwitch © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 4 to change without notice. enable_distibuted_routing = True programs DVR flow handling external network compute nodes on external network
- 5. API extension Adds ‘distributed’ attribute to ‘router’ object • Can be set by admin user through the API • Global default is set as “router_distributed” in neutron.conf • Default is False • The attribute is only visible to admin tenant in GET • Cannot be updated • Work in progress to allow update from False to True © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 5 to change without notice.
- 6. DVR – East-West (subnet-to-subnet) “VM1-1 QRouter-X S1.1 S2.1 br-int patch-tun eth0 ARP for gw kept local © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 6 to change without notice. “VM2-1 br-int patch-tun eth0 QRouter-X S1.1 S2.1 same gw IP/MAC on each node no remote bcast in to routers
- 7. DVR – North-South (floating IP) “VM1-1 “VM2-1 br-int QRouter-Y qr rfp-y floating IP namespace br-ex external-vlan eth0 QRouter-X qr rfp-x © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 7 to change without notice. floating IP NAT in router ns default route via FIP-NS local addressing connects QR to FIP-NS floating IP host routes pointing to QR proxy-arp for floating IPs on br-ex (future option: BGP route injection) fpr-x fpr-y fg-u
- 8. DVR – North-South (default SNAT) br-int snat-Y qr qg-x br-ex external-vlan eth0 snat-X qr qg-x © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 8 to change without notice. default SNAT in snat namespace default route via br-ex
- 9. Database router_extra_attributes router_id string uuid distributed boolean dvr_host_macs host string 255 mac_address string 32 ml2_dvr_port_bindings - port binding for all the ports associated to a DVR identified by router_id port_id string uuid host string router_id string uuid vif_type string vif_details string vnic_type string profile string cap_port_filter boolean driver string segment string status string csnat_l3_agent_bindings router_id string uuid l3_agent_id string uuid host_id string csnat_gw_port_id string uuid © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 9 to change without notice.
- 10. config file options neutron.conf [DEFAULT] router_distributed = False dvr_base_mac = fa:16:3f:00:00:00 ovs_neutron_plugin.ini [agent] enable_distributed_routing = False # Make the l2 agent run in dvr mode l3_agent.ini [DEFAULT] agent_mode = legacy # legacy, dvr, or dvr_snat © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 10 to change without notice.
- 11. OVS Flow Handling How to Distribute the Router’s Internal Port • ARP Requests to Router Port are Blocked from the Tunnel • These ARP requests should only be seen by the local port • Source Mac is Mapped to Host Mac on Overlay Network • All traffic generated by the • A mac address is allocated for each compute host • Mapping must be done on both ends of the tunnel • Destination Mac Blocked from Overlay • These should go to the local port • They would create mac ambiguity in the overly • L2 Pre-Population is Required • “Prevent(s) multiple unicast of routed packets destined to remove VMs.” © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 11 to change without notice.
- 12. DVR Limitations © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 13. Default SNAT still centralized snat-Y qr qg-x br-ex external-vlan eth1 “VM1-1 qrouter-X S1.1 S2.1 br-int patch-tun eth0 br-int patch-tun eth0 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 13 to change without notice.
- 14. Floating IP Namespace • Pros • Eliminates Need for Public Address/IR • Keeps IR Macs Off External Net • Cons • Extra Complexity in L3 Agent • Still Consumes a Public Address / CN © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 14 to change without notice. QRouter-Y qr rfp-y fpr-x fpr-y floating IP namespace fg-u eth0 QRouter-X qr rfp-x br-ex external-vlan
- 15. Heavy L2 Integration • Led to Initial Dependence on OVS and Tunnel Protocols • Mitigate Scope Creep • Distributed Port Concept Needs to be Abstracted © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 15 to change without notice.
- 16. Firewall as a Service (FWaaS) Complexity • External Net Connects to Hypervisors • FWaaS is Needed There Too. • Asymmetric Routing Problem • E/W Routing “VM1-1 QRouter-X S1.1 S2.1 br-int patch-tun eth0 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 16 to change without notice. “VM2-1 br-int patch-tun eth0 QRouter-X S1.1 S2.1
- 17. Contributing DVR © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 18. Initial Development on Havana Pros and Cons • Stable Code Base • No Risk of External Regression • Very Large Effort to Integrate • Upstream Moves Quickly • Subject to Regression • Comm. Standard Enforcment • Code Style • No Demand for Unit or Functional Tests © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 18 to change without notice.
- 19. Initial Development on Havana If We had to do it Over Again… • Contribute Smaller, More Focused Patches to Trunk • Start with Pure Refactoring Needed to Ease Development • Develop Unit Tests for Code that will be Modified • Move Gradually Beyond Refactoring to Other Improvements • Divide Implementation According to Themes • Develop Unit Tests (TDD) and Functional Tests to Prevent Regression © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 19 to change without notice.
- 20. Divided in to 7 Patches • Division According to Component • Patches Added Unused Code for Later Patches • Indicates there are themes that cross patch boundaries • Each Patch Had Multiple Active Authors • Indicates possibly more than one theme in the patch © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 20 to change without notice.
- 21. Handling Multiple Changes • Dependency Order Not Linear • Should it be? © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 21 to change without notice.
- 22. Handling Multiple Changes It is Never Linear!!! © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 22 to change without notice.
- 23. Handling Multiple Changes Enemy Number One! • Automatic Rebase Feature • Default behavior of “git review” • Most of the Time it is Disruptive • Sometimes it is Destructive © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 23 to change without notice.
- 24. Handling Multiple Changes Enemy Number One! • Clobbered API Extension!! • Happened More than Once • Use --no-rebase Always • Rebase on Merge Conflict • Work from the bottom up • Merge Faster • Smaller, more focused patches • Continuous community involvement © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 24 to change without notice.