SRECon-Europe-2017: Networks for SREs

Editor's Notes

1. So today I want to briefly talk about what this talk is about and what I hope to achieve by the end of this session. I then want to do a quick review some of the basics of the internet and networks. Then talk about 3 specific advances of networks.
2. NOTE: So what’s the problem we’re trying to solve in this space: If there’s one thing I would like you all to get out of this talk, it is: Don’t trust any part of the network
3. Tier 1: AT&T Level3 Tata Telecom Italia Telefonica
4. We have our Layer 7 application layer which the application protocols that we use daily HTTP, DNS, SSH, SMTP and somewhat importantly the BGP protocol We have the Layer 4 Transport layer, which is our where our TCP & UDP protocols live We have the Layer 3 IP or Internet Layer, this is there the IP protocol lives, the ICMP protocol, but also a number of other important routing protocols including IPSEC, OSPF & RIP We have our Layer 2 data-link layer, this layer provides the functional means to transfer data between entities. This is where the Ethernet protocol (802.3) protocol is And finally we have the physical layer which we’ll talk about in a few minutes
5. So in the last 10 years or so we’ve finally started to see an advancement in the implementation of networks, particularly in the following areas Clos Networks Advancement of network speeds Eventual implementation of IPv6 within networks and on the internet Multi-homed internet connections Using BGP as an Interior Routing Protocol All of these things have brought their own set of unique challenges to the way we operate the network, but also the applications we as SRE’s run underneath them. So let’s talk about these
6. Clos Networks, named after Charles Clos who formalized this design in 1952. The Clos Network design actually started out as a multi-stage switching system for telephone systems. Funnily enough, the original “key advantage” of this design was to increase capacity and reduce bottlenecks in switching devices.
7. Fast-forward approximately 60 years, Network Engineers started to use Clos topology in datacenter networks. In a fashion similar to what you see on the screen. The interesting thing about the typical implementation of the Clos (Spine/Leaf) topology is that instead of it being a switching network (A Layer 2 network), It’s a Routed Network (A Layer 3 network).
8. As an aside, Clos networks can be represented a number of different ways. In the three representations shown here, The spine planes are all connected in the same way, just arranged differently.
9. So now we have traffic being routed across multiple links (no L2 protocols here: spanning tree or LACP here). We are using what’s known as Equal Cost Multipath Routing or ECMP. So what does it mean to us as SRE’s? Simply put, how you go from server A to B (within a datacenter or fabric) could be 16, 64 or 256 various paths. Making ”why are my packets not making it to Server B”, a difficult problem to troubleshoot. According to the Paris-Traceroute research paper, ECMP flows are load-balanced using a set of five fields (Source/ Destination IP’s, Ports and Type of Service). Unfortunately, unless you have a SDN controller that’s aware of these flows, it’s not possible to identify the path of application traffic in real time. So where does that leave us for troubleshooting poor connectivity between servers. Unfortunately, for the most part, traditional tools like ping, traceroute and even MTR aren’t useful. Using the default options on these options will only let you discover 1 path out of potentially 100’s. There are two utilities that have made ground in this area: Dublin-traceroute which draws paths Fbtracert, by Facebook which is built ontop of Go Hopefully in the near future, we can bring a similar utility to LinkedIn
10. As you can see, since the 1990’s, we’ve been growing our LAN network speeds every few years. In the space of 20 years, we’ve gone from 10Mb Ethernet over copper wires to 100Gb over optical fibers. As internet backbone speeds have grown, so have the speeds on our desktops and of course on our servers.
11. So to think you’re going to get 10Gb out of the box is somewhat of a pipedream unfortunately, there are some optimizations and forethought required.
12. So for this to work harmoniously together, there’s three components that need to work together NIC Linux Kernel Network Switches
13. So when you look at the NIC side of the equation, there’s so many variables Suggest you check out Joe Damato’s talk from Monitorama 2016 where he talks about why statistic collection for network devices in Linux is probably wrong.
14. Standard TCP congestion control relies on packet loss to detect congestion DCTCP
15. Standard TCP congestion control relies on packet loss to detect congestion DCTCP
16. Standard TCP congestion control relies on packet loss to detect congestion DCTCP
17. http://sophiedogg.com/funny-ipv6-words/
18. http://www.networkworld.com/article/2692482/ipv6/infographic-ipv4-vs-ipv6.html
19. http://www.tcpipguide.com/free/t_IPv6AutoconfigurationandRenumbering.htm
20. 1. Don’t implicitly trust the network.  Remember that the network rarely hard-fails, most of the failures are partial and troublesome to debug