DC/OS 1.8 Container Networking

© 2016 Mesosphere, Inc. All Rights Reserved.
DC/OS 1.8
NETWORKING
@SARGUN
1
Sargun Dhillon, July 2016

WHO AM I?
2

AGENDA
3
• How, and why did we go here?
• How does DC/OS bring you closer to the ideal?
• Some of the future

A BRIEF
HISTORY OF
NETWORKS
IN THE DC
4

ORGANIZATIONS CIRCA 2007
5
•Before DevOps was first heard
•Clear diﬀerentiation of ownership
•The datacenter was owned by a the NOC
•Deployment of services was done by sysadmins in the operations group
•Developers operated without access to production
•Production deployments gated by QA, Operations

SOFTWARE CIRCA 2007
6
•Diﬀerent services glued together via CORBA, XML-RPC, SOAP
•No one was really consciously doing microservices
•Networks were static, giant layer 2 domains
•Load Balancing provided by hardware
•Everyone ran their own datacenter
•EC2 in its infancy, only a year prior has the term “Cloud” began to become popular
•Systems statically partitioned

© 2016 Mesosphere, Inc. All Rights Reserved. 7
SaaS continued to grow at
an incredible rate

There became a race to ship faster

We kept the software alive
By feeding it
With Sysadmins

We kept the machines alive
By feeding them
With Blood

This wasn’t working

ORGANIZATIONS 2008+
12
•We began seeing a gradual shift in the industry where lines between QA, Dev, and
Ops were blurring
•Devops term coined in 2008, first DevOpsDay in 2009
•Gradual adoption of the cloud, fewer organizations owning their own datacenters
•Either networking was outsourced to the cloud, or typically remained in a small
internal organization
•Needed to reduce ratio of operators to servers

SOFTWARE CIRCA 2008+
13
•Popularization of Open Source tooling to automate much of traditional operations, and QA
•Jenkins / Hudson
•Puppet / Chef
•Capistrano
•Popularization of stacks requiring with more complex operational requirements
•Nutch / Hadoop
•NoSQLs
•Still statically partitioned machines
•Networks still sacred territory

CIRCA 2011
14
•Much of what’s been happening for the past half-decade hits networking
•Much of this falls under the term “SDN” (Software Defined Networking) or “NFV” (Network
Function Virtualization)
•Hastened by the adoption of VMs in the enterprise in the hype cycle
•Openflow promises to fix everything
•Major adoptions of the cloud by startups as well as enterprise
•Virtualization begins to become mainstream as a mechanism of consolidating workloads
•The invention of the “private cloud”
•DotCloud / Docker funded by Y-combinator a year earlier
•Term “Microservice” coined

CIRCA 2013
15
• Docker becomes instant hit and brings containers to the forefront
• Dynamic partitioning begins to make in-roads
• Google releases Omega paper
• Apache Aurora open sourced
• Microservice counts explode, demanding collocation of workloads for eﬀiciency
• Mesosphere Founded
• Site Reliability Engineering begins to popularize and further blur the lines between
Dev, Ops, and QA

Everything was changing

Why?

Business Value

BENEFITS
20
•Reduction in cost of goods sold
•Smaller engineer to server ratio
•Linear, or super linear growth rate of engineering team to servers is unsustainable
•Smaller engineer to capability ratio, where capability includes:
•Features
•Throughput
•Better User Experience
•Better availability
•Quicker release to features

But at what cost?

Complexity

OLD WORLD
23

NEW WORLD
24

DIVING DEEPER
25

Paxos?
Raft?
Ω Failure Detector?
Pods?
Wat?
Sidecars?
Etcd?
Zookeeper?
VxLan?

Performance

REDIS PERFORMANCE
28

THE DC/OS
APPROACH
29

CORE TENANTS
30
•DC/OS must be agnostic to the underlying environment
•AWS / Azure / GCE / Softlayer as the lowest common
denominators
•DC/OS should require no to minimal changes to the
code in order to work
•DC/OS should provide similar services to existing
environments
•Fixed load balancers
•Security
•IP/Container
•We do not want to require a change in organization
procedures

CURRENT
SERVICES
PROVIDED
31
• Service Discovery
• Mesos-DNS
• Navstar*
• Spartan*
• Load Balancing
• Minuteman*
• Accessibility
• Octarine*
• IP Per Container
• Control Plane
• Lashup* *Project Name

SERVICE DISCOVERY
PROJECTS:
MESOS-DNS
NAVSTAR
SPARTAN
32

MESOS-DNS / NAVSTAR / SPARTAN
33
• Simple service discovery mechanism that exposes service locations over DNS
• Service names, and locations exposed
• via SRV records.
• via A records
• Typically requires modification of downstream code
• Good for bootstrap

HIGH LEVEL
34

EXTENDED
USAGE
35

INTEGRATIONS
36

SPARTAN
37
• DNS proxy that’s closely coupled with Navstar
• Raises availability by doubling work
• Makes DNS 2N, 2N+1, or N+1 systems act as such
• Reduces latency at scale
• Dual dispatches the query, and waits for first response
A Jeﬀ Dean Jig

SPARTAN
38

LOAD
BALANCING
PROJECTS:
MINUTEMAN
39

MINUTEMAN
40
•Low-Overhead TCP load balancing
•Low-overhead during continuous TCP connection
•Pay balancing cost upfront
•Inflicts minimal overhead on non-load balanced traﬀic
•Fault-tolerance period aims to be <100ms

FUNCTIONALLY
41

VIRTUAL
NETWORKS
PROJECTS:
NAVSTAR
45

VIRTUAL NETWORKS
46
•Base DC/OS 1.8 functionality
•With custom Mesos module
•Provides IP/Container out of the box
•Utilizes oﬀ the shelf encapsulation
•VXLan
•Artisanal controller built at Mesosphere: Navstar

50’000 FOOT VIEW
47

DIVING IN
48

ACCESSIBILITY
PROJECTS:
OCTARINE
49

OCTARINE
50
• Transparent HTTP Proxy
• Automatically integrates with Mesos-DNS Resolves SRV records
• SOCKS Proxy
• Automatic OpenVPN Proxy
• Currently leverages master SSH access for ACLs
• Soon will integrate with DC/OS ACLs

USE CASE
51

INTERACTION DIAGRAM
52

OCTARINE: BENEFITS
53
• Provides Day 0 access to DC/OS services
• With security
• Without internet exposure
• Without task pinning
• Works without custom software
• Works without infrastructure modification

CONTROL PLANE
PROJECTS:
LASHUP
54

We had a problem

Computers

Sometimes
They Break

Sometimes
Many Break

Sometimes
You don’t know

Before Lashup

90+ Second Resolution
For
10% failure

You’d Need to Be Ultron To Keep Track
Of It All

Let’s Distribute the Problem

A Mess

Academia

Connected Graph
Hyparview

Constant Adaptive
Health Checks

Dealing with Failure
Hyparview

Hyparview

Throw Some Link-State
Routing At it

Free Multicast

Sprinkle On Some CRDTs

PROJECT LASHUP
75
•A novel distributed systems SDK that provides:
•Failure detection
•Membership
•Multicast Delivery
•Strongly-eventually consistent data storage
•Powers:
•Minuteman VIP dissemination
•Minuteman node liveness checks
•Overlay routing
•DNS Synchronization

THE FUTURE
76

FUTURE PLANS?
77
• Security
• Encryption in flight
• Task-level Microsegmentation and Filtering
• Further research required:
• QoS between services
• “Zero-overhead” NFV

Zero Overhead NFV:
Rewrite the OS
at
The Syscall Layer

Zero Overhead NFV:
~/linux$ sudo samples/bpf/test_probe_write_user
Server bound to: 127.0.0.1:35707
Client connecting to: 255.255.255.255:5555
Server received connection from: 0.0.0.0:44804
Client's peer address: 127.0.0.1:35707

Zero Overhead NFV:
~/linux$ sudo strace -e ... samples/bpf/test_probe_write_user
bind(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
getsockname(3, {sa_family=AF_INET, sin_port=htons(42085), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
Server bound to: 127.0.0.1:42085
Client connecting to: 255.255.255.255:5555
connect(7, {sa_family=AF_INET, sin_port=htons(5555), sin_addr=inet_addr("255.255.255.255")}, 16) = 0
accept(3, {sa_family=AF_INET, sin_port=htons(50016), sin_addr=inet_addr("127.0.0.1")}, [16]) = 8
Server received connection from: 0.0.0.0:50016
getpeername(7, {sa_family=AF_INET, sin_port=htons(42085), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
Client's peer address: 127.0.0.1:42085

ZERO-OVERHEAD NFV
81
•Implemented using kernel probes
•JIT at runtime
•Allows standard BSD API to work
•In preliminary testing: Undetectable overhead

ZERO-OVERHEAD NFV
82
•We’re still taking the VM approach; Why not take the container approach to NFV?
•Lie to the program
•Manipulate the syscalls
•Win big
•Current Status:
•Research Project
•Upstreamed first patches due for the 4.8 Kernel

DC/OS 1.8 NETWORKING
83
•Core functionality:
•IP/CT
•Internal Service Discovery
•Load Balancing
•3rd Party Service Integration
•Upcoming Features:
•Security
•External Load Balancing
•Future research going on

DC/OS 1.8
NETWORKING
@SARGUN
84

DC/OS 1.8 Container Networking

Related slideshows

More Related Content

DC/OS 1.8 Container Networking