How to Leverage Go for Your Networking Needs

I’m Sneha.
I’m a software engineer at DigitalOcean.

DigitalOcean
cloud-hosting company
14 data centers
1.5 million droplets
k8saas, dbaas, lbaas, storage, VPC

“What is
Go?”
k8s, Prometheus

“What is
Go?”
k8s, Prometheus
DHCP, Gateways

“What is Go?”
k8s, Prometheus
DHCP, Gateways
LB, port scanner
PIPOD

Why use Go for networking services?

Concurrency
Easy To Fuzz-Test
Server-SIDE
Great for CLI TOOLS
EASY TO WRITE
REST/RPC SERVICES

But when might Go NOT be the
answer?

High-performance
packet-processing

DigitalOcean has a monorepo and
shared packages.

hvflowd
hvaddrd
OvS
br0
hvflowctl
RNS
AddFlows
OpenFlow
SetParameters
bond0
ethX
ethX addr0
bolt
DHCPv4 NDP
gRPC
DHCPv6
tapX dropletX
hvaddrd traffic
hvaddrctl
AddFlows
internet traffic
Hypervisor
main
Using gRPC and CLI
tools

Performance was already good
enough.

OSI Open-systems Interconnect
Model

The Layers: Data Encapsulation

socket: internal endpoint for sending
or receiving data on a computer
network

TCP (stream socket): provides a
connection-oriented, sequenced flow of
data

UDP (datagram socket): provides a
connectionless flow of data

How do we read a packet?
How do we read a frame?

raw socket: a socket that allows sending
and receiving of data without
protocol-specific transport layer
formatting

raw socket: IP layer (3
socket(AF_INET,SOCK_RAW,protocol)
packet socket: link layer (2
socket(AF_PACKET,SOCK_RAW,protocol)

bpf (berkeley packet filter)
interface:
raw link-layer interface on most
unix-based systems

bpf filters also used with raw
sockets or tcpdump

“The best way to learn is
simply by doing.”
 Me

Loadbalancer
LB
A device distributing traffic
across a number of backends.
https://gbhackers.com/load-balancer-reverse-proxy/

Building a layer-7 LB
1. Use HTTP protocol.
2. Accept client-side connections.
3. Pass client-side request to one of the
backends.
4. Return server-response back to the client.

HTTP
HyperText Transfer
Protocol
An application layer (7
protocol:
● request-response based
● stateless
● Has different methods
GET, PUT, etc)
● Uses TCP or UDP sockets
under-the-hood
network deﬁnition
Source:
https://www.ntu.edu.sg/home/ehchua/programming/webprogramming/HTTP_Basic
s.html

http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
...
})
log.Fatal(http.ListenAndServe(":8080", nil))
1. ListenAndServe

n := rand.Intn(len(backends))
r.URL.Host = backends[n]
r.URL.Scheme = "https"
req, err := http.NewRequest(r.Method, r.URL.String(), r.Body)
...
res, err := client.Do(req)
...
}
2. Read and send request to
backend

...
w.WriteHeader(res.StatusCode)
_, err = io.Copy(w, res.Body)
if err != nil {
log.Printf("error writing response to client: %v", err)
}
...
}
3. Return response to client

Building a layer-4 LB
1. Use TCP protocol (via a streaming
socket).
2. Accept client-side connections.
3. Open backend connection.
4. Use one goroutine to shuttle packets from
the client to the backend.
5. Use another goroutine to shuttle packets
from the backend to the client.

TCP Transmission
Control Protocol
Connection-oriented protocol
with 3 phases:
● Connection establishment
TCP handshake)
● Data transfer
● Connection termination
network deﬁnition
https://www.researchgate.net/figure/Three-way-Handshake-in-TCP-Connection-
Establishment-Process_fig7_313951935

GO FEATURE
github.com/oklog/run

listener, err := net.Listen("tcp","127.0.0.1:9090")
if err != nil {
log.Fatal(err)
}
for {
conn, err := listener.Accept()
if err != nil {
log.Printf("error accepting client conn: %v", err)
continue
}
go handleConn(conn)
}
1. Open and bind client TCP socket

func handleConn(clientConn net.Conn) {
n := rand.Intn(len(backends))
backendConn, err := net.Dial("tcp", backends[n])
if err != nil {
log.Printf("error %s: %v", backends[n], err)
return
}
...
}
2. Open and connect backend TCP
socket

var g run.Group
{
g.Add(func() error {
return copy(clientConn, backendConn)
}, func(error) {
clientConn.Close()
backendConn.Close()
})
}
{
g.Add(func() error {
return copy(backendConn, clientConn)
}, ...)
}
err = g.Run()
3. Configure rungroups

func copy(from net.Conn, to net.Conn) error {
for {
readBytes := make([]byte, 1024)
n, err := from.Read(readBytes)
if err != nil {
...
}
if _, err = to.Write(readBytes[:n]); err != nil {
...
}
}
}
4. Read and copy

Loadbalancer IRL
● Exact mechanism of handling connections and/or requests
depends on the layer.
● Layer-4 loadbalancers should be more efficient:
○ Won’t necessarily create a new backend connection per
client conn.
○ Use NAT and munging incoming packets.
● Better loadbalancing algorithms.
● Client stickiness.

Nmap
Linux utility that scans hosts
and attempts to determine
open UDP/TCP ports
network deﬁnition

Building a TCP port-scanner
1. Select range of ports to scan.
2. Try to open and connect a TCP socket to a
remote address (and port)..
3. Print results.

addr := fmt.Sprintf("%s:%d", hostname, port)
conn, err := net.DialTimeout("tcp", addr, timeout)
if err != nil {
fmt.Printf("port %d closed: %vn", port, err)
} else {
fmt.Printf("port %d openn", port)
conn.Close()
}
1. Open and connect TCP socket

conSema := make(chan struct{}, 10)
var wg sync.WaitGroup
for i := 1; i <= 65535; i++ {
wg.Add(1)
conSema <- struct{}{}
go func(port int) {
...
wg.Done()
<-conSema
}(i)
}
wg.Wait()
2. Use waitgroup and channels

Nmap IRL
● Checks UDP and TCP ports on local or remote
hosts.
● Host discovery
● OS detection
● Auditing security of a firewall

DHCP Protocol
● Dynamic Host Configuration
Protocol is used by routers to
allocate IP addresses to
network interfaces
● DHCPv6 uses NDP and
DHCPv4 uses ARP
network deﬁnition
https://study-ccna.com/dhcp-dns/

Need link layer MAC addresses.

Building a DHCP server:
1. Open and bind a raw socket to an interface.
2. Read data from socket into bytes buffer.
3. Unmarshal into DHCP message and retrieve
sender hardware address.
4. Switch between handlers based on message
type.
5. Validate DHCP request message and craft
response.
6. Unicast response back to sender.

GO FEATURE
github.com/mdlayher/raw

GO FEATURE
filename := "eth-packet-socket"
typ := unix.SOCK_RAW
if cfg.LinuxSockDGRAM {
filename = "packet-socket"
typ = unix.SOCK_DGRAM
}
sock, err := unix.Socket(unix.AF_PACKET, typ, 0)
...
f := os.NewFile(uintptr(sock), filename)
sc, err := f.SyscallConn()
...
n, addr, err = unix.Recvfrom(int(fd), p, flags)
github.com/mdlayher/raw/raw_linux.go

GO FEATURE
// Try to find an available BPF device
for i := 0; i <= 10; i++ {
bpfPath := fmt.Sprintf( "/dev/bpf%d", i)
f, err = os.OpenFile(bpfPath, os.O_RDWR, 0666)
if err == nil {
break
}
if perr, ok := err.(*os.PathError); ok {
if perr.Err.(syscall.Errno) == syscall.EBUSY
{
continue
}
}
return nil, err
}
github.com/mdlayher/raw/raw_bsd.go

GO FEATURE
syscall
os
golang.org/x/sys/unix

ifi, err := net.InterfaceByName(iface)
if err != nil {
return nil, err
}
pc, err := raw.ListenPacket(ifi, uint16(ethernet.EtherTypeIPv4),
&raw.Config{
// Don't set any timeouts to avoid syscall busy
// loops, since this server will run forever anyway.
NoTimeouts: true,
})
1. Open and bind raw socket

b := make([]byte, 1500)
for {
n, from, err := s.pc.ReadFrom(b)
if err != nil {
...
continue
}
buf := make([]byte, n)
copy(buf, b)
workC <- request{
buf: buf,
from: from,
}
}
2. Read data into buffer

var workWG sync.WaitGroup
workWG.Add(Workers)
workC := make(chan request, Workers)
for i := 0; i < Workers; i++ {
go func() {
defer workWG.Done()
for r := range workC {
s.serve(r.buf, r.from)
}
}()
}
3. Worker handles request

func (h *handler) serveDHCPv4(ctx context.Context, req
*dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr)
{
...
if !from.EthernetSource.Equal(reqMAC) {
...
return h.shouldNAK(req.Type, to, broadcast)
}
...
}
4. Validate request

func (h *handler) serveDHCPv4(ctx context.Context, req
*dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr) {
...
res, err := h.buildResponse(ctx, params, req, from)
if err != nil {
topError(span, err, "failed to build response")
return h.shouldNAK(req.Type, to, broadcast)
}
res.Broadcast = broadcast
return res, to
...
}
5. Build and send response

DHCP Summary:
1. DHCP dynamic host configuration
protocol used to dynamically assign IP
address.
2. Use raw sockets.
3. Jump down to link-layer to do source MAC
validation.

Go can be good for building
networking primitives.

Layer 3/2:
raw package
os,syscall
golang.org/x/sys/unix

sync package
github.com/oklog/run package
goroutines
channels

Ohai!
@snehainguva
sneha.inguva@gmail.com

Sources
● Port Scanner: https://github.com/si74/portscanner
● Layer 7 loadbalancer: https://github.com/si74/layer7lb
● Layer 4 loadbalancer: https://github.com/si74/tcpproxy
● Raw package: https://github.com/mdlayher/raw
● Godocs: https://godoc.org/
● Golang: https://godoc.org/

How to Leverage Go for Your Networking Needs

Related slideshows

More Related Content

How to Leverage Go for Your Networking Needs