add bridge networking

[tonistiigi]

The worker currently uses host networking. Move this to use a bridge. Example in https://gist.github.com/42a6ca6b8f21af1bead05095aa97681c

buildd can reuse docker0 if it exists, or one can be passed in with a flag.

[kunalkushwaha]

Is this still required or it need to be fixed in context of distributed Buildkit #62 ?

/cc @tonistiigi @AkihiroSuda

[tonistiigi]

Yes, I don't think this is related to #62. We will not expect the workers to create processes that can reach each other. This is more a security feature, to make sure all processes are properly sandboxed.

[tonistiigi]

@AkihiroSuda I saw you were working on slirp go bindings? What's your take on using slirp as a default for buildkit?

[AkihiroSuda]

https://twitter.com/_AkihiroSuda_/status/970665536492158976

I'd use slirp for enabling NAT in unprivileged netns+tap.
Processes within this netns (such as BuildKit) would be unaware of slirp.

I think we should only use slirp for rootless mode, because it is slow.
Even for rootless mode, I'd suggest using lxc-user-net SUID binary for network-intensive workload.

rough benchmark: When I downloaded Ubuntu ISO from the internet using VDE slirp (https://github.com/AkihiroSuda/runrootless/tree/b4dc7bfd80ea05aedd4e92fffbb1d2562572ba47/misc/vde), it tooked 55s while it tooked 47s on the native networking. (Although there is a big space for performance optimization.)

[AkihiroSuda]

Processes within this netns (such as BuildKit) would be unaware of slirp.

p.s. I think BuildKit would use CNI (bridge) plugin by default, for both rootful mode and rootless mode.

[ciaranmcnulty]

This would be useful for us; we build a few wiremock-based images together in a bake, and as part of their builds they run unit tests that bind Java services to local ports and test some requests+responses. Without CNI these can contend; we've ended up moving them out of the default group and explicitly building in series

[sipsma]

Just a note, @tonistiigi mentioned in slack that the blocker on enabling bridge mode in the moby/buildkit images is the performance cost of initializing network namespaces w/ CNI. This could be alleviated by adding support for a pool of pre-initialized, re-usable namespaces.