Fabio rapposelli pks-vmug
- 4. Containers 101
4
Container Host (VM)
Developer
Dev Host (VM)
UBUNTU
JAVA
TC SERVER
{APP}
KERNEL
CONTAINERCONTAINER
Portable
Container Image
`docker run –d myimage`
CONTAINER
• Reliable Packaging
• Server/VM Density
• Fast Time To Launch
• Built for CI/CD
CONFIDENTIAL
- 5. Kubernetes 101 (CaaS)
5
K8s Cluster
Worker
`kubectl apply –f myapp.yml`
Worker
kube-proxy
Master
& ETCD kube-proxy
Service: nodeport | ingress | LB
POD POD
Load Balancer
URL Request:
myapp.foo.com/k8siscool
Docker
Registry
Developer
Containers @ Scale
Master
& ETCD
Master
& ETCD
CONFIDENTIAL
- 6. Pivotal Cloud Foundry 101 (PaaS)
6
war
Availability Zone 1 Availability Zone 2 Availability Zone 3
Staging
Root
FS
Build
Pack
war
`cf push`
Drop
let
A
I
A
I
myapp.foo.com
*.foo.com = NSX Edge Vip
NSX Edge
PCF Routing PCF Routing PCF Routing
LB Pool Members
“Here is my source code
Run it on the cloud for me
I do not care how”
URL Request:
myapp.foo.com
Developer
CONFIDENTIAL
- 9. 9
Code Analysis Testing
Commit Code
Changes
Staging Production Zero Downtime Upgrades
AUTOMATED
PIPELINE
SPEED
Releasing smaller things more
often will reduce complexity and
improve time-to-market
QUALITY
We embed testing early in the lifecycle
to surface problems sooner, avoiding
last minute issues and helping us be
more responsive to change
AGILITY
Let’s push updates on a
regular basis without ANY
downtime to improve
customer experience and
shorten time-to-market
AUTOMATION
Let’s integrate tools and
automate processes from
testing, to builds & deployment
CI/CD CI/CD CI/CD CI/CD CI/CD
SOFTWARE DEVELOPMENT LIFECYCLE
Agile methods help drive Digital Transformation
Problem to Solve, Faster Time To Value …
Drive Business Value into Production Faster and Safer
CONFIDENTIAL
- 10. Multiple Use Cases Dictate Multiple Workloads and Approaches
10
Container Instance (CI) Container Service (CaaS) Application Platform (PaaS)
IaaS
CONFIDENTIAL
CONTAINERS BATCHES
DATA SERVICES MICROSERVICESMONOLITHIC
APPLICATIONS
The Goal:
Pick the Right
Approach for
the Workload
CONFIDENTIAL
- 11. IaaS
Choosing the Right Tool for the Job
11
Developer
Provides
Tool
Provides
Container
Service
Container Orchestration
Container Scheduling
Primitives for Routing,
Logs & Metrics
CONTAINER IMAGES,
TEMPLATES, DEPLOYMENTS
Application
Platform
APPLICATION CODE
Container Service
Container Image & build
L7 Network & Routing
Logs, Metrics, Monitoring
Services Marketplace
Team, Quotas & Usage
Container
Instance
CONTAINER IMAGE
Container Runtime
Primitives for Network and
Storage
Container Instance
CONFIDENTIAL
CONFIDENTIAL
- 12. IaaS
Choosing the Right Tool for the Job
12
Developer
Provides
Tool
Provides
Container
Service
Container Orchestration
Container Scheduling
Primitives for Routing,
Logs & Metrics
CONTAINER IMAGES,
TEMPLATES, DEPLOYMENTS
Application
Platform
APPLICATION CODE
Container Service
Container Image & build
L7 Network & Routing
Logs, Metrics, Monitoring
Services Marketplace
Team, Quotas & Usage
Container
Instance
CONTAINER IMAGE
Container Runtime
Primitives for Network and
Storage
Container Instance
CONFIDENTIAL
Application Specificty
Higher flexibility, lower automation, more DIY
CONFIDENTIAL
- 13. IaaS
Choosing the Right Tool for the Job
13
Abstraction
Container
Service
CONTAINER IMAGES,
TEMPLATES, DEPLOYMENTS
Application
Platform
APPLICATION CODE
Container
Instance
CONTAINER IMAGE
CONFIDENTIAL
Pivotal Container Service
Pivotal Cloud Foundry
Elastic Runtime
BOSH
vSphere Integrated
Containers
CONFIDENTIAL
- 15. Purpose-built container service to operationalize Kubernetes
for the multi-cloud enterprises and service providers
Fully Supported Kubernetes
Runs on vSphere and VMC
Unified VM + Containers on SDDC
Deep Integration with NSX
Hardened, Production-grade
HA, Security, Multi-tenancy, Tools
VMware and Pivotal Collaborate to Deliver
VMware Pivotal Container Service (VMware PKS)
- 16. Fault-tolerance for
masters, workers,
and etcd nodes
Auto-scaling of
masters, workers,
and etcd nodes
Routine health
checks and self-
healing of cluster
LCM includes rolling
upgrades to ensure
workload uptime &
application of CVEs
ScalingHigh Availability Health Checks
& Healing
Lifecycle
Management
VMware PKS – Solving Day-2 Operational Challenges
- 17. 17
BOSH
VMware GCP Azure Openstack AWS
Container Infrastructure for Cloud-Native Apps
Rapidly deliver and operationalize next generation apps
Container
Registry
Kubernetes on BOSH (Kubo)
NSX-T
GCP
Service
Broker
masteretcd workermasteretcd worker
PKS Controller
- 18. Who is PKS built for?
18
IT
Operator
– PRE (Platform Reliability
Engineering)
– Deploy, Scale, Operate
Platform
– Innovation of Business
Capability as Cloud
native Apps
– Develop, Deploy, Scale,
Monitor Apps
– Physical Infrastructure is
Operated
– Network & Security
Control Policy is defined
• Platform Reliability Engineers
– Platform is Reliable
– Capacity Is planned for
– Platform is Secured & Controlled
– Platform is Auditable
– Application Dev/Ops owners are Agile
• Application Dev/Ops owner
– Automate Everything
– Agile
* Role Shift
– It is common to see the VI Admins (IT Ops), becoming the Platform Reliability Engineer
Cloud Native Applications at scale can & should
be kept running by a 2 Pizza Team mentality
(DevOps in Action) Application
Dev/Ops Owner
Platform
Reliability Engineer
CONFIDENTIAL
- 19. 19
BOSH
VMware GCP Azure Openstack AWS
Container
Registry
Kubernetes on BOSH (Kubo)
NSX-T
GCP
Service
Broker
masteretcd workermasteretcd worker
PKS Controller
PKS Technical Overview
- 21. PKS
BOSH
K8S-1
Work
er
Worker
K8S-2
BOSH
Agent
BOSH
Agent
K8s-api
Team A
K8s-api
KUBO
BOSH
Release
(tgz)
DAY 2 Ops
- Auto/Manual Rebuild
- Auto/Manual Repair
- Manual Scale
- Patch & Upgrade
- Control & Audit OPS Events
NAMESPACE_1: TEAM A
NAMESPACE_2: TEAM B
Team C
Team B
NAMESPACE_1: DEFAULT
DAY 1 Ops
DEPLOY
OperateK8s+RunApps/Containers
UI
&
API
Worker
Application
Dev/Ops Owner
Application
Dev/Ops Owner
Application
Dev/Ops Owner
Work
er
MASTER
WorkerMASTER
ETCD
WorkerMASTER
ETCD
MASTER
MASTER
ETCD
Platform
Reliability Engineer
Self Service K8s
BOSH Day 2
1.7 -> 1.8
1.7 -> 1.8
PKS Controller
CONFIDENTIAL 21
- 22. 22
BOSH
VMware GCP Azure Openstack AWS
Container
Registry
Kubernetes on BOSH (Kubo)
NSX-T
GCP
Service
Broker
masteretcd workermasteretcd worker
PKS Controller
PKS Technical Overview
- 23. 23
Need Harbor screenshot
• user management & access control
• role-based access control
• AD/LDAP integration
• Security vulnerability scanning
(Clair)
• content trust - image signing
• policy based image replication
• audit and logs
• Restful API
• open-source under Apache 2
license
Harbor – Enterprise Grade Private Registry
CONFIDENTIAL
- 26. Harbor – Use Cases
PKS Stemcell
CVE in Root File
System of Container
CVE Exec Layer: TC
Server
CVE on the Container
Host OS
Vulnerability in
Code{}
Restage Applications
CVE FOUND
!!!
BOSH
CVE & Update Patching
• Patch OS Level via Stemcells
• Harbor Scans Images for
Vulnerability (Clair)
• Address CVE in minutes/hours
versus days/weeks
Application
Dev/Ops Owner
Platform
Reliability Engineer
OS CVE
FOUND !!!
Patched
Stemcell
Patched
Stemcell
Patched
Worker(s)
CONFIDENTIAL 26
- 27. 27
BOSH
VMware GCP Azure Openstack AWS
Container
Registry
Kubernetes on BOSH (Kubo)
NSX-T
GCP
Service
Broker
masteretcd workermasteretcd worker
PKS Controller
PKS Technical Overview
- 28. WorkerWorkerWorker
K8s
Master
K8s
Master
Kubernetes Components
• K8s Cluster Consists of Master(s)
and Nodes
• K8s Master Components
– API Server
– Scheduler
– Controller Manager
– Dashboard
• K8s Node Components
– Kubelet
– Kube-Proxy
– Containers Runtime (Docker for PKS 1.0)
28
Controller
Manager
K8s API
Server
Key-Value
Store
dashboard
Scheduler
K8s Nodes
kubelet c runtime
Kube-proxy
> _
Kubectl
CLI
K8s Master(s)
POD POD
Application
Dev/Ops Owner
CONFIDENTIAL
- 29. K8s POD
Kubernetes Pod – Networking Basics
Special
‘Pause’ container
(‘owns’ the IP stack)
10.24.0.0/16
10.24.0.2
nginx
tcp/80
mgmt
tcp/22
logging
udp/514
IPC
External IP Traffic
• A Pod is a group of one
or more co-located
containers that share
an IP address, PID
namespace and/or
Data Volumes
29CONFIDENTIAL
- 31. NSX-T & PKS Components
NSX Container Plugin (NCP)
• NCP is a software component
provided by VMware in form of a
container image, e.g. to be run as a
K8s Pod.
• NCP is build in a modular way, so
that individual adapters can be
added for different CaaS and PaaS
systems
31CONFIDENTIAL
- 32. PKS & NSX-V • PKS supported with NSX-V or without NSX
• Flannel overlay.
• 1 Flat SDN Overlay per Cluster
• 1 Large CIDR “10.200.0.0/16”
• Each worker node routes a
subnet for Pods across
• Example: 10.200.1.0/24
• No integrated North South Load
Balancing
• No Integrated Security Policy
32
K8s Cluster
K8s Cluster
Namespace 1 Namespace 2 Namespace 3
VXLAN Network
Namespace 1 Namespace 2 Namespace 3
• NSX-T
• Multiple Logical Switches (L2 Domain)
per Namespace
• Routable as NAT or No-NAT
• Integrated Load Balancing (NSX-T 2.1)
• Integrated Security Policy
CONFIDENTIAL
- 33. PKS w/ NSX-T & NSX-V
• NSX-V and NSX-T Can coexist.
• Dedicated Clusters for
NSX-T Managed Hosts
• Can Share a common
vCenter backplane
33
NSX-T
Managed
Common vCenter
w/ NSX-v
managed Hosts
CONFIDENTIAL
- 34. NSX-T & PKS Operational Tools
34
NSX-T Traceflow
NSX-T Operational Tools
• Traceflow
• Port Mirroring
• Port Connection Tool
• Spoofguard
• Syslog
• Port Counters
• IPFIX
CONFIDENTIAL
- 35. 35
BOSH
VMware GCP Azure Openstack AWS
Container
Registry
Kubernetes on BOSH (Kubo)
NSX-T
GCP
Service
Broker
masteretcd workermasteretcd worker
PKS Controller
PKS Technical Overview
- 36. GCP Service Broker
K8s Cluster
WorkerWorker
kube-proxy
Master
& ETCD
kube-proxy
Service: nodeport | ingress | LB
POD PODMaster
& ETCD
Master
& ETCD
Application
Dev/Ops Owner
Broker: GCP
SVC Catalog
Controller
Broker: X
Broker: X
K8s Secrets:
`kubectl cs pubsub –n my_pubsub`
• Self Service consumption of
GCP services.
• Operator controlled via plan &
subscriptions
• Service bindings (credentials &
connection urls) stored as K8s
Secrets
- 38. Project Hatchway : https://github.com/vmware/vsphere-storage-for-kubernetes
ESXi
vCenter
vSphere Storage for Kubernetes
38
ESXi
K8s Worker (Container Host)
Datastore1dataVol.vmdk
K8s kubelet
# vi vsphere.conf
POD
Tools,
Libs, SW
Redis
DB
K8s API
ESXi
<Add Flags & Restart Ctrlr, API, Kubelets>
--cloud-provider=vsphere
--cloud-config=vsphere.conf
# systemctl restart kubelet.service
kind: PersistentVolume
spec:
capacity: [storage: 16Gi]
storageClassName: slow
K8s Volume
PersistentVolumeClaim
K8s vSphere
Cloud provider
Container
CONFIDENTIAL
- 40. PKS Telemetry – On vSphere
Who needs what?
40
Infra K8s Containers Apps Application
Dev/Ops Owner
Platform
Reliability Engineer
vRLI
vRops Wavefront
CONFIDENTIAL
- 41. Monitoring & Logging
41
METRICS
LOGS
Metrics & Logs emit from
many Sources:
• IaaS (vSphere)
• PKS K8s Platform
• Applications
• NSX
• Physical & Logical
Platform Reliability
Engineer MUST leverage
ALL of them
PKS Control
IaaS
CONFIDENTIAL
- 42. Deamon
Set
Deamon
Set
vRLI Logging w/ PKS
POD vRLI
POD
vRLI
• App Logging
• System Logging
– OS & Processes not
run in Containers
App Logging
• Per App Only
Sidecar
• App Logging @ Pod level
POD
Daemon
Set
(PODs)
vRLI
POD
LOGGER
DOCKERDDOCKERD
vRLI
DaemonSet
• App Logging @ Cluster level
• Cluster Logging
Dockerd
• App Logging @ Cluster level
• Cluster Logging
• Not handled in K8s API
SyslogD
Platform
Reliability Engineer
Application
Dev/Ops Owner
&
CONFIDENTIAL 42
- 43. Wavefront & PKS
K8s Monitoring Integration w/
Wavefront by VMware
Wavefront Integration can be
deployed as containers within the
K8s Cluster
– Proxy
– Heapster
• Comprehensive Dashboards
– SaaS
• APM for the Developer
• Cluster KPIs for the Operator
• Integrated with PKS
Image source: https://www.wavefront.com/surf-container-wave-join-wavefront-container-world-santa-clara/
Platform
Reliability Engineer
Application
Dev/Ops Owner
CONFIDENTIAL 43
- 44. 44
vRops & PKS (Operations & Monitoring)
vRealize Operations & K8s
• Operator KPIs
• Single Pane for SDDC & K8s
clusters monitoring
• vRLI Integrated
• Alert on K8s KPIs
• Entity Relationship
• Capacity Planning
• Integrated with PKS
Platform
Reliability Engineer
CONFIDENTIAL
- 45. CATALOG
Entitlements, Approvals, Policies
CD PIPELINE
Developers, CI/CD LOB Users
MANAGEMENT&OPERATIONS
PRIVATE CLOUD
OR DATA CENTER
PUBLIC
CLOUD
BRANCH/EDGE
COMPUTE
APP FRAMEWORKS
PAAS FAAS
GLOBALLY CONSISTENT INFRASTRUCTURE AS CODE
IAAS ORCHESTRATION
BLUEPRINT
CLOUD APIs
CLOUD APIs
Consume native K8s
services from PKS
1
BLUEPRINTS & ITERATIVE
DEVELOPMENT
Compose applications using
simplified YAML iteratively &
Deploy to K8s
2
INTEGRATED CATALOG
AND PIPELINE
Catalog for self-service
provisioning of PKS K8s &
applications pipelines for
CI/CD
3
vRA & PKS (Automation)
Application
Dev/Ops Owner
Platform
Reliability Engineer
CONFIDENTIAL 45
- 46. vRNI & PKS (Security & Analytics) – Post 1.1
Platform
Reliability Engineer
vRealize Network Insight & K8s
• Plan Security Policy based on
knowledge of actual traffic
patterns
• Continuously monitor & audit
network security compliance
• Complete Network Visibility and
Troubleshooting
• Accelerate micro-segmentation
deployment
CONFIDENTIAL 46
- 48. 48
BOSH
VMware GCP Azure Openstack AWS
Container
Registry
Kubernetes on BOSH (Kubo)
NSX-T
GCP
Service
Broker
masteretcd workermasteretcd worker
PKS Controller
PKS Technical Overview
Editor's Notes
- Walk Thru of a Container 101
Describe benefits of containers and establish common understanding for K8s discussion.
- With announcements today about PKS lets look a little at how K8S is different from PCF
From the Developer point of view:
I check my code in just like if I were pushing to PCF
But in addition to application artifacts, the pipeline is going to build an image for me …
In this visual we have a K8S cluster already running docker as the backend container engine, so our CI/CD pipeline will build a docker image for us and post it to a registry, in this case VMware Harbor
Afterwhich, the pipeline will instantiate a K8S deployment to run our docker image based application as a set of pods in a replica set in case a worker note goes offline.
The developer can than create a ‘service’ that gives worker nodes (or any external node) running the kube-proxy service the ability to route to where those pods are and access the apps/microservices running in them.
Ingress routing from external is similar to that of CF with an external DNS map being required to forward requests to 1 or more worker nodes running kube-proxy
One of the key differences is that Kubernetes isn’t opinionated on how the container image should be built, this give more flex to the developers but in some cases can make things more difficult for operators as we’ll see later on in the presentation
Agility is why developers want it
- Lets walk thru what makes PCF so Powerful ….
From the Developer point of view:
I write my code {}
I check it into a repository
A CI/CD pipeline then builds & tests my code, then outputs an ‘artifact’. In this visual, we will use a java app, so it’s a war.
The pipeline then ‘pushes’ the artifact to PCF to stage
From here its all up to the platform ….
Staging occurs, where an image called a ‘droplet’ is built by combining a (1) a read only root filesystem , (2) a buildpack that is a tarball that contains the exec components like tc server for example to run a java app, (3) and the app artifact
After staging, the app can now be run. For example if we say that we want 2 instances of the application, PCF will launch 2 containers using the same droplet image we just compiled and schedule them across CF Availability Zones. This gives us the ability to keep our app up if an AZ were to go offline.
PCF also creates a route map for our application so when a request is forwarded to it, the request can be routed to the correct containers. PCF calls these containers Application Instances or AIs
Developers also benefit from a rich set of buildpacks in the platform support many application dev frameworks. Even .net apps with Windows Container hosts are supported by PCF.
Agility is why developers want it
- Application Purchases Will Increasingly Be "Build," Not "Buy"
Gartner predicts that by 2020, 75 percent of application purchases supporting digital business will be "build," not "buy." Gartner's research shows that many organizations already favor a new kind of "build" that does not include out-of-the-box solutions, but instead is a combination of application components that are differentiated, innovative and not standard software or software with professional services (for customization and integration requirements), or solutions that are increasingly sourced from startups, disrupters or specialized local providers.
http://www.gartner.com/newsroom/id/3119717
- Adopting Agile processes is a key driver to help a business digitally transform. Software truly is eating the world.
The key for these business is changing not only the way apps are coded, for example cloud native/12 factor) but also the processes by which they are built and operationalized
Speed: Compose apps as micro services to allow more scalable and rapid development. Work for smaller releases to reduce sprints
Automation: Automate everything. It reduces risk and increases speed
Quality: Test Driven coding, tests should be part of the pipeline, if a fault is found, tests go back into the pipeline.
Agility: Release often, design apps and pipelines to allow for frequent pushes.
-
By making the first task on any software effort “delivery” - deploy the code somewhere, even if it doesn’t do anything.
And then keep doing that every time you change anything…
- In the ‘New Stack” required for an agile world , the Developer and the Operator need to act as 1, or at least a 1 pizza team (or 2 pizza if they are hungry). Sort of like the acronym Devops
This means that just like the Developer needs everything API Driven & self service from the platform, the Platform Operator also needs everything API driven & self service from his infrastructure. The Devops team cant lob stuff over the fence, they own it!!!!
- API server: Target for all operations to the data model. External API clients like the K8s CLI client, the dashboard Web-Service, as well as all external and internal components interact with the API server by ’watching’ and ‘setting’ resources
Scheduler: Monitors Container (Pod) resources on the API Server, and assigns Worker Nodes to run the Pods based on filters
Controller Manager: Embeds the core control loops shipped with Kubernetes. In Kubernetes, a controller is a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state
Etcd: Is used as the distributed key-value store of Kubernetes
Watching: In etcd and Kubernetes everything is centered around ‘watching’ resources. Every resource can be watched in K8s on etcd through the API Server
Kubelet: The Kubelet agent on the Nodes is watching for ‘PodSpecs’ to determine what it is supposed to run
Kubelet: Instructs Container runtimes to run containers through the container runtime API interface
Docker: Is the most used container runtime in K8s. However K8s is ‘runtime agnostic’, and the goal is to support any runtime through a standard interface (CRI-O)
Rkt: Besides Docker, Rkt by CoreOS is the most visible alternative, and CoreOS drives a lot of standards like CNI and CRI-O
Kube-Proxy: Is a daemon watching the K8s ‘services’ on the API Server and implements east/west load-balancing on the nodes using NAT in IPTables
- POD: A pod (as in a pod of whales or pea pod) is a group of one or more containers
Networking: Containers within a pod share an IP address and port space, and can find each other via localhost. They can also communicate with each other using standard inter-process communications like SystemV semaphores or POSIX shared memory
Pause Container: A service container named ‘pause’ is created by Kubelet. Its sole purpose is to own the network stack (linux network namespace) and build the ‘low level network plumbing’
External Connectivity: Only the pause container is started with an IP interface
Storage: Containers in a Pod also share the same data volumes
Motivation: Pods are a model of the pattern of multiple cooperating processes which form a cohesive unit of service
- (click) Configure a vSphere Cloud Provider Manifest. Provide key info …. like vCenter Creds & default datastores
(click) Restart all core K8s components & add new flags to enable vSphere Cloud Provider (API, K8s Ctrlr, & Kubelets
(click) Create a K8s Persistent volume
kubectl cmd applys the yaml via the K8s API…
The Kubelet picks up the work and uses the configured Storage provider
The Persistent Volume is created on the Datastore (can even optionally pass vSAN Storage Classes for SBPM)
(click) The vmdk is represented as a K8 PersistentVolume
A running POD can now make a PersistentVolumeClaim and mount the volume
https://vmware.github.io/vsphere-storage-for-kubernetes
- A PCF deployment will emit various logs & metrics from various sources.
- How do we modernize IT and Applications across multiple clouds and multiple platforms:
1. Make the cloud easy: Create /Deploy/ OOTB content / integrations for a private cloud
A.) Easy deploy(LCM)
B.) Quick TTV(OOTB dashboard, sizing, workflows, Integrations) infoblox, snow, puppet, teraform, OOTB content
C.) SaaS services
2. Simplify dev consumption: Unified consumption model across all clouds
A.) Globally consistent IaaS (API)
B.) Blueprints and Iterative dev
C.) Integrated catalog of services and pipeline
3. Consistent, unified ops: Unified Ops for all apps across platforms
A.) Closed loop workload scheduling (Automatically place and re-balance VMs)
B.) Realtime full-stack troubleshooting and monitoring (wavefront) (extra slide)
C.) App intelligence (bringing together infra and apps, NI, apps, infra metrics) (possible extra slide)
- MG: Add Opsman experience here