Cncf kanister.pptx

Kanister: Application-Level
Data Operations on Kubernetes

© 2021 Kasten by Veeam. Confidential information. All rights reserved. All trademarks are the property of their respective owners.
Abstract
For stateful, cloud-native applications, data operations must often be performed by tools with
semantic understanding of the data. The volume-level primitives provided by orchestrators are
not sufficient to support data workflows like backup/recovery of complex, distributed databases.
To bridge this gap between operational requirements for these applications and Kubernetes, the
open source project Kanister was created. Kanister is a framework to support application-level
data management in Kubernetes. It lets developers define relationships between tools and
applications, and then makes running those tools in Kubernetes simple. Kanister is managed
through Kubernetes API objects called CustomResourceDefinitions, and all interactions with
Kanister take place through Kubernetes tools and APIs. In short, Kanister allows administrators
and automation to perform data operations at the Kubernetes level regardless of the complexity
of the application.
In this live webinar, Kanister contributors present how it is used and will demo protection
operations on a live MongoDB cluster. This webinar is targeted towards developers and ops
teams interested in stateful applications in Kubernetes. kanister.io.

Introductions

The Challenge: Layers of Operation
• Day 2 services have become a here and now problem
• Data Management functions such as backup and disaster recovery along with
application mobility.
• Cloud-Native applications, microservices use multiple data services (MongoDB,
Redis, Kafka, etc.)
• Storage technologies to store state and are typically deployed in multiple
locations (regions, clouds, on-premises).

Physical Storage

File, Block & Object

Data Services

Stateful Application

The Challenge: Flavours of Data Management
• What level of protection is enough?
• Backup & Restore
• Application Mobility
• Disaster Recovery
• Compliance Requirements
• Freedom of choice

Storage-Centric Snapshots
• Physical Storage Layer exercised
• Crash-Consistent
• Dependant on error handling capability
Fastest option for Backup & Recovery *
*BUT… Same storage system as
production / transaction-level granularity

Storage-Centric with Data Service Hooks
Same as before but now with hooks into
your Data services
1. Freeze & Flush the Data Services layer
2. Initiate a Storage-layer snapshot
3. Unfreeze the Data Services layer
4. Record completion and status of the
Snapshot process
Fast option for Backup & Recovery *
*BUT… Same storage system as production

Data Service Centric
• Storage Efficient – database aware and
db specific compression
• No Dependency on underlying storage
• Recovery can be complicated
Good option for backup and consistency,
recovery complex but we are now getting
data out of band.

Application-Centric
• Focus is on business continuity of an
application
• Backup can be easy but restore also needs
to be easy.
• Higher level of consistency and flexibility
• Freedom of choice

Data Management Summary
• Storage-centric snapshots provided by the underlying file or block
storage
• Storage-centric with data service hooks that spans across storage and
data services layers,
• Data service-centric approaches that uses database specific utilities, and
finally
• Application-centric that exercises all the above capabilities in a
coordinated manner.

Need for Application-Consistent Data Management
• Shortcomings of Crash-Consistent Storage Snapshots
• Logical Backups
• mysqldump, pg_dump, mongodump etc.
• Provider Specific API Calls
• RDS Snapshot APIs, Operator CRs
• Application Quiesce/Unquiesce
• e.g. FLUSH TABLES WITH READ LOCK
• Advanced Scenarios
• Backup MongoDB Secondaries

Data Protection Workflows are Complex
• One application includes many domains
• Cluster Admins != App developers != Database Administrators
• Difficult to separate concerns
• Many moving parts
• Targets: Object Storage, Vendor Targets
• Types of Backups: logical dumps, volume snapshots
• Application lifecycle: Scale down/up when restoring

KANISTER allows domain experts to capture application specific data
management tasks in blueprints which can be easily shared and extended.

Kanister Framework for App-level Data Management
• Kanister Controller
• Operator responsible for Kubernetes Custom Resources and state
management
• Blueprints
• Define workflows for backup, restore and delete operations
• ActionSets
• Run an action to backup, restore and delete
• Profiles
• Define target destination for backups or sources for restores

Kanister CLI Tools
• kanctl
• CLI to create Kanister Profile CRs and ActionSets
• kando
• CLI used within containers to push and pull backup data to and from an
object store location

Example Blueprint
apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
name: mongodb-blueprint
actions:
backup:
# Store backup information
outputArtifacts:
cloudObject:
keyValue:
path: '/mongodb-replicaset-backups/{{ .StatefulSet.Name }}/rs_backup.gz'
# Use Kanister functions to perform operations
phases:
- func: KubeTask
args:
namespace: ‘{{ .StatefulSet.Namespace }}’
image: kanisterio/mongodb:0.68.0
command:
- bash
- -c
- mongodump --oplog --gzip --archive --host ${host} -u root -p “${dbPassword}”

Example ActionSet
kind: ActionSet
metadata:
generateName: s3backup-
namespace: kanister-controller
spec:
# Select an action from a Blueprint
actions:
- name: backup
blueprint: mongodb-blueprint
# Select a Kubernetes resource to run the action on
object:
kind: StatefulSet
name: mongodb-replicaset
namespace: mongodb
# Select a Profile to use as the source/destination for the action
profile:
kind: profile
name: example-profile
# Status set by the Kanister controller
status:
actions:
- artifacts:
cloudObject:
keyValue:
path: '/mongodb-replicaset-backups/mongodb-replicaset/rs_backup.gz'

Example Profile
kind: Profile
metadata:
generateName: s3profile-
# Object Store location
location:
type: s3Compliant
bucket: kanister-backup
# Credentials for the Object Store location
credential:
type: keyPair
keyPair:
idField: example_key_id
secretField: example_secret_access_key
secret:
apiVersion: v1
kind: Secret
name: example-secret
namespace: example-namespace

Execution Walkthrough
Controller
Blueprint
Database
Workload

ActionSet
Controller
Blueprint
Database
Workload

ActionSet
Controller
Blueprint
Kanister
Function
Database
Workload

ActionSet
Controller
Blueprint
Kanister
Function
Database
Workload
Object Storage/
Cloud Snapshot

ActionSet
Controller
Blueprint
Database
Workload
Object Storage/
Cloud Snapshot

Kanister Functions
Custom Logic
• KubeExec
• KubeExecAll
• KubeTask
Resource Lifecycle
• Scale up/down workload
• KubeTask with kubectl command
Handle PVC
• Backup/Restore/DeleteData
• PrepareData
Volume Snapshots
• Create/Restore/Delete snapshots
Amazon RDS
• Create/Restore/Delete snapshots
• ExportSnapshotToRegion

Providers Supported
Object Storage
• AWS S3
• S3 Compliant
• Azure Blob
• Google Cloud Storage
Block/File Storage (in-tree)
• AWS EBS/EFS
• Azure Disk
• Google Persistent Disk
• IBM Disk
• CSI

Roadmap / New Features
• File Storage destinations for backups
• Encryption, deduplication and compression support with kando
• Kanister functions to manage data in Data Service Operators like K8ssandra

Next Steps

Closing
Please look at the project
Feedback & Contributions
Spread the word
An extensible open-source framework for
application-level data management on
Kubernetes

Cncf kanister.pptx

More Related Content

Cncf kanister.pptx