CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf

Confidential / © Harness Inc. 2022
CNCF On-Demand Webinar:
Project Updates
with LitmusChaos

Agenda
● What’s New with LitmusChaos 2.0?
● iFood Use Case: Resilient Online Food Ordering services
● HaloDoc Use Case: Cloud-Native Chaos Engineering applied to Healthcare
● LitmusChaos Demo: Making a cloud-native e-Commerce application Resilient

What’s new with Litmuschaos
● Improve probe addition in workflows
● Improvements to tune workflow to customize your scenarios further
● Ability to upgrade a chaos delegate
● More secure RBAC with updates to APIs
● Added support for basic argo workflows
● Support for Kubernetes 1.22.0 and above

● Support for IPv6
● Ensuring better testing with added E2E test suit
● Ability to skip SSL verification
● Enabling better debugging with enhanced server logs
● Migration of project collection from litmus to auth database
● Internal code refactor of authentication server for improved security

● Enhancements in CMD probe
● Hardened litmus alpine image for all Litmuschaos tools
● E2E Pipeline to monitor all pipeline builds
● New experiment additions like AWS AZ experiment, Azure Disk Loss, etc

iFood: What’s it About?
● One of the most demanded Latin American online food ordering and food
delivery platform delivering more than 60 million orders each month.
● iFood was founded in 2011, with the aim to provide an online food delivery portal
that develops an innovative system so users can order deliveries on the internet
with no hassle.
● With over 80% of the market share geographically, iFood covers most cities and
regions in Brazil, especially in Brazil’s financial center Sao Paulo.

● Due to its growing popularity, iFood’s decision to break the existing
monolith architecture into several micro services brought scalability
but it also came with complexity and additional costs.
● Faults like database servers going out of business, messaging
brokers crashing, the entire region of a cloud provider going down
due to a power outages, and network bandwidth dropping sharply
without notice were some of the challenges.
● Tightening up reliability by continuously doing load tests and bare
minimum chaos experiments but the solutions lacked specific use
case driven functionality.
● Requirement to know which user performed what chaos
experiment to enable better RBAC control on production.
● Current chaos engineering solutions being used were not
automated and had limited number of experiments thus iFood
wanted to eliminate manual cost.
Challenges faced by iFood

79
M
inutes
T
h
e
a
v
e
r
a
g
e
d
o
w
n
t
i
m
e
f
o
r
a
n
o
u
t
a
g
e
w
a
s
7
9
m
i
n
u
t
e
s
$
8
4
,
6
5
0
T
h
e
c
o
s
t
o
f
d
o
w
n
t
i
m
e
a
v
e
r
a
g
e
d
a
t
$
8
4
,
6
5
0
U
S
p
e
r
h
o
u
r
Actual cost of downtime

How LitmusChaos helps iFood?
● Providing lots of Chaos Experiments suiting their requirements with a
declarative approach which helps them customize the Chaos Engine
further.
● Giving fine grained RBAC controls to authenticate users to
LitmusChaos and restrict which services a developer can target to
inject chaos.
● Ability to construct a workflow as a cron and even save it as a template
for later use to aid with easier automation and auto chaos after a
specified interval.

HaloDoc: What’s it About?
● One of the most popular all-around healthcare application in Indonesia.
● A rapidly-growing startup founded in 2016, whose mission is to simplify
and bring quality healthcare across Indonesia.
● Partnership with more than 4,000 pharmacies in over 100 cities to bring
medicine to people’s doorsteps.
● Provides premium appointment service that partners with more than 500
hospitals, allowing patients to book a doctor’s appointment inside our
application.

● A hybrid application infrastructure, including a managed
Kubernetes cloud with a cloud-native communication
framework utilising AWS managed services such as RDS,
Lambda and S3.
● Ensuring continuous system reliability, including all its
dependencies, in an Agile delivery model to prevent service
downtimes stemming from a host of failure causes.
● Chaos engineering was found suitable to supplement the
existing QA with comprehensive automated test suites and
periodic performance testing/analysis to make the platform
robust.
Service Reliability Challenges

How LitmusChaos helps HaloDoc?
Wide Range of Experiments
Litmus offers extensive fault types ranging
from k8s to public and private clouds and even
bare metal.
Observability Driven
Litmus integrates with Prometheus TSDB
for providing experiment metrics and
generates pragmatic logs, metrics and
events data for fully understanding the
impact of the chaos.
Kubernetes Native
HaloDoc uses k8s as the underlying
platform for a majority of the business
services. Hence a k8s-native chaos
engineering tool offers better integration.
Chaos Scenario Definition
Simple and declarative chaos scenarios
that can simulate complex real-life failure
scenarios for almost every use-case.
Security Focused
Litmus provides RBAC for teams and
admin control for limiting blast radius. It
also offers fine-grained control for the
execution of experiment resources.
Vibrant Community
Litmus has a flourishing community with
approachable maintainers for technical
oversight and feedback mitigation.

How LitmusChaos helps HaloDoc?
● Discovery of system weaknesses and unexpected system behaviour when subjugated to individual experiments.
● Simulation of real-life failure conditions with automated hypotheses validation using chaos scenarios and Litmus probes.
● Continuous validation in staging environment using event-triggered chaos injection with Litmus EventTrackerPolicies while using
GitOps for chaos scenarios.
● Conducting automated game days in production, while keeping track of the fault type and load conditions devised based on
usage and traffic patterns.
● Results are fed into a data lake for further analysis while reports from the ChaosCenter, especially the resilience score, is also
leveraged for high-level views.

Concepts, New Terms, and Definitions
●
●
●
Terms Slide

Feature Architecture, if applicable
●
●
●
Architecture Slide

More Information
● Link to spec
● Link to epic
● Link to public documentation
● If behind feature flag, name of feature flag
● New feature is available in release version:

Demonstration
Show us how to use the new
feature.

Topic
● In order to use this feature, what are the steps to onboard a user?
●
Onboarding Path Slide

Topic
● List common problems.
● List the ways to troubleshoot those problems.
●
●
Troubleshooting Slide

Topic
● List common best practices.
●
●
●
Best Practices Slide

Topic
●
●
●
Roadmap and Timeline Slide

Topic
●
●
●
Supplemental Resources Slide

Please write 2-4 questions to test our knowledge on your presentation.
● Question 1:
○ Correct answer:
○ Wrong answer:
○ Wrong answer:
○ Wrong answer:
● Question 2:
○ Correct answer:
○ Wrong answer:
○ Wrong answer:
○ Wrong answer:
(Optional) Questions Slide

● Question 3:
○ Correct answer:
○ Wrong answer:
○ Wrong answer:
○ Wrong answer:
● Question 4:
○ Correct answer:
○ Wrong answer:
○ Wrong answer:
○ Wrong answer:
(Optional) Questions Slide

/harnessio
/harnessinc
Contact us on:
www.harness.io
Follow us on

CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf

Recommended

Recommended

More Related Content

Similar to CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf

Similar to CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf (20)

More from LibbySchulze

More from LibbySchulze (20)

Recently uploaded

Recently uploaded (20)

CNCF On-Demand Webinar_ LitmusChaos Project Updates.pdf