SlideShare a Scribd company logo
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
The “Zen” {
Paige Cruz, Chronosphere
[of Python Exemplars]
}
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
01 Exemplars Explained
02 Example Exemplars
03 Exemplar Enlightenment
Table Of ‘Contents’ {
}
talk.json
00 The Zen of Python
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
00 {
[The Zen of Python]
}
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
hello.py
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
The Zen of Python (aka PEP 20)
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
…
There should be one-- and preferably only
one --obvious way to do it.
…
The Zen of Python (aka PEP 20)
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
��
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
01 {
[Exemplars Explained]
}
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
02 {
Example Exemplars:
OpenTelemetry
}
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
— OpenTelemetry Metrics Data Model Spec
“An exemplar is a recorded value that
associates OpenTelemetry context to a
metric event within a Metric”
definition.txt
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Value
Filtered Attributes
< Unix Epoch nanoseconds >
Observation Timestamp
OpenTelemetry ‘Exemplars’{
}
Trace
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
OpenTelemetry Metrics {
}
Metric Point
Sum ✅
Gauge ✅
Histogram /
Exponential
Histogram
✅
Exemplars?
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
02 {
Example Exemplars:
Prometheus
}
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
— OpenMetrics Specification
< “Exemplars are references to data
outside of the MetricSet. A common use
case are IDs of program traces.” >
definition.txt
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
< Integer or 64-bit float >
< traceID, spanID <=128 UTF-8 chars >
Value
LabelSet
< Unix Epoch seconds >
Timestamp
OpenMetrics ‘Exemplars’{
}
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Prometheus Metrics {
}
Metric Type
Counter ✅
Histogram ✅
Gauge ❌
Summary ❌
Exemplars?
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
traces
metrics
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
def do_something():
with tracer.start_as_current_span("do_something") as span:
data = call_api()
filtered = parse_data(data)
trace_id = '{:032x}'.format(span.get_span_context().trace_id)
TASK_COUNTER.inc(1, exemplar={"trace_id": trace_id, "trace_url":
f"http://localhost:16686/trace/{trace_id}"})
return filtered
script.py
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
FROM prom/prometheus:v2.48.0
ADD prometheus.yml /etc/prometheus
ENTRYPOINT [ "prometheus" ]
CMD [ "--config.file=/etc/prometheus/prometheus.yml",
"--enable-feature=exemplar-storage" ]
Buildfile-prom
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
02 {
[Example Exemplars]
Prometheus < > OTel
}
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
musings.md
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
03 {
Exemplar Enlightenment
}
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Exemplars {
}
01 Traces are discoverable in typical workflows
02 Jumpstart good/bad trace comparisons
03 Bridge for (potentially) siloed telemetry
the_good.txt
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
OpenMetrics Spec
OTel Compatibility with Prom & OpenMetrics
@paigerduty
1
2
3
4
5
6
7
8
9
10
11
12
13
14
CREDITS: This presentation template was
created by Slidesgo, including icons by
Flaticon, and infographics & images by Freepik
Thanks <3 {
paigerduty
@hachyderm.io
@chronopshere.io
www.paigerduty.com
Get in touch
}

More Related Content

More from Paige Cruz

There's No Place Like Production
There's No Place Like ProductionThere's No Place Like Production
There's No Place Like Production
Paige Cruz
 
Taming Feral DevOps
Taming Feral DevOps Taming Feral DevOps
Taming Feral DevOps
Paige Cruz
 
SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power
SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of PowerSRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power
SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power
Paige Cruz
 
Pushing Observability Uphill - The Single “Pain” of Glass
Pushing Observability Uphill - The Single “Pain” of GlassPushing Observability Uphill - The Single “Pain” of Glass
Pushing Observability Uphill - The Single “Pain” of Glass
Paige Cruz
 
Power Up with Podman
Power Up with PodmanPower Up with Podman
Power Up with Podman
Paige Cruz
 
Intro to Instrumentation
Intro to InstrumentationIntro to Instrumentation
Intro to Instrumentation
Paige Cruz
 
From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
From Cardinal(ity) Sins to Cost-Efficient Metrics AggregationFrom Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
Paige Cruz
 
99.9% of Your Traces are Trash
99.9% of Your Traces are Trash99.9% of Your Traces are Trash
99.9% of Your Traces are Trash
Paige Cruz
 
3rd Wave Observability: Open or Bust
3rd Wave Observability: Open or Bust 3rd Wave Observability: Open or Bust
3rd Wave Observability: Open or Bust
Paige Cruz
 

More from Paige Cruz (9)

There's No Place Like Production
There's No Place Like ProductionThere's No Place Like Production
There's No Place Like Production
 
Taming Feral DevOps
Taming Feral DevOps Taming Feral DevOps
Taming Feral DevOps
 
SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power
SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of PowerSRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power
SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power
 
Pushing Observability Uphill - The Single “Pain” of Glass
Pushing Observability Uphill - The Single “Pain” of GlassPushing Observability Uphill - The Single “Pain” of Glass
Pushing Observability Uphill - The Single “Pain” of Glass
 
Power Up with Podman
Power Up with PodmanPower Up with Podman
Power Up with Podman
 
Intro to Instrumentation
Intro to InstrumentationIntro to Instrumentation
Intro to Instrumentation
 
From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
From Cardinal(ity) Sins to Cost-Efficient Metrics AggregationFrom Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
 
99.9% of Your Traces are Trash
99.9% of Your Traces are Trash99.9% of Your Traces are Trash
99.9% of Your Traces are Trash
 
3rd Wave Observability: Open or Bust
3rd Wave Observability: Open or Bust 3rd Wave Observability: Open or Bust
3rd Wave Observability: Open or Bust
 

Recently uploaded

Dev Dives: Mining your data with AI-powered Continuous Discovery
Dev Dives: Mining your data with AI-powered Continuous DiscoveryDev Dives: Mining your data with AI-powered Continuous Discovery
Dev Dives: Mining your data with AI-powered Continuous Discovery
UiPathCommunity
 
Call Girls Firozabad ☎️ +91-7426014248 😍 Firozabad Call Girl Beauty Girls Fir...
Call Girls Firozabad ☎️ +91-7426014248 😍 Firozabad Call Girl Beauty Girls Fir...Call Girls Firozabad ☎️ +91-7426014248 😍 Firozabad Call Girl Beauty Girls Fir...
Call Girls Firozabad ☎️ +91-7426014248 😍 Firozabad Call Girl Beauty Girls Fir...
jiaulalam7655
 
Chapter 6 - Test Tools Considerations V4.0
Chapter 6 - Test Tools Considerations V4.0Chapter 6 - Test Tools Considerations V4.0
Chapter 6 - Test Tools Considerations V4.0
Neeraj Kumar Singh
 
Database Management Myths for Developers
Database Management Myths for DevelopersDatabase Management Myths for Developers
Database Management Myths for Developers
John Sterrett
 
Chapter 4 - Test Analysis & Design Techniques V4.0
Chapter 4 - Test Analysis & Design Techniques V4.0Chapter 4 - Test Analysis & Design Techniques V4.0
Chapter 4 - Test Analysis & Design Techniques V4.0
Neeraj Kumar Singh
 
Metadata Lakes for Next-Gen AI/ML - Datastrato
Metadata Lakes for Next-Gen AI/ML - DatastratoMetadata Lakes for Next-Gen AI/ML - Datastrato
Metadata Lakes for Next-Gen AI/ML - Datastrato
Zilliz
 
CNSCon 2024 Lightning Talk: Don’t Make Me Impersonate My Identity
CNSCon 2024 Lightning Talk: Don’t Make Me Impersonate My IdentityCNSCon 2024 Lightning Talk: Don’t Make Me Impersonate My Identity
CNSCon 2024 Lightning Talk: Don’t Make Me Impersonate My Identity
Cynthia Thomas
 
Brightwell ILC Futures workshop David Sinclair presentation
Brightwell ILC Futures workshop David Sinclair presentationBrightwell ILC Futures workshop David Sinclair presentation
Brightwell ILC Futures workshop David Sinclair presentation
ILC- UK
 
Chapter 1 - Fundamentals of Testing V4.0
Chapter 1 - Fundamentals of Testing V4.0Chapter 1 - Fundamentals of Testing V4.0
Chapter 1 - Fundamentals of Testing V4.0
Neeraj Kumar Singh
 
Move Auth, Policy, and Resilience to the Platform
Move Auth, Policy, and Resilience to the PlatformMove Auth, Policy, and Resilience to the Platform
Move Auth, Policy, and Resilience to the Platform
Christian Posta
 
Chapter 5 - Managing Test Activities V4.0
Chapter 5 - Managing Test Activities V4.0Chapter 5 - Managing Test Activities V4.0
Chapter 5 - Managing Test Activities V4.0
Neeraj Kumar Singh
 
An Introduction to All Data Enterprise Integration
An Introduction to All Data Enterprise IntegrationAn Introduction to All Data Enterprise Integration
An Introduction to All Data Enterprise Integration
Safe Software
 
Chapter 3 - Static Testing (Review) V4.0
Chapter 3 - Static Testing (Review) V4.0Chapter 3 - Static Testing (Review) V4.0
Chapter 3 - Static Testing (Review) V4.0
Neeraj Kumar Singh
 
Call Girls Chennai ☎️ +91-7426014248 😍 Chennai Call Girl Beauty Girls Chennai...
Call Girls Chennai ☎️ +91-7426014248 😍 Chennai Call Girl Beauty Girls Chennai...Call Girls Chennai ☎️ +91-7426014248 😍 Chennai Call Girl Beauty Girls Chennai...
Call Girls Chennai ☎️ +91-7426014248 😍 Chennai Call Girl Beauty Girls Chennai...
anilsa9823
 
What is an RPA CoE? Session 4 – CoE Scaling
What is an RPA CoE? Session 4 – CoE ScalingWhat is an RPA CoE? Session 4 – CoE Scaling
What is an RPA CoE? Session 4 – CoE Scaling
DianaGray10
 
Introduction to ThousandEyes AMER Webinar
Introduction  to ThousandEyes AMER WebinarIntroduction  to ThousandEyes AMER Webinar
Introduction to ThousandEyes AMER Webinar
ThousandEyes
 
this resume for sadika shaikh bca student
this resume for sadika shaikh bca studentthis resume for sadika shaikh bca student
this resume for sadika shaikh bca student
SadikaShaikh7
 
Lessons Of Binary Analysis - Christien Rioux
Lessons Of Binary Analysis - Christien RiouxLessons Of Binary Analysis - Christien Rioux
Lessons Of Binary Analysis - Christien Rioux
crioux1
 
Cyber Recovery Wargame
Cyber Recovery WargameCyber Recovery Wargame
Cyber Recovery Wargame
Databarracks
 
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
dipikamodels1
 

Recently uploaded (20)

Dev Dives: Mining your data with AI-powered Continuous Discovery
Dev Dives: Mining your data with AI-powered Continuous DiscoveryDev Dives: Mining your data with AI-powered Continuous Discovery
Dev Dives: Mining your data with AI-powered Continuous Discovery
 
Call Girls Firozabad ☎️ +91-7426014248 😍 Firozabad Call Girl Beauty Girls Fir...
Call Girls Firozabad ☎️ +91-7426014248 😍 Firozabad Call Girl Beauty Girls Fir...Call Girls Firozabad ☎️ +91-7426014248 😍 Firozabad Call Girl Beauty Girls Fir...
Call Girls Firozabad ☎️ +91-7426014248 😍 Firozabad Call Girl Beauty Girls Fir...
 
Chapter 6 - Test Tools Considerations V4.0
Chapter 6 - Test Tools Considerations V4.0Chapter 6 - Test Tools Considerations V4.0
Chapter 6 - Test Tools Considerations V4.0
 
Database Management Myths for Developers
Database Management Myths for DevelopersDatabase Management Myths for Developers
Database Management Myths for Developers
 
Chapter 4 - Test Analysis & Design Techniques V4.0
Chapter 4 - Test Analysis & Design Techniques V4.0Chapter 4 - Test Analysis & Design Techniques V4.0
Chapter 4 - Test Analysis & Design Techniques V4.0
 
Metadata Lakes for Next-Gen AI/ML - Datastrato
Metadata Lakes for Next-Gen AI/ML - DatastratoMetadata Lakes for Next-Gen AI/ML - Datastrato
Metadata Lakes for Next-Gen AI/ML - Datastrato
 
CNSCon 2024 Lightning Talk: Don’t Make Me Impersonate My Identity
CNSCon 2024 Lightning Talk: Don’t Make Me Impersonate My IdentityCNSCon 2024 Lightning Talk: Don’t Make Me Impersonate My Identity
CNSCon 2024 Lightning Talk: Don’t Make Me Impersonate My Identity
 
Brightwell ILC Futures workshop David Sinclair presentation
Brightwell ILC Futures workshop David Sinclair presentationBrightwell ILC Futures workshop David Sinclair presentation
Brightwell ILC Futures workshop David Sinclair presentation
 
Chapter 1 - Fundamentals of Testing V4.0
Chapter 1 - Fundamentals of Testing V4.0Chapter 1 - Fundamentals of Testing V4.0
Chapter 1 - Fundamentals of Testing V4.0
 
Move Auth, Policy, and Resilience to the Platform
Move Auth, Policy, and Resilience to the PlatformMove Auth, Policy, and Resilience to the Platform
Move Auth, Policy, and Resilience to the Platform
 
Chapter 5 - Managing Test Activities V4.0
Chapter 5 - Managing Test Activities V4.0Chapter 5 - Managing Test Activities V4.0
Chapter 5 - Managing Test Activities V4.0
 
An Introduction to All Data Enterprise Integration
An Introduction to All Data Enterprise IntegrationAn Introduction to All Data Enterprise Integration
An Introduction to All Data Enterprise Integration
 
Chapter 3 - Static Testing (Review) V4.0
Chapter 3 - Static Testing (Review) V4.0Chapter 3 - Static Testing (Review) V4.0
Chapter 3 - Static Testing (Review) V4.0
 
Call Girls Chennai ☎️ +91-7426014248 😍 Chennai Call Girl Beauty Girls Chennai...
Call Girls Chennai ☎️ +91-7426014248 😍 Chennai Call Girl Beauty Girls Chennai...Call Girls Chennai ☎️ +91-7426014248 😍 Chennai Call Girl Beauty Girls Chennai...
Call Girls Chennai ☎️ +91-7426014248 😍 Chennai Call Girl Beauty Girls Chennai...
 
What is an RPA CoE? Session 4 – CoE Scaling
What is an RPA CoE? Session 4 – CoE ScalingWhat is an RPA CoE? Session 4 – CoE Scaling
What is an RPA CoE? Session 4 – CoE Scaling
 
Introduction to ThousandEyes AMER Webinar
Introduction  to ThousandEyes AMER WebinarIntroduction  to ThousandEyes AMER Webinar
Introduction to ThousandEyes AMER Webinar
 
this resume for sadika shaikh bca student
this resume for sadika shaikh bca studentthis resume for sadika shaikh bca student
this resume for sadika shaikh bca student
 
Lessons Of Binary Analysis - Christien Rioux
Lessons Of Binary Analysis - Christien RiouxLessons Of Binary Analysis - Christien Rioux
Lessons Of Binary Analysis - Christien Rioux
 
Cyber Recovery Wargame
Cyber Recovery WargameCyber Recovery Wargame
Cyber Recovery Wargame
 
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
Call Girls Kochi 💯Call Us 🔝 7426014248 🔝 Independent Kochi Escorts Service Av...
 

The "Zen" of Python Exemplars - OTel Community Day