Exploring the power of OpenTelemetry on Kubernetes
•
3 likes•2,135 views
Modern cloud-native applications are incredibly complex systems. Keeping the systems healthy and meeting SLAs for our customers is crucial for long-term success. In this session, we will dive into the three pillars of observability - metrics, logs, tracing - the foundation of successful troubleshooting in distributed systems. You'll learn the gotchas and pitfalls of rolling out the OpenTelemetry stack on Kubernetes to effectively collect all your signals without worrying about a vendor lock in. Additionally we will replace parts of the Prometheus stack to scrape metrics with OpenTelemetry collector and operator.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Exploring the power of OpenTelemetry on Kubernetes
Similar to Exploring the power of OpenTelemetry on Kubernetes (20)
More from Red Hat Developers
More from Red Hat Developers (20)
Recently uploaded
Recently uploaded (20)
Exploring the power of OpenTelemetry on Kubernetes
- 1. 1 Exploring the power of OpenTelemetry on Kubernetes Pavol Loffay Principal Software Engineer Benedikt Bongartz Senior Software Engineer
- 2. 2 ● Pavol Loffay ● Principal Software Engineer ● OpenTelemetry, Jaeger maintainer ● Follow me @ploffay Bio
- 3. 3 ● Benedikt Bongartz ● Senior Software Engineer ● Working on OpenTelemetry ● Contact ○ @frzifus@fosstodon.org ○ frzifus:matrix.org Bio
- 4. 4 ● History observability ● Intro to OpenTelemetry (OTEL) ○ Instrumentation ○ Collector ● OpenTelemetry Kubernetes operator ○ Instrumentation CR ○ Collector CR ● Data collection use-cases - Traces, Metrics, Logs ● What is next? Agenda
- 5. A peek into history of OSS observability technology advancement and less vendor locking
- 6. 6 OSS distributed tracing history X-Trace 2007 Dapper paper 2010 Zipkin 2012 OpenTracing 2015 Jaeger 2017 OpenTelemetry DataDog 2019 New Relic 2020 OpenCensus 2016 Hypetrace Grafana Tempo 2020 SigNoz 2021 Apache SkyWalking 2015 Data model bespoke APIs SDK Collector auto-instrumentation Spec API Instrumentation libraries some instrumentation reusable instrumentation libraries Agents/ auto-instrumentation
- 7. OpenTelemetry a.k.a. OTEL Open-source data collection project
- 8. 8 ● Open source! ● Cloud Native Computing Foundation (CNCF) ● Vendor neutral telemetry data collection ● Specification, API, SDK, data model - OTLP, auto-instrumentation, collector ● Helm chart, Kubernetes operator
- 9. 9 OpenTelemetry is only data collection It is not platform, storage nor query API
- 10. OpenTelemetry instrumentation How telemetry data is created
- 12. ● Manual / Explicit ○ Better control ○ Might have better performance ○ Requires code change, recompilation, redeploy ● Direct integration in runtimes - e.g. Quarkus, Wildfly ● Auto-instrumentation / agent ○ Easy to roll out ○ Wide framework coverage 12 Instrumentation java -javaagent:/opt/javaagent.jar -jar target/demo-0.0.1-SNAPSHOT.jar tracing.enabled = true io.opentelemetry:opentelemetry-api:1.22.0 io.opentelemetry.instrumentation:opentelemetry-grpc-1.6:1.19.1
- 13. ● Java ● C# ● NodeJS ● Python ● PHP ● Golang 13 OpenTelemetry auto-instrumentation java -javaagent:/opt/javaagent.jar -jar target/demo-0.0.1-SNAPSHOT.jar JAVA_TOOL_OPTIONS = -javaagent:/otel-auto-instrumentation/javaagent.jar CORECLR_ENABLE_PROFILING, CORECLR_PROFILER_PATH, DOTNET_ADDITIONAL_DEPS, DOTNET_SHARED_STORE NODE_OPTIONS = --require /otel-auto-instrumentation/autoinstrumentation.js PYTHONPATH = /otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation//otel-auto-instrumentation
- 14. OpenTelemetry collector How telemetry data is collected and exported to a platform
- 15. ● Written in Golang ● Docs - opentelemetry.io/docs/collector/ ● Two distributions ○ Core - opentelemetry-collector-releases/distributions/otelcol ○ Contrib - opentelemetry-collector-releases/distributions/otelcol-contrib ○ Build your own ● Many components open-telemetry/opentelemetry-collector-contrib ● Distributed as: binary, container image, APK/DEB/RPM 15 OpenTelemetry collector
- 16. 16 Instrumentation and collector app/workload/pod (auto) Instrumentation API/SDK RPC framework (e.g. servlet) Jaeger OpenTelemetry Zipkin OpenCensus Batching PII filter Re-labeling Custom Splunk Kafka Jaeger Batching Data drop Collector Receivers Processors Exporters
- 17. OpenTelemetry Kubernetes Operator Manages OpenTelemetry collector and Instrumentation
- 18. ● https://github.com/open-telemetry/opentelemetry-operator ● Kubernetes operator Custom Resource Definitions (CRDs): ○ opentelemetrycollectors.opentelemetry.io / otelcol ○ instrumentation.opentelemetry.io / otelinst ● The operator can be installed via OLM, Helm chart or K8s manifests 18 OpenTelemetry operator
- 19. Collector CRD
- 20. ● Deployment modes: .spec.mode ○ sidecar, deployment, daemonset, statefulset ● Auto scaling or kubectl scale --replicas=5 otelcols/simplest ● Expose outside of the cluster .spec.ingress ● Use custom collector image .spec.image 20 OpenTelemetry Operator - Collector Kind
- 21. 21 OTEL-Collector configuration apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel spec: mode: deployment config: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:14250 exporters: logging: loglevel: debug service: extensions: [headers_setter, ...] pipelines: traces: processors: [batch,...] receivers: [otlp] exporters: [logging] # Data src: traces, metrics, logs processors: # <optional> batch: send_batch_size: 10000 timeout: 10s resourcedetection/openshift: detectors: openshift timeout: 2s override: false extensions: # <optional> headers_setter: - key: X-Scope-OrgID from_context: tenant_id - key: User-ID value: user_id sidecar.opentelemetry.io/inject: "true"
- 22. 22 OpenTelemetry Operator - Collector Kind kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel spec: config: | receivers: otlp: protocols: grpc: http: jaeger: protocols: grpc: thrift_compact: processors: exporters: jaeger: endpoint: jaeger-collector.headless:14250 service: pipelines: traces: receivers: [otlp, jaeger] processors: [] exporters: [jaeger] EOF sidecar.opentelemetry.io/inject: "true"
- 23. 23 OpenTelemetry Operator - Collector Kind - Processor processors: k8sattributes: # Add k8s attributes attributes: # Delete db.table attribute actions: - key: db.table action: delete resource: # Add k8s cluster name attributes: - key: k8s.cluster.name from_attribute: k8s-cluster action: insert Other use cases: ● Tail base sampling ● Derive RED metrics from traces
- 25. 25 OpenTelemetry Operator - Instrumentation Kind apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my-instrumentation spec: exporter: endpoint: http://otel-collector:4317 propagators: - tracecontext - baggage - b3 sampler: type: parentbased_traceidratio argument: "0.25" resource: attributes: k8s.cluster.name: test-1-23 instrumentation.opentelemetry.io/inject-java: "true" instrumentation.opentelemetry.io/inject-python: "true" instrumentation.opentelemetry.io/inject-nodejs: "true" instrumentation.opentelemetry.io/inject-dotnet: "true" instrumentation.opentelemetry.io/inject-sdk: "true" instrumentation.opentelemetry.io/container-names: "app"
- 27. 27 OpenTelemetry Architecture - Tracing Pod Container 1 OTEL-Client OpenTelemetry API Instrumentation Application push Platform (Jaeger/Loki/Splunk…) Platform instrumentation.opentelemetry.io/inject-java: "true" apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my–java-instrumentation spec: exporter: . . .
- 28. 28
- 29. 29
- 30. 30
- 31. 31
- 33. 33 Collector Platform (Prometheus/Thanos/Splunk…) Platform Pod 2 Node 1 OTEL-Collector configuration Pod 1 Pull Push
- 34. 34 OTEL-Collector configuration apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel spec: targetAllocator: enabled: true allocationStrategy: least-weighted replicas: 3 serviceAccount: ta prometheusCR: enabled: true serviceMonitorSelector: team: backend-1 config: | receivers: prometheus: config: scrape_configs: - job_name: ‘otel-collector’ scrape_interval: 10s static_configs: ...
- 35. 35 OTEL-Collector configuration apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel spec: mode: statefulset targetAllocator: enabled: true allocationStrategy: least-weighted replicas: 3 serviceAccount: ta prometheusCR: {} config: | receivers: prometheus: config: scrape_configs: - job_name: ‘otel-collector’ scrape_interval: 10s static_configs: - targets: [`0.0.0.0:8888`] target_allocator: endpoint: http://my-ta-svc interval: 30s collector_id: collector-1
- 36. 36 OTEL-Collector configuration … receivers: prometheus: config: global: scrape_interval: 1m scrape_timeout: 10s evaluation_interval: 1m scrape_configs: - job_name: otel-collector honor_timestamps: true scrape_interval: 10s scrape_timeout: 10s metrics_path: /metrics scheme: http follow_redirects: true http_sd_configs: - follow_redirects: false url: http://metrics-ta-svc:80/jobs/otel-collector/targets?collector_id=$POD_NAME spec.targetAllocator.enabeld: "true"
- 37. 37 OTEL-Collector configuration Be aware of: ● Prometheus does not support UTF-8 labels [prometheus#11700] [grafana#42615] Therefore OTEL-Prometheus-Remote-Write converts metrics to Prom Naming convention. [otel#normalization] ● Other useful metric receiver [host-metrics]
- 39. 39 ● For metrics and traces OpenTelemetry takes the approach of a clean-sheet design, specifies a new API and provides full implementations of this API in multiple languages. ● Our approach with logs is somewhat different. For OpenTelemetry to be successful in logging space we need to support existing legacy of logs and logging libraries, while offering improvements and better integration with the rest of observability world where possible. OpenTelemetry logs
- 40. 40 OpenTelemetry Operator - collecting logs ● File log receiver ○ Available in the “contrib” docker image ○ Example config - opentelemetry-collector-contrib/otel-collector-config.yml ○ Includes: /var/log/pods/*/*/*.log ○ Set of operators to parse data: json_parser, regex_parser, move… ○ Collectors resource attributes: namespace, pod name/UID/restarts, container name ● Fluentforwardreceiver - receive data from
- 41. 41 DaemonSet collector Platform (Loki/OpenSearch/Splunk…) Platform Pod 1 Node 1 Pod 2 Pod 3 DaemonSet collector Pod 1 Node 2 Pod 2 Pod 3 DaemonSet collector Pod 1 Node 3 Pod 2 Pod 3 Kubernetes cluster
- 42. 42 OpenTelemetry Operator - collecting logs, CR apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel-logs spec: mode: daemonset image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0. 69.0 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers volumeMounts: - mountPath: /var/log name: varlog readOnly: true - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true
- 43. 43 OpenTelemetry Operator - collecting logs, logging exporter 1 2023-01-19T16:28:48.675Z info ResourceLog #0 Resource SchemaURL: Resource attributes: -> k8s.namespace.name: Str(cert-manager) -> k8s.pod.name: Str(cert-manager-cainjector-857ff8f7cb-xmr6p) -> k8s.container.restart_count: Str(0) -> k8s.pod.uid: Str(53574bad-1d91-4880-ab87-8f5a88bfffee) -> k8s.container.name: Str(cert-manager) ScopeLogs #0 ScopeLogs SchemaURL: InstrumentationScope LogRecord #0 ObservedTimestamp: 2023-01-19 16:28:48.674124625 +0000 UTC Timestamp: 2023-01-18 12:58:21.446092114 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"log":"I0118 12:58:21.445823 1 controller.go:178] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcede finition "msg"="Starting EventSource" "source"="u0026{{%!s(*v1.Certificate=u0026{{ } { 0 {{0 0 u 003cnilu003e}} u003cnilu003e u003cnilu003e map[] map[] [] [] []} {u003cnilu003e u003cnilu003e u003cnilu003e [] [] [] [] u003cnilu003e u003cnilu003e { } false [] u003cnilu003e u003cnilu003e u003cnilu003e []} {[] u003cnilu003e u003cnilu003e u003cnilu00 3e u003cnilu003e u003cnilu003e u003cnilu003e u003cnilu003e}}) %!s(*cache.informerCache=u0026{0xc000438380}) %!s(chan error=u003cnilu003e) %!s(func()=u003cnilu003e)}}"n","stream":"stderr","time":"2023-01-18T12:58:21.446092114Z"})
- 44. 44 OpenTelemetry Operator - collecting logs, logging exporter 2 Attributes: -> log.file.path: Str(/var/log/pods/cert-manager_cert-manager-cainjector-857ff8f7cb-xmr6p_53574bad-1d91-4880-ab 87-8f5a88bfffee/cert-manager/0.log) -> log: Str(I0118 12:58:21.445823 1 controller.go:178] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-custo mresourcedefinition "msg"="Starting EventSource" "source"="&{{%!s(*v1.Certificate=&{{ } { 0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} {<nil> <nil> <nil> [] [] [] [] <nil> <nil> { } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0xc000438380}) %!s(chan error=<nil>) %!s(func()=<nil>)}}" ) -> time: Str(2023-01-18T12:58:21.446092114Z) -> log.iostream: Str(stderr) Trace ID: Span ID: Flags: 0
- 45. What is next?
- 46. ● OpenTelemetry Operator ○ Auto-instrumentation for Webservers ● Profiling vision see 0212-profiling-vision.md ● OpAMP: Open Agent Management Protocol see open-telemetry/opamp-spec 46 What is next in OpenTelemetry?
- 47. Thank you 47 Pavol Loffay @ploffay Benedikt Bongartz @frzifus