🤔 Observability Ports: Why They Matter and Who's Accountable? 👉 Observability ports allow controlled access to the internal state of a data product while preserving the information hiding and encapsulation principles on which its design is based. 🤓 The information about the internal state exposed by a data product is generally called observations or signals. 👨⚖️ 👩⚖️ The governance team defines the type and semantics of the signals that each data product must produce. It also defines the APIs of the observability ports through which the generated signals are exposed to external observers. 👷♂️ 👷♀️ The platform team defines the format of the managed signals and the protocol for serializing and sharing them. Based on this protocol, it also provides development teams with a sidecar that collects the signals generated by the data product and makes them available through observability ports that conform to the APIs defined by the governance team. 👨🔧 👩🔧 Finally, the product team instruments the internal product components with appropriate sensors capable of generating the signals required by governance policies and communicates them to the sidecar provided by the platform team to expose them externally. 🤚 Multiple external tools can therefore use the information exposed by the observability ports to manage the operation of a data product instantiated in a runtime environment. To facilitate interoperability with existing external tools, it is recommended to adopt the Open Telemetry Protocol (OTLP) as the signal management protocol. #TheDataJoy #dataproducts #dataobservability
Great summary. Indeed, a data product is not just publishing the data. It should also connect correctly to all observability solutions to allow monitoring.
Governance team probably needs to be much more “product-like” than it usually is
Hanaa Ali
1w
Can you elaborate on the observability port consumers. I see the importance of identifying the consumers but how would you integrate them without creating duplication of data. Also if this is automated, is there a need for a governance team? - Thanks
Imran H.
2w
Hassan Abbas Abul Qasim
Partner and CTO at Quantyca & Co-founder at Blindata
2wSome related resources: 📌 Signals: Traces, Metrics & Logs https://opentelemetry.io/docs/concepts/signals/ 📌 OpenTelemetry Protocol https://opentelemetry.io/docs/specs/otel/protocol/ 📌 Semantic Conventions https://opentelemetry.io/docs/concepts/semantic-conventions/