Expensive Observability: The Cardinality Challenge - OpenObservability Talks S3E02

We all collect logs, metrics and perhaps traces and other data types, in support of our observability. But this can get expensive pretty quickly, especially in microservices based systems, in what is commonly known as “the cardinality problem”.  

On this episode of OpenObservability Talks I’ll host Ben Sigelman, co-founder and the GM of Lightstep, to discuss this data problem and how to overcome it. Ben architected Google’s own planet-scale metrics and distributed tracing systems (still in production today), and went on to co-create the open-source OpenTracing and OpenTelemetry projects, both part of the CNCF.

The episode was live-streamed on 12 July 2022 and the video is available at https://youtu.be/gJhzwP-mZ2k

OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube.

We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and pitch in with your comments and questions on the live chat.
https://www.twitch.tv/openobservability
https://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg

Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/

Show Notes:

The difference between monitoring, observability and APM
What comprises the cost of observability
How common is the knowledge of cardinality and how to add metrics
Controlling cost with sampling, verbosity and retention
Lessons from Google’s metrics and tracing systems
Using metric rollups and aggregations intelligently
Semantic conventions for logs, metrics and traces
OpenCost project
New research paper by Meta on schema-first approach to application telemetry metadata
OTEL code contributions - published stats

Resources:

Monitoring vs. observability: https://twitter.com/el_bhs/status/1349406398388400128
The two drivers of cardinality: https://twitter.com/el_bhs/status/1360276734344450050
Sampling vs verbosity: https://twitter.com/el_bhs/status/1440750741384089608
Observing resources and transactions: https://twitter.com/el_bhs/status/1372636288021524482

Socials:

Twitter: https://twitter.com/OpenObserv
Twitch: https://www.twitch.tv/openobservability
YouTube: https://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg