Zack Kayser and Ethan Gunderson, Software Engineers at Cars Commerce, join the Elixir Wizards to share their expertise on telemetry and observability in large-scale systems. Drawing from their experience at Cars.com—a platform handling high traffic and concurrent users—they discuss the technical and organizational challenges of scaling applications, managing microservices, and implementing effective observability practices.The conversation highlights the pivotal role observability plays in diagnosing incidents, anticipating system behavior, and asking unplanned questions of a system. Zack and Ethan explore tracing, spans, and the unique challenges introduced by LiveView deployments and WebSocket connections.They also discuss the benefits of OpenTelemetry as a vendor-agnostic instrumentation tool, the significance of Elixir’s telemetry library, and practical steps for developers starting their observability journey. Additionally, Zack and Ethan introduce their upcoming book, Instrumenting Elixir Applications, which will offer guidance on integrating telemetry and tracing into Elixir projects.Topics Discussed:Cars.com’s transition to Elixir and scaling solutionsThe role of observability in large-scale systemsUncovering insights by asking unplanned system questionsManaging high-traffic and concurrent users with ElixirDiagnosing incidents and preventing recurrence using telemetryBalancing data collection with storage constraintsSampling strategies for large data volumesTracing and spans in observabilityLiveView’s influence on deployments and WebSocket behaviorMitigating downstream effects of socket reconnectionsContextual debugging for system behavior insightsObservability strategies for small vs. large-scale appsOpenTelemetry for vendor-agnostic instrumentationLeveraging OpenTelemetry contrib libraries for easy setupElixir’s telemetry library as an ecosystem cornerstoneTracing as the first step in observabilityDifferentiating observability from business analyticsProfiling with OpenTelemetry Erlang project toolsThe value of profiling for performance insightsMaking observability tools accessible and impactful for developersLinks Mentionedhttps://www.carscommerce.inc/https://www.cars.com/ https://hexdocs.pm/telemetry/readme.html https://kubernetes.io/ https://github.com/ninenines/cowboyhttps://hexdocs.pm/bandit/Bandit.html https://hexdocs.pm/broadway/Broadway.html https://hexdocs.pm/oban/Oban.html https://www.dynatrace.com/ https://www.jaegertracing.io/ https://newrelic.com/ https://www.datadoghq.com/ https://www.honeycomb.io/https://fly.io/phoenix-files/how-phoenix-liveview-form-auto-recovery-works/ https://www.elastic.co/ https://opentelemetry.io/ https://opentelemetry.io/docs/languages/erlang/ https://opentelemetry.io/docs/concepts/signals/traces/ https://opentelemetry.io/docs/specs/otel/logs/ https://github.com/runfinch/finchhttps://hexdocs.pm/telemetry_metrics/Telemetry.Metrics.html https://opentelemetry.io/blog/2024/state-profiling https://www.instrumentingelixir.com/ https://prometheus.io/ https://www.datadoghq.com/dg/monitor/ts/statsd/ https://x.com/kayserzl https://github.com/zkayser https://bsky.app/profile/ethangunderson.com https://github.com/open-telemetry/opentelemetry-collector-contrib Special Guests: Ethan Gunderson and Zack Kayser.