Observability systems: design the telemetry path before the dashboard

Observability is a data system before it is a set of dashboards. The critical design work happens in the path between a workload producing a signal and an operator using that signal to make a decision.

This guide connects the vendor-neutral telemetry architecture described in selected work with reusable design principles for logs, metrics, and traces.

Begin with operational questions

Instrumentation should answer a known question: is a service meeting its objective, where is latency introduced, which dependency failed, or what changed before an incident? Collecting data without a decision in mind creates cost without reliable diagnostic value.

Define service objectives, failure modes, and investigation paths first. From those, derive the signals, dimensions, retention, and access requirements.

Use a telemetry contract

A telemetry contract makes data consistent across teams and tools. At minimum, define:

• service and deployment identity;
• environment and ownership attributes;
• trace and request correlation fields;
• event severity and error semantics;
• privacy classification and redaction requirements;
• schema version and compatibility rules.

OpenTelemetry provides vendor-neutral APIs, semantic conventions, and a collector model. It does not remove the need for organisational conventions; it gives those conventions a portable foundation.

Separate collection, routing, and storage

Treat the telemetry path as distinct stages. Collection receives and normalises data near the workload. Routing applies policy, enrichment, sampling, and destination selection. Storage and analysis systems serve different operational and retention needs.

This separation reduces vendor coupling. A team can change an analysis destination without rebuilding every application integration. It also creates clear control points for regional handling, cost limits, and security policy.

Design for pressure and partial failure

Telemetry volume often rises during the incident that operators most need to understand. The path therefore needs explicit behaviour for backpressure, buffering, retries, sampling, and data loss. Document which signals are durable, which can be sampled, and how operators detect a degraded pipeline.

Monitor the observability system itself: queue depth, dropped records, processing latency, cardinality growth, export failures, and configuration drift are first-class service indicators.

Control cardinality and cost at the source

Unbounded identifiers in metric labels can make a monitoring system expensive or unstable. Define approved dimensions and move high-cardinality detail into logs or traces when appropriate. Apply retention and sampling according to the value and sensitivity of the signal rather than using one policy for all telemetry.

Make ownership visible

Every service and telemetry schema needs an owner. An operator should be able to move from an alert to the responsible team, current runbook, recent deployments, and relevant service objectives without searching across disconnected systems.

The related notes on distributed cloud, multi-cloud architecture, and serverless computing describe deployment models that make a portable telemetry contract especially valuable.

Sources and further reading

• OpenTelemetry documentation
• OpenTelemetry semantic conventions
• Google SRE workbook: monitoring