Observability Beyond Logs, Metrics, and Traces

Key Takeaways
- Aggregated metrics and disconnected logs fail to answer unknown-unknown failure modes in distributed systems.
- High-cardinality structured events provide rich context (user ID, tenant ID, query duration, host) on every transaction.
- Alert on customer-impacting Service Level Objectives (SLOs) rather than noisy server-level resource utilization.
- OpenTelemetry standardizes cross-language distributed tracing and telemetry collection without vendor lock-in.

The Limitation of Traditional Observability

For a decade, the industry treated logs, metrics, and traces as the "three pillars of observability." However, in modern cloud architectures with hundreds of microservices, serverless functions, and ephemeral containers, traditional tooling breaks down:

• Metrics aggregate away vital context: A metric tells you P99 latency is 3.2 seconds, but averages out which specific tenants or query types caused the spike.

• Unstructured logs are expensive and unsearchable: Millions of plain text log lines create massive ingestion bills and take minutes to search during active outages.

• Disconnected traces lack business context: Traces show which service took 800ms, but fail to show the customer tier, shopping cart size, or input parameters.

High-Cardinality Structured Events

True observability is the ability to understand the internal state of a system based solely on its external outputs—especially for questions you did not anticipate when writing the code.

High-cardinality events attach rich contextual dimensions to every single request:

{
  "timestamp": "2026-08-25T17:15:32.401Z",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "service": "billing-service",
  "environment": "production",
  "tenant_id": "org_98234",
  "user_id": "usr_4410",
  "plan_tier": "enterprise",
  "http_route": "/v1/invoices/generate",
  "http_status": 504,
  "duration_ms": 3210.4,
  "db_queries_count": 42,
  "db_duration_ms": 3150.2,
  "db_shard": "aurora-pg-shard-03",
  "error": "database query deadline exceeded"
}

With high-cardinality data, querying WHERE http_status = 504 GROUP BY db_shard instantly identifies that 100% of timeouts are localized to aurora-pg-shard-03, eliminating hours of guesswork.

Observability Maturity Framework

Moving to SLO-Based Alerting

Stop waking up engineers for "CPU utilization > 80%." High CPU is often healthy during batch processing. Instead, alert on Service Level Objectives (SLOs) that directly impact users:

• Availability SLO: 99.9% of payment authorizations return HTTP 200 within 500ms over a rolling 30-day window.

• Burn-Rate Alerting: Trigger a page only when the error budget is burning fast enough to exhaust 10% of the monthly budget within 1 hour.

Frequently Asked Questions

What does high-cardinality mean in database systems?

High-cardinality refers to fields with millions of unique values, such as user IDs, UUIDs, or email addresses. Traditional time-series databases crash when indexing high-cardinality data, requiring modern columnar stores (like ClickHouse or Honeycomb) designed for wide events.

How does OpenTelemetry prevent vendor lock-in?

OpenTelemetry provides open-source, vendor-neutral SDKs and collectors. Your application code instruments traces and metrics once, and the OpenTelemetry Collector routes telemetry to Datadog, Grafana, Jaeger, or S3 via simple YAML configuration.

How much does high-cardinality observability cost?

By replacing noisy string logs with single wide canonical events per request and applying dynamic head/tail sampling, organizations frequently reduce their observability data volume by 40-60% while dramatically improving query speed.