A user reports that some action is "slow" in production. Application logs show no errors. The CPU dashboard is green. And yet something is off somewhere in the chain between the HTTP request and the response. This is the classic symptom of an application that has logs but no observability: you can read what happened at one specific point, but you can't reconstruct the full path of a request across the system. This post covers the three pillars (logs, metrics, traces), what each one reveals and what it hides, and how to combine them to diagnose instead of guess.
The problem: each pillar answers a different question
The three pillars of observability are not interchangeable:
- Logs answer "what exactly happened, at this moment, in this service?". Rich in context, but expensive to store and hard to correlate across services without a shared identifier.
- Metrics answer "what's the overall trend?" (error rate, p95 latency, throughput). Cheap to store long-term (aggregated), but say nothing about an individual request.
- Distributed traces answer "where did the time go, for this specific request, across every service it passed through?". They tie logs and metrics together by giving the thread for a single request.
The classic trap: a team that only has centralized logs (ELK, Loki) believes it's "observable," but without correlation by request identifier (trace_id) across services, it can only reconstruct the path of a slow request by manually cross-referencing timestamps - an exercise that takes hours and breaks the moment two requests overlap in time.
Why it happens: instrumentation by silo
Each service logs on its own, with its own format, without propagating context to the services it calls. An HTTP call from service A to service B carries no information that would let the two logs be linked after the fact. The total latency perceived by the user is the sum of several segments (network time, database time, a third-party service call, serialization), and without distributed tracing, that breakdown exists nowhere - it has to be mentally reconstructed, hoping intuition is right.
Diagnosing it: instrument before you have the problem
Observability isn't added after the incident, it's added before. OpenTelemetry has become the de facto standard for instrumenting logs, metrics and traces in a unified way, with exporters to most backends (Jaeger, Grafana Tempo, Datadog, and others).
Propagate trace context between services
// Node.js / Express - minimal OpenTelemetry instrumentation
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: 'http://otel-collector:4318/v1/traces',
}),
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
With auto-instrumentation enabled, every outgoing HTTP call automatically propagates the traceparent header (W3C Trace Context format) to the called service, which continues the same trace instead of starting a new one.
Link logs and traces through a shared identifier
// Structured logger (pino) injecting the current trace_id into every log line
const pino = require('pino');
const { trace } = require('@opentelemetry/api');
const logger = pino({
mixin() {
const span = trace.getActiveSpan();
if (!span) return {};
const { traceId, spanId } = span.spanContext();
return { trace_id: traceId, span_id: spanId };
},
});
logger.info({ orderId: '8421' }, 'Order created successfully');
// => {"level":30,"trace_id":"a1b2c3...","span_id":"d4e5f6...","orderId":"8421","msg":"Order created successfully"}
From that point on, any application log can be found starting from a slow trace in Jaeger or Tempo, and vice versa: you can click on a trace_id in an error log to see the full trace of the request that produced it.
Manually instrument the critical segments
Auto-instrumentation captures HTTP calls and standard SQL queries, but not internal business logic. For an expensive step (a recommendation calculation, a call to a complex business rule), a manual span gives the necessary granularity:
const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('order-service');
async function calculateShippingCost(order) {
return tracer.startActiveSpan('calculate-shipping-cost', async (span) => {
span.setAttribute('order.id', order.id);
span.setAttribute('order.itemCount', order.items.length);
try {
const cost = await shippingRulesEngine.evaluate(order);
span.setAttribute('shipping.cost', cost);
return cost;
} catch (err) {
span.recordException(err);
span.setStatus({ code: 2, message: err.message }); // ERROR
throw err;
} finally {
span.end();
}
});
}
Visualized in Jaeger, this span shows up as a distinct segment in the request's timeline, with its own duration, making it immediately visible whether it represents 5ms or 2 seconds out of a 2.1-second total request.
Metrics: detection, not diagnosis
Metrics (via Prometheus, for instance) serve to detect that a problem exists, not to explain it. An alert on p95 latency triggers the investigation; the individual trace of a slow request, captured at the same moment, provides the explanation.
// Prometheus histogram for per-endpoint latency
const client = require('prom-client');
const httpDuration = new client.Histogram({
name: 'http_request_duration_seconds',
help: 'Duration of HTTP requests',
labelNames: ['method', 'route', 'status_code'],
buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 2, 5],
});
app.use((req, res, next) => {
const end = httpDuration.startTimer({ method: req.method });
res.on('finish', () => {
end({ route: req.route?.path ?? 'unknown', status_code: res.statusCode });
});
next();
});
The point of explicit buckets is being able to compute percentiles (p50, p95, p99) on the Prometheus/Grafana side without storing every individual value, which would be unmanageable in volume for a high-traffic service.
Alerting on error budgets, not raw thresholds
A common failure mode once metrics are in place is alerting on a raw threshold ("fire if p95 latency exceeds 500ms") that either pages constantly on noise or stays silent during a slow but steady degradation that never crosses the line. Alerting against a Service Level Objective (SLO) and its error budget gives a more stable signal, because it reacts to the rate at which the budget is being consumed rather than to a single noisy spike.
# Prometheus recording rule: fraction of requests slower than the 300ms SLO threshold
groups:
- name: slo-latency
rules:
- record: slo:requests_slow:ratio_rate5m
expr: |
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))
# Alert fires only when the error budget burns fast enough to exhaust it
# well before the end of the 30-day window, which filters out brief blips
- alert: LatencySLOBurnRateHigh
expr: slo:requests_slow:ratio_rate5m > 0.02
for: 15m
labels:
severity: page
annotations:
summary: "Latency SLO burn rate is abnormally high over the last 15 minutes"
The for: 15m clause matters as much as the threshold itself: without it, a transient spike during a deploy triggers a page that nobody needs to act on, which trains the on-call rotation to ignore alerts. Combined with traces, this kind of alert tells you that something degraded and roughly when it started; the trace for a request captured during that window tells you where in the call chain the time went.
The tradeoffs
Full distributed tracing has a real cost: the volume of data to store, the CPU overhead of instrumentation (generally low with OpenTelemetry, but not zero), and extra operational complexity (an OTel collector to maintain, retention to manage). Tracing 100% of requests on a very high-traffic service quickly gets expensive in storage; the common practice is sampling, keeping 100% of error traces and a reduced percentage of successful ones.
// Status-based sampling: keep all errors, 10% of the rest
const { TraceIdRatioBasedSampler, ParentBasedSampler } = require('@opentelemetry/sdk-trace-base');
const sdk = new NodeSDK({
sampler: new ParentBasedSampler({
root: new TraceIdRatioBasedSampler(0.1),
}),
// ...
});
When "standard" observability isn't enough
For intermittent performance issues that can't be reproduced on demand, even a detailed trace can show nothing if sampling didn't happen to capture the problematic occurrence. In that case, continuous profiling (via tools like Pyroscope or Parca) complements the three pillars by giving a code-level view (which function is burning CPU) over a time window, without waiting for a specific request to trigger an alert.
The point to keep: logs explain an event, metrics detect a trend, traces connect the two across services. Without context propagation (a shared trace_id) between these three pillars, each one stays an island that has to be cross-referenced by hand - exactly the work observability is supposed to eliminate.
