observability-and-instrumentation

Observability and Code Instrumentation

Skill Overview

The observability and code instrumentation skill helps developers add logging, metrics, tracing, and alerting capabilities to production code, making system behavior visible and diagnosable from the outside. If a feature goes live without telemetry data, the first bug report from a user becomes an archaeological excavation rather than a simple query.

Applicable Scenarios

Diagnosing production incidents: When a production issue is reported but the existing data cannot reveal what happened, telemetry data is needed to quickly identify the root cause.

Monitoring newly released features: When building any feature that will run in production—including new services, APIs, background jobs, or external integrations—ensure that system behavior is observable.

Reviewing alert rule configurations: When setting up or reviewing alert rules, ensure that alerts are based on symptoms experienced by users rather than internal system causes, thereby avoiding alert fatigue.

Core Capabilities

Structured logging and telemetry design: Guide developers in writing structured logs with stable event names and machine-readable fields, enforce the propagation of correlation IDs, and ensure that every log is queryable and correlatable. Also help select the right type of telemetry signal—logs answer “what exactly happened,” metrics answer “how often and how fast,” and traces answer “where was the time spent.”

RED/USE monitoring metrics and cardinality control: Apply the RED method to request-driven services (Rate, Errors, Duration latency histograms), and the USE method to resources (Utilization, Saturation, Errors). Strictly control label cardinality to prevent unbounded values such as user IDs, raw URLs, and error messages from causing monitoring backends to fail.

Symptom-based alerting and OpenTelemetry integration: Alerts must be based on symptoms perceived by users (error rate >1%, p99 latency >2s), not system causes (CPU at 85%, disk at 70%). Each alert should link to an actionable runbook. Use OpenTelemetry to enable zero-code automatic instrumentation for distributed tracing, supporting HTTP, gRPC, and common database clients to ensure complete, uninterrupted tracing of cross-service requests.

Common Questions

When should monitoring be added to code?

Monitoring should be added while writing the feature code, not after the feature is “complete.” Monitoring is part of the feature, just like testing. If a feature goes live without telemetry, you will be unable to determine what happened during the first production incident. Adding monitoring then will cost far more than implementing it correctly during development.

How can problems be located quickly during a production incident?

First, ensure that every request has a Correlation ID propagated through all logs, traces, and cross-service calls. Then follow the complete path of an individual request in the tracing UI and examine the time spent at each hop. At the same time, query event fields in structured logs (such as payment_failed and errorCode) and the p95/p99 latency of RED metrics to determine whether the issue is caused by business logic, a third-party timeout, or resource saturation. Without this telemetry data, troubleshooting is reduced to guesswork and repeated trial-and-error changes.

Why can too many alerts cause a team to become desensitized?

Because alerts based on internal system causes—such as CPU utilization, memory usage, or restart counts—rather than user symptoms may trigger frequently while the system is operating normally, while genuine user-facing failures go undetected. Alert fatigue causes operations personnel to habitually ignore all alerts. The correct approach is to alert only on user-perceivable symptoms, such as error rates, latency, and queue backlogs, and ensure that every alert is actionable. If the default response is “Ignore it; it will self-heal,” that alert should be removed.