observability

I instrument services because I don’t want performance and reliability to be a guessing game. In Service instrumentation, I wrap the work in ActiveSupport::Notifications.instrument and emit a stable payload (things like IDs and counts, not giant blobs

Sometimes you want a quick diagnostic without wiring a full metrics stack. expvar exposes variables at /debug/vars in a standard JSON format. I use it for a handful of counters like “requeststotal” and “jobsfailed” when I’m bootstrapping a service or

When multiple services call the same upstream, I like a custom http.RoundTripper to centralize cross-cutting behavior: inject headers, measure duration, and apply consistent redaction rules. This keeps call sites clean and prevents copy/paste mistakes

net/http/pprof is incredibly useful during performance incidents, but it should never be open to the public internet. I register the pprof handlers on a separate mux and wrap them with Basic Auth (or, better, your real auth middleware). The important

Comprehensive request logging provides visibility into API usage patterns, performance bottlenecks, and security incidents. I log structured JSON that includes request method, path, parameters (sanitized to exclude passwords), response status, duratio

Structured logging outputs JSON-formatted log entries for machine parsing. Each log line includes timestamp, level, message, and contextual fields like request_id, user_id, and service. Structured logs enable powerful queries in Elasticsearch through

I use ActiveSupport::CurrentAttributes any time I need request-scoped context without turning my app into a tangle of thread-local globals. In Current, I define the attributes I care about (member, request_id, remote_ip) and I use the resets hook to c

Noisy error reporting without context turns into a pile of untriaged alerts. I initialize Sentry early, set the release from the git SHA, and tag the environment so I can separate staging from production. I also set a sample rate and capture only what

APIs are easier to operate when errors are structured and consistent. Wrap errors into a problem-details style response with a stable type and request_id so support can quickly trace issues.

Production visibility requires more than basic request logging. I instrument critical code paths using ActiveSupport::Notifications to publish custom metrics that monitoring services consume. Each instrumented block publishes events with timing data,

Observability provides insight into database health and performance. I monitor key metrics—queries per second, connection count, cache hit ratio. Slow query logs identify performance problems. Query latency percentiles show user experience. Lock wait

Prometheus collects and stores time-series metrics via a pull model. It scrapes /metrics endpoints at configured intervals. The prometheus.yml defines scrape_configs with target discovery. static_configs list fixed targets while kubernetes_sd_configs