operations

Retries can become infinite noise. For certain failure classes, record the payload + error in a DLQ table and alert/triage. This keeps queues healthy and makes failures actionable.

A real health check tests the dependencies you care about (DB, Redis). Keep it fast and don’t make it do expensive queries. Use it for load balancers and alerts.

Full reindexes can be long and fragile. Add checkpoints (last processed id), process in batches, and make it resumable. That turns a scary operation into a routine one.

Sometimes you need a one-time maintenance operation outside normal schema changes. Use a small table to track “run once” tasks so reruns are safe and the operation is visible.

Backfills often fail because we accidentally load full records and associations. Use select to fetch only needed columns and find_each to keep memory flat. This is basic, but it’s where outages come from.

Counter caches drift (deleted records, backfills, manual SQL). A repair job that recomputes counts safely is invaluable. It’s the kind of operational code you’re glad you wrote the first time a dashboard is wrong.

When cache structures change, you want to invalidate safely without flushing the world. Use a namespace version key (per feature) and incorporate it into cache keys.

When downstream systems degrade, jobs pile up and amplify outages. Add a simple “queue depth guard” so non-critical jobs skip or reschedule instead of making the backlog worse.

Admin actions are production sharp edges. Require a typed confirmation string, log actor + request_id, and run the dangerous work in a background job. This reduces accidental incidents.

Direct deletes can be risky and slow. Mark records for deletion, then sweep in batches in a maintenance job. This gives you observability and a rollback window.

Long-running jobs can hit stale connections. Wrap work in with_connection and consider verify! before heavy DB usage. This reduces “PG::ConnectionBad” noise during long maintenance tasks.

Sometimes you need “only one runner globally” (backfills, refresh jobs). A Redis mutex with TTL avoids deadlocks if the process dies. It’s not perfect, but it’s a solid pragmatic tool.