Rails Background Jobs: Why Customers Shouldn’t Wait for Work That Can Run Later

Every extra second between a click and a response erodes trust and kills conversions. Background jobs fix this by moving slow, non-essential work out of the request cycle. Your Rails app responds instantly. The heavy lifting happens later, invisibly.

This is how we actually think about background job architecture on client work: what belongs in a job, how to choose a queue backend, and the failure modes that only show up once real traffic hits.

Why this matters to users right now

Picture a Rails e-commerce app on Black Friday. Thousands of customers check out at once, and each checkout triggers a payment confirmation, a fraud check, a receipt email, and an invoice PDF. Run all of that inline and the checkout page grinds to a halt: spinners, timeouts, double-submitted forms, abandoned carts.

Now the same app with background jobs. The controller processes the payment, commits the order, and returns a confirmation page immediately. Everything else, the fraud check, the receipt, the PDF, runs in the background queue. The work still happens. The customer just isn’t the one waiting for it.

Inline: click → server does all the work → server responds (slow, risky under load). Background: click → server does the critical work → responds immediately → workers pick up the rest from a queue (fast, resilient).

Customers should never wait for PDF generation, bulk emails, image processing, or third-party API calls when that work can run later without touching what they see. Modern Rails (7.1 through 8.0) ships Active Job as the standard abstraction and Solid Queue as the built-in backend as of Rails 8. Sidekiq remains the most widely used Redis-based alternative for high-throughput work, and both share the same job class structure and enqueue API.

What a background job actually is

A background job is code executed outside the request cycle by a separate worker process. Three pieces make up the system: job classes (Ruby classes inheriting from ApplicationJob, with a perform method defining the work), a queue backend that stores and schedules jobs, and worker processes that pull jobs off the queue and run them. Three flavors exist: one-off jobs triggered immediately, scheduled jobs that run at a future time, and recurring jobs that run on a schedule like a nightly sales report.

Active Job sits between your application code and whatever queue backend you choose, standardizing the job class and enqueue API so your code stays portable. You define a perform method and it handles serialization, deserialization, and routing to the right queue. Implement jobs once against the Active Job API and keep the freedom to swap backends later, move from Solid Queue to Sidekiq or back, and your job classes don’t change. Worth knowing: queue names and positive-integer priorities, callbacks (before_perform, after_perform, around_perform) for instrumentation, error hooks (retry_on, discard_on), and integration with Action Mailer’s deliver_later and GlobalID for safe record serialization. From Rails 8 onward Solid Queue is the default backend; earlier apps commonly run Sidekiq or Delayed Job depending on scale.

The problems this actually solves for users

Run everything inline and users hit spinning loaders lasting 5 to 30 seconds, request timeouts, double-submits from frustrated re-clicks, and cascading failures when a flaky third-party API blocks the entire request. Tasks that belong in a job instead: welcome emails, receipts, PDF or CSV exports, image processing, CRM syncs, webhook delivery, fraud checks against external APIs. A SaaS analytics platform is a good example: a detailed report might take two minutes to generate, so instead of making the user wait, the app responds immediately, “your report is being prepared, we’ll notify you when it’s ready,” while a job does the heavy lifting. Controllers with response times above a few hundred milliseconds are the first candidates for this treatment.

Choosing a queue backend

Solid Queue is the default from Rails 8.0. It stores jobs in your existing SQL database, MySQL or PostgreSQL, so there’s zero external dependency beyond what your app already runs. It supports delayed jobs, concurrency controls, and recurring tasks out of the box, with throughput around 800 to 1,200 jobs per minute and polling-based pickup latency from 100ms to a few seconds.

Sidekiq needs a Redis server. It pushes and pops jobs through Redis, hitting 5 to 15ms pickup latency and 5,000 to 10,000+ jobs per minute. Workers are plain Ruby classes including Sidekiq::Job, backed by a mature community, a built-in web dashboard, and metrics, though some advanced features like batch jobs and enterprise rate limiting sit behind paid tiers. Redis itself adds operational overhead: monitoring, backups, high availability.

Where we actually land on this: most of the Rails apps we work with predate Rails 8 and already run Redis, so Sidekiq is still what we default to on real client work. It’s proven, our own infrastructure is built around it, and there’s rarely a strong reason to disrupt a stable production job system just to modernize it. For a genuinely new Rails 8 project, we’re open to starting with Solid Queue. Migrating an existing app off a working Sidekiq setup purely to switch backends is a case-by-case call, not a default move. The operational risk of a job-system migration has to earn its payoff.

Designing what goes inside a job

What you put inside a job determines whether the system stays reliable or becomes a source of mysterious failures. A few rules we don’t compromise on: one job, one responsibility, send an email, process an image, sync a record, nothing more. Pass IDs, not objects, user_id: 42 rather than the full User instance, so the job looks up fresh data instead of working from something stale. Design for idempotency, because jobs run more than once due to retries, deploys, or duplicate enqueues, and a job that isn’t safe to run twice will eventually double-charge someone or send a duplicate email. Keep transactions short to reduce lock contention, and process large datasets in batches instead of loading thousands of rows into a single job.

We routinely refactor “god jobs,” monolithic classes trying to do everything at once, into smaller, composable classes during maintenance work. It’s one of the most common things we find in a legacy job system.

Queues, priorities, and separating workloads

Not every job is equally urgent. A payment confirmation matters more than a weekly analytics rollup, and queue structure is how you enforce that. queue_as routes different job classes to different queues: critical for payments and fraud checks, default for standard work, mailers for email, low_priority for exports and cleanup, external_api for anything hitting a third party with unpredictable latency.

queue_with_priority adds finer control. Solid Queue honors positive integer priorities and queue order, so a payment job in the same queue as a report job can always run first. Sidekiq processes queues in the order listed in its configuration, giving priority to whatever’s listed earlier. Use queue_name_prefix and environment separation so a job enqueued in staging never gets picked up by a production worker. We formalize queue naming conventions early on any project scaling its job architecture, because ad-hoc queue sprawl is much harder to untangle later than to prevent now.

Error handling and retries

Jobs fail: API timeouts, missing records, database write errors. Active Job handles this with retry_on and discard_on, retrying with increasing delays or silently dropping jobs where the referenced record no longer exists. Sidekiq layers its own retry logic on top, retrying failed jobs up to 25 times over roughly 21 days, so if you’re running it through the Active Job adapter, configure the two intentionally rather than letting both retry independently.

Database-backed backends like Solid Queue record failures in dedicated tables, giving you an audit trail until someone retries or discards them. A few rules of thumb: rescue only expected exceptions and let unexpected ones bubble up to your error tracker, log the job id, queue name, and domain identifiers like order_id, never silently swallow an error, and review failed jobs regularly since patterns often point at upstream problems nobody’s noticed yet.

The transactional bug that catches almost everyone once

Jobs should only be triggered after a database transaction commits. This is one of the most damaging mistakes in background job systems, and we run into it regularly on client Rails apps.

Here’s the shape of it: a controller creates an order inside a transaction and enqueues a SendReceiptJob in that same block. Because Sidekiq, or any Redis-backed queue, can pick the job up almost instantly, it’s entirely possible for the job to start before the surrounding transaction has actually committed. The job queries for the order, finds nothing yet, and fails, or worse, the transaction rolls back later and the job already sent a receipt for an order that technically doesn’t exist. It’s intermittent by nature, which is exactly what makes it painful to track down after the fact instead of obvious up front.

Rails gives you clean ways around it: after_commit callbacks ensure a job only enqueues once the transaction is durably written, enqueue_after_transaction_commit ties enqueue timing to transaction completion explicitly, and when Solid Queue shares your app’s database, enqueuing can participate in the same transaction for strong consistency. For financial or compliance work, always enqueue after commit. A receipt for a rolled-back payment is worse than a slightly delayed one, and this race condition is nearly impossible to reproduce reliably in development, which is why it’s non-negotiable on anything fintech-adjacent.

Scheduling and recurring jobs

Some work needs to run later, some needs to run repeatedly. wait and wait_until schedule individual jobs for future execution, ReminderJob.set(wait: 10.minutes).perform_later(user.id) for example. Solid Queue supports recurring tasks defined in a source-controlled config/recurring.yml, nightly cleanup, weekly summaries, hourly cache warmups, daily renewal checks, with its scheduler ensuring exactly one job gets enqueued per interval even with multiple workers running. Keep recurring definitions in version control rather than manual server configs, make sure only one scheduler instance runs, and monitor for schedule drift. We frequently replace hand-written cron scripts with Rails-native recurring jobs during modernization work, mostly because a forgotten cron entry is a bug waiting to happen.

Capacity planning and observability

Running jobs reliably at scale takes deliberate planning, not just adding workers. Worker count and concurrency needs to match your database connection pool, or you’ll see connection exhaustion: a web server using 20 connections and workers using 30 against a database that allows 40 means timeouts on both sides. Long-running jobs need explicit timeouts, especially around network calls, so a single stuck request doesn’t block a worker indefinitely, and query efficiency inside jobs matters as much as anywhere else in the app.

The metrics worth watching: jobs executed per minute, failure rate, average execution time, queue backlog per queue, and job wait time (enqueue to execution start), your leading indicator that workers can’t keep up. For the Sidekiq-backed apps that make up most of our client work, the Sidekiq Web UI is what we reach for day to day, processed counts, failure counts, queue latency, and retries all visible without a separate tool. For Solid Queue apps, Mission Control gives comparable visibility. Either way, integrate job metrics into your existing APM stack. A backed-up queue should show up in the same dashboard where you watch response times, not in a separate tool nobody checks.

Best practices in one place

One responsibility per job, a class named ProcessOrderAndSendEmailAndUpdateCrmJob is a red flag on sight. Design for idempotency using database constraints, unique keys, or status flags. Pass simple arguments, IDs and small hashes, never large payloads or complex objects. Name things clearly, SendWelcomeEmailJob tells an operator exactly what it does, ProcessJob tells them nothing. Centralize error handling, logging, and instrumentation in ApplicationJob instead of duplicating it across every job class. Log the job id, queue name, and domain identifiers on every line, since those details are what let an on-call engineer figure out what happened at 3am. And in multi-tenant apps, scope every job to the correct tenant boundary explicitly. Getting that wrong is a common failure mode in its own right, and one of the uglier ones to clean up after the fact.

How we approach this work

Essence Solusoft designs, implements, and scales background job architecture for SaaS, e-commerce, and enterprise Rails applications. Depending on what a client needs, this shows up as a focused audit of an unreliable job system, part of a larger build, ongoing staff augmentation alongside an existing team, or consulting during a Rails 8 upgrade. We’ve done this on apps processing a few hundred jobs a day and apps processing millions a month, and the architecture has to match the scale it’s actually serving, not the scale someone hopes for in a pitch deck.

Rails background jobs turn a blocking, fragile app into a responsive one. Instead of a loading screen while the server generates a PDF or calls an API, you return control instantly and let workers handle the rest. It doesn’t take a rewrite, it takes auditing your own app for slow actions, checking controllers where response times spike, and treating inline email delivery, synchronous API calls, and report generation as candidates worth moving off the request cycle. The apps that stay responsive under pressure are the ones where someone decided, early, that customers shouldn’t wait for work that can run later. We’ve helped tune Rails performance this way often enough that it’s usually the first thing we look at on a new engagement.

Talk to us about your Rails background job architecture.