How We Cut a Rails CSV Download from 60s → 1.5s (40x Faster)No background jobs, no batch processing, just a rewrite of the export logic

1. The Typical "Offload the Heavy Work" Options We Considered First

Faced with "a process taking over a minute inside an HTTP request," a Rails developer's reflexes usually converge on one of the following two options. We started by considering both of them on this project, too.

1-1. Option A: Batch Processing (Report Table / Pre-Generation)

In this approach, a nightly batch generates the CSV file (or a report table holding the aggregated results) ahead of time, and the download request simply returns it. The HTTP side of the download effectively becomes a file transfer, so it completes almost instantly.

On this project, however, there were structural reasons that made Option A a poor fit. The system was originally designed so that both the report screens and the downloads are generated on the fly from the same queries; there are no report tables or pre-generated files anywhere. Introducing a pre-generation batch just for the monthly report download would create an inconsistent design where only the monthly report goes through a batch while every other report hits the queries directly.

On top of that, since we were planning a daily batch, there would be a constant freshness problem: whenever someone pulled "last month" at the start of a month, data near the last day would only be reflected up to the time the batch ran (in effect, you could only ever get data "up to yesterday"). Given that these are monthly aggregates, having a few days around the month boundary be off is hard to accept from a business standpoint.

• Pros: Permanently free from HTTP timeouts; minimal load at download time
• Cons: Diverges from the existing system's design philosophy (query-based); data freshness degrades to daily granularity
• Implementation effort: A few days to a week (one-time)
• Maintenance effort: Ongoing for the life of the project: monitoring the nightly batch, recovering from failures, and managing the report table lifecycle

1-2. Option B: Background Jobs (Adding Solid Queue as a Gem)

Here, clicking the download button kicks off a background job, and once it finishes, the file is handed to the user via an email notification or by placing it on S3. This completely decouples the work from the HTTP timeout.

The typical setup is Sidekiq + Redis, but we didn't go that way on this project, because adding Redis to the stack means one more service to manage. Our first choice instead was adding Solid Queue as a gem (Solid Queue is the ActiveJob backend adopted as the default in Rails 8; on this project's Rails 7.x environment, it can be added as a gem. It's database-backed, so no Redis is needed).

Even with Solid Queue, though, going async requires the same design changes as Sidekiq does.

• Pros: Free from HTTP timeouts; enables concurrent processing; with Solid Queue, no extra services like Redis
• Cons: Always-on worker processes, a retry design for failed jobs, a completion-notification mechanism, and front-end changes
• Implementation effort: A few days to a week (lighter than Sidekiq + Redis)
• Maintenance effort: Ongoing for the life of the project: process monitoring, job table lifecycle management, and compatibility checks on major Rails / gem upgrades

1-3. The Real Question for Both: Is the Complexity Worth the Payoff?

More than the implementation effort, both options add "maintenance effort" that stays with you for the life of the project (batch monitoring, always-on workers, looking after job tables, Redis upkeep, and so on). Something felt off about a structure that piles on ongoing costs without actually solving the underlying problem.

When something feels off like that, we make a habit of putting the decision on hold and taking time to re-examine how the process is structured. Rush into a major overhaul, and you often realize later that "a much smaller change would have done it."

2. Putting It on Hold and Taking a Fresh Look at the Structure

We shelved the major overhaul and first set out to isolate what was actually slow.

2-1. What the Measurements Showed: The Export Logic Was the Real Bottleneck

When we broke the download down and measured how long the query alone took versus how long the CSV export alone took, the result was surprising. Even the query for a full year came back in about 19 seconds. The aggregate queries were already covered by indexes, so the SQL itself was fast enough.

Yet the existing implementation timed out on a full year (i.e., at least 60 seconds), which means most of the slowness was on the CSV export side. The export's processing time grew far faster than linearly as data increased, and that's a problem offloading to a background job wouldn't fix at the root (it would take just as long wherever it ran).

At this point our approach shifted: "Going async won't remove the underlying bottleneck; fixing the export itself is the better path."

2-2. How the CSV Export Was Structured, and Why It Slowed Down Non-Linearly

The existing implementation looked roughly like this (generalized; in this project, rows is an array of row data already aggregated upstream):

def export
  rows = @report_data[:rows]  # aggregated data array built upstream
  csv_data = CSV.generate do |csv|
    csv << headers
    rows.each do |row|
      csv << build_row(row)
    end
  end
  send_data csv_data, filename: "export.csv"
end

This implementation hid two factors that degrade non-linearly with data volume.

• Internal string concatenation via csv << row inside the CSV.generate block: every time the internal buffer grows, it tends to trigger memory reallocation and copying, and naive String concatenation in Ruby is known to degrade roughly O(n²) relative to the cumulative length N
• Sending everything at once with send_data: send_data puts the entire CSV string it receives into the response body via Rack in one piece. Because it holds the whole CSV (proportional to N) in memory before pushing it out all at once, peak memory grows and TTFB (time until the first byte comes back) gets worse

"Building the entire CSV string" and "sending it all at once" stacked up serially as the memory and processing time of a single response.

That's what was really behind "60 seconds for two months, a timeout for a full year." It's the structural reason why, when the data grew 6x, processing time grew by far more than 6x.

3. Implementing the Fix: Streaming with Enumerator + response_body

To address the bottleneck identified in §2, we made a single change: stream the CSV out one line at a time. No additional services or gems such as Solid Queue, Sidekiq, or Redis; the change lives entirely within the controller (and a concern for reuse).

3-1. Before / After Code

Here is a generalized version of the actual code.

Before:

def export
  rows = @report_data[:rows]
  csv_data = CSV.generate do |csv|
    csv << headers
    rows.each do |row|
      csv << build_row(row)
    end
  end
  send_data csv_data, filename: "export.csv"
end

After:

def export
  stream_csv_response(
    "export.csv",
    build_csv_enumerator(@report_data[:rows])
  )
end

private

# extract into a concern if used across multiple controllers
def stream_csv_response(filename, enumerator)
  response.headers["Content-Type"]        = "text/csv; charset=UTF-8"
  response.headers["Content-Disposition"] = %(attachment; filename="#{filename}")
  response.headers["Cache-Control"]       = "no-cache"
  response.headers["X-Accel-Buffering"]   = "no"  # buffering must be disabled when behind nginx
  response.headers.delete("Content-Length")       # size is unknown when streaming
  self.response_body = enumerator
end

def build_csv_enumerator(rows)
  Enumerator.new do |yielder|
    yielder << CSV.generate_line(headers)
    rows.each do |row|
      yielder << CSV.generate_line(build_row(row))
    end
  end
end

3-2. What Each Change Does

The key points are easier to follow if you split the changes into three groups: the core performance fix, the prerequisites for streaming to work correctly, and cleanup for reusability.

The core performance fix: these two are the heart of it

• self.response_body = Enumerator.new {|yielder| ... }: Rails' mechanism for streaming responses. Each time the Enumerator yields a line with yielder << row, the content up to that point flows through Rack into the HTTP response. The client receives the first byte sooner (shorter TTFB), and Rails never has to accumulate the whole CSV in memory
• CSV.generate_line: A utility that CSV-escapes a single row and returns it as a string. It sidesteps the internal buffer reallocation problem of the CSV.generate block (which degrades roughly O(n²)), letting each row be handled as an independent, short string

Prerequisites for streaming to work correctly

• X-Accel-Buffering: no: Tells reverse proxies such as nginx not to buffer the response. Without it, no matter how much Rails yields, the client receives everything in one burst. It serves the same purpose behind CDNs like Cloudflare
• Removing Content-Length: When streaming, the total response size isn't known in advance, so you must not send Content-Length. Rails middleware may add it automatically, so we remove it explicitly

Cleanup for reusability (a decision separate from performance)

• Extracting a concern: This project had similar CSV exports in multiple controllers (the admin panel and the client-facing screens), so we moved stream_csv_response and build_csv_enumerator into a concern along the lines of EmissionCsvStreaming and include it in both controllers

Note that in this project rows is row data already aggregated upstream, so we don't use find_each. When streaming raw records directly (such as a CSV of hundreds of thousands of transaction records or more), there's room to extend this to fetch in batches with something like Model.find_each(batch_size: 1000) and yield as you go.

3-3. Implementation Time: Done in a Few Hours

The code change came to effectively 10 to 20 lines and was done in a few hours, review included. Compared with batch processing (a few days to a week) or background jobs (a few days to two weeks to introduce Solid Queue / Sidekiq), the change was an order of magnitude smaller.

4. Results: 40x Faster for Two Months, and a Full Year Now Completes

We measured before and after in the same environment with the same data ranges.

4-1. Processing Time by Data Volume

Here are three screenshots of the actual measurements (Chrome DevTools Network tab). The Waiting for server response value is the key metric here.

4-2. Breakdown: The Export Cost Squeezed Down to Almost Nothing

Breaking down the post-fix 20 seconds for a full year makes the structural change clear. As measured in §2-1, the one-year query alone takes about 19 seconds, so of the 20-second total, the export accounts for only the remaining 1 second or so.

With the export cost cut by more than 40x, processing time is now effectively bounded by query execution time. As data grows, the query's linear growth translates directly into overall growth (i.e., it scales at roughly O(n)), and there's no longer any room for the export side to blow up non-linearly as it did before. This is a structural improvement you would never get by offloading to a background job.

5. Always Verify the Results Are Correct

The most important thing to watch for in any performance fix is "it got faster, but did the output change?" On this project, we exported two months of CSV with both the Before and After implementations and confirmed with diff that they were identical (including record order, columns, line endings, and character encoding).

Reporting in particular is an area where correctness trumps speed. Adding "confirm the output hasn't changed" to your checklist is a simple habit, but skip it and you risk aggregates silently drifting.

6. Lessons: Rethink the Process Before You "Offload" It (With Caveats)

6-1. Before Reflexively Reaching for "Offload," Measure to Pinpoint the Bottleneck

Reflexes like "slow process → make it async" or "heavy query → fix the SQL" are healthy in themselves, but if you skip the step of isolating the real bottleneck through measurement first, you risk fixing the wrong thing. On this project the bottleneck was the export logic, not the SQL, so offloading to a background job would have taken just as long. Regardless of years of experience, skipping the "verify by measuring" step lowers the quality of your decisions.

6-2. Always Build In a "Review the Structure" Phase Before a Major Overhaul

Simply making it a habit, when something feels off, not to commit to a major overhaul on the spot but to hold off for a day or two and re-read the structure, lets you catch cases like this one where a small change does the job. If 30 minutes of measurement might save you days or weeks of major rework, measuring is the better investment. Batch processing and background jobs may only cost implementation effort once, but they then add "maintenance effort" that stays with you for the life of the project (process monitoring, looking after job tables, batch windows and failure recovery, and so on). It's worth asking every time whether that is in proportion to the original goal (speeding up a download feature).

6-3. When "Offloading" Is Still the Right Call

We didn't go with offloading on this project, but it's a solution we'd adopt immediately if the conditions were right. On other projects, these can be the first choice.

When batch processing is the right call

• Even after optimization, processing still takes on the order of minutes (several minutes or more)
• The data doesn't need to be real-time (as of the previous day is good enough, etc.)
• The data volume is so extreme that no rewrite will make synchronous processing fit
• Multiple users share the same aggregated results (i.e., one pre-generation run can serve everyone)

When background jobs are the right call

• Even after optimization, processing takes 10 minutes or more (more than synchronous HTTP can bear)
• The system already has other background processing, with worker processes already running (Solid Queue / Sidekiq, etc.)
• A "job finished" notification mechanism (email, Slack, SSE, etc.) already exists, so the added cost is low
• Failed jobs need to be retried safely (which presupposes an idempotent design)

On this project, "1.5 seconds for two months, 20 seconds even for a full year" kept us within the bounds of synchronous HTTP, so we judged that not adding operational moving parts outweighed the benefits of going async or adding a pre-generation batch. If data grows and times creep back into the 30-second-to-one-minute range, that's when we plan to revisit going async.