javascript 70 lines · 3 tabs

Stream a Large CSV Export in Express with Backpressure and an Async Row Generator

Shared by codesnips Aug 2026

3 tabs

function escapeCell(value) {
  if (value === null || value === undefined) return '';
  const str = String(value);
  if (/[",\n\r]/.test(str)) {
    return '"' + str.replace(/"/g, '""') + '"';
  }
  return str;
}

function toCsvLine(fields, columns) {
  return columns.map((col) => escapeCell(fields[col])).join(',') + '\r\n';
}

module.exports = { escapeCell, toCsvLine };

async function* streamRows(db, { batchSize = 1000 } = {}) {
  let cursor = 0;
  for (;;) {
    const rows = await db.query(
      'SELECT id, email, plan, created_at FROM users ' +
      'WHERE id > $1 ORDER BY id ASC LIMIT $2',
      [cursor, batchSize]
    );
    if (rows.length === 0) return;
    for (const row of rows) {
      yield row;
    }
    cursor = rows[rows.length - 1].id;
    if (rows.length < batchSize) return;
  }
}

module.exports = { streamRows };

const { once } = require('events');
const { streamRows } = require('./rowGenerator');
const { toCsvLine } = require('./csvTransform');

const COLUMNS = ['id', 'email', 'plan', 'created_at'];

async function exportUsersCsv(req, res, next) {
  res.setHeader('Content-Type', 'text/csv; charset=utf-8');
  res.setHeader(
    'Content-Disposition',
    'attachment; filename="users-export.csv"'
  );

  let aborted = false;
  res.on('close', () => { aborted = true; });

  try {
    res.write(COLUMNS.join(',') + '\r\n');

    for await (const row of streamRows(req.app.locals.db)) {
      if (aborted) return;
      const ok = res.write(toCsvLine(row, COLUMNS));
      if (!ok) {
        await once(res, 'drain');
      }
    }

    res.end();
  } catch (err) {
    if (res.headersSent) {
      res.destroy(err);
    } else {
      next(err);
    }
  }
}

module.exports = { exportUsersCsv };

3 files · javascript Explain with highlit

Exporting large datasets as CSV is a classic place where naive code falls over: building the whole file in memory before sending it can exhaust the heap on tables with millions of rows. This snippet shows the streaming alternative, where rows flow from the database through a transform and out to the HTTP response one chunk at a time, keeping memory flat regardless of dataset size.

In rowGenerator.js, streamRows is an async generator that pages through the database with a keyset cursor rather than OFFSET. Each iteration fetches a bounded batchSize, yields rows individually with yield, and advances cursor to the last id seen. Keyset pagination is used instead of LIMIT/OFFSET because offset scans get slower as the offset grows, while WHERE id > cursor stays fast and stable even when rows are inserted mid-export. Because it is an async generator, backpressure is natural: the consumer pulls the next batch only when it is ready.

csvTransform.js wraps escapeCell and toCsvLine to turn a plain object into a correctly quoted CSV line. Fields containing commas, quotes, or newlines are wrapped in double quotes and internal quotes are doubled, which is the RFC 4180 rule that prevents a stray comma from shifting every column.

The heart of it is exportController.js. The handler sets Content-Type and a Content-Disposition attachment header so browsers download a named file, writes the header row, then loops the generator with for await. The crucial detail is checking the boolean returned by res.write: when it returns false the kernel buffer is full, so the code awaits a drain event via once(res, 'drain') before continuing. Without this, a fast producer and a slow client would let Node buffer unbounded data in memory, defeating the whole purpose.

The controller also guards edge cases: res.on('close', ...) aborts the loop if the client disconnects mid-download, and errors are handled differently depending on whether headers were already flushed, since a 500 status cannot be sent once streaming has begun. This pattern is the right tool whenever the export could plausibly exceed available memory or take long enough that a client should see bytes immediately.

Related snips

typescript

export interface RetryOptions {
  retries: number;
  baseMs: number;
  maxMs: number;
  signal?: AbortSignal;
  onRetry?: (attempt: number, delay: number, err: unknown) => void;

Exponential backoff with jitter for retries

typescript reliability retry

by codesnips 2 tabs

ruby

require "csv"

class PeopleCsvStream
  include Enumerable

  HEADERS = %w[id full_name email signed_up_at plan].freeze

Resilient CSV Export as a Streamed Response

rails performance streaming

by codesnips 3 tabs

ruby

# Vulnerable: user input is concatenated directly into SQL.
email = params[:email]
password = params[:password]

sql = "SELECT * FROM users WHERE email = '#{email}' AND password_hash = '#{password}'"
user = ActiveRecord::Base.connection.execute(sql).first

SQL injection prevention with unsafe and safe query patterns

sql-injection owasp database

by Kai Nakamura 3 tabs

typescript

export type Settled<R> =
  | { status: 'fulfilled'; value: R }
  | { status: 'rejected'; reason: unknown };

export interface ConcurrencyOptions {
  limit: number;

Simple concurrency limiter for batch operations

node concurrency async

by codesnips 2 tabs

typescript

import { randomBytes, createHash } from "crypto";
import jwt from "jsonwebtoken";
import { RefreshTokenStore } from "./store";

const ACCESS_SECRET = process.env.ACCESS_SECRET!;
const ACCESS_TTL = "15m";

JWT access + refresh token rotation (conceptual)

security node jwt

by codesnips 3 tabs

sql

-- EXPLAIN ANALYZE (actual execution statistics)
EXPLAIN ANALYZE
SELECT u.username, COUNT(o.id) AS order_count
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE u.created_at >= '2024-01-01'

Advanced query optimization techniques

database optimization query-performance

by Maria Garcia 2 tabs

Share this code

Here's the card — post it anywhere.

Stream a Large CSV Export in Express with Backpressure and an Async Row Generator — share card

Link copied