Image Pipeline System Design: Resizing, Formats, Cache Keys and Signed URLs

Your pages get slow and expensive, because every visitor downloads megabytes they never see. Image pipeline system design fixes this by making the right-sized file once and serving it from cache many times.

Picture a common setup. A second-hand furniture marketplace lets sellers upload phone photos, and buyers browse listing cards on cheap Android phones. The setup is made up, but every measured number below came from our own test runs.

Our test photo, a generated stand-in for a phone photo, was 3000 by 2000 pixels and 1,633,303 bytes as a JPEG. The listing card shows it 320 pixels wide. The same photo as a 320-pixel AVIF was 1,953 bytes, about 836 times smaller.

Four things go wrong without a pipeline. Pages ship the full photo to every card. Every browser gets JPEG, even ones that show smaller formats. Some phone photos appear sideways after resizing. Anyone can ask the resizer for any size and keep your servers busy.

A separate SevyDevy guide on image optimisation covers the browser side: which format and size a page should ask for. This guide is the server side, the system that makes and serves those files.

Ten words carry the whole guide. Each one is tied to the marketplace's listing photos.

• Original: the photo exactly as the seller uploaded it. You keep it and never change it.
• Variant: a copy of the original that is resized, re-encoded or both, such as a 320-pixel WebP for the listing card.
• Transform: the work that makes a variant, which is decode, resize and encode.
• Decode and encode: decoding turns a compressed file into raw pixels; encoding compresses pixels into a file format such as JPEG, WebP or AVIF.
• Object storage: a cloud store for files, called objects, kept in buckets and fetched by name over HTTP.
• CDN: a content delivery network, many caching servers close to users that keep copies of responses.
• Cache key: the label a cache uses to decide whether two requests can share one stored response.
• Content negotiation: the server picking one of several versions of a resource from request headers.
• Accept and Vary: Accept is the request header listing formats the browser wants; Vary is the response header naming headers that changed the answer.
• HMAC: a hash-based message authentication code, a short fingerprint computed from a message and a secret key.

Image pipeline system design is the plan for storing originals, making variants and delivering them through caches. It has five parts: upload, original storage, a transform service, variant storage and a CDN.

The upload path writes the original once. The transform service makes variants with an image library such as libvips, used through the sharp package for Node.js, or Pillow for Python. Variants land in their own bucket, and the CDN serves them.

Precompute the few sizes you always need, and make the rest on the fly with a cache in front. Most teams end up with this mix, because each approach fails in a different way.

The naive first try is to precompute everything: every width your designers might want, in every format. It works on day one. Then a redesign adds a 480-pixel card, and you must reprocess every photo ever uploaded.

On the fly means the first request for a size runs the transform, then the result is cached. Nothing is wasted on sizes nobody views. The cost moves to the first visitor, who waits for the resize.

Our numbers show both costs. Three widths in three formats made nine variants of one photo. They totalled 129,729 bytes, about 7.9 percent of the original. Making them took about 4.7 and 5.8 seconds on one thread in two runs.

AVIF was the smallest at every width, and it also scored a higher PSNR, a simple pixel-error measure, than WebP. Real photos will give different numbers. AVIF was also the slowest, taking about 4 to 12 times longer than WebP across both runs.

That slowness decides the design. A first 320-pixel WebP took 72 milliseconds in that run. A first 1280-pixel AVIF took nearly 2 seconds, which a waiting buyer feels. So precompute the hot sizes, especially AVIF, and let rare sizes fall back to on the fly.

How a precompute job usually starts

Object stores can announce new uploads. Amazon S3 event notifications, for example, can send a message to a queue, a topic, a function or an event bus when an object is created.

Two details from the S3 docs shape the worker. Notifications are usually delivered in seconds but can take a minute or longer. They are designed to be delivered at least once, so the same upload can arrive twice.

1. 1

   Store the original

   The upload API writes the original under a new, unique name and returns that name to the app.

2. 2

   Receive the event

   A worker reads the object-created message from the queue. Repeats are expected, so the job must be safe to run twice.

3. 3

   Make the hot variants

   Build the sizes and formats you always need, such as 320 and 640 pixels in AVIF and WebP.

4. 4

   Write each variant under its cache key

   Running twice then writes the same file to the same name, which is harmless.

5. 5

   Keep the on-the-fly path

   A buyer may open the listing before the event arrives. The request path must still be able to build the variant itself.

It reads the Accept header the browser sends with the image request and picks the best format named there. If no modern format is named, it sends JPEG, which every browser shows.

HTTP calls this proactive negotiation. RFC 9110, the HTTP semantics standard, lets Accept list media types, format names such as image/webp, with optional q weights from 0 to 1. A weight of 0 means not acceptable. A range such as image/* covers every image type.

The other way is to let the page choose. The picture element lists several sources with a type attribute, and the browser skips any type it cannot show. The browser decides, and each format gets its own URL, so caches need no extra rules.

What browsers send, and where the published list lags

MDN keeps a list of the Accept values browsers send for images. We ran each one through our format picker. The picker sends AVIF or WebP only when the browser names it explicitly.

Two results surprise people. MDN's listed Safari Accept value does not name AVIF, though MDN says Safari supports AVIF from version 16.1. That entry looks stale: a 2024 WebKit bug report shows Safari 18.2 naming image/avif, and our picker chose AVIF for it. So log the Accept headers your real visitors send before trusting any published list.

The second is image/*. By the standard, it covers AVIF too. But Safari before macOS Big Sur sends image/*, and MDN says WebP needs Big Sur. Treating image/* as permission to send AVIF would break those images.

A cache key is everything that must match before a cache reuses a stored response. For a variant, that is the original's version, the transform options and the chosen format.

RFC 9111, the HTTP caching standard, says the key is at least the request method and the URL. When a response carries Vary: Accept, a cache must also match the Accept header before reusing it.

Think of a library catalogue. A book's call number names the title, the edition and the copy type: hardback, paperback or ebook. Here the title is the original photo and the edition is its version. The copy type is the format.

Where the picture breaks: a library never prints a new copy when you ask for one it lacks. An on-the-fly pipeline does exactly that, which is why the key must also include every option that changes the pixels.

Why raw Vary: Accept splits the cache

A cache that applies Vary: Accept literally can store one copy per distinct Accept string. Our nine Accept strings would make nine stored copies of the same photo. Only three different files exist: AVIF, WebP and JPEG.

RFC 9110 itself lists this as a cost of proactive negotiation: it limits the reuse of responses in shared caches. The fix is to turn Accept into a small bucket before it reaches the key.

Our key function, shown in the server section below, sorts the options first. In our run, options given as fit then width and as width then fit produced the same key. Changing the original's version from v1 to v2 produced a new key.

What CDNs let you put in the key

CDN support differs, so check your plan. Cloudflare's default key is the full URL plus a few headers. As of 29 October 2026, adding custom headers to the key is an Enterprise feature there.

Cloudflare also has a feature called Vary for images, on Pro plans and above as of the same date. It parses Accept and caches each format variant separately. It needs Vary: Accept from your origin and a file extension in the URL path.

Amazon CloudFront's docs make the general trade-off plain: fewer values in the cache key means more cache hits. Without such a feature, put the format in the URL yourself, for example by using the picture element.

Keep originals in object storage under a name that never gets reused, and treat them as read-only. Variants are a cache you can rebuild; originals are the only copy of what the seller gave you.

Never overwrite an original in place. If a seller replaces a photo, store it under a new name or version. The cache key then changes, and no CDN keeps serving the old picture under the old address.

That rule is what makes long caching safe. Our server sends Cache-Control: public, max-age=31536000, immutable, which keeps a response fresh in caches for a year. RFC 8246 defines immutable as a promise that the response will not change during that time.

Keeping originals also pays off later. When a better encoder arrives, you can rebuild variants from the originals. You cannot rebuild quality from an old 320-pixel JPEG.

Because an unsigned resize URL lets anyone make your servers do unlimited work. They can change the width to 4000, then 4001, then 4002, and every new URL is a cache miss that runs a transform.

Open-source image servers say this in their docs. Thumbor's security page warns that a user can ask for other sizes in loops. imgproxy says signing stops attackers from launching a denial-of-service attack by requesting many different resizes.

The fix is to sign the path. Your app server, which knows the secret key, computes an HMAC of the transform path and puts it in the URL. The image server recomputes it and refuses any URL where the two differ.

// Signs and verifies transform paths with HMAC-SHA256 (Node.js crypto only).
const crypto = require('crypto');

function sign(path, key) {
  return crypto.createHmac('sha256', key).update(path).digest('base64url');
}

function verify(url, key) {
  const slash = url.indexOf('/', 1);
  const given = Buffer.from(url.slice(1, slash));
  const path = url.slice(slash);
  const expected = Buffer.from(sign(path, key));
  if (given.length !== expected.length) return null;
  return crypto.timingSafeEqual(given, expected) ? path : null;
}

module.exports = { sign, verify };

We signed the path for a 640-pixel listing photo with a fake test key, then changed only the width to 4000. The original URL verified. The tampered URL, a wrong key and a too-short signature all returned null, meaning rejected.

The comparison uses crypto.timingSafeEqual, which Node.js documents as a constant-time compare suited to HMAC digests. It throws if the two inputs differ in length, which our run confirmed. That is why verify checks the length first.

How real servers do it

imgproxy computes HMAC-SHA256 over a salt plus the path and encodes it as URL-safe Base64, a text form of bytes that is safe inside URLs. Its key and salt are hex-encoded, and signature checking is off until you set them. Thumbor documents HMAC with SHA1.

A signature only proves your app made the URL. If your app signs any width a user asks for, the attack is back. Keep a short allowlist of widths too. In our test, a correctly signed 999-pixel URL still got 400 Bad Request.

Join the pieces in this order: verify the signature, check the size, pick the format, build the key, then serve or transform. Our test server did this in about 50 lines of Node.js with sharp 0.35.5.

The format picker is the piece most teams get wrong, so here it is in full. It trusts only formats the browser names.

// Picks avif, webp or jpeg from an Accept header.
// A modern format is used only when the browser names it explicitly with a q weight above 0.
function pickFormat(accept) {
  const named = {};
  for (const part of String(accept || '').toLowerCase().split(',')) {
    const [type, ...params] = part.split(';').map(function (s) { return s.trim(); });
    const q = params.find(function (p) { return p.startsWith('q='); });
    named[type] = q ? parseFloat(q.slice(2)) : 1;
  }
  if (named['image/avif']) return 'avif';
  if (named['image/webp']) return 'webp';
  return 'jpeg';
}

module.exports = { pickFormat };

The cache key function sorts the options, joins them with the original's name and version and the format, then hashes the result into a short name.

// Builds one cache key per output file: original version + sorted options + chosen format.
const crypto = require('crypto');

function cacheKey(source, options, format) {
  const opts = Object.keys(options).sort()
    .map(function (k) { return k + '=' + options[k]; }).join(',');
  const raw = source + '|' + opts + '|fmt=' + format;
  const hash = crypto.createHash('sha256').update(raw).digest('hex').slice(0, 16);
  return { raw, hash };
}

module.exports = { cacheKey };

1. 1

   Verify

   Call verify on the request URL. A null result means 403 Forbidden and no work at all.

2. 2

   Check the size

   Parse the width from the path and reject anything outside 320, 640 and 1280 with 400 Bad Request.

3. 3

   Negotiate

   Run pickFormat on the Accept header to get avif, webp or jpeg.

4. 4

   Build the key

   Call cacheKey with the original's name and version, the width and the format.

5. 5

   Serve or transform

   On a hit, return the stored bytes. On a miss, read the original, resize without enlarging, encode, store, then return.

6. 6

   Send the right headers

   Set Content-Type to the chosen format, Vary: Accept, and the one-year immutable Cache-Control.

We called the running server with Accept values from MDN's list. Each format used sharp's own default quality: 50 for AVIF and 80 for WebP and JPEG.

What just happened? One URL produced three different files, one per format bucket, and each was cached under its own key. Both bad URLs were refused before any image work ran.

To check your own server, request the same signed URL twice with curl and the same Accept header. The first response should show a miss, and the second a hit with identical bytes. Send no Accept header and Content-Type should read image/jpeg.

Without protection, every waiting request runs its own transform of the same variant. This is a cache stampede: a popular listing goes live and many buyers ask for the same new size together.

Pause and predict: 20 buyers ask for the same new size in the same instant, with no protection. How many resizes run?

We fired 20 identical requests at once for a size not yet stored. With a plain check-then-transform, all 20 ran the resize. With an in-flight map, which holds a handle to each transform already running, only one ran.

All 20 requests in the second run got 200 with the same bytes. They simply waited on the first transform. This pattern is often called single flight or request coalescing.

An in-memory map only protects one server process. With many image servers, a stampede can still run one transform per server. Precomputing hot sizes, or a shared lock in a store such as Redis, closes that gap.

The SevyDevy guide on caching from first principles covers stampedes in general. For images the stakes are higher than usual, because one AVIF miss took up to about 2.7 seconds of processing time in our runs.

Four defaults in sharp 0.35.5 changed our outputs. Pillow 13 has a similar pixel limit. Each one is documented, but easy to miss until a seller complains.

The same quality number means different things per format

We encoded the 640-pixel photo at quality 75 in both formats. WebP gave 5,904 bytes and a PSNR of 39.0 dB. AVIF gave 7,335 bytes and 44.5 dB, so it was bigger and sharper.

sharp's own defaults differ for this reason: 50 for AVIF, 80 for WebP and JPEG. Keep one quality setting per format in your server, and never pass one number to all three.

Phone photos turn sideways

Many photos carry an EXIF orientation tag. EXIF is the metadata block cameras write, and this tag tells viewers to rotate the stored pixels before display. sharp's docs say all metadata is removed by default, including that tag.

We resized a 400 by 200 test image tagged to rotate 90 degrees. By default the result was 100 by 50 with the tag gone, so it would display sideways. With autoOrient set to true, it came out 100 by 200, upright.

libvips' own thumbnail operation rotates by default, so defaults differ even between tools built on libvips. Test with a real tagged phone photo before launch.

Small originals get blown up

Asking for 1280 pixels from a 500-pixel original produced an upscaled 1280-pixel file with no extra detail by default. With withoutEnlargement set to true, sharp kept it at 500 pixels.

Tiny files can hide huge images

A decompression bomb is a small file that expands into an enormous image when decoded. Our 20000 by 20000 PNG was only 1.2 MB, yet it held 400 million pixels.

Pillow only warns between the limit and twice the limit, and raises an error above that. Keep these limits on, check pixel size at upload, and run transforms with a timeout. A separate SevyDevy guide explains the Pillow 13 security update for upload handling.

Walk from the upload to the buyer's screen, and name one trade-off at each step. The reasons behind each choice matter more than library names.

1. 1

   Clarify the scale and sizes

   Ask how many uploads a day, which display sizes exist, and which browsers matter. These decide precompute versus on the fly.

2. 2

   Store originals once

   Object storage, unique names, never overwritten, with backups. Variants live in a separate bucket and can be rebuilt.

3. 3

   Choose the variant strategy

   Precompute hot sizes from upload events, which can repeat, and make rare sizes on the fly behind a cache.

4. 4

   Negotiate the format

   Bucket Accept into AVIF, WebP or JPEG, send Vary: Accept, or let the picture element choose by URL.

5. 5

   Design the cache key

   Original version, sorted transform options and format bucket, served with long immutable caching.

6. 6

   Stop abuse

   HMAC-signed transform URLs, a size allowlist, pixel limits and timeouts on every transform.

7. 7

   Handle the stampede

   Coalesce identical misses in each server and precompute the sizes every listing needs.