Implementing ETags in REST APIs: Caching and Concurrency, Step by Step

Why ETags Matter in REST APIs

ETags are entity tags: small validators the server attaches to a representation so clients can ask “has this changed?” without re‑downloading it. Used well, ETags:

• Cut bandwidth and latency with conditional GETs returning 304 Not Modified.
• Prevent lost updates with optimistic concurrency using If-Match and 412 Precondition Failed.
• Play nicely with CDNs and reverse proxies for efficient cache validation.

This tutorial shows how ETags work, how to design them, and how to implement them in popular stacks (Node/Express, FastAPI, Spring Boot), with curl tests and deployment tips.

Core HTTP Concepts (Quick Ref)

• ETag: An opaque validator string for a specific representation, usually quoted, e.g., "686897696a7c876b7e".
• Strong vs weak:
  • Strong: byte‑for‑byte identical representation; no W/ prefix.
  • Weak: semantically equivalent but not byte‑identical; prefix with W/.
• Client conditional headers:
  • If-None-Match: “Send me the resource only if its ETag doesn’t match these.” Typical for GET to get 304.
  • If-Match: “Only perform this write if the resource’s ETag matches these.” Typical for PUT/PATCH/DELETE to avoid overwrites.
• Typical responses:
  • 304 Not Modified for cache validation hits (no body).
  • 412 Precondition Failed when a precondition (e.g., If-Match) fails.

ETags are validators (revalidation) and complement freshness (Cache-Control, Expires). You can (and often should) return both an ETag and Last-Modified for broader client support.

Designing Your ETag Strategy

Choose one of these patterns based on your data model and infrastructure:

1. Content hash (strong ETag)

   • Compute a hash (e.g., SHA‑256) of the exact response bytes. Changes in whitespace, encoding, or formatting change the ETag.
   • Pros: Precise. Cons: Must recompute after each change and per variant/encoding.

2. Version/rowversion (strong or weak)

   • Use a stable server‑maintained version field (e.g., DB rowversion, updated_at + primary key).
   • Common format: Etag = hash(concat(version,id)). If your representation may vary by formatting, use weak: W/"v123".

3. Canonical representation hash (strong)

   • Serialize the resource to a canonical form (e.g., stable JSON ordering, normalized whitespace, chosen charset) and hash that.
   • Balances precision with stability across platforms.

4. Aggregates/collections

   • Hash a compact manifest: highest version + count + IDs hash. Use weak ETags for lists that change order or include transient fields.

Guidelines:

• Treat ETags as opaque to clients. Never encode sensitive info or guessable IDs.
• Quote your ETag values: ETag: "...". Use W/ prefix for weak: ETag: W/"...".
• If intermediaries compress responses, either:
  • Compute ETags on the final encoded bytes and send Vary: Accept-Encoding, or
  • Use weak ETags to avoid strong‑validator mismatches across encodings.

Conditional Request Flows

• Cache validation (GET):

  1. Client: GET /items/42
  2. Server: 200 OK, body, ETag: "abc123".
  3. Client later: GET /items/42 with If-None-Match: "abc123".
  4. Server unchanged: 304 Not Modified (no body).

• Optimistic concurrency (PUT/PATCH/DELETE):

  1. Client fetches resource and its ETag: "abc123".
  2. Client updates with If-Match: "abc123".
  3. If server’s current ETag is different (someone else updated), return 412 Precondition Failed.

Special cases:

• If-None-Match: * means “only if resource does not exist” (create‑if‑absent).
• If-Match: * means “only if resource exists”.

Implementation: Node.js (Express)

Below is a minimal pattern showing both validation (GET) and concurrency (PUT). It uses:

• A canonical JSON string for strong ETags.
• A simple in‑memory store for the example (swap with your DB).

const express = require('express');
const crypto = require('crypto');
const app = express();
app.use(express.json());

const store = new Map();
// Seed item
store.set('42', { id: '42', name: 'Flux Capacitor', qty: 1, updatedAt: 0 });

function canonicalJson(obj) {
  // Deterministic key order
  const keys = Object.keys(obj).sort();
  const entries = keys.map(k => [k, obj[k]]);
  return JSON.stringify(Object.fromEntries(entries));
}

function makeStrongEtag(repr) {
  const hash = crypto.createHash('sha256').update(repr).digest('base64url');
  return '"' + hash + '"';
}

app.get('/items/:id', (req, res) => {
  const it = store.get(req.params.id);
  if (!it) return res.sendStatus(404);

  const body = canonicalJson(it);
  const etag = makeStrongEtag(body);

  // Conditional GET using If-None-Match
  const inm = req.headers['if-none-match'];
  if (inm && inm === etag) return res.status(304).set('ETag', etag).end();

  res.set('ETag', etag).type('application/json').send(body);
});

app.put('/items/:id', (req, res) => {
  const id = req.params.id;
  const current = store.get(id);
  if (!current) return res.sendStatus(404);

  const currentEtag = makeStrongEtag(canonicalJson(current));
  const ifMatch = req.headers['if-match'];
  if (!ifMatch || ifMatch !== currentEtag) {
    return res.status(412).set('ETag', currentEtag).json({ error: 'Precondition failed' });
  }

  const next = { ...current, ...req.body, updatedAt: Date.now() };
  store.set(id, next);
  const body = canonicalJson(next);
  const etag = makeStrongEtag(body);
  res.set('ETag', etag).type('application/json').send(body);
});

app.listen(3000, () => console.log('ETag demo on :3000'));

Notes:

• Always echo the current ETag in 412 responses so clients can refetch and retry.
• Add Cache-Control according to your freshness policy (e.g., Cache-Control: private, max-age=0, must-revalidate).

Implementation: Python (FastAPI)

from fastapi import FastAPI, Request, Response, HTTPException
from hashlib import sha256
import json, time

app = FastAPI()

store = {"42": {"id": "42", "name": "Flux Capacitor", "qty": 1, "updatedAt": 0}}

def canonical(obj):
    return json.dumps(obj, separators=(",", ":"), sort_keys=True)

def etag_of(body: str) -> str:
    return '"' + sha256(body.encode("utf-8")).hexdigest() + '"'

@app.get("/items/{item_id}")
async def get_item(item_id: str, request: Request):
    it = store.get(item_id)
    if not it:
        raise HTTPException(status_code=404)
    body = canonical(it)
    etag = etag_of(body)
    inm = request.headers.get("if-none-match")
    if inm and inm == etag:
        return Response(status_code=304, headers={"ETag": etag})
    return Response(content=body, media_type="application/json", headers={"ETag": etag})

@app.put("/items/{item_id}")
async def put_item(item_id: str, request: Request):
    it = store.get(item_id)
    if not it:
        raise HTTPException(status_code=404)
    current_etag = etag_of(canonical(it))
    if_match = request.headers.get("if-match")
    if not if_match or if_match != current_etag:
        raise HTTPException(status_code=412, headers={"ETag": current_etag})
    patch = await request.json()
    it.update(patch)
    it["updatedAt"] = int(time.time())
    body = canonical(it)
    return Response(content=body, media_type="application/json", headers={"ETag": etag_of(body)})

Implementation: Java (Spring Boot)

Spring provides helpers for conditional requests. Two handy options:

• ShallowEtagHeaderFilter: computes a weak ETag over the response body automatically.
• Manual control using WebRequest.checkNotModified and ResponseEntity.eTag(...) for strong/weak as you choose.

Example with manual ETag and conditional handling:

@RestController
@RequestMapping("/items")
public class ItemController {

  private final ItemRepo repo; // your persistence layer

  @GetMapping("/{id}")
  public ResponseEntity<String> get(@PathVariable String id, WebRequest request) {
    var it = repo.find(id).orElse(null);
    if (it == null) return ResponseEntity.status(404).build();

    String body = canonicalJson(it); // implement stable ordering
    String etag = '"' + sha256(body) + '"';

    if (request.checkNotModified(etag)) {
      return null; // Spring will return 304 with ETag
    }
    return ResponseEntity.ok()
        .eTag(etag)
        .contentType(MediaType.APPLICATION_JSON)
        .body(body);
  }

  @PutMapping("/{id}")
  public ResponseEntity<String> put(@PathVariable String id, @RequestBody Map<String,Object> patch, @RequestHeader(value = "If-Match", required = false) String ifMatch) {
    var it = repo.find(id).orElse(null);
    if (it == null) return ResponseEntity.status(404).build();

    String currentBody = canonicalJson(it);
    String currentEtag = '"' + sha256(currentBody) + '"';

    if (ifMatch == null || !ifMatch.equals(currentEtag)) {
      return ResponseEntity.status(412).eTag(currentEtag).build();
    }

    it.applyPatch(patch);
    repo.save(it);

    String body = canonicalJson(it);
    String etag = '"' + sha256(body) + '"';

    return ResponseEntity.ok().eTag(etag).contentType(MediaType.APPLICATION_JSON).body(body);
  }
}

To enable ShallowEtagHeaderFilter globally (weak ETags), register it as a @Bean in your configuration; Spring will add ETag: W/"..." automatically for responses.

Adding Freshness: Cache-Control That Works

ETags validate; Cache-Control governs how long clients can reuse cached data without asking:

• For user‑specific resources: Cache-Control: private, max-age=0, must-revalidate.
• For public cacheable resources (e.g., product catalog snapshots): Cache-Control: public, max-age=60, stale-while-revalidate=30, stale-if-error=86400.
• For CDNs, add s-maxage distinct from max-age to control intermediary behavior.

Always pair with Vary when the representation changes by header (e.g., Vary: Accept, Accept-Encoding, Authorization?—but be careful: varying on Authorization can explode cache keys; many CDNs treat authenticated responses as uncacheable).

Compression and Strong Validators

A strong ETag must change if the representation changes, including content codings like gzip. Options:

• Compute ETags on the post‑compression bytes per Accept-Encoding and send Vary: Accept-Encoding.
• Or switch to weak ETags (W/"...") when intermediaries might transform content.

If you terminate compression at the edge (CDN), let the CDN handle ETags and forward them transparently, or pin servers to generate content‑encoded bodies consistently.

Collections and Pagination

For GET /items?page=2&size=50, build the ETag from a compact manifest:

• Highest item version in the page
• Count
• Stable page key (e.g., hash(query))

Use weak ETags for lists likely to change order or include transient metadata.

Testing With curl

• First fetch:

curl -i http://localhost:3000/items/42

• Validate with If-None-Match (replace with the ETag you received):

curl -i http://localhost:3000/items/42 \
  -H 'If-None-Match: "abc123"'

Expected: 304 Not Modified with no body.

• Optimistic PUT (success):

curl -i -X PUT http://localhost:3000/items/42 \
  -H 'Content-Type: application/json' \
  -H 'If-Match: "abc123"' \
  --data '{"qty":2}'

• Optimistic PUT (fail due to mismatch):

curl -i -X PUT http://localhost:3000/items/42 \
  -H 'Content-Type: application/json' \
  -H 'If-Match: "stale"' \
  --data '{"qty":3}'

Expected: 412 Precondition Failed and current ETag in headers.

Deployment and Ops Considerations

• Multi‑node consistency: Ensure all nodes compute the same ETag for the same resource. Prefer canonical serialization + stable hashing or store the ETag/version alongside the record.
• Reverse proxies/CDNs: Forward ETag, If-None-Match, and If-Match. Configure Cache-Control and Vary carefully.
• Partial responses: If you support Range, a strong ETag should still validate the complete representation. Send Accept-Ranges: bytes accordingly.
• HEAD requests: Serve HEAD identically to GET (same headers including ETag) but without a body.
• Logging/metrics: Track 304 and 412 rates to validate cache effectiveness and concurrency protection.

Common Pitfalls (and Fixes)

• Hashing non‑canonical JSON so key order/spacing flips ETags unnecessarily → Use canonical JSON or model/localized formatting.
• Including dynamic, non‑semantic fields in the hash (e.g., generatedAt) → Exclude or normalize before hashing.
• Strong ETags with downstream compression causing mismatches → Use weak ETags or compute on encoded bytes; add Vary: Accept-Encoding.
• Using timestamps alone as validators → Risk of clock skew and granularity issues; combine with content or monotonic versions.
• Leaking internals: ETags that reveal row IDs or predictable versions → Use opaque hashes.
• Ignoring If-Match on writes → Clients can overwrite each other’s changes. Enforce optimistic concurrency.

Minimal Checklist

• Choose ETag type (strong vs weak) per endpoint.
• Implement canonicalization and hashing or a stable version field.
• Return ETag on 200/201/204 (and even 404 when useful).
• Honor If-None-Match for GET/HEAD and return 304 when matched.
• Enforce If-Match for PUT/PATCH/DELETE; return 412 on mismatch.
• Add sensible Cache-Control and Vary headers.
• Test with curl and monitor in production.

Wrapping Up

ETags are simple but powerful. By separating freshness (Cache-Control) from validation (ETag) and embracing conditional requests, you trim bandwidth, speed up clients, and gain safe, optimistic concurrency. Start with a canonical representation + strong ETag for single resources, weak ETags for collections, and expand across your API with confidence.