Designing REST APIs with Partial Response Field Selection (fields, select, $select)

Overview

Partial response field selection lets API clients specify exactly which properties they want returned, reducing payload size, latency, and server work. Instead of always sending full resources, an API can honor a query like ?fields=id,name (or ?select=id,name) and return only those fields. Google’s APIs, JSON:API, and OData popularized this capability under slightly different names (fields, sparse fieldsets, $select), but the goal is the same: stop overfetching.

This article explains why partial responses matter, common syntax choices, HTTP and caching implications, implementation patterns across databases and ORMs, security considerations, and concrete examples.

Why partial responses matter

• Performance and cost: Smaller JSON means fewer bytes on the wire, faster TLS handshakes completion, and lower egress bills—especially at scale and on mobile networks.
• Latency: Less server-side serialization and fewer database columns projected can shave milliseconds from critical paths.
• Battery and bandwidth: Mobile and IoT clients benefit disproportionately from smaller responses.
• Product agility: Teams can add new optional fields without bloating all client responses.

Design goals

• Predictable: The same query should always produce the same shape given the same resource version.
• Safe by default: Never leak sensitive fields; enforce allow-lists and authorization per-field.
• Cacheable: Play nicely with HTTP caches and downstream CDNs.
• Composable: Work with other query capabilities like pagination, filtering, sorting, and expansion of related resources.

Popular syntaxes

There’s no single standard, but four well-trodden patterns exist:

1. fields (Google-style)

• Query param: fields
• Dot-notation for nesting, parentheses for sub-selections
• Examples:
  • /v1/users/123?fields=id,name
  • /v1/users/123?fields=id,name,profile(photo/url,height,width)

2. JSON:API sparse fieldsets

• Per-type param: fields[users]=id,name
• For compound documents with include=
• Examples:
  • /users/123?fields[users]=id,name
  • /articles?include=author&fields[articles]=title&fields[people]=name

3. OData $select

• Param: $select
• Examples:
  • /Users(123)?$select=Id,DisplayName
  • /Orders?$select=Id,Total&$expand=Customer($select=Id,Name)

4. Simple select

• Param: select
• Examples:
  • /users/123?select=id,name

Pick one pattern and stick to it across your API portfolio. If you already expose JSON:API or OData semantics, reuse those instead of inventing new ones.

Semantics and rules of thumb

• Defaults: Return a compact, stable default field set when fields is absent. Include identifiers and links (e.g., id, self) for navigability.
• Unknown fields: Reject with 400 and an error describing the unknown field(s). Silent ignores make debugging hard.
• Nested selection: Support nested paths only where relationships are legitimately embed-able. Example: ?fields=id,name,profile(photo/url).
• No wildcards by default: Avoid * unless you can enforce strong allow-lists and version stability.
• Aliasing: Skip output aliases unless you have a strong use case; they complicate caching and documentation.
• Consistent casing: Use the same field casing strategy as your resource model (snake_case or camelCase) to avoid confusion.

Working with relationships: include vs expand

• include: Pull related resources as full, separate objects (e.g., JSON:API include=). Pair with sparse fieldsets per type.
• expand: Inline a related object inside the parent (e.g., ?expand=owner&fields=owner(id,name)).

Choose one style. If you offer both, define precedence: for example, expand controls embedding while fields controls which properties of the embedded object appear.

HTTP and caching considerations

• Cache keys: Vary the cache key by the fields parameter so different projections don’t collide. Set Vary: fields in responses if you rely on shared caches.
• ETags: Compute ETags on the final serialized body. Different field sets must yield different ETags.
• Compression: Smaller payloads still benefit from gzip/br, but partial responses compound the savings.
• Pagination: Always return pagination metadata (e.g., next, prev, total if applicable) regardless of fields. Consider these part of the envelope, not fields.
• Rate limiting: If limits count bytes or response weight, partial responses reduce client cost—document how.

Security and governance

• Allow-list only: Maintain a per-resource allow-list of exposable fields. Never pass client-specified field names directly to serialization without checking.
• Field-level authorization: Some fields may require scopes/roles (e.g., email, internalNotes). Deny with 403 for unauthorized fields, or omit with a warning depending on policy.
• PII and sensitive defaults: Sensitive fields must never be in the default set. Make clients explicitly ask, and require scopes.
• Audit: Log the requested fields along with principal and endpoint for forensic analysis.

Database and ORM strategies

Goal: Align projection at the storage layer with the selected fields to avoid fetching and serializing unused data.

• SQL projection: Build a SELECT list of columns from the field tree. Be mindful of ORM conveniences that still fetch all columns unless configured.
  • Postgres/SQL examples: SELECT id, name FROM users WHERE id = $1;
  • ORM hints:
    • Django: .only('id', 'name') or .defer(...)
    • SQLAlchemy: load_only(User.id, User.name)
    • Hibernate/JPA: projections with Criteria or DTO queries
• NoSQL projection:
  • MongoDB: { projection: { id: 1, name: 1 } }
  • Document stores benefit greatly from projections due to large blobs.
• N+1 risk with nested fields: Limit inline expansions or batch load relations (JOINs, prefetch_related, select_related).
• Computed fields: Cache or precompute heavy fields; require explicit opt-in.

Building a field tree (implementation sketch)

1. Parse: Convert fields=id,name,profile(photo/url) into a normalized tree:

{
  id: true,
  name: true,
  profile: { photo: { url: true } }
}

2. Validate: Walk the schema and compare to allow-lists and authz rules.
3. Plan: Produce a projection plan per persistence layer (columns, joins, aggregations).
4. Serialize: Use the same tree to drive selective serialization.

Error handling and DX

• 400 Bad Request: Unknown or malformed fields. Include a machine-readable error code and a details.unknownFields array.
• 403 Forbidden: Field requested but not permitted for the caller.
• 413 Payload Too Large: If you cap field count or nested depth, explain the limits.
• Hints: Return Accept-Fields or a link to schema docs to aid discovery.

Example error payload:

{
  "error": {
    "code": "UNKNOWN_FIELDS",
    "message": "One or more requested fields are not available.",
    "details": { "unknownFields": ["foo", "profile.secret"] }
  }
}

OpenAPI documentation

• Document the fields parameter as a string with examples and a link to your field grammar.
• Provide schema component snippets showing which properties are selectable and which require scopes.
• Use x-* vendor extensions or description blocks to enumerate allowed fields.

Example (OpenAPI 3.1 excerpt):

parameters:
  - in: query
    name: fields
    schema:
      type: string
      example: id,name,profile(photo/url)
    description: Comma-separated list of fields to include. Use dot-notation for nesting and parentheses for sub-selections.

End-to-end examples

cURL

# Basic projection
curl \
  -H "Authorization: Bearer <token>" \
  "/v1/users/123?fields=id,name"

# Nested selection with expansion
curl \
  "/v1/users/123?expand=profile.photo&fields=id,name,profile(photo/url,width,height)"

JavaScript (Node + fetch)

const res = await fetch('/v1/users/123?fields=id,name,profile(photo/url)', {
  headers: { Authorization: `Bearer ${token}` }
});
const user = await res.json();

Python (requests)

import requests
r = requests.get(
    'https://api.example.com/v1/users/123',
    params={'fields': 'id,name,profile(photo/url)'},
    headers={'Authorization': f'Bearer {token}'}
)
print(r.json())

SQL-oriented projection (pseudo)

SELECT id, name, profile_photo_url AS "profile.photo.url"
FROM users
LEFT JOIN photos ON photos.id = users.profile_photo_id
WHERE users.id = $1;

MongoDB projection

db.users.find({ _id: ObjectId("...") }, { id: 1, name: 1, "profile.photo.url": 1 })

Compatibility with existing conventions

• Google APIs (fields): Rich syntax with parentheses and wildcards; widely used in Google Cloud and YouTube Data APIs.
• JSON:API (sparse fieldsets): Best when your API already follows JSON:API; pairs with include.
• OData ($select, $expand): Strong, standard vocabulary for enterprise scenarios.
• Facebook/Meta Graph API (fields): Similar to Google’s approach; often combined with edges and paging.

Choose the dialect that matches your audience and tooling. Where clients already use OData or JSON:API, adopting their semantics reduces friction.

GraphQL comparison

GraphQL’s query shape inherently specifies exact fields, which eliminates overfetching by design. Partial responses in REST mimic that benefit while retaining HTTP semantics, CDN friendliness, and simpler infrastructure. If you already run REST at scale, partial responses offer a pragmatic middle ground without adopting a new server runtime.

Testing and observability

• Contract tests: For each endpoint, verify that requesting a field set returns exactly and only those fields.
• Golden files: Snapshot minimal and typical projections; protect them with CI.
• Metrics: Emit counters for requested field count, depth, and unknown field rejections. Use them to prune dead fields.
• Tracing: Annotate spans with selected fields to tie payload size to performance.

Migration and versioning

• Start small: Ship a compact default (e.g., id, name, self).
• Opt-in growth: Add new fields behind explicit selection.
• Deprecation: Warn (via headers or log events) when clients request soon-to-be-removed fields; remove across a version boundary.
• Backward compatibility: Ensure old clients that omit fields still receive a stable default.

Implementation checklist

• Grammar: Choose syntax (fields, select, $select, or JSON:API) and document it.
• Validation: Parse → normalize → allow-list → authorize → cap size/depth.
• Projection: Compile DB/ORM projection and relation prefetch plan.
• Serialization: Drive output strictly from the validated field tree.
• HTTP: Set Vary: fields, compute field-aware ETags, keep pagination metadata.
• Security: Enforce scopes, never expose sensitive defaults, log field requests.
• Docs and SDKs: Provide examples and helper utilities for building queries.

Conclusion

Partial response field selection is a low-risk, high-reward enhancement to REST APIs. It trims bytes, improves latency, and gives clients control—all while preserving the simplicity and operability of HTTP. By adopting a consistent syntax, enforcing allow-lists and authorization, and aligning storage projections with requested fields, you can deliver a faster, safer API without a rewrite.