Content Negotiation for Consolidated Machine-Readable Representations

1.1. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

This document uses the terms "resource", "representation", "content negotiation", "client", and "server" as defined by HTTP semantics [RFC9110].¶

A machine-readable representation is a representation whose format and structure are intended primarily for automated processing rather than direct human presentation. Examples include Markdown, JSON, XML, CSV, and other structured or lightly structured formats.¶

A consolidated representation is a representation selected in response to Prefer: return=consolidated. It is intended to present a resource in a form more suitable for the requesting application, for example by omitting presentation chrome, restructuring information, combining related material, or providing additional context. The term describes the requested representation form; it does not imply a particular media type, or accuracy, completeness, equivalence, neutrality, or trustworthiness.¶

A navigational HTML representation is an HTML representation intended primarily for human navigation in a web browser. It typically includes navigation, layout, styling hooks, scripts, advertising, related links, and other material used for presentation or site traversal.¶

An automated client is an HTTP client that retrieves representations for automated processing, whether directly or on behalf of another system. Examples include crawlers, search systems, retrieval systems, monitoring tools, and agent infrastructure.¶

A publisher is the party responsible for making a resource and its representations available. In this document, the term is used informally and may refer to a site operator, origin server operator, content provider, or other party controlling publication of the representation.¶

2.1. Accept Header

Clients indicate preferred media type using the Accept header. The examples below prefer Markdown or JSON over HTML, reflecting the primary use case for consolidated machine-readable representations:¶

Accept: text/markdown;q=0.9, text/html;q=0.8

or¶

Accept: application/json;q=0.9, text/html;q=0.8

Clients MAY specify multiple formats with appropriate quality values.¶

2.2. Prefer Header

Clients indicate desire for consolidated content using the Prefer header with the return preference [RFC7240]:¶

Prefer: return=consolidated

2.3. Combined Request

A complete request combines both headers:¶

GET /documentation HTTP/1.1
Host: example.org
Accept: text/markdown;q=0.9, text/html;q=0.8
Prefer: return=consolidated

3.1. Honouring Preferences

Servers receiving requests with Prefer: return=consolidated SHOULD provide consolidated representations when practical. Servers that honour the preference MUST include Preference-Applied in the response:¶

HTTP/1.1 200 OK
Content-Type: text/markdown
Preference-Applied: return=consolidated
ETag: "consolidated-v1-a3f8b2"
Vary: Accept, Prefer

The Vary header MUST include both Accept and Prefer to ensure proper caching behaviour by intermediaries (proxies, CDNs). Other headers (such as Accept-Encoding) MAY also appear in Vary as appropriate. Without appropriate Vary headers, caches may incorrectly serve consolidated representations to clients that did not request them, or vice versa.¶

3.2. Content Structure

Consolidated representations SHOULD differ in structure and organisation from their navigational HTML counterparts. Servers SHOULD:¶

• Consolidate related content from multiple pages into hierarchical sections¶

• Organise information by semantic relationships rather than navigation structure¶

• Include appropriate context for understanding without navigation chrome¶

• Preserve information fidelity while restructuring for machine consumption¶

• Focus on coherent topics rather than consolidating entire sites¶

Very small sites MAY consolidate all content into a single resource. Larger sites SHOULD create multiple focused consolidated resources, each addressing a specific topic or information need. Content that is not directly relevant to understanding the primary topic SHOULD be excluded.¶

3.3. Discovery

Publishers MAY advertise the availability of alternate representations using HTML <link> elements in the same manner as feed discovery:¶

<link rel="alternate" type="text/markdown"
      href="/products/catalogue"
      title="Product Catalogue (Markdown)">
<link rel="alternate" type="application/ld+json"
      href="/products/catalogue"
      title="Product Catalogue (Linked Data)">

This allows automated agents to discover available formats without relying solely on content negotiation. The href attribute points to the resource URL, and the type attribute indicates the available media type. These alternate representations may support consolidation via the Prefer header, or may simply be format conversions - the <link> element advertises format availability either way. Publishers providing alternate representations for multiple resources MAY include multiple sets of <link> elements.¶

Representations MAY also advertise alternate formats of the same resource within themselves. For example, a markdown representation might include in its frontmatter:¶

alternate-formats:
  - type: application/ld+json
    title: Product Catalogue (Linked Data)
  - type: text/html
    title: Product Catalogue (HTML)

This enables format discovery for agents that bypass HTML parsing and directly request machine-readable representations. The URL for alternate formats is the same as the current resource.¶

3.4. When Consolidation is Not Practical

Servers MAY decline to provide consolidated representations by serving the standard representation without the Preference-Applied header.¶

6.1. Consolidated Representation Trust

Clients MUST treat consolidated representations as untrusted input. A consolidated representation is a publisher-authored representation of a resource, not an independent assessment of the resource's quality, relevance, authority, neutrality, or completeness.¶

Consolidated representations are expected to differ from navigational HTML. They SHOULD omit navigation, advertising, decorative material, repeated boilerplate, and other content intended primarily for human presentation. They SHOULD also restructure, summarise, combine, and emphasise information in ways intended to make the resource easier for automated systems to process. These differences are the purpose of consolidation.¶

Clients MUST initially treat any response selected for automated consumption as potentially hostile. They MUST NOT treat the presence of Preference-Applied: return=consolidated, or the existence of a machine-readable representation, as a trust or quality signal.¶

Clients SHOULD apply the same or stronger provenance, corroboration, abuse-detection, prompt-injection, and content-integrity controls that they apply to equivalent content obtained from navigational HTML or other sources.¶

6.2. Content Integrity

While this specification does not introduce new attack vectors, consolidated representations may amplify the impact of existing content integrity issues. A single poisoned consolidated resource in a cache could affect more automated systems than poisoning individual pages, and similarly, malicious content from origin server compromise may have broader impact when consolidated. Publishers should re-evaluate their content integrity measures.¶

8.1. Normative References

[RFC9110]

Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110, June 2022, <https://www.rfc-editor.org/rfc/rfc9110>.

[RFC9111]

Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., "HTTP Caching", STD 98, RFC 9111, DOI 10.17487/RFC9111, June 2022, <https://www.rfc-editor.org/rfc/rfc9111>.

[RFC7240]

Snell, J., "Prefer Header for HTTP", RFC 7240, DOI 10.17487/RFC7240, June 2014, <https://www.rfc-editor.org/rfc/rfc7240>.

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/rfc/rfc2119>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

8.2. Informative References

[RFC9113]

Thomson, M., Ed. and C. Benfield, Ed., "HTTP/2", RFC 9113, DOI 10.17487/RFC9113, June 2022, <https://www.rfc-editor.org/rfc/rfc9113>.

[RFC9114]

Bishop, M., Ed., "HTTP/3", RFC 9114, DOI 10.17487/RFC9114, June 2022, <https://www.rfc-editor.org/rfc/rfc9114>.

A.1. Why This Matters

Automated systems increasingly access web content: search engines, AI training systems, research tools, and monitoring services. These systems must fetch multiple pages to assemble complete information, repeatedly crawl sites to detect changes, and employ heuristics to distinguish content from presentational markup. This inefficiency costs money: publishers serve largely redundant traffic, consumers fetch and process bloated responses, and both produce suboptimal results.¶

This specification provides a mechanism for publishers to serve consolidated, machine-optimised representations directly. Publishers reduce costs and load, and consumers improve information quality and efficiency.¶

A.2. Beyond Simple Conversion

While format conversion alone reduces bandwidth and parsing overhead, consolidated representations provide something more valuable: a direct way to communicate relevance and context to automated systems. Rather than forcing machines to scrape multiple pages and infer relationships, consolidated representations explicitly state what information belongs together (see Example Transformation below for concrete illustrations).¶

By consolidating related information, publishers guide automated systems to the complete picture while implicitly indicating, by omission, what is not relevant. This benefits both parties: machines get better information with less work, publishers reduce load while maintaining control over how their content is understood.¶

A.3. Quantifiable Benefits

Based on typical modern web architectures, implementing consolidated representations can yield substantial operational improvements. The figures in this section are illustrative estimates intended to convey likely magnitudes of effect, not results of formal benchmarking.¶

Server load can, in many cases, drop by on the order of 70-90% per information retrieval session, as a single consolidated resource replaces five to twenty individual page fetches. This reduction cascades through the infrastructure: fewer database queries, less server-side rendering overhead, and diminished load on application servers.¶

Bandwidth consumption often decreases proportionally. Consolidated representations in machine-readable formats are typically significantly smaller than equivalent HTML pages, lacking navigation chrome, advertisements, and presentational markup. A single ETag check can replace multiple page checks for change detection, further reducing transfer overhead. Network egress costs decline accordingly.¶

Caching efficiency can also improve markedly. Rather than caching dozens of individual pages, CDNs and intermediaries cache single consolidated resources. Cache hit rates increase, invalidation simplifies, and CDN costs drop.¶

Illustrative example: Consider a documentation site receiving 10,000 automated agent visits daily. Suppose each visit currently fetches an average of 10 pages to assemble complete information (typical for documentation traversal), representing 100,000 daily requests. With consolidated representations reducing this to one request per visit, the site serves 90,000 fewer requests daily.¶

Modern documentation pages of even modest visual complexity can easily reach 1 MB once navigation, styling, and scripts are included, while equivalent consolidated markdown representations might plausibly be 350 KB for the same underlying content. In this scenario, bandwidth drops from approximately 100 GB daily (HTML) to 3.5 GB daily (consolidated markdown) - a reduction of roughly 2.9 terabytes monthly. Actual figures will vary by site design and traffic patterns, but these orders of magnitude are realistic for many contemporary sites.¶

Automated systems benefit as well. Consolidated representations provide clearer information structure, reducing parsing errors from complex HTML and JavaScript execution. Deliberate consolidation supplies context that scattered pages cannot, improving comprehension and result quality.¶

A.4. Operational Benefits

The benefits of explicit negotiation are not limited to reduced server load, improved cache efficiency, and lower bandwidth use. A publisher can measure requests containing Prefer: return=consolidated, responses carrying Preference-Applied: return=consolidated, and cache behaviour varying on Accept and Prefer. These are stable protocol signals rather than inferences from client identity.¶

Client-detection approaches require servers to decide whether a request comes from a recognised crawler, agent, or intermediary. That decision may depend on User-Agent strings, IP ranges, reverse DNS, CDN-provided bot classifications, allow lists, or deployment-specific heuristics. Each additional rule increases operational complexity and creates another failure mode: an unrecognised automated client receives navigational HTML, a recognised identity is imitated, or a cache cannot safely reuse a response because representation selection depends on implicit client classification.¶

Explicit negotiation separates representation selection from access control, authorisation, commercial policy, abuse prevention, and telemetry. Servers may still apply those policies, but the request for a machine-readable representation is expressed independently of the client's identity. Unknown clients that implement the protocol can request the same representation form without being pre-registered, and publishers can evaluate support using ordinary HTTP request and response data.¶

These benefits are difficult to express as universal percentages, but they are operationally significant. They reduce dependence on brittle client-detection rules, improve cacheability, simplify measurement, and provide a cleaner path for new automated clients to interoperate with publishers.¶

B.1. Relationship to Earlier Machine-Readable Web Approaches

Various initiatives have attempted to make web content more machine-accessible. The Semantic Web / RDF / Linked Data efforts, XHTML and XML-based publishing approaches, and embedded structured data approaches like Microdata and Schema.org have been under development for decades, yet broad and consistent adoption has not materialised. Feed formats like RSS and Atom achieved significant adoption but remained separate from the standard web browser model, requiring dedicated client software or an aggregator. In practice, this separation was vulnerable: when major implementations were discontinued, the ecosystem fragmented and recovery proved difficult.¶

These approaches addressed real problems, and some remain useful in particular domains. The deployment lesson is narrower: mechanisms that require publishers, clients, and intermediaries to adopt a separate architectural model have struggled to displace ordinary HTTP deployment practice, even where the model is technically well-designed.¶

This specification therefore takes a different approach by using only existing HTTP mechanisms.¶

• Uses existing HTTP content negotiation - RFC 9110¶

• Uses existing Prefer header - RFC 7240¶

• Uses existing media types - for example, text/markdown (RFC 7763), application/json, application/xml¶

• No new protocols, no new standards, no new infrastructure¶

Unlike approaches that prescribe specific data models or require adoption of complex frameworks, this specification leaves content organisation entirely to implementers. The challenge shifts from technical implementation to editorial judgement: what information belongs together, what context is needed, what can be omitted.¶

B.2. Contemporary Approaches

B.3. Manipulation of Machine-Readable Representations

Machine-readable representations create incentives. Once automated systems use a representation to classify, rank, summarise, cite, recommend, or otherwise act on a resource, publishers and intermediaries have an incentive to optimise that representation for the consuming system.¶

This is not new. Search engines produced keyword stuffing, hidden text, doorway pages, cloaking, link farms, and other attempts to influence ranking systems. Social platforms produced optimisation for preview cards, sharing heuristics, recommendation systems, and engagement metrics. Machine-readable representations intended for automated agents are now producing the same pattern: representations are being created or modified specifically to influence downstream automated behaviour.¶

Consolidated representations make the optimisation target more explicit. That does not create a new class of risk. Sites can already provide different content to automated clients through client detection, bot-specific routing, cloaking, or other response-selection mechanisms. In those deployments, the consuming system may receive content that differs materially from the representation served to ordinary human navigation, without any protocol-level indication that this has occurred.¶

This specification moves that risk into an explicit representation layer. A client can request a consolidated representation, and a server can indicate that such a representation was supplied. That improves protocol clarity, but it does not prove that the representation is accurate, complete, equivalent, benign, or faithful to any other representation of the resource.¶

A publisher can decide what context to include, what to omit, what to emphasise, and how to structure the resource for automated processing. A hostile or compromised origin can go further: it can serve a machine-readable representation that has little or no substantive relationship to the navigational HTML representation of the same resource. Those behaviours are possible with or without this specification; the difference is that explicit negotiation makes the representation boundary visible.¶

This is a deployment risk, not a protocol anomaly. The protocol mechanism only identifies a requested representation of a resource; it does not attest to the accuracy, completeness, neutrality, or intent of that representation. Consumers therefore need independent quality, provenance, corroboration, abuse-detection, and content-integrity controls.¶

Optimisation is inevitable where automated systems create economic or reputational incentives. The purpose of this specification is to avoid confusing representation availability with trustworthiness, and to ensure that clients have an explicit mechanism for requesting machine-readable representations without relying on client identity or ad-hoc detection.¶

C.1. Markdown Example: Product Website

Consider a typical small business website with navigational structure:¶

Site structure (navigational):
/               (landing page: hero image, value proposition,
                 social proof testimonials, call-to-action buttons)
/features/      (feature list with marketing copy)
/features/a/    (detailed feature A with screenshots)
/features/b/    (detailed feature B with screenshots)
/pricing/       (pricing tiers with comparison table)
/contact/       (contact form, office locations, map)
/docs/          (technical documentation)

A consolidated representation for the root resource provides an overview with links to detailed consolidated resources:¶

GET / HTTP/1.1
Accept: text/markdown;q=0.9
Prefer: return=consolidated

# Product Overview
[Value proposition and core description from landing page.
 Hero images, testimonials, and CTAs omitted]

High-level feature summary with key capabilities.

For detailed information:
- Features: /features/
- Pricing: /pricing/
- Documentation: /docs/

[Note: Links point to same URLs; consolidated representations
 served based on Accept/Prefer headers]

A consolidated representation for the features resource provides comprehensive technical detail:¶

GET /features/ HTTP/1.1
Accept: text/markdown;q=0.9
Prefer: return=consolidated

# Features

## Feature A
### What it does
[Consolidated from /features/ and /features/a/ - technical
 description of capabilities]

### How it works
[Implementation details from /docs/ where relevant]

### Pricing
[Relevant pricing tier information for this feature, consolidated
 from /pricing/]

### Technical requirements
[System requirements, API details, integration notes]

## Feature B
[Similar comprehensive structure]

Related resources:
- Complete technical documentation: /docs/
- Pricing comparison: /pricing/

Note the key differences from simple page conversion:¶

• Multiple consolidated views: Different URLs provide different semantic organisations of the same underlying content¶

• Deep consolidation: Feature pages pull in relevant pricing, documentation, and technical details¶

• Semantic restructuring: Content organised by "what/how/requirements" rather than mirroring site navigation¶

• Selective omission: Marketing copy, testimonials, decorative elements excluded¶

• Preserved navigation: Links to other consolidated resources maintained for context¶

This illustrates the key principle: consolidation is about semantic organisation and selective inclusion of substantive information, not mechanical conversion of all page content. Each consolidated resource provides a complete, contextual view optimised for understanding that specific topic, drawing from multiple source pages as needed.¶

C.2. JSON Example: Financial News Article

Consolidated representations are not limited to text/markdown. Consider a financial news website:¶

Site structure (navigational):
/article/12345      (news article with ads, related links)
/stock/ACME         (stock price chart and basics)
/company/acme-corp  (company profile)
/filings/acme-q3    (SEC filing summary)

A consolidated JSON representation for the article provides structured data combining relevant financial information:¶

GET /article/12345 HTTP/1.1
Accept: application/json;q=0.9
Prefer: return=consolidated

HTTP/1.1 200 OK
Content-Type: application/json
Preference-Applied: return=consolidated
ETag: "article-12345-consolidated-v2"
Vary: Accept, Prefer

{
  "alternateFormats": [
    {
      "type": "text/markdown",
      "title": "Article (Markdown)"
    }
  ],
  "article": {
    "headline": "ACME Corp Reports Strong Q3 Results",
    "published": "2024-12-16T14:30:00Z",
    "summary": "ACME Corp exceeded analyst expectations...",
    "content": "[Article text without ads/chrome]"
  },
  "financial_data": {
    "stock_symbol": "ACME",
    "current_price": 142.50,
    "change_percent": 8.3,
    "market_cap": "125B",
    "as_of": "2024-12-16T20:00:00Z"
  },
  "company": {
    "name": "ACME Corporation",
    "sector": "Technology",
    "employees": 15000,
    "founded": 1995
  },
  "quarterly_results": {
    "period": "Q3 2024",
    "revenue": "8.2B",
    "revenue_growth": 12.5,
    "eps": 1.85,
    "eps_expected": 1.72
  },
  "related": {
    "stock_details": "/stock/ACME",
    "company_profile": "/company/acme-corp",
    "sec_filings": "/filings/acme-q3"
  }
}

This JSON consolidation pulls key financial metrics from separate stock ticker, company profile, and filing pages, presenting them alongside the article content in a structured format optimised for automated analysis. The same article URL serves both human-readable HTML and machine-readable consolidated JSON based on content negotiation.¶

D.1. Client Implementation

Clients can begin requesting consolidated representations immediately. The protocol is designed for graceful degradation: if a server does not support consolidated representations, it will simply return the standard representation (typically HTML), which clients already handle. In conforming deployments, unsupported preferences are ignored rather than treated as errors. Early adoption benefits clients immediately whenever any publisher implements support, with no cost when publishers have not yet done so.¶

Clients SHOULD request text/markdown by default, as it handles the vast majority of web content effectively. When requesting resources known to contain structured data (API endpoints, financial feeds, datasets), clients SHOULD request application/json instead. These formats have widespread parser support and cover nearly all use cases. Clients can refine their format preferences as publisher implementations mature and specific needs emerge.¶

Clients SHOULD:¶

1. Include Prefer: return=consolidated in requests where consolidated content would be beneficial¶

2. Specify appropriate Accept headers (text/markdown by default for text-heavy resources, application/json for known structured resources)¶

3. Check for Preference-Applied: return=consolidated in responses to confirm support¶

4. Fall back to standard content processing when the preference is not honoured¶

D.2. Server Implementation

Implementation requires three steps:¶

1. Parse the Prefer header and recognise return=consolidated¶

2. Serve machine-readable format when requested via Accept header¶

3. Generate appropriate consolidated content for the publisher's use case¶

Existing web servers, frameworks, or WAFs that already parse the Prefer header may need minor updates to recognise the return=consolidated value. Beyond that, the technical implementation is straightforward. Publishers SHOULD log when consolidated representations are served for analytics and capacity planning; logging the Preference-Applied response header provides one straightforward approach.¶

Publishers SHOULD begin by implementing text/markdown for text-heavy content and application/json for structured data. These formats have widespread parser support, are straightforward to generate, and align with client expectations. Publishers may evolve their format offerings as the ecosystem matures and specific consumer needs emerge, but starting with these pragmatic defaults ensures immediate interoperability.¶

The real work lies in content decisions: which information to consolidate, how to structure it, what context to include, what to omit. These decisions depend on site architecture, content type, and audience needs. This specification provides the mechanism, publishers provide the judgement. (Large language models may prove surprisingly capable at making these editorial decisions, should publishers wish to automate the process.)¶

Publishers can begin with minimal investment; even simple format conversion of existing pages provides immediate bandwidth and load reduction benefits. The incremental adoption approach (below) allows publishers to start small and expand based on observed value.¶

D.3. Incremental Adoption

Implementing consolidated representations does not require a complete site overhaul. Publishers can adopt this specification incrementally:¶

Phase 1: Simple Format Conversion Start by serving machine-readable versions of existing pages (ignoring the Prefer header initially). This provides immediate bandwidth and parsing benefits. A simple conversion tool can generate markdown from HTML with minimal effort.¶

Phase 2: Analyse Access Patterns Monitor which pages automated agents fetch together. Collect statistics on common access patterns: which documentation pages are read sequentially, which product pages are accessed alongside pricing information, and so on.¶

Phase 3: Create Targeted Consolidated Resources Based on usage patterns, create consolidated representations for high-traffic combinations. A site might start with just two or three strategic consolidated resources covering the most common information retrieval patterns.¶

Phase 4: Expand as Beneficial Add consolidated representations where server load or bandwidth justify the effort.¶

D.4. Format Selection

Initial adoption¶

Publishers and consumers SHOULD begin with text/markdown for text-heavy content (documentation, articles, blogs, guides) and application/json for structured data (financial information, API responses, datasets, metrics). These formats provide widespread parser availability, straightforward implementation, clear specifications, and existing adoption by automated consumers. Starting with these formats ensures immediate interoperability.¶

Publishers MAY use alternative formats where compelling domain-specific reasons exist. For example, application/xml for complex structured documents where XML tooling provides clear value, or text/csv for tabular data in domains with established CSV conventions.¶

Format evolution¶

This specification is deliberately format-agnostic to accommodate future evolution. Publishers and consumers SHOULD NOT treat markdown and JSON as permanent requirements. They are pragmatic starting points that work today. Should new formats emerge with compelling advantages, this specification supports their adoption through discovery and standard content negotiation. Migration is possible without specification change.¶

Selection principle¶

Keep it simple. Use formats with widespread support, clear specifications, and proven parser availability. Novel formats require compelling justification from major consumers demonstrating concrete benefits that outweigh implementation burden.¶

D.5. Validation and Testing

Analogous to how social media platforms provide preview tools for OpenGraph meta tags, organisations consuming consolidated representations SHOULD provide validation tools allowing publishers to verify their implementations. Such tools should provide metrics and feedback beyond what simple command-line tools offer: consolidation quality assessment, structure analysis, and guidance on whether the representation meets consumer needs. Only consumers possess the technical capability to evaluate whether consolidated representations suit their processing requirements.¶