Internet-Draft agents-txt June 2026
Cardillo Expires 14 December 2026 [Page]
Workgroup:
Independent Submission
Internet-Draft:
draft-car-agents-txt-wellknown-00
Published:
Intended Status:
Informational
Expires:
Author:
K. Cardillo
Independent

AGENTS.TXT: Capability Declarations for Web Agents

Abstract

This document requests registration of two Well-Known URIs under the "/.well-known/" path: "agents.txt" and "agents.json". These URIs define a machine-readable capability declaration format: a positive statement of what web agents CAN do on a site -- which endpoints are sanctioned for agent use, which protocols (REST, MCP, A2A, GraphQL, WebSocket) are supported, what authentication mechanisms are expected, and what rate limits the site advertises.

This is distinct from "robots.txt", which uses a restriction syntax to declare what crawlers may not do. Where "robots.txt" expresses prohibition, "agents.txt" expresses capability -- a sanctioned channel for agent interaction that is otherwise routinely blocked by bot detection, CAPTCHAs, and rate limiters because no positive declaration surface exists.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 14 December 2026.

▲

Table of Contents

1. Introduction

Automated AI agents increasingly interact with websites to perform tasks on behalf of users: searching product catalogs, retrieving structured data, executing transactions, and calling APIs. These agents are routinely blocked by bot detection systems, CAPTCHAs, and rate limiters because no sanctioned channel for agent interaction exists.

Simultaneously, website operators have no standard mechanism to declare which agent behaviors they support, which endpoints are designed for machine access, or how agents should authenticate.

"agents.txt" addresses this gap. It is an opt-in capability declaration file, served at a well-known location, that communicates to AI agents:

1.1. Relationship to Existing Standards

"agents.txt" is complementary to, and does not replace, existing standards:

robots.txt:

Declares crawling restrictions. "agents.txt" declares what agents are explicitly permitted to do. Both files may coexist.

llms.txt [LLMSTXT]:

Provides human-readable content for LLMs to read. "agents.txt" declares machine-callable endpoints and capabilities.

security.txt [RFC9116]:

Declares security vulnerability disclosure contacts. "agents.txt" declares AI agent interaction policies.

OpenAPI:

Documents individual API endpoints in detail. "agents.txt" is a discovery layer; it may reference OpenAPI specifications per capability.

MCP (Model Context Protocol):

A protocol for AI tools. "agents.txt" can declare MCP endpoints, making them discoverable without prior configuration.

1.3. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

2. The "agents.txt" Well-Known URI

2.1. Location

The "agents.txt" file MUST be served at:

https://example.com/.well-known/agents.txt

Agents SHOULD also check the root path as a fallback:

https://example.com/agents.txt

The "/.well-known/agents.txt" path takes precedence when both exist.

The file MUST be served over HTTPS in production deployments. HTTP is permitted only in development or testing environments.

The file MUST be served with Content-Type "text/plain; charset=utf-8".

2.2. Format

The "agents.txt" file uses a block-based key-value format inspired by "robots.txt". Each line contains a key, a colon, and a value. Lines beginning with "#" are comments. Indented lines (two or more spaces, or one or more tabs) belong to the preceding block.

A minimal "agents.txt" file:

# agents.txt
Spec-Version: 1.0
Site-Name: Example Store
Site-URL: https://example.com

Capability: product-search
  Endpoint: https://example.com/api/search
  Method: GET
  Protocol: REST
  Auth: none
  Rate-Limit: 60/minute
  Description: Search the product catalog

Allow: /api/*
Disallow: /admin/*

Agent: *

2.3. Header Fields

Spec-Version (REQUIRED):

The specification version. MUST be "1.0" for documents conforming to this specification.

Generated-At (OPTIONAL):

ISO 8601 timestamp of when the file was generated.

Declaration-Type (OPTIONAL):

One of "platform" or "agent". Default: "platform". A platform declaration states what agents may do on this site (the standard case). An agent declaration, published by an agent operator at the operator's own domain, states what the operator's agent does on external platforms.

Operates-On (OPTIONAL):

URL of a platform this agent operates on. MAY appear multiple times. Expected when Declaration-Type is "agent"; not used in platform declarations.

2.4. Site Fields

Site-Name (REQUIRED):

Human-readable name of the site or service.

Site-URL (REQUIRED):

Canonical HTTPS URL of the site.

Site-Description (OPTIONAL):

Brief description of the site.

Site-Contact (OPTIONAL):

Contact email address for agent-related inquiries.

Site-Privacy-Policy (OPTIONAL):

URL of the site's privacy policy.

2.5. Capability Blocks

A Capability block declares a single action available to agents. Capability identifiers MUST consist of lowercase letters, digits, and hyphens only.

Capability (REQUIRED):

Identifier for this capability.

Endpoint (REQUIRED):

Full HTTPS URL of the endpoint.

Protocol (REQUIRED):

The interaction protocol. One of: REST, MCP, A2A, GraphQL, WebSocket.

Method (OPTIONAL):

HTTP method for REST endpoints. Default: GET.

Auth (OPTIONAL):

Authentication type. One of: none, api-key, bearer-token, oauth2, hmac. Default: none. Servers MUST NOT include actual credentials in this field.

Auth-Endpoint (OPTIONAL):

URL where agents obtain authentication tokens. MUST be present when Auth is "bearer-token" or "oauth2".

Auth-Docs (OPTIONAL):

URL of human-readable documentation describing the authentication flow for this capability.

Scopes (OPTIONAL):

Comma-separated list of OAuth2 scopes required by this capability.

Rate-Limit (OPTIONAL):

Advisory rate limit in the format "N/window" where window is one of: second, minute, hour, day. Agents SHOULD respect declared limits. Servers MUST enforce limits independently.

Description (OPTIONAL):

Human-readable description of the capability.

OpenAPI (OPTIONAL):

URL to an OpenAPI specification document describing the endpoint.

Param (OPTIONAL):

Declares one parameter of a REST endpoint. MAY appear multiple times within a Capability block. The value uses the form:

name (location, type[, required]) [- description]

: where "location" is one of "query", "path", "header", or "body"; "type" is one of "string", "integer", "number", or "boolean"; the literal token "required", when present, marks the parameter as required; and the free-text description after "-" is optional. Example:

Param: q (query, string, required) - Search query

Fields not defined in this document MUST be ignored by parsers, to permit forward-compatible extension.

2.6. Access Control Fields

Allow (OPTIONAL):

Glob pattern for paths agents may access. Semantics follow "robots.txt" conventions.

Disallow (OPTIONAL):

Glob pattern for paths agents MUST NOT access.

More specific patterns take precedence over less specific patterns. When no access control is declared, only paths referenced by capabilities are implicitly permitted.

2.7. Agent Policy Blocks

Agent blocks declare per-agent policies. The wildcard "*" declares the default policy for all agents.

Agent: *

Agent: claude
  Rate-Limit: 200/minute
  Capabilities: product-search, store-assistant

Agent identifiers SHOULD match the first token of the agent's User-Agent header (case-insensitive).

Capabilities (OPTIONAL within an Agent block):

Comma-separated list of capability identifiers this agent is permitted to use. If omitted, all declared capabilities are permitted.

Agent-Declaration (OPTIONAL within an Agent block):

URL of the agent operator's own "agents.txt" file (a declaration with Declaration-Type "agent"). Enables cross-referencing between a platform's grant of capabilities and the agent operator's published statement of what the agent does and where it operates.

3. The "agents.json" Well-Known URI

3.1. Location

The JSON companion file MUST be served at:

https://example.com/.well-known/agents.json

The file MUST be served with Content-Type "application/json; charset=utf-8".

3.2. Format

The JSON format contains equivalent information to "agents.txt" in a typed JSON structure suitable for direct consumption by programmatic clients. The "agents.txt" file MAY reference the JSON file via:

Agents-JSON: https://example.com/.well-known/agents.json

A minimal "agents.json" document:

{
  "specVersion": "1.0",
  "generatedAt": "2026-02-01T00:00:00.000Z",
  "site": {
    "name": "Example Store",
    "url": "https://example.com"
  },
  "capabilities": [
    {
      "id": "product-search",
      "description": "Search the product catalog",
      "endpoint": "https://example.com/api/search",
      "method": "GET",
      "protocol": "REST",
      "auth": { "type": "none" },
      "rateLimit": { "requests": 60, "window": "minute" }
    }
  ],
  "access": {
    "allow": ["/api/*"],
    "disallow": ["/admin/*"]
  },
  "agents": {
    "*": {}
  }
}

Field semantics are identical to those defined in Section 2 for the text format.

4. Agent Behavior

4.1. Discovery

Agents SHOULD fetch "/.well-known/agents.txt" and/or "/.well-known/agents.json" before interacting with an unfamiliar site.

Agents SHOULD prefer the JSON format when both are available, as it is more precisely typed and unambiguous.

Agents SHOULD cache the capability declaration for the duration declared by the HTTP Cache-Control header. Implementations SHOULD use a minimum cache TTL of 60 seconds to reduce server load.

4.2. Identification

Agents SHOULD identify themselves via the User-Agent HTTP header when calling capability endpoints. The agent name in the User-Agent header is matched (case-insensitively) against Agent blocks to apply per-agent policies.

4.3. Rate Limiting

Agents SHOULD respect Rate-Limit declarations as advisory limits. Servers MUST enforce rate limits independently and MUST NOT rely on agents to self-enforce.

5. Server Behavior

5.1. Caching

Servers SHOULD serve "agents.txt" and "agents.json" with appropriate Cache-Control headers. A max-age of 300 seconds (5 minutes) is RECOMMENDED for most deployments.

5.2. CORS

Servers SHOULD include the following headers to permit cross-origin discovery:

Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, OPTIONS

5.3. Security Considerations

Capability declarations MUST NOT include actual credentials, API keys, tokens, or secrets of any kind. The Auth and Auth-Endpoint fields describe mechanisms only.

Servers MUST enforce all declared restrictions (rate limits, access control, agent policies) independently of the declarations in "agents.txt". The file is advisory to agents; it is not a trust boundary.

Agents MUST validate that capability endpoints use HTTPS before sending authentication credentials.

Site owners SHOULD review their capability declarations periodically to ensure they accurately reflect current server capabilities and access policies.

6. IANA Considerations

6.1. Well-Known URI Registration: "agents.txt"

This document requests registration of the following Well-Known URI in the "Well-Known URIs" registry established by [RFC8615]:

URI suffix:

agents.txt

Change controller:

Kayla Cardillo

Specification document(s):

This document.

Related information:

Text-format capability declaration file for AI agent discovery.

6.2. Well-Known URI Registration: "agents.json"

URI suffix:

agents.json

Change controller:

Kayla Cardillo

Specification document(s):

This document.

Related information:

JSON-format capability declaration file for AI agent discovery. Companion format to agents.txt.

7. References

7.1. Normative References

[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/rfc/rfc2119>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/rfc/rfc8174>.
[RFC8615]
Nottingham, M., "Well-Known Uniform Resource Identifiers (URIs)", RFC 8615, DOI 10.17487/RFC8615, , <https://www.rfc-editor.org/rfc/rfc8615>.
[RFC9110]
Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110, , <https://www.rfc-editor.org/rfc/rfc9110>.

7.2. Informative References

[A2A-AGENT-CARD]
"A2A Agent Card", , <https://a2aproject.github.io/>.
[LLMSTXT]
"llms.txt", , <https://llmstxt.org>.
[MCP]
"Model Context Protocol", , <https://modelcontextprotocol.io/specification>.
[MCP-SERVER-CARD]
"Model Context Protocol Server Card", , <https://modelcontextprotocol.io>.
[RFC9116]
Foudil, E. and Y. Shafranovich, "A File Format to Aid in Security Vulnerability Disclosure", .
[ROBOTS]
"Robots Exclusion Protocol", , <https://www.rfc-editor.org/rfc/rfc9309>.
[SRIJAL-AGENTS-POLICY]
"AGENTS.TXT: Strict Policy File for Automated Clients", Work in Progress, Internet-Draft, draft-srijal-agents-policy-00 (expired), , <https://datatracker.ietf.org/doc/draft-srijal-agents-policy/>.

Appendix A. Example: E-Commerce Site 

# agents.txt
Spec-Version: 1.0
Generated-At: 2026-02-01T00:00:00Z
Site-Name: Outdoor Supply Co.
Site-URL: https://outdoorsupply.example
Site-Description: Gear for outdoor adventures
Site-Contact: agents@outdoorsupply.example

Capability: product-search
  Endpoint: https://outdoorsupply.example/api/search
  Method: GET
  Protocol: REST
  Auth: none
  Rate-Limit: 60/minute
  Description: Search the product catalog
  Param: q (query, string, required) - Search query
  Param: limit (query, integer) - Max results, default 20
  Param: category (query, string) - Filter by category

Capability: store-assistant
  Endpoint: https://outdoorsupply.example/mcp
  Protocol: MCP
  Auth: bearer-token
  Auth-Endpoint: https://outdoorsupply.example/auth/token
  Description: Full store interaction via MCP

Allow: /api/*
Allow: /mcp
Disallow: /admin/*
Disallow: /internal/*

Agent: *
Agent: claude
  Rate-Limit: 200/minute
  Capabilities: product-search, store-assistant

Appendix B. Acknowledgments

The "agents.txt" format draws on the design of "robots.txt" [ROBOTS], "security.txt" [RFC9116], and OpenAPI for structural inspiration. The MCP protocol reference is to the Model Context Protocol specification.

Author's Address

Kayla Cardillo
Independent