AGENTS.TXT: Strict Policy File for Automated Clients

AGENTS.TXT: Strict Policy File for Automated Clients Independent

srijaldutta.official+agentstxt@gmail.com

This document specifies the AGENTS.TXT protocol, a strict plaintext policy file for automated clients, bots, and crawlers. It defines directives, top-line hash verification, optional parameters, and mandatory failure behavior for malformed files. Malformed files are treated as fully restrictive to prevent unintended access.

AGENTS.TXT is a strict policy file format for automated clients, similar in purpose to robots.txt but providing more control over client behavior. Malformed files are treated as completely restrictive. All AGENTS.TXT traffic validation is based on a SHA-256 hash (FIPS 180-4) of the canonical directive content.

The canonical path for the file is /agents.txt. Files must be served as UTF-8 with content-type text/plain (HTTP/1.1 Semantics).

The first non-comment, non-empty line MUST be the hash line, starting with '*' followed by the lowercase SHA-256 hex digest of the file excluding the hash line and comments (SHA-1 comparison for historical reference). Subsequent lines are directives: /status ALLOW /dashboard ALLOW limit=50 /admin DISALLOW

Lines starting with '#' are comments and ignored for hash computation and parsing. Metadata such as version, generated-by, or grace-period may be included.

Any hash missing, hash mismatch, or directive syntax error MUST result in treating the entire site as restricted (RFC 2119 requirements). Cached copies MUST be invalidated.

Each directive line has the format: <path> <action> [params...] <path> starts with '/', <action> is ALLOW or DISALLOW, and optional params are key=value pairs (URI syntax).

Compute SHA-256 over UTF-8 bytes of the file after removing the hash line, comments, and blank lines. Join remaining lines with '\n' for hashing.

Strict malformed-file behavior ensures accidental exposure does not occur. Site operators must ensure valid files to prevent clients from blocking themselves (TLS Best Practices).

# version: 1.0 *e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 \ #sample. may not be accurate. /status ALLOW /dashboard ALLOW limit=50 /admin DISALLOW

Clients SHOULD follow HTTP client best practices and API crawler guidelines when interpreting AGENTS.TXT directives. Use of AGENTS.TXT aims to reduce accidental site disruption (Bot traffic management).

Key words for use in RFCs to Indicate Requirement Levels SHA-256 Secure Hash Standard Robots.txt: History, Use, and Standardization Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content US Secure Hash Algorithm 1 (SHA1) Uniform Resource Identifier (URI): Generic Syntax Recommendations for Secure Use of Transport Layer Security (TLS) and Datagram TLS (DTLS) Bot Traffic and Management Best Practices HTTP Client Best Practices for Automated Agents Guidelines for Secure API Crawlers

Srijal Dutta Email: srijaldutta.official+agentstxt@gmail.com