A JSON-Based Format for Publishing IP Ranges of Automated HTTP Clients

2.1. Top-Level Object Definition

An IP range publication file MUST be a single JSON object. The text encoding of the file MUST be UTF-8. This top-level object serves as the root container for metadata relevant to the file and the list of IP prefixes.¶

The top-level object contains the fields defined in Table 1.¶

2.2. The prefixes Array

The prefixes member of the top-level object MUST contain an array of JSON objects. Each of these objects is a Prefix Object, as defined in Section 2.3.¶

To simplify implementation for consumers, there MUST be a single, unified array for both IPv4 and IPv6 prefixes.¶

2.3. The Prefix Object

Each object within the prefixes array represents a single IP address range and its associated metadata.¶

A Prefix Object MUST contain exactly one of either the ipv4Prefix or ipv6Prefix field. An object containing both or neither of these fields is invalid and MUST be ignored by consumers. The fields are defined in Table 2.¶

2.4. Extensibility

The JAFAR format is extensible. Both the top-level object and Prefix Objects MAY include additional fields not defined in this specification. Consumers MUST ignore any fields they do not recognize.¶

3.1. Fetching and Caching

Consumers SHOULD fetch the machine-readable IP range publication file from a stable URL provided by the publisher. The file location MUST be disclosed by the publisher of the file on the page that describes the crawler as specified by [CBCP].¶

Publishers SHOULD update the file when there is any change to the prefixes, or at least every 24 hours, even if the only update is to creationTime. Consumers SHOULD implement a polling mechanism to check for updates at a reasonable interval, such as once every 24 hours. Consumers MUST NOT poll more frequently than once per hour unless explicitly permitted by the publisher's documentation or HTTP caching headers.¶

To minimize server load for the publisher and reduce unnecessary bandwidth usage for the consumer, consumers MUST respect standard HTTP caching headers specified in [HTTP-CACHING] that may be present in the response, such as Cache-Control, ETag, and Last-Modified. Publishers SHOULD provide these headers to facilitate efficient caching.¶

3.2. Handling Format Versioning

To ensure long-term stability and allow for future evolution of this specification, consumers MUST inspect the version optional parameter of the application/jafar+json media type, as specified in the Content-Type HTTP header. The version is expressed as "major.minor". If the version parameter is absent, consumers SHOULD assume the latest stable version of this specification they are programmed to handle.¶

• Major Version Changes: A change in the major version number (e.g., from "1.0" to "2.0") indicates a non-backward-compatible change to the specification. If a consumer encounters a file advertised with a major version number greater than the major version it is programmed to handle, the consumer MUST NOT attempt to parse the file. It SHOULD treat this situation as an error and MAY continue to use its last known valid list until it can be updated to support the new version. This prevents the misinterpretation of data from a significantly altered schema.¶

• Minor Version Changes: A change in the minor version number (e.g., from "1.0" to "1.1") indicates the addition of new features or fields that are backward-compatible. For example, a new OPTIONAL field might be added to the Prefix Object. A consumer programmed to handle version "1.0" MUST be able to correctly parse a file with version "1.1". The minor version number increases numerically independently of the major version number, for instance: 1.9 -> 1.10 -> 1.11. The consumer MUST ignore any unrecognized fields or properties within the JSON objects.¶

3.3. Prefix Aggregation and Specificity

A publication file MAY contain overlapping IP address ranges. For instance, a publisher might list a broad range like 198.51.100.0/22 with a generic service tag, and also list a more specific range within it, such as 198.51.100.0/24, with a more specific service tag.¶

When a consumer evaluates a specific IP address against the list, it MAY match multiple Prefix Objects. In such cases, the consumer's logic SHOULD use the data from the Prefix Object with the most specific CIDR range (i.e., the one with the largest prefix length) for that IP address.¶

4.1. Example 1: Basic Publication

This example demonstrates a minimal, valid file for a publisher with a single type of automated client. It uses only the required fields, making it simple to generate and consume.¶

{
  "creationTime": "2025-08-15T14:30:00Z",
  "prefixes": [
    {
      "ipv4Prefix": "66.249.64.0/20"
    },
    {
      "ipv4Prefix": "34.64.0.0/12"
    },
    {
      "ipv6Prefix": "2001:4860:4000::/36"
    }
  ]
}

4.2. Example 2: Advanced Publication with Multiple Services

This example illustrates a more complex file from a large cloud provider that uses optional fields to provide richer context. It differentiates between IP ranges used for a general web crawler, a specialized ads crawler, and a user-triggered fetching service. This allows consumers to implement more granular access policies.¶

{
  "creationTime": "2025-08-15T14:30:00Z",
  "prefixes": [
    {
      "ipv4Prefix": "66.249.64.0/24",
      "services": ["ExampleCloud-Crawler", "ExampleCloud-Ads"]
    },
    {
      "ipv6Prefix": "2001:4860:4860::/48",
      "services": ["ExampleCloud-Fetcher"]
    }
  ]
}

4.3. Example 3: Aggregated Publication by a Third Party

This example shows how a third-party security provider or an open-source project could aggregate IP lists from multiple publishers into a single file. The service field is used to attribute each prefix to its original source, allowing consumers to apply policies based on the original publisher. This reflects the use case of services that provide curated lists of known bots.¶

{
  "synctoken": "20260410223000",
  "creationTime": "2026-04-10T22:30:00Z",
  "notes": "Aggregated feed of verified automated clients. Attribution is maintained via the services array. Data refreshed every 24 hours.",
  "prefixes": [
    {
      "ipv4Prefix": "192.0.2.0/24",
      "services": [
        "SearchEngine-A-Crawler",
        "SearchEngine-A-ImageBot"
      ]
    },
    {
      "ipv4Prefix": "198.51.100.0/24",
      "services": [
        "SocialMedia-B-Preview"
      ]
    },
    {
      "ipv6Prefix": "2001:db8:abc::/48",
      "services": [
        "TechCo-C-HealthCheck",
        "TechCo-C-Ads"
      ]
    }
  ]
}