Personal Data Portability Archive

3.1.1. Data portability

A main use case for the novel format is to allow exporting the full user data managed by services or software products into a simple file (or set of files) which is under full control of the user.¶

The user might use such export for backup, archiving, or for importing when switching to another service or software (i.e., migration).¶

Depending on the type of data, exporting/importing can be a time-consuming process. Particularly for the case of switching services, PDPA should allow to minimize the time period during which a user cannot use the origin system but also the destination system is not yet ready.¶

3.1.2. Incremental backup

Beyond snapshot backups/exports, the format should optionally allow for incremental backups.¶

There are at least two scenarios for this:¶

• Software which wants to keep a permanent, incremental mirror of user data (e.g., for instant export or restore)¶
• Users regularly exporting changes in data managed by services or software products¶

3.1.3. Synchronization

Data portability does not just allow users to switch from one service to another one, but to let users benefit from 3rd party services getting a copy of their data (at the request of the user). Simple synchronization features could make this much better.¶

For example, current online systems that allow importing contacts are not often suited to maintaining one's address book on two systems. Re-importing a contact into a system that already has that contact often results in duplicating the exact same contact, whether or not there have been edits, making repeated synchronization practically infeasible. It should be easy to do a significantly better job of this with some attention to object IDs and modification timestamps.¶

We however do not attempt to solve two-way synchronization via export files. It would require significant additional work to allow two systems, neither of which is agreed-upon to be the source of truth, to reliably synchronize changes from both. In comparison, solving one-way synchronization only requires agreed-upon usage of existing fields and values.¶

3.1.4. Dataset exchange

PDPA should be usable to exchange and share larger data sets than just one user, or to share a single user's data outside the context where the user knows what it is and where it came from.¶

Potential applications of this are:¶

• The ability to exchange test data and known mailstore state, e.g. for conformance testing or internal functional tests¶
• Legal discovery and forensics use cases may benefit from a standard export format, such that investigators can expect a great deal of consistency when collecting data from different systems.¶
• Researchers may be able to collect archive files through data donations and use as input to research.¶

While these use cases may suggest small features that would help these use cases be successful, supporting more advanced features required by these use cases is not a priority. For example, we do not attempt to cryptographically solve provenance for use in legal and forensics use cases.¶

3.1.5. Data persistence

The format MAY be used as a development-time active persistence layer for user data in, e.g., email clients or applications. It is not intended as, or suitable for, a production-level persistence layer.¶

3.2.1. Email standards compatibility

Data formats should aim for compatibility with JMAP data formats for the sake of interoperability and synergies in software libraries.¶

Dedicated JMAP API methods for exporting and importing the format described here, or for related server-to-server transfer protocols are out of the scope of this document.¶

Due to its specifics and ubiquitous usage, the Internet Message Format [RFC5322]; latest revision of [RFC2822]/[RFC822]) should be the core of representing individual email data.¶

This specification should ideally describe mappings between PDPA and existing mailbox persistence schemes such as Maildir or MBOX [RFC4155].¶

3.2.2. Interoperability

It should be mostly possible to use personal data exports from one system with different software or services. When a source exports personal data it can include all the information it would need for a fully-functional import, however destination systems running different software may not be able to import all of that information (especially if it includes non-standard features) and use it exactly the same way. This specification does not attempt to achieve perfect interoperability between diverse systems, but instead to make reasonable trade-offs.¶

3.2.3. Extensibility

This format should be extensible to accommodate types of personal data not explicitly mentioned or foreseen when writing these specs.¶

3.2.4. Flexible granularity

This format should allow flexible granularity in two ways:¶

1)

It should enable easy access to separate types of data (e.g., emails vs. contacts), e.g. to allow for partial imports or exports¶

2)

While ideally representable as a single file, archives may also span several files due to reasons such as file size restrictions or incremental generation logic.¶

(The ability for a user to export and/or backup an entire email account requires some accommodation of large amounts of data and risks of interruptions in downloads. Splitting exports into multiple files during export is one possible solution.)¶

3.2.5. Accessible for local tooling

PDPA should allow easy access for local tools (e.g., CLIs). While this may sound obvious, it is a key factor for the intended versatility of the format.¶

3.2.6. Efficiency

Since certain kinds of personal data might involve large quantities of data, major use cases for PDPA should be realizable in an efficient manner.¶

For now, this is stated as an abstract guiding principle. Its actual dimensions and trade-offs need to be refined while evolving this specification.¶

6.1.1. Archive section

6.1.2. Dataset section

6.1.3. Datasource section

6.3.1. Email

Each IMAP/JMAP folder is represented as subdirectory under "mail" directory. For example, the folder INBOX would be represented as "mail/INBOX", and the folder "Archive/2024/2024-12" would be represented as "mail/Archive/2024/2024-12".¶

(In examples above "/" is the IMAP hierarchy delimiter)¶

Folder names are encoded in UTF-8.¶

TODO: how to signal removal of a folder in an incremental archive? Need to add some kind of tombstone mechanism.¶

Each folder metatadata is described by "folder.json", which has the following format:¶

CDDL description of "folder.json" is included below:¶

;; /// Or possibly use ranges for 2 types below?
u32 = uint .size 4
u64 = uint .size 8

; uid => message filename map
uid_map = {* uid => filename}
uid = u32
filename = tstr

flags_map = {* uid => flags}
flags = [tstr]

modseq_map = {* uid => modseqs}
modseqs = u64


folder_info = {
  ? allowed_keywords: [tstr],
  last_uid: u32,
  ? highest_modseq: u64,
  ? recent_uid: u32,
  uidvalidity: u32,

  is_subscribed: bool,

  ; See RFC 6154 (SPECIAL-USE). E.g. "inbox", "sent", "drafts", "junk", etc.
  ? role: tstr,

  ; See Section 2, RFC 8621.
  ? sort_order: u32,



  ; See Section 3.5 of RFC 4314. For example "rwiptsldaex"
  ? myrights: tstr,

  ; Maps UIDs to the corresponding relative file names (names of .eml files)
  uids: uid_map,

  ; Maps UIDs to the corresponding array of flags/keywords (as strings)
  flags: flags_map,

  ; Maps UIDs to the corresponding modseqs (u64). See RFC 7162.
  ? modseqs: modseq_map,

  ; Original folder name if the name can't be represented in filesystem
  ? original_name: tstr,

  ; Can include information about partial export or filter used
  ; in human readable UTF-8 text
  ? comment: tstr,

  ;; The following attributes are only used for incremental exports:
  ? removed: [u32],
}

EML (.eml) file for each message, in order to avoid reconstructing them. Names of EML files are referenced from the "folder.json" file.¶

Example of folder.json (full export):¶

{
  "allowed_keywords": ["$Forwarded", "$MDNSent", "$ismailinglist"],
  "last_uid": 16,
  "highest_modseq": 6371729,
  "recent_uid": 15,
  "uidvalidity": 1107190787,
  "is_subscribed": true,
  "role": "sent",
  "sort_order": 1,
  "myrights": "rwiptsldaex",

  "uids": {
     1: "msg-1.eml",
     3: "msg-3.eml",
     15: "imported-ABC.eml"
  },

  "flags": {
     1: ["$seen"],
     3: ["$seen", "$flagged"],
     15: ["$answered", "$forwarded"]
  }
}

Example of folder.json that shows incremental changes from the previous export shown above. In this example 2 messages with UIDs 3 and 15 were removed. Message with UID 1 has updated flags. Several new messages were added, some of them are with flags set.¶

{
  "allowed_keywords": ["$Forwarded", "$MDNSent", "$ismailinglist"],
  "last_uid": 21,
  "highest_modseq": 6371845,
  "recent_uid": 20,
  "uidvalidity": 1107190787,
  "is_subscribed": true,
  "role": "sent",
  "sort_order": 1,
  "myrights": "rwiptsldaex",

  "uids": {
     1: "msg-1.eml",
     17: "msg-17.eml",
     19: "msg-19.eml",
     20: "20.eml",
     21: "21.eml"
  },

  "flags": {
     1: ["$seen", "$answered"],
     17: ["$seen", "$answered", "$forwarded"],
     19: ["$seen"],
     20: [],
     21: []
  },

  "removed": [3, 15]
}

6.3.2. Contacts

VCard [RFC6350] has long been the basis for address book and contact data representation in structured data files. The specifications for JSContact [RFC9553] and JMAP for Contacts [RFC9610] do a bunch of the work to explain how to do this in JSON, and in particular RFC9610 explains how to express references between objects (e.g. an address book and a contact in that address book) which is useful for a full export that can have its references reconstructed. This section explains how to use the fields and structures of those specifications within a PDP Archive export.¶

{
   "@type": "ContactCard",
   "version": "1.0",
   "uid": "22B2C7DF-9120-4969-8460-05956FE6B065",
   "id":
   "updated": "2021-10-31T22:27:10Z",
   "kind": "individual",
   "addressBookIds": [
      "062adcfa-105d-455c-bc60-6db68b69c3f3"
   ]
   "name": {
       "components": [
         { "kind": "given", "value": "John" },
         { "kind": "surname", "value": "Doe" }
       ],
       "isOrdered": true
   },

   "relatedTo": {
      "urn:uuid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6": {
         "relation": {
           "friend": true
         }
       }
   },

   "notes": {
     "n1": {
       "note": "Open office hours are 1600 to 1715 EST, Mon-Fri",
       "created": "2022-11-23T15:01:32Z",
       "author": {
         "name": "John"
       }
     }
   }
}

{
   "@type": "ContactCard",
   "kind": "group",
   "name": {
     "full": "The Doe family"
   },
   "uid": "urn:uuid:ab4310aa-fa43-11e9-8f0b-362b9e155667",
   "updated": "2021-10-31T22:27:10Z",
   "members": {
     "urn:uuid:03a0e51f-d1aa-4385-8a53-e29025acd8af": true,
     "urn:uuid:b8767877-b4a1-4c70-9acc-505d3819e519": true
   }
}

6.3.3. Using RFC9610 address book objects

The VCard [RFC6350] specifications never defined a representation for address books. Nor did JSContact [RFC9553]. JMAP for Contacts [RFC9610] does. Its model is clearly that of non-exclusive collection membership: a Contact item may appear with the same UID in multiple Address Books, and if the Contact item with that UID is updated in one it is updated in the other also.¶

Individual address book objects are returned in JMAP protocol messages with protocol wrappers. It is the items inside the "list" element inside "AddressBook/get" that are nearly ready to be represented as individual files in a PDPArchive. However, some things are missing: * uid is called 'id' in JMAP for Contacts but this specification REQUIRES uid. * updated is required * The @type of AddressBook should be included within the data¶

This example copies the examples in RFC9610 so that interoperability between this spec and that one is clear.¶

A file with an arbitrary name like address-book1.json would contain:¶

{
   "@type": "AddressBook",
   "uid": "062adcfa-105d-455c-bc60-6db68b69c3f3",
   "updated": "2020-01-09T14:32:01Z",
   "name": "Personal",
   "description": null,
   "sortOrder": 0,
   "isDefault": true,
   "isSubscribed": true,
   "shareWith": {
     "3f1502e0-63fe-4335-9ff3-e739c188f5dd": {
       "mayRead": true,
       "mayWrite": false,
       "mayShare": false,
       "mayDelete": false
     }
   },
   "myRights": {
     "mayRead": true,
     "mayWrite": true,
     "mayShare": true,
     "mayDelete": false
   }
}

address-book2.json would contain:¶

{
   "@type": "AddressBook",
   "uid": "cd40089d-35f9-4fd7-980b-ba3a9f1d74fe",
   "updated": "2020-01-09T14:32:01Z",
   "name": "Autosaved",
   "description": null,
   "sortOrder": 1,
   "isDefault": false,
   "isSubscribed": true,
   "shareWith": null,
   "myRights": {
     "mayRead": true,
     "mayWrite": true,
     "mayShare": true,
     "mayDelete": false
   }
}

Note that the first example includes a shareWith value, showing that the user's AddressBook has been shared with in this case one other principal with the id "3f1502e0-63fe-4335-9ff3-e739c188f5dd".
This information can be exported and may be quite useful in case of backup/restore use cases. However, it may not be useful in other administrative domains where the same concept of principals does not allow the Principal ID to be resolved against the correct account. In any case, the object referred to by this Principal ID is not itself given representation in the PDP Archive export.¶

6.3.4. Calendar events, tasks and groups

JSCalendar [RFC8984] is the basis for representing events, tasks and groups in JSON. This section explains how to export individual events and tasks within an archive. JMAP for Calendars (https://datatracker.ietf.org/doc/draft-ietf-jmap-calendars/) does provide some additional considerations when producing calendar data from a JMAP system or to be consumed by a JMAP system, so it is also a normative reference.¶

Note on CalDAV compatibility: Although CalDAV servers are fairly common, they support the older VEVENT and VTODO syntax. This specification requires the JSCalendar syntax instead. Either way, a server building a personal data archive is likely transforming an internal implementation-specific relational data format to an export format.¶

Note on ETag: CalDAV servers use the event's UID to identify the same object, and use ETags to identify changed events, so that a CalDAV client may make sure it has the version a server has before it updates an item, solving the lost-update problem. Since this specification doesn't attempt to solve the lost-update problem as well as client-server protocols can, and since JSCalendar does not include the ETag of a calendar event in any way, this specification does not include any requirements for ETags.¶

Notes on specific fields:¶

• The globally unique uid property is mandatory in JSCalendar and MUST be included. See JMAP Calendars [ref todo] section 1.4.1 for when the uid property can appear the same for multiple recurrences of the same underlying event.¶
• The updated property is mandatory in JSCalendar and MUST be included.¶
• The sequence value is optional in JSCalendar but SHOULD be included if available.¶
• The @type property for one of these items MUST be "Event", "Task" or "Group".¶
• Recurrence rules SHOULD be fully exported, unless it's clear from the use case or user request that the destination for the data wants expanded recurrences within a specific time period.¶
• The calendarIds field defined in JMAP Calendars is REQUIRED in order to match up events to the calendar they are supposed to appear in.
  ¶

For example, a file called event1.json could contain:¶

{
  "@type": "Event",
  "uid": "2a358cee-6489-4f14-a57f-c104db4dc2f2",
  "updated": "2020-01-09T14:32:01Z",
  "title": "Board Meeting",
  "start": "2024-10-25T09:00:00",
  "timeZone": "Europe/London",
  "duration": "PT1H30M",
  "participants": {
    "1": {
      "@type": "Participant",
      "name": "Jane Doe",
      "sendTo": {
        "mailto": "jane@example.com"
      },
      "roles": {
        "attendee": true
      }
    }
  },
  "calendarIds": {
    "062adcfa-105d-455c-bc60-6db68b69c3f3": true
  }
}

The event object includes a calendarIds property, which links it to the calendar collection it belongs to.¶

6.3.5. Calendar Collection Items

Calendar collection items are built using JMAP for Calendars [REF TODO - still I-D].¶

If a system exports events belonging to calendars, it SHOULD also export the referenced Calendar objects.¶

A file with an arbitrary name, such as calendar1.json, in a directory (e.g., \calendars\calendar2) would contain the calendar's metadata:¶

{
  "@type": "Calendar",
  "uid": "062adcfa-105d-455c-bc60-6db68b69c3f3",
  "updated": "2020-01-09T14:32:01Z",
  "name": "Work Calendar",
  "color": "#123456",
  "sortOrder": 0,
  "isDefault": true,
  "isSubscribed": true,
  "myRights": {
    "mayRead": true,
    "mayWrite": true,
    "mayShare": true,
    "mayDelete": false
  }
}

The uid value here corresponds to the ID used in the calendarIds property of the individual event item.¶

{
  "@type": "Task",
  "uid": "7b0f69a6-6e3e-4f1b-85d8-c89b43d2f2a1",
  "updated": "2022-11-23T15:01:32Z",
  "title": "Submit Quarterly Report",
  "status": "in-progress",
  "priority": 1,
  "due": "2024-12-31T23:59:59Z"
}

6.3.6. Notes

• VJournal is first defined in iCalendar (now [RFC5545]).¶
• VJournal also used in CalDAV¶
• However, they are NOT used in https://jmap.io/spec-calendars.html¶

Do we even have a JSON format for notes defined?¶

6.3.7. Files

6.3.8. Other

(LMD Note: I think this might better fit in an out of scope section - I think out of scope sections are useful for statements that explain why scope is limited. that's assuming we all agree that groups, freebusy and timezones are left out.)¶

Groups as defined in JSCalendar are NOT part of this archive format. Groups in JSCalendar can combine events and tasks in a container. This specification, for consistency and simplicity, uses folders and requires individual objects to be in separate files.¶

VFREEBUSY objects are not part of this archive format. Calendar software can calculate freebusy time from event data. Use cases that are not satisfied by this limitation could extend this archive format but understanding what it means to backup, restore, export or import freebusy data would need to be fleshed out.¶

VTIMEZONE objects are not part of this archive format. Timezones are more likely to be system objects referred to by calendar objects in modern calendar systems, than objects which are exchanged across systems.¶

6.4.1. Always include 'uid' and 'updated'

These requirements apply to JSContact [RFC9553], JSTask and JSCalendar [RFC8984] objects when exported or imported using the formats in this specification, because these all have 'uid' and 'updated' values.¶

Requirements:¶

• exporters MUST include the UID and updated fields.
  ¶
• The 'updated' field MUST be exported in UTC and interpreted in UTC. Accurate system time is important.
  ¶
• importers MUST use the UID in an imported object, if the importer is creating a new object, rather than invent a new UID.¶
• importers MUST search for existing objects with the same UID, and if the object in storage is similar enough (see Note below) to the import data, the importer SHOULD NOT change the object and MUST NOT update its 'updated' timestamp.
  ¶

Recommendations:¶

• Importers SHOULD use caution with fields that are system-updated, especially frequently updated. Such fields SHOULD NOT change the value of 'updated' that is exported or used to decide whether to update an object during an import operation. See note below on 'updated'.¶
• Importers SHOULD apply common sense in updating internal or implementation-specific fields. This specification does not require the importer to include, omit, handle or disregard values for fields that it believes are internally-generated or implementation-specific. For example, a system in the role of exporter might export an event object with a video-conference room ID in a custom field. It can decide that it is sensible to export that value as a URL for external use. Later, the same system or one with code written for compatibility could import that event with the video-conference URL, and it would be sensible to avoid overwriting its own knowledge of the room ID with the URL.¶
• When importing a changed or new object with a UID and 'updated' value, the importer SHOULD set the 'updated' value to the one imported. Thus, if a Contact is updated on Jan 1, exported on Jan 2 and imported on Jan 3, the new or updated imported contact would show an 'updated' value of Jan 1.¶

Note on similar enough: This specification requires nuance in order to allow both reasonably consistent synchronization and reasonable behavior in a wide variety of use cases and implementations. The language above is intended to give implementors both guidance and wiggle room. For example, the importer could convert a DTSTART time from UTC to the user's local time and save it as the displayed start time. Later, re-importing the same object with the same UID, the importing code could be smart enough to realize that the time hasn't actually changed, and avoid changing the 'updated' timestamp or creating a conflicting event. This logic could be implemented by saving separate fields (imported time vs display time), by keeping a log of updates (log entry stating that the system auto-converted start time from X to Y), or by other clever algorithms. Thus, the clever implementation can avoid the appearance of an object that changes every time the calendar is synchronized.¶

Note on updated: The definition of 'updated' in JSContact [RFC9553] is not rigorous or nuanced. "when the data in the Card was last modified" could refer to several instances of the card -- its internal implementation, its representation in an email share, its representation in an HTTP GET response [RFC6352]. It's not specified whether 'updated' is the same as REV in VCard [RFC6350], which is defined differently. Neither definition explicitly covers vendor-specific fields. Thus, this specification makes additional recommendations for handling 'updated':¶

• The value of 'updated' SHOULD only change when two conditions hold: the end-user makes a decision to change a value of a user-visible field, AND the export of the JSContact shows a different value.
  ¶
• Thus, non-user-visible fields like 'version' could be changed without causing the 'updated' value to change. A value such as 'language' could be set without changing 'updated' (if an implementation infers the language tag and begins to include 'fr-CA' as the language value in exports instead of no language, nevertheless this doesn't change the user-visible content).¶
• If implementations need to manage the synchronization of vendor-specific fields, a vendor-specific field like 'example.com:updated' can be used rather than affect the user-visible synchronization made possible by 'updated'. Implementations could possibly also handle 'updated' differently when used for export/import using the formats in this specification, than when the same field is handled in other code paths.
  ¶

We recognize that this understanding of 'updated' is highly judgement-dependent. The same field can change in one way and cause a change to 'updated' and in another way may not (the example of server inferring the language is 'fr-CA' vs the user explicitly setting it). It is likely to be frustrating to protocol designers and implementors (as it is to the authors of this specification) that the definition is so wobbly. We'd love to know of better solutions that work with the status quo.¶

6.4.2. Synchronizing address books

Build on [RFC9610]¶

6.4.3. Synchronizing mailbox folders

Because servers may differ in which characters they support in folder names, how many levels deep folders may be created, and even in what separator character is used to indicate folder hierarchy, difficulties in synchronizing folder names will definitely arise. Folder names that are not likely to be widely supported in other systems should be translated for export, because if the exporting system has a consistent translation algorithm, then even if the mailbox name looks different in the importing system it will still be imported consistently.¶

Systems that support mailbox IDs MUST include them in exports. Systems that do not (though it's strongly encouraged) SHOULD use the full mailbox name as the unique identifier value.¶

TODO: Also it would be good to include a "display name" in case the server has had to translate the mailbox name for compatibility. E.g. a server that has a mailbox named "%L33T%", but knows the "%" should not be exported because many servers forbid the "%", would translate the name consistently to _pc_L33Tpc or another set of safe characters and include a display name of "%L33T%" for reference and debugging.¶

6.4.4. Blobs and files?

Reference [RFC9404]?¶