OPVault design

The OPVault format was introduced in December 2012 and shortly thereafter became the default format for syncing with iCloud and Dropbox.

The Agile Keychain format was introduced in 2008 as a successor to using macOS Keychain integration. It proved to be much more reliable for syncing and gave us flexibility in design, efficient and reliable syncing, and portability across a variety of platforms. We designed it not only to withstand threats from 2008 but future threats as well.

Changes in available technology allowed us to improve on that design. And we again designed against threats that may not exist today, but which may develop in the coming years.

There are many design features that we keep in the OPVault format. We continue to:

• Use JSON files for most everything.
• Rely on existing cryptographic libraries, and do everything we can to avoid rolling our own. In particular, we use the CommonCrypto libraries for Mac and iOS.
• Use PBKDF2 for key derivation, and we’ve dramatically simplified other portions of our key derivation.
• Have as little information as necessary decrypted at any one time when the data is “unlocked”.
• Be open about our data format, implementations, and design.

However, we also took the opportunity to make many improvements.

Improvements over Agile Keychain

Fewer separate files

The Agile Keychain format used a separate file for each item. Experience has taught us not only that having each item in its own file creates difficulties for some filesystems, but that it also brings a penalty with data syncing as well. There is an overhead for syncing each individual file regardless of its size. Having hundreds or thousands of small files led to inefficient syncing in some cases. On the other hand, we don’t want a single monolithic file, which would introduce its own problems for syncing 1Password data across systems.

We combine items into 16 bands depending on the item’s unique identifier (UUID. For example, all items that have a UUID beginning with “C” will be listed in the file band_C.js. Band files range from band_0.js to band_F.js. If there are no items in a particular band, there will be no band file for it.

We also design a data format that is appropriate for each platform and have a process that syncs the OPVault with a format that is most appropriate for the application. We had already been doing this in some situations with earlier versions of 1Password, but in OPVault it is a fundamental design decision. The file formats used for syncing are never directly used by the local application.

We will refer to the particular data files and formats used directly by apps on particular platforms as the “local data”. The local formats will use the same encryption design as discussed here, but will (typically) be SQLite files. But this document just focuses on OPVault, which is the format of data that will be used for syncing and storage across networks, initially with iCloud.

Encrypting everything

The Agile Keychain kept some information (most notably Location and Title) unencrypted so that these could be used to search for or identify a particular item, whereas the more sensitive content could remain encrypted. With the Agile Keychain format, the 1Password classic extension could identify and list potential matches for a website without having to be “unlocked”. With the OPVault format, we have moved away from that. The user must unlock the data with their vault password before they can see a list of Logins.

Overview data and full content

However, even when 1Password is unlocked, we want to keep as little sensitive data decrypted as possible at any one time. So in the new format we make a distinction between “overview” data and the full encrypted content. Overview data contains the information that is used to sort, index, and search for items in the keychain. We can decrypt the overview data for all items after the vault password is entered, but keep the details for each item encrypted until the particular item’s content is needed.

Unencrypted metadata

Some metadata remains unencrypted:

• which folder an item is in
• what category (Login, Credit Card, …) an item belongs to
• creation time
• modify time
• last sync time

The item UUIDs are fully available, which can be used to determine how many attachments, if any, an item has associated with it. The UUID of any folder an item belongs to is unencrypted, and thus an attacker can determine which items are in the same folder.

Ciphersuites

We use Encrypt-then-MAC authenticated encryption everywhere we use encryption. The MAC is HMAC-SHA256 and encryption is AES-CBC using 256-bit keys. Key derivation is uses PBKDF2-HMAC-SHA512. More detail about these choices will be presented in the relevant sections on key derivation and item encryption.

In this document we will refer to “blocks of data”. Unless otherwise stated, blocks are the length of AES blocks, 128 bits (16 bytes).

Authenticated encryption and data integrity

When the Agile Keychain format was developed, chosen ciphertext attacks (CCA) were seen as theoretical. Furthermore, the primary threat to people who use 1Password was thought to be from an attacker stealing the data once and pursuing an off-line attack. It did not anticipate an attacker who could tamper with user data that would be subsequently processed by the legitimate owner.

Instead of trying to design against particular CCAs or particular mischief that can be done through data manipulation, we authenticate everything we can. Authenticated encryption is used whenever we encrypt, and HMACs are calculated over the elements in each item. The item is rejected if the MAC does not verify. The Encrypt-then-MAC construction is better thought of as “Verify-and-only-then-Decrypt”.

Building blocks

Keys

We handle a variety of keys, the purpose and derivation of which will be described in later sections.

1. Vault password (OK, this one isn’t a “key”).
2. Derived encryption key.
3. Derived MAC key.
4. Master encryption key.
5. Master MAC key.
6. Overview encryption key.
7. Overview MAC key.
8. Item encryption key (item specific).
9. Item MAC key (item specific).

As an overview of the role of these various keys, the secret content of each item is encrypted a unique pair of keys (encryption and MAC) for that item. The relationship between an item key and the vault password can be summarized in song:

Each item key’s encrypted with the master key
And the master key’s encrypted with the derived key
And the derived key comes from the MP
Oh hear the word of the XOR
Them keys, them keys, them random keys (3x)
Oh hear the word of the XOR

All of the encryption keys are 256-bit keys for use with AES. This has been a highly requested change. Although 128-bit keys remain more than sufficiently resistant to any brute force attack, moving to 256-bit keys has the side effect of leading us to use a SHA512 in our key derivation, which may add meaningful strength defenses against password crackers. Additionally, as the systems that 1Password normally runs on are much more powerful than they were years ago, the additional overhead of using 256-bit keys is no longer a reason not to.

We are aware of the key schedule concerns surrounding 256-bit AES keys, and we will continue to keep an eye on attacks against it. We can modify our format and cipher suites to return to 128-bit keys if necessary. The MAC keys are also all 256-bit keys.

All keys (other than the key encryption keys derived from the vault password) are created using cryptographically appropriate pseudorandom number generation.

UUIDs

As in the Agile Keychain format, each item is associated with a universally unique identifier, the UUID. These are 128-bit numbers that are chosen as RFC 4122 Version 4 UUIDs when an item is first created. In what follows, we will often use the term “UUID” to refer to the uppercase hexadecimal representations of a UUID. It should be clear from context when we mean the hexadecimal representation and when we mean the number itself.

Because each UUID is chosen at random, it contains no information about the content of an item. These UUIDs reveal no information about the creators system other that than the fact that they are RFC 4122 Version 4 UUIDs. When a user modified information in an item the UUID remains the same, although the time stamp associated with it will change.

opdata01

All encrypted data with the exception of some encrypted keys is a specific format we are calling “opdata01”.

The first 8 bytes of the data are the string “opdata01”. The next 8 bytes contain the length in bytes of the plaintext as a little endian unsigned 64 bit integer. The next 16 bytes are the randomly chosen initialization vector.

The plaintext is padded using the following scheme.

If the size of the plaintext is an even multiple of the block size then 1 block of random data is prepended to the plaintext. Otherwise, between 1 and 15 (inclusive) bytes of random data are prepended to the plaintext to achieve an even multiple of blocks.

The data – excluding the IV but including the prepended padding – is encrypted using AES in CBC mode with a 256-bit encryption key.

The HMAC-SHA256 is computed over the entirety of the opdata including header, length, IV and ciphertext using a 256-bit MAC key. The 256-bit MAC is not truncated. It is appended to the ciphertext.

Depending on where opdata is stored within data files it may be base64 encoded.

Implementation

On the Mac and on iOS we use the CommonCrypto libraries, using kCCHmacAlgSHA256 and kCCAlgorithmAES128. We set up the cryptor with CCCryptorCreate(kCCEncrypt, kCCAlgorithmAES128, 0, encryptionKey, 32, IV, &cryptor) and the MAC with CCHmacInit(&context, kCCHmacAlgSHA256, MACKey, kCCKeySizeAES256)

Rationale

We perform authenticated encryption for the reasons described in the authenticated encryption section. CommonCrypto does not, at this time, offer any direct authenticated encryption modes, so we use Encrypt-then-MAC composition.

We do not use kCCOptionPKCS7Padding as this can lead to padding oracle CCAs (although our use of Encrypt-then-MAC should prevent all CCAs). Instead we do our own random padding as described above. By putting the padding up front, this has the effect of acting like a smaller, additional, IV.

The IETF draft for authenticated encryption with associated data was not available in time for our first version of this, but we will certainly consider the simpler padding proposed there in the future.

Random number generation

1Password uses the Cryptographically Secure Pseudo-Random Number Generators (CSPRNG) provided by the respective operating systems for all its random number needs. On Mac and iOS, that is SecRandomCopyBytes(), on Windows that is CryptGenRandom(), and on Android that is SecureRandom(). These are used to create the master keys (encryption and MAC), overview keys (encryption and MAC), and PBKDF2 salt when a new vault is created. Subsequently, random numbers are used for item keys (encryption and MAC), initialization vectors for CBC encryption, random CBC padding, and UUIDs.

Directory layout

The top level directory is called onepassword_data and will contain a directory for each profile. Default is the only profile so far. The default folder will contain 16 band files, a folders.js file, a profile.js file, and any number of attachments.

Here is the layout for a sample OPVault:

{
onepassword_data
-- default
	|-- 026AA7B7333B4F925F16DE9E21B912B7_5754B83288A34CD39DE64B45C2F05A9D.attachment
	|-- 6F8CDF100CC99FD55053B77492D97487_072A1462CBDE4E2488FB2DA16D96B84B.attachment
	|-- 7BAFBD42CA6F487C94DEC95C187A88A7_C10EA572E11246019A76C4657E302C8B.attachment
	|-- BD1B7C38FDC647E188E172DE2C06C391_A766BB1A9D484174B0D3453E8142118C.attachment
	|-- EE2B42ABEB4D4276AEA51006B3A9C47C_CC20BEA583014AD38E2899853CC48DB8.attachment
	|-- F6A28725D3F0F4DCB857C62A10E612F6_E390C5572D1848E3B128C416162CA60B.attachment
	|-- band_0.js
	|-- band_1.js
	|-- band_2.js
	|-- band_3.js
	|-- band_4.js
	|-- band_5.js
	|-- band_6.js
	|-- band_7.js
	|-- band_8.js
	|-- band_9.js
	|-- band_A.js
	|-- band_B.js
	|-- band_C.js
	|-- band_D.js
	|-- band_E.js
	|-- band_F.js
	|-- folders.js
	-- profile.js
}

Band files

An individual band file, say, band_4.j, will contain all the items with UUIDs that begin with ‘4’.

The contents of a band file containing two Login items may look like:

{
ld({
  "468B1E24F93B413DAD57ABE6F1C01DF6": {
    "category": "001",
    "created": 1325483950,
    "d": "b3BkYXRhMDFcAQ ...",
    "folder": "C8CE328220DF4157961787FBA30DAB96",
    "hmac": "30d0rq4tFQvU++mIOElzzI1zilZS0M4Ya10TXmsFz2c=",
    "k": "KGRbxc/qWol4sbs7bwwctO ...",
    "o": "b3BkYXRhMDGCAAAAAAAAAB0ZLRV9xZidd ...",
    "tx": 1347560906,
    "updated": 1325483950,
    "uuid": "468B1E24F93B413DAD57ABE6F1C01DF6"
  },
  "4E36C011EE8348B1B24418218B04018C": {
    "category": "001",
    "created": 1325483950,
    "d": "b3BkYXRhMDFcAQAAAAAAAIyTActXjlxJEgx ...",
    "hmac": "T/8VMqYTr8WJDJqPrJyJl+bFcmS28dn2ZBICwS6d+Tw=",
    "k": "87VRNWy3Pj6 ...",
    "o": "b3BkYXRhMDFyAAAAAAAAADkKPiRdEPFXmLQG9T ...",
    "tx": 1347560906,
    "updated": 1325483951,
    "uuid": "4E36C011EE8348B1B24418218B04018C"
  }
});
}

Items

And individual item within a band file may look like:

{
"468B1E24F93B413DAD57ABE6F1C01DF6": {
    "category": "001",
    "created": 1325483950,
    "d": "b3BkYXRhMDFcAQ ...",
    "folder": "C8CE328220DF4157961787FBA30DAB96",
    "hmac": "30d0rq4tFQvU++mIOElzzI1zilZS0M4Ya10TXmsFz2c=",
    "k": "KGRbxc/qWol4sbs7bwwctO ...",
    "o": "b3BkYXRhMDGCAAAAAAAAAB0ZLRV9xZidd ...",
    "tx": 1347560906,
    "updated": 1325483950,
    "uuid": "468B1E24F93B413DAD57ABE6F1C01DF6"
  }
}

The item is labeled by its UUID which is also one of the elements within the data. The MAC is calculated over all the items (in C lexicographic order) except for hmac itself and folder. The folder is excluded because there may be legitimate instances of an item changing its folder when the MAC key is unavailable.

category

Item category indicates whether the item is a Login (“001”), Secure Note (“003”), Credit Card (“002”), and so on. These are three digit decimal numbers.

Rationale

category is left unencrypted so that a list of, say, Email items can be presented without having to decrypt the overviews of every item in within the keychain. If category was specified within the encrypted overviews then it would be necessary to decrypt every overview to identify the particular items to be listed.

created

Unix time of item creation. All time stamps are Unix time and are written as an ASCII decimal number. Internal representation of Unix time stamps involves data types that are based on 64-bit integers and so will handle dates beyond January 2038.

d

This is the encrypted details for the item. It is base64 encoded opdata. It is encrypted using the item keys in “k”.

fave

The presence of fave indicates that the item is to be listed among “favorites”. The value, an ASCII representation of an unsigned long integer is used as a sort index, with lower numbers sorted first.

It is useful to have large gaps in the sort indices, so that items added later to favorites can be inserted in the sort order without having to renumber existing items. For example, the item to be sorted first may have a sort index of 1000 and the item to be sorted second may have a sort index of 2000. The allows someone to add a new favorite item to be sorted be sorted between those two, sort index 1500, without having to modify the existing items.

Rationale

fave, like category, is left unencrypted so that a list of favorites can be displayed without having to decrypt the overviews of every item in the keychain. If fave were to be among the overview data, then the overviews of all items would need to be decrypted to identify which items are favorites.

folder

This is the UUID of any folder an item may belong to.

hmac

This MAC (HMAC-SHA256) is computed over all the elements of the item with the exception of the hmac itself. It’s computed over the item elements and their values. Very pseudo code:

{
elements = Sort_with:@selector(compare:);
for (element in elements) {
    if (element == "hmac") continue;
    CCHmacUpdate(&context, [element UTF8String], [element length]);
    *data = [dict[element] description];
    CCHmacUpdate(&context, [data UTF8String], [data length]);
}
}

The HMAC is computed using SHA256 using the overview MAC key, and it is base64 encoded.

k

This is the encrypted item and MAC keys. It is encrypted with the master encryption key and authenticated with the master MAC key.

The last 32 bytes comprise the HMAC-SHA256 of the IV and the encrypted data. The MAC is computed with the master MAC key. The data before the MAC is the AES-CBC encrypted item keys using unique random 16-byte IV.

The decrypted object is defined by:

typedef struct {
  uint8_t crypto_key[32];
  uint8_t mac_key[32];
};

This is base64 encoded.

Rationale

One reason to provide a separate key for each item is to make sure that only a safe amount of data is encrypted under one key. Although we don’t expect item details to grow particularly large, any attachment to an item is also encrypted under the item key.

o

This is the encrypted overview for the item. It is base64 encoded opdata, encrypted with the overview encryption and MAC keys.

There are still cases where we need to find items without having to decrypt all the data. So we separate the data in each item into “overview data” and “encrypted data”. In the clients, the overview data may remain unencrypted (in memory) for the entire time that 1Password is unlocked, whereas the restricted data for any item is only briefly decrypted.

The decrypted overview data includes the URLs (yes, that is plural) associated with an item, the Title associated with the item, and some additional information depending on the category of an item. For a Login, the username for the item is included. For a Password, the creation time is included. For a secure note, the first 80 bytes of the note are included. For an Identity, the full name is included. For a Credit Card, the masked credit card number is included.

trashed

If "trashed": true appears in the item, it indicates that the item has been moved to the Archive. The default for trashed is false, so the absence of it indicates that the item is not in the Archive.

tx

This is the “transaction timestamp” (Unix time) indicating when this particular item was last modified or added to the current database through syncing. When an item is modified or added to the database through syncing, this item will be updated to reflect that. This along with the updated timestamp are used to manage syncing.

updated

This is a Unix timestamp indicating when the contents of the item were last modified.

uuid

This is the UUID of the item. Its inclusion within the item makes sure that it is included when the hmac is calculated.

folders.js

The file folders.js has a similar structure to a band file, and each folder listed has a structure very similar to an item in the band file. For security purposes, it is worth noting that the folder name and the search query for defining a smart folder are encrypted with the overview key. In this version, other than the actual encrypted data, there is no MAC for each folder nor for the list of folders themselves.

Here is a sample entry:

{
"379A3A7E5D5A47A6AA3A69C4D1E57D1B": {
    "created": 0,
    "overview": "b3BkYXRhMDES ...",
    "tx": 1373753421,
    "updated": 0,
    "uuid": "379A3A7E5D5A47A6AA3A69C4D1E57D1B"
}
}

The overview is encrypted with the overview keys, and contains the name of the folder for ordinary folders.

In this sample, it is clear that this folder was created and last modified at the very beginning of time itself. (Or it was merely created in the pre-beta version of 1Password used to create this data sample.)

Some folders may also contain "smart": true,, indicating that it is a smart folder. The overview in those cases will contain, in addition to its name, a definition of the search predicate used to define the smart folder. The specification of those is still in flux and is not, at this point, expected to be portable.

profile.js

This file contains the encrypted master and overview keys. Here is a sample:

{
var profile={
    "lastUpdatedBy":"Dropbox",
    "updatedAt":1325483930,
    "profileName":"default",
    "salt":"LlfotX7aTZnfL+AjJJYARA==",
    "passwordHint":"fred",
    "masterKey":"b3BkY ...",
    "iterations":227272,
    "uuid":"FFF0089247244BDEAAE714B9AC6FF129",
    "overviewKey":"b3BkYXRhMD ...",
    "createdAt":1347560904};
}

The masterKey and the overviewKey are in base64 encoded opdata objects. iterations is the number of PBKDF2 iterations. salt is base64 encoded 16 byte random salt. These will be described in more detail in the section on key derivation.

lastUpdatedBy is used to help make the syncing process more efficient. The password hint is stored unobfuscated. We did not feel that we could devise an obfuscation system that could sufficiently resist reverse engineering for the very limited gain of obfuscating the hint.

Key derivation

We have been able to simplify key derivation because we no longer have multiple security levels in the Agile Keychain format, where each security level required its own master key. The key derived from the user’s vault password is used to authenticate and decrypt the masterKey and the overviewKey in profile.js.

From vault password to derived encryption and MAC keys

The vault password is converted to a UTF-8 null terminated string. This is then used in the Common Crypto key derivation function CCKeyDerivationPBKDF, using PBKDF2, the salt, and HMAC-SHA512 as the pseudorandom function to derive the encryption key and the MAC key. The first 256-bits are the derived encryption key and the remaining 256-bits are the derived MAC key.

Using SHA2-512 serves two purposes:

1. It gives us enough raw material to for both an encryption key and a MAC key (each 256-bits).
2. By using a wide hash (512 bits) in PBKDF2, we make it more difficult for GPUs which are constrained by their memory access limitations.

At the moment our choice of PBKDF2 over scrypt is largely motivated by the fact PBKDF2 is available in the system libraries for the platforms that we use. We may migrate to scrypt, or some other successor to PBKDF2, in future.

From derived keys to master keys and beyond

The derived keys are used to decrypt the contents of masterKey and overviewKey. The masterKey and the overviewKey are base64 encoded opdata objects.

The plaintext of the masterKey is 256 bytes of data selected randomly when the keychain was first created. The plaintext of the overviewKey is 64 bytes selected randomly when the keychain was first created.

The 256 byte (2048 bit) plaintext content of the masterKey is then hashed with SHA-512. The first 32 bytes (256-bits) of the resulting hash are the master encryption key, and the second 32 bytes are the master hmac key.

The master keys (encryption and MAC) are used to encrypt the item specific item keys within each item. These in turn are used to encrypt the data within the item and also any attachment associated with the item.

The overview keys (encryption and MAC) are derived in the same way as the master keys. The contents of overviewKey is verified and decrypted with the derived keys; the result is then hashed with SHA-512; the first 32 bytes of the hash output will be the overview encryption key; the last 32 bytes of the hash output will be the overview hmac key.

The overview key is used for encrypting the overviews in each item as well as the overviews in for each folder.

Attachments

Attachment files are named with the UUID of the item that they are attached to followed by an underscore and then followed by the UUID of the attachment itself. The file is then given the extension .attachment.

The contents can be divided into four sections.

1. The header
2. The meta-data (up to 2^16^ bytes)
3. The encrypted icon (up to 2^32^ bytes)
4. The encrypted data (up to 2^32^ bytes)

header

Counting from zero

metadata

This is then followed by a JSON object containing the meta data (displayed here with whitespace for legibility, but there is no whitespace in the actual attachment file).

{
{
    "itemUUID":"EE2B4 ...",
    "contentsSize":277153,
    "external":true,
    "updatedAt":1328222369,
    "txTimestamp":1347487884,
    "overview":"b3BkYXRhMDEiAAAAAAAAACMP4CJz/gLE4lwhnyahSY0jBL ...",
    "createdAt":1328222369,
    "uuid":"CC20BE..."
}
}

encrypted icon

This is opdata. In attachment files these are not base64 encoded.

encrypted contents

This is opdata, encrypted with the associated item and MAC keys. In attachment files these are not base64 encoded.

Sample data and further details

For those who want to explore and analyze this further, we have posted sample data.

Although we don’t anticipate publishing source code for manipulating 1Password keychains, others, unaffiliated with AgileBits, have done so. You can read more about this in:

• You have secrets; we don’t, why our data format is public