Exploring 1Password's Crypto With Go

Posted on 30 May 2015

1Password is a simple and convenient application to keep your passwords safe. Im a big fan since 2013 when i got super tired of maintaining lots of passwords and decided to finally end the struggle of remembering the damn thing for each account. It paid off and i now have over 100+ passwords and secure notes. 1Password also provides an iPhone app with TouchID support. Handy!

Until now i had absolutely zero interest in how 1Password works under the hood. My previous post also involved some data encryption so i decided to dig in and research how does 1Password manage to keep its secrets. It also does not hurt to know more things about cryptography and its applications to software development in general.

Vault / Agile Keychain

First things first: where's the stuff? Under the hood 1Password keeps all its data in a single Sqlite3 database, called Vault. If you're using OSX, vault files are located at the following path:

ls -al ~/Library/Containers/2BUA8C4S2C.com.agilebits.onepassword-osx-helper/Data/Library/Data

-rw-r--r--@  1 sosedoff  staff   421888 May 28 20:43 OnePassword.sqlite
-rw-r--r--@  1 sosedoff  staff    32768 May 29 20:02 OnePassword.sqlite-shm
-rw-r--r--@  1 sosedoff  staff  1697472 May 30 08:39 OnePassword.sqlite-wal

OnePassword.sqlite is the main database file, OnePassword.sqlite-wal is file used for Write Ahead Logging and OnePassword.sqlite-shm is a temp file used to share memory during updates. It's highly recommended to not mess with the database directly and use sync feature instead, which is designed to export 1password's data to cloud providers. Sqlite database is intended for internal usage only.

Agile Keychain in a directory named as "*.agilekeychain". Its houses a copy of data stored in sqlite database: master encryption keys and all your passwords. Keychain is made to be very portable and simple. Indeed, one could use Dropbox sync to update keychain on all devices. In most cases agile keychain is also referred to as a vault.

Let take a look at directory structure:

$ ls ~/Dropbox/1password/1Password.agilekeychain/data/default

-rw-r--r--@  1 sosedoff  staff  6550 Nov 29  2013 .1password.keys
-rw-r--r--@  1 sosedoff  staff  6550 Nov 29  2013 1password.keys
-rw-r--r--@  1 sosedoff  staff  9849 May 28 10:01 contents.js
-rw-r--r--@  1 sosedoff  staff  6014 Nov 29  2013 encryptionKeys.js
-rw-r--r--@  1 sosedoff  staff   645 Apr 12 12:00 02B3B2D9239B4F75B0B76FEE21041A31.1password
-rw-r--r--@  1 sosedoff  staff  1469 Sep  7  2014 05DF594381C440BFA8F3149F9CC1F71F.1password
-rw-r--r--@  1 sosedoff  staff   630 Nov 10  2014 06ADBCB6993A42BCA4B04CEEDCE1988A.1password
-rw-r--r--@  1 sosedoff  staff   607 Dec  2  2013 0792B0BBE0D24AFFBD850F1A97AA4B3E.1password

There are two files .1password.keys and 1password.keys which are exactly the same and hold master encryption keys for the whole keychain in xml plist. format. Another file encryptionKeys.js represents a JSON version of the same encryption keys, which is easier to read and work with.

Lets take a look at encryptionKeys.js contents:

{
  "SL3": "6F08ADB92B284CF5B021ED5411450A59",
  "SL5": "830EF6E33D8947648FFF28C4C52863BF",
  "list": [
    {
      "data": "U2FsdGVkX1+arfeNeAr3bjLcYIsOqtF2w98XN6FH7bo...",
      "validation": "U2FsdGVkX191D55NSlStJT6NImLPRwCuFcWOc...",
      "level": "SL5",
      "identifier": "830EF6E33D8947648FFF28C4C52863BF",
      "iterations": 83333
    },
    {
      "data": "U2FsdGVkX18ghmg2cVZaTqrhXvJLjN0uI8O7y6Ka9Xcs...",
      "validation": "U2FsdGVkX18QMP1Hhi5BK1TLIzfUes1j23hdcs...",
      "level": "SL3",
      "identifier": "6F08ADB92B284CF5B021ED5411450A59",
      "iterations": 83333
    }
  ]
}

SL3 and SL5 properties in the json data represent security level identifiers used to associate encryption key with a password item. list property holds the detailed collection of keys used to encrypt all passwords in the keychain. data property of a key is a base64-encoded decryption key which is encrypted by a key derived from your master password using PBKDF2 with number of iterations. The master password itself is not being stored anywhere and comes from a user during keychain "unlock". validation property represents a copy of decryption key encrypted with itself.

Rest of the keychain directory is filled with *.1password files. These are actual secret files with your account details, etc. Example:

{
  "uuid": "02B3B2D9239B4F75B0B76FEE21041A31",
  "updatedAt": 1428858020,
  "locationKey": "example.com",
  "securityLevel": "SL5",
  "contentsHash": "82b86c8e",
  "title": "My Password",
  "location": "example.com",
  "encrypted": "U2FsdGVkX19DVMD68Wv7dp5m\/IP\/JUMLD++itV\/GWrQ5+...",
  "txTimestamp": 1428858020,
  "createdAt": 1428857993,
  "typeName": "webforms.WebForm"
}

Most of the password file properties are plaintext, such as security level used to reference master encryption key, and other metadata. What we're interested in is the encrypted attribute which represents a password's encrypted JSON payload. Its content is encrypted with a 1024-byte randomly generated key.

Lets Decrypt

Now that we understand the basics, lets try to decrypt keychain programmatically using Go. Language choice is purely for experimental purpose. Instead of testing code on an existing vault, for safety reasons i created a new vault with a simple master password and then exported it into agile keychain. Keychain only includes one password item with the following:

  • Login: gmail.com
  • Password: mygmailpassword

After the export i grabbed all necessary data form keychain files:

key        = U2FsdGVkX18O742tiin24L289EipX0yfOvS24X+XPnS2MRItGW1qQr4Lw5qd6dMYGa7zhbMUY1K/bbO0i7X0gwP/QgxCOHlyCnosq9yH0SIP7uQ8StAE1gxWaY2GM42oCIaKgXS3uvvSrtKZOyJizmgnMcA4KR5fNbfRHQoiZaWpT+2L+eckYeglN/+FN4YuAwgREzydBbzwGrJkDwWGuaHdXVBVRUMNjqJy7lP7gQ8iIVh6dPR1V733cgckPWLY6BaWs4CMrFu+/UOuPSvtnDXGjHCEXzwLjPKNKCvTF2h585Nl3zlMw/G/ZSlg0sjdy0MHOjIMMRjTKidzICNbNZgEKBe7BlW/MUBmaV4Vca+om2I8kcnC/7fBOevmZG810/zWlsPgt6nStroIMFeq1Vmqc4GnFa2GZcpKIpD7QJwAfcTfN0ALBlXgaXzO0/3bZduN2rMhoUygVPGncEcsnMfFKyknbuUXW/RHNxoOdS3k7n18yi6ZoQAeV+AVeKgHCinrGoKmmWHNTyvnZ/AXtkceuLcu99PaZidAErQdTsxxtCAUfPD++2Qs1LKwr/J9z1cnJM7TybBB+J4uNyWVLIV4WhyHGL4SZmCT2Apf0ZbqPWFKaSkelg96S5xQ0+epMxv4r6qsZoIBBJjijboWfl6UTgSzqcaDeQUbajhKTEZl65zqq00y3BG51i6AuLpjkhKNVitc/Xhu1dOT5YEja3wGsSLL2axSqSk2PG80PmprHduTkLArQkeEihED3UhtMVC40CvRcjr17FTxRjkDI2iZfDYWIxOIFDUZ1LPwiJXP3Egoi+HRFSZqQ5Uyv6WaET3a+drQFHHw+bliWYbRodx/qqb8t6ubnox/i9iaQ+eYqRXrKz58uXqBcJBhpa3+uU+Vj92qBBBxcCReZ+X5xYf9Kv8JOmjturFG1rG31qPrFuEKT+6xNSYa8qWP7rk3uKo/SZES03f4fwxDmP1/SxZtVtufWqkvvYM7ml6zwOj6fY2+65+3x2a3A/stAVtYjWLtqJbbzjzj2nSEcXr7+QN3TP0gRotbU+iLwgkrP97FPnj8ri3vmnQgqquvKu5C4dqNo6lwQuB/PaS9MF+vfHJlkDaTbk0SqfiWwjSnGJ4wQvGUCxSP2GUio4HoMmyuBy7gQ/UsU9ijuoVt/Geu3OwTpa8B7mmiWzcd5cwq/kjxJ9l460dhcyvLfoSWCDRVyvQd2kDAPRxqZ3FPKu4RNqOXr5EnqMqCeA3p6kp+copCgQUEEJxduisgYzNxdjgViWYbYzOgepKCq0jqOIrg7Rud3vD152VhbLXORyqmoYVZ3sNWTMpPCdD1mO2zkierl2JZRf78EpYUq6rzdGS9ZXApYSMH0iRIrC9rxHdHiMNPm5WVU6clVIDbcvyhMoGu
password   = U2FsdGVkX19rzutpu67A2hGyjdoLGSnWEtTGcfSKK7w4SUATx2y3bB0HdvXpTrODvJjAa73eJUDF+oRMYVyh6k9EIBCW8XI+1aKbtQGNVnJtRoW7xrpkl3TdgC++x0e4UPjB+LBA79LK8gHTr4Ad2Kd06dWTxwpqAgsLU3VI6ZUXvoctUk9ffxCHzZocQBBHnEInovrp+/kUHZxYWZkS6/2wZiY60PH9gYyatenh472mNgS9h0sH3FW38HFVKudhxj4uIiYr+gO3Y6XxTq9th7ykhV4FwpYXL9xbKptE2H8=
iterations = 100000

Workflow

Our objective is to get code to produce the same login and password, that way we know it works. Lets overview all steps before we dig into any code:

  1. First, convert key (A) and password (B) data from base64 to raw bytes.
  2. Extract salt (C) from decoded key (A) data.
  3. Derive a new key by applying PBKDF2 with master password, extracted salt (C) and number of iterations.
  4. Extract an AES-128 key and initialization vector (IV) from derived key.
  5. Decrypt original key (A) data with extracted AES key and IV from previous step.
  6. Extract password salt (D) from decoded password (B) data.
  7. Derive an AES-128 key and IV from decrypted key (from step 5) and salt (D) using MD5.
  8. Decrypt password data with extracted AES key and IV from previous step.

Code

Instead of going to review each chunk of code individually, i put together a code snippet that implements decryption mechanisms:

package main

import (
  "bytes"
  "crypto/md5"
  "crypto/sha1"
  "encoding/base64"
  "encoding/json"
  "log"
  "strings"

  "github.com/spacemonkeygo/openssl"
  "golang.org/x/crypto/pbkdf2"
)

func main() {
  masterpass := []byte("pa$$w0rd")
  encodedKey := `U2FsdGVkX18O742tiin24L289EipX0yfOvS24X+XPnS2MRItGW1qQr4Lw5qd6dMYGa7zhbMUY1K/bbO0i7X0gwP/QgxCOHlyCnosq9yH0SIP7uQ8StAE1gxWaY2GM42oCIaKgXS3uvvSrtKZOyJizmgnMcA4KR5fNbfRHQoiZaWpT+2L+eckYeglN/+FN4YuAwgREzydBbzwGrJkDwWGuaHdXVBVRUMNjqJy7lP7gQ8iIVh6dPR1V733cgckPWLY6BaWs4CMrFu+/UOuPSvtnDXGjHCEXzwLjPKNKCvTF2h585Nl3zlMw/G/ZSlg0sjdy0MHOjIMMRjTKidzICNbNZgEKBe7BlW/MUBmaV4Vca+om2I8kcnC/7fBOevmZG810/zWlsPgt6nStroIMFeq1Vmqc4GnFa2GZcpKIpD7QJwAfcTfN0ALBlXgaXzO0/3bZduN2rMhoUygVPGncEcsnMfFKyknbuUXW/RHNxoOdS3k7n18yi6ZoQAeV+AVeKgHCinrGoKmmWHNTyvnZ/AXtkceuLcu99PaZidAErQdTsxxtCAUfPD++2Qs1LKwr/J9z1cnJM7TybBB+J4uNyWVLIV4WhyHGL4SZmCT2Apf0ZbqPWFKaSkelg96S5xQ0+epMxv4r6qsZoIBBJjijboWfl6UTgSzqcaDeQUbajhKTEZl65zqq00y3BG51i6AuLpjkhKNVitc/Xhu1dOT5YEja3wGsSLL2axSqSk2PG80PmprHduTkLArQkeEihED3UhtMVC40CvRcjr17FTxRjkDI2iZfDYWIxOIFDUZ1LPwiJXP3Egoi+HRFSZqQ5Uyv6WaET3a+drQFHHw+bliWYbRodx/qqb8t6ubnox/i9iaQ+eYqRXrKz58uXqBcJBhpa3+uU+Vj92qBBBxcCReZ+X5xYf9Kv8JOmjturFG1rG31qPrFuEKT+6xNSYa8qWP7rk3uKo/SZES03f4fwxDmP1/SxZtVtufWqkvvYM7ml6zwOj6fY2+65+3x2a3A/stAVtYjWLtqJbbzjzj2nSEcXr7+QN3TP0gRotbU+iLwgkrP97FPnj8ri3vmnQgqquvKu5C4dqNo6lwQuB/PaS9MF+vfHJlkDaTbk0SqfiWwjSnGJ4wQvGUCxSP2GUio4HoMmyuBy7gQ/UsU9ijuoVt/Geu3OwTpa8B7mmiWzcd5cwq/kjxJ9l460dhcyvLfoSWCDRVyvQd2kDAPRxqZ3FPKu4RNqOXr5EnqMqCeA3p6kp+copCgQUEEJxduisgYzNxdjgViWYbYzOgepKCq0jqOIrg7Rud3vD152VhbLXORyqmoYVZ3sNWTMpPCdD1mO2zkierl2JZRf78EpYUq6rzdGS9ZXApYSMH0iRIrC9rxHdHiMNPm5WVU6clVIDbcvyhMoGu`
  encodedPassword := `U2FsdGVkX19rzutpu67A2hGyjdoLGSnWEtTGcfSKK7w4SUATx2y3bB0HdvXpTrODvJjAa73eJUDF+oRMYVyh6k9EIBCW8XI+1aKbtQGNVnJtRoW7xrpkl3TdgC++x0e4UPjB+LBA79LK8gHTr4Ad2Kd06dWTxwpqAgsLU3VI6ZUXvoctUk9ffxCHzZocQBBHnEInovrp+/kUHZxYWZkS6/2wZiY60PH9gYyatenh472mNgS9h0sH3FW38HFVKudhxj4uIiYr+gO3Y6XxTq9th7ykhV4FwpYXL9xbKptE2H8=`
  iterations := 100000

  // First, we need to convert key and password data from base64 to raw bytes using
  // helper method used to make sure there are no escape "\" chars in the key.
  key, err := base64decode(encodedKey)
  if err != nil {
    log.Fatalln("key is not base64:", err)
  }

  password, err := base64decode(encodedPassword)
  if err != nil {
    log.Fatalln("password is not base64:", err)
  }

  // Encrypted key data is salted. To determine if key is salted you can compare
  // its first 8 bytes "Salted__". The rest 8 bytes are actual salt data.
  keySalt := key[8:16]
  keyData := key[16:]

  // Next step is to get an encryption key and IV by applying PBKDF2 to master password
  derivedKey := pbkdf2.Key(masterpass, keySalt, iterations, 32, sha1.New)

  // Now we need to extract AES key and IV from newly derived key
  aesKey := derivedKey[0:16]
  aesIv := derivedKey[16:32]

  // After we got AES key and IV we can use it to decrypt our key contents
  keyRaw, err := decrypt(keyData, aesKey, aesIv)
  if err != nil {
    log.Fatalln("unable to decrypt key:", err)
  }

  // If key decryption is successful, lets use that key to decrypt our password contents.
  // First, we would need to get salt and update password data
  passwordSalt := password[8:16]
  passwordData := password[16:]

  // Now, lets derive AES key and IV from password contents
  passwordKey, passwordIv := deriveKey(keyRaw, passwordSalt)

  // And finally, decrypt password contents
  passwordRaw, err := decrypt(passwordData, passwordKey, passwordIv)
  if err != nil {
    log.Fatalln("unable to decrypt password:", err)
  }

  // Print out the result
  printJson(passwordRaw)
}

func decrypt(data, key, iv []byte) ([]byte, error) {
  cipher, err := openssl.GetCipherByName("aes-128-cbc")
  if err != nil {
    return nil, err
  }

  ctx, err := openssl.NewDecryptionCipherCtx(cipher, nil, key, iv)
  if err != nil {
    return nil, err
  }

  cipherbytes, err := ctx.DecryptUpdate(data)
  if err != nil {
    return nil, err
  }

  finalbytes, err := ctx.DecryptFinal()
  if err != nil {
    return nil, err
  }

  cipherbytes = append(cipherbytes, finalbytes...)
  return cipherbytes, nil
}

func deriveKey(password []byte, salt []byte) (key []byte, iv []byte) {
  rounds := 2
  data := append(password, salt...)
  md5Hashes := make([][]byte, rounds)
  sum := md5.Sum(data)

  md5Hashes[0] = append([]byte{}, sum[:]...)

  for i := 1; i < rounds; i++ {
    sum = md5.Sum(append(md5Hashes[i-1], data...))
    md5Hashes[i] = append([]byte{}, sum[:]...)
  }

  return md5Hashes[0], md5Hashes[1]
}

func base64decode(data string) ([]byte, error) {
  sanitized := strings.Replace(data, `\`, "", -1)
  return base64.StdEncoding.DecodeString(sanitized)
}

func printJson(data []byte) {
  var buff bytes.Buffer
  if err := json.Indent(&buff, data, "", "  "); err == nil {
    log.Printf("%s\n", buff.Bytes())
  }
}

If you have Go installed, just save the file and run it with:

go run myfile.go

And the result is:

{
  "URLs": [
    {
      "label": "",
      "url": "gmail.com"
    }
  ],
  "fields": [
    {
      "value": "mygmailpassword",
      "name": "password",
      "type": "P",
      "designation": "password"
    },
    {
      "value": "",
      "name": "username",
      "type": "T",
      "designation": "username"
    }
  ]
}

It worked, we see that 1Password keeps login data as another json payload with includes more metadata. The reason is pretty simple - you can securely store multiple values in a single login and json is a good fit for that.