A guide to back up and recover 2FA tokens from FreeOTP (Android)

freeotp_backup.md

Backing up and recovering 2FA tokens from FreeOTP

NOTE: THIS MAY NOT WORK ANYMORE - SEE COMMENTS

Backing up FreeOTP

Using adb, create a backup of the app using the following command:

adb backup -f freeotp-backup.ab -apk org.fedorahosted.freeotp

org.fedorahosted.freeotp is the app ID for FreeOTP.

This will ask, on the phone, for a password to encrypt the backup. Proceed with a password.

Manually extracting the backup

The backups are some form of encrypted tar file. Android Backup Extractor can decrypt them. It's available on the AUR as android-backup-extractor-git.

Use it like so (this command will ask you for the password you just set to decrypt it):

abe unpack freeotp-backup.ab freeotp-backup.tar

Then extract the generated tar file:

$ tar xvf freeotp-backup.tar
apps/org.fedorahosted.freeotp/_manifest
apps/org.fedorahosted.freeotp/sp/tokens.xml

We don't care about the manifest file, so let's look at apps/org.fedorahosted.freeotp/sp/tokens.xml.

Reading tokens.xml

The tokens.xml file is the preference file of FreeOTP. Each <string>...</string> is a token (except the one with the name tokenOrder).

The token is a JSON blob. Let's take a look at an example token (which is no longer valid!):

<?xml version='1.0' encoding='utf-8' standalone='yes' ?>
<map>
	<!-- ... -->
	<string name="Discord:[email protected]">{&quot;algo&quot;:&quot;SHA1&quot;,&quot;counter&quot;:0,&quot;digits&quot;:6,&quot;imageAlt&quot;:&quot;content://com.google.android.apps.photos.contentprovider/-1/1/content%3A%2F%2Fmedia%2Fexternal%2Ffile%2F3195/ORIGINAL/NONE/741876674&quot;,&quot;issuerExt&quot;:&quot;Discord&quot;,&quot;issuerInt&quot;:&quot;Discord&quot;,&quot;label&quot;:&quot;[email protected]&quot;,&quot;period&quot;:30,&quot;secret&quot;:[122,-15,11,51,-100,-109,21,89,-30,-35],&quot;type&quot;:&quot;TOTP&quot;}</string>
</map>

Let's open a python shell and get the inner text of the XML into a Python 3 shell. We'll need base64, json and html in a moment:

>>> import base64, json, html
>>> s = """{"algo":"SHA1","counter":0,"digits":6,"imageAlt":"content://com.google.android.apps.photos.contentprovider/-1/1/content%3A%2F%2Fmedia%2Fexternal%2Ffile%2F3195/ORIGINAL/NONE/741876674","issuerExt":"Discord","issuerInt":"Discord","label":"[email protected]","period":30,"secret":[122,-15,11,51,-100,-109,21,89,-30,-35],"type":"TOTP"}"""

We decode all those HTML entities from the XML encoding:

>>> s = html.unescape(s); print(s)
{"algo":"SHA1","counter":0,"digits":6,"imageAlt":"content://com.google.android.apps.photos.contentprovider/-1/1/content%3A%2F%2Fmedia%2Fexternal%2Ffile%2F3195/ORIGINAL/NONE/741876674","issuerExt":"Discord","issuerInt":"Discord","label":"[email protected]","period":30,"secret":[122,-15,11,51,-100,-109,21,89,-30,-35],"type":"TOTP"}

What we specifically need from this is the secret. It's a signed byte array from Java... Let's grab it:

>>> token = json.loads(s); print(token["secret"])
[122, -15, 11, 51, -100, -109, 21, 89, -30, -35]

Now we have to turn this into a Python bytestring. For that, these bytes need to be turned back into unsigned bytes. Let's go:

>>> secret = bytes((x + 256) & 255 for x in token["secret"]); print(secret)
b'z\xf1\x0b3\x9c\x93\x15Y\xe2\xdd'

Finally, the TOTP standard uses base32 strings for TOTP secrets, so we'll need to turn those bytes into a base32 string:

>>> code = base64.b32encode(secret); print(code.decode())
PLYQWM44SMKVTYW5

There we go. PLYQWM44SMKVTYW5 is our secret in a format we can manually input into FreeOTP or Keepass.

All combined:

import html, json, base64
def decode(s):
    s = html.unescape(s)
    token = json.loads(s)
    secret = bytes((x + 256) & 255 for x in token["secret"])
    code = base64.b32encode(secret)
    print(code.decode())

And for every token in the tokens.xml file:

import xml.etree.ElementTree, json, base64
tree = xml.etree.ElementTree.parse('tokens.xml')
tokens = tree.getroot().findall('string')
for t in tokens:
    n = t.get('name') 
    if n == "tokenOrder":
        continue
    else:
        print(n)
    token = json.loads(t.text)
    secret = bytes((x + 256) & 255 for x in token["secret"])
    code = base64.b32encode(secret)
    print(code.decode())

Also if you don't want to go through backup/unpack, you can just cat data/data/org.fedorahosted.freeotp/shared_prefs/tokens.xml from the adb shell. At least, works on my phone… path or way to get a shell may vary I suppose.

Thanks Man ! You helped me a lot to transfer an older account from FreeOTP to another TOTP app. This fucking app has no easy transfer option.
Thanks again ! thanks thanks thanks thanks 😽

@AnatomicJC glad it was helpful :)

Hi. Little suggestion to have it create the full totp string:

import xml.etree.ElementTree, json, base64
import urllib.parse
import json

tree = xml.etree.ElementTree.parse('./org.fedorahosted.freeotp/sp/tokens.xml')
tokens = tree.getroot().findall('string')

for t in tokens:
    n = t.get('name')
    if n == "tokenOrder":
        continue
    token = json.loads(t.text)
    secret = bytes((x + 256) & 255 for x in token["secret"])
    code = base64.b32encode(secret).decode()
    issuer = token.get('issuerExt', '')
    print('otpauth://{type}/{issuer}?{args}'.format(
        type=token['type'].lower(),
        issuer=urllib.parse.quote(
            ':'.join(filter(None, (issuer, token['label'])))),

        args=urllib.parse.urlencode({
            'secret': base64.b32encode(secret).decode('utf-8'),
            'algo':   token['algo'],
            'digits': token['digits'],
            'period': token['period']
        })
    ))

Not my code but it works and you may combine it with qrencode to create a lot of png ready to be scanned:

mkdir images && python export.py | while read A; do 
  qrencode -o images/$(echo -n $A | md5sum | awk '{print $1}').png $A
done

When backing up, I received a file in which I can’t understand where the secret is

<?xml version='1.0' encoding='utf-8' standalone='yes'?>
<map>
	<string name="bedef5fb-1752-4fc5-80f2-f9ea8a551b51">
		{&quot;key&quot;:&quot;{\&quot;mCipher\&quot;:\&quot;AES\/GCM\/NoPadding\&quot;,\&quot;mCipherText\&quot;:[46,-7,16,-72,110,-85,15,-24,23,29,61,102,-38,127,60,-101,0,-86,-83,-63,-93,101,23,85,-81,74,-62,-111,109,64,-55,122,-94,80,123,104],\&quot;mParameters\&quot;:[48,17,4,12,88,92,127,-56,98,50,24,-50,-36,-50,-32,-47,2,1,26],\&quot;mToken\&quot;:\&quot;HmacSHA1\&quot;}&quot;}</string>
	<string name="bedef5fb-1752-4fc5-80f2-f9ea8a551b51-token">
		{&quot;algo&quot;:&quot;SHA1&quot;,&quot;digits&quot;:6,&quot;issuerExt&quot;:&quot;Forti
		VPN&quot;,&quot;issuerInt&quot;:&quot;Forti
		VPN&quot;,&quot;label&quot;:&quot;[email protected]&quot;,&quot;period&quot;:30,&quot;type&quot;:&quot;TOTP&quot;}</string>
	<string name="masterKey">
		{&quot;mAlgorithm&quot;:&quot;PBKDF2withHmacSHA512&quot;,&quot;mEncryptedKey&quot;:{&quot;mCipher&quot;:&quot;AES/GCM/NoPadding&quot;,&quot;mCipherText&quot;:[-81,26,67,112,48,-66,68,22,70,117,26,-14,-18,-76,45,-55,-89,93,-112,23,110,82,-91,-47,-83,79,-63,-122,-25,18,-72,87,-15,106,-103,-9,76,-106,71,-35,-37,47,-31,116,125,124,-72,13],&quot;mParameters&quot;:[48,17,4,12,49,105,-120,-75,38,-47,1,-48,45,-16,46,107,2,1,16],&quot;mToken&quot;:&quot;AES&quot;},&quot;mIterations&quot;:100000,&quot;mSalt&quot;:[119,-4,-69,-28,84,-38,74,-17,17,14,120,37,-2,-67,-68,49,22,8,-102,12,55,91,-33,-111,-123,111,44,-92,77,121,87,5]}</string>
</map>

@sanodin

Looks like they changed the format quite a bit.

Looking at this, you have three entries: a [uuid], a [uuid]-token, and a masterKey. It seems similar to what was there before, except that now the secret is encrypted.

{"key": "{\"mCipher\":\"AES/GCM/NoPadding\",\"mCipherText\":[46,-7,16,-72,110,-85,15,-24,23,29,61,102,-38,127,60,-101,0,-86,-83,-63,-93,101,23,85,-81,74,-62,-111,109,64,-55,122,-94,80,123,104],\"mParameters\":[48,17,4,12,88,92,127,-56,98,50,24,-50,-36,-50,-32,-47,2,1,26],\"mToken\":\"HmacSHA1\"}"}
{"algo": "SHA1", "digits": 6, "issuerExt": "Forti   VPN", "issuerInt": "Forti   VPN", "label": "[email protected]", "period": 30, "type": "TOTP"}
{"mAlgorithm": "PBKDF2withHmacSHA512", "mEncryptedKey": {"mCipher": "AES/GCM/NoPadding", "mCipherText": [-81, 26, 67, 112, 48, -66, 68, 22, 70, 117, 26, -14, -18, -76, 45, -55, -89, 93, -112, 23, 110, 82, -91, -47, -83, 79, -63, -122, -25, 18, -72, 87, -15, 106, -103, -9, 76, -106, 71, -35, -37, 47, -31, 116, 125, 124, -72, 13], "mParameters": [48, 17, 4, 12, 49, 105, -120, -75, 38, -47, 1, -48, 45, -16, 46, 107, 2, 1, 16], "mToken": "AES"}, "mIterations": 100000, "mSalt": [119, -4, -69, -28, 84, -38, 74, -17, 17, 14, 120, 37, -2, -67, -68, 49, 22, 8, -102, 12, 55, 91, -33, -111, -123, 111, 44, -92, 77, 121, 87, 5]}

In order:

• The first object appears to be an HmacSHA1 TOTP secret, but it's been encrypted with AES/GCM/NoPadding. The ciphertext bytes are [46, -7, 16, -72, 110, -85, 15, -24, 23, 29, 61, 102, -38, 127, 60, -101, 0, -86, -83, -63, -93, 101, 23, 85, -81, 74, -62, -111, 109, 64, -55, 122, -94, 80, 123, 104] and parameter bytes [48, 17, 4, 12, 88, 92, 127, -56, 98, 50, 24, -50, -36, -50, -32, -47, 2, 1, 26].
• The second object is the metadata on the entry. {'algo': 'SHA1', 'digits': 6, 'issuerExt': 'Forti VPN', 'issuerInt': 'Forti VPN', 'label': '[email protected]', 'period': 30, 'type': 'TOTP'}
• The third object is, I think, the key that was used to encrypt that first object. I'm ... not sure why it's embedded in there. It seems to have been generated by the SecretKeyFactory class in java, if that helps you track it down.

This is the master key data from your xml:

{'mAlgorithm': 'PBKDF2withHmacSHA512', 'mEncryptedKey': {'mCipher': 'AES/GCM/NoPadding', 'mCipherText': [-81, 26, 67, 112, 48, -66, 68, 22, 70, 117, 26, -14, -18, -76, 45, -55, -89, 93, -112, 23, 110, 82, -91, -47, -83, 79, -63, -122, -25, 18, -72, 87, -15, 106, -103, -9, 76, -106, 71, -35, -37, 47, -31, 116, 125, 124, -72, 13], 'mParameters': [48, 17, 4, 12, 49, 105, -120, -75, 38, -47, 1, -48, 45, -16, 46, 107, 2, 1, 16], 'mToken': 'AES'}, 'mIterations': 100000, 'mSalt': [119, -4, -69, -28, 84, -38, 74, -17, 17, 14, 120, 37, -2, -67, -68, 49, 22, 8, -102, 12, 55, 91, -33, -111, -123, 111, 44, -92, 77, 121, 87, 5]}

I would try to recreate the key from that, but if I'm reading this correctly, the key itself is encrypted -- I would wager, by Android's own encryption utilities; if that's the case, you'd need to really mess around your phone to decrypt that.

Of course, if it is the case, I'm kinda wondering why they're doing this round-about "encrypt the secret, but store the key next to it, but encrypt that key" thing. Maybe I'm missing something.

Good luck

Thanks for the answer. When backing up and restoring, a password is used, which is entered during the initial installation of the application, and during backup, it is encrypted with this password, then when I transfer the backup to another phone, I enter the password, otherwise I can’t restore

Is the password specific to the app, or is it at the android level? If it's the former you might be able to decrypt the key with just your password but I'm not sure... let me try something, this gave me an idea.

I create the password for the application myself, during installation
Immediately sorry for the letter in a personal, I wrote you a password to the mail and the original xml

@sanodin courtesy of GPT4:

import json
import base64
import xml.etree.ElementTree as ET
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.backends import default_backend

bytelist_to_bytes = lambda bytelist: bytes((x + 256) & 255 for x in bytelist)


def decrypt(cipher_text, key, parameters):
    nonce = parameters[:12]
    aad = parameters[12:]
    cipher_text, tag = cipher_text[:-16], cipher_text[-16:]
    cipher = Cipher(
        algorithms.AES(key), modes.GCM(nonce, tag), backend=default_backend()
    )
    decryptor = cipher.decryptor()
    decryptor.authenticate_additional_data(aad)
    return decryptor.update(cipher_text) + decryptor.finalize()


password = b"your_password_here"

# Read XML file
tree = ET.parse("test.xml")
root = tree.getroot()

# Extract masterKey and key data from XML
for elem in root.iter("string"):
    if elem.get("name") == "masterKey":
        master_key_data = json.loads(elem.text)
    elif not elem.get("name").endswith("-token"):
        totp_key_data = json.loads(elem.text)

# Extract and process masterKey components
mAlgorithm = master_key_data["mAlgorithm"]
mEncryptedKey_cipherText = bytelist_to_bytes(
    master_key_data["mEncryptedKey"]["mCipherText"]
)
mEncryptedKey_parameters = bytelist_to_bytes(
    master_key_data["mEncryptedKey"]["mParameters"]
)
mIterations = master_key_data["mIterations"]
mSalt = bytelist_to_bytes(master_key_data["mSalt"])

# Derive master_key using PBKDF2
kdf = PBKDF2HMAC(
    algorithm=hashes.SHA512(),
    length=32,
    salt=mSalt,
    iterations=mIterations,
    backend=default_backend(),
)
master_key = kdf.derive(password)

# Decrypt master_key
decrypted_master_key = decrypt(
    mEncryptedKey_cipherText, master_key, mEncryptedKey_parameters
)

# Extract and process TOTP secret key components
key_cipherText = bytelist_to_bytes(json.loads(totp_key_data["key"])["mCipherText"])
key_parameters = bytelist_to_bytes(json.loads(totp_key_data["key"])["mParameters"])

# Decrypt TOTP secret key
decrypted_totp_secret = decrypt(key_cipherText, decrypted_master_key, key_parameters)

print("Decrypted TOTP secret:", decrypted_totp_secret)

Unfortunately, it doesn't work -- even with the password you sent. I get cryptography.exceptions.InvalidTag. Maybe I messed up extracting the data from the xml, or it was altered in some way. Or maybe GPT-4 is just wrong :)

Updated the code to read directly from the XML (still via GPT4. Insane). That other XML file you sent me doesn't work either, unfortunately (it does appear you changed some bytes, but the original did not help)

I can’t get the xml out of the phone, adb creates an empty 47 kb archive, if you do it with a password, then an archive of 549 bytes is created, but I can’t unpack it anymore. It will turn out only by means of the application itself to make a backup, but it is in the form in which I sent it to you.
By the way, the very first one that I showed, I pulled out of the emulator

FreeOTP got "Export to json" option now, info for new comers.