Skip to content

Instantly share code, notes, and snippets.

@jleclanche

jleclanche/ngdp.md

Created June 1, 2018 23:06
Show Gist options
  • Save jleclanche/91f2f5c0f2042a81db1c61464ae6d459 to your computer and use it in GitHub Desktop.
Save jleclanche/91f2f5c0f2042a81db1c61464ae6d459 to your computer and use it in GitHub Desktop.
Download ZIP
NGDP docs
Raw
ngdp.md

Definitions

  • Keys: a key is a 16-byte md5 hash. When used as a string, it is encoded as 32 hexadecimal characters. There are two types of keys:
  • Content Key, ckey: equal to the MD5 hash of the file's contents
  • Encoded Key, ekey: equal to the MD5 hash of the file's header

Default patch server:

  • http://{region}.patch.battle.net:1119/hsb

Endpoints

{patch-server}/{game-id}/versions

Returns version information for each region encoded as pipe-separated values:

  • Region: 2 character region code, e.g. eu, us, etc.

  • BuildConfig: content key of the build configuration file

  • CDNConfig: content key of the CDN configuration file

  • BuildId: a decimal representation of the current version for use by the game

  • VersionsName: a string representation of the current version for display to the user

{patch-server}/{game id}/cdns

Returns cdn information for each region encoded as pipe-separated values:

  • Name: 2 character region code, same as Region above

  • Path: the path prefix used with this cdn entry, e.g. tpr/pro

  • Hosts: a list of hostnames separated by spaces. Multiple hosts are listed to allow the client to increase throughput by downloading in parallel from multiple hosts. {cdn-url} = http://{host}/{path}

{cdn-url}/config/{key}

Used for the build, CDN and patch config files.

All {cdn-url} paths with {key} have the key formatted as aa/bb/aabbxxx... For example, the encoding file with ekey 74bb7a28a1c0ab3d1c2cc2aba192aad2 has the URL {cdn-url}/config/74/bb/74bb7a28a1c0ab3d1c2cc2aba192aad2.

{cdn-url}/data/{key}

Used for data archives and files

{cdn-url}/patch/{key}

Used for patch archives and files

{ekey-path}.index

Returns an archive index file

Files

Build Configuration

A text file accessed via the ckey provided in the /versions endpoint, encoded as simple {key} = {value} entries separated by newlines.

  • root: ekey of the root file
  • download: ekey of the download file
  • install: ekey of the install file
  • encoding: (ckey, ekey) of the encoding file
  • encoding-size: sizes (decoded, encoded) of the encoding file
  • build-name: Descriptive string for the build
  • build-product: Product name
  • build-uid: Product shortcode, same as the {game-id} portion of patch urls
  • patch: ekey of the patch file
  • patch-size: size of the patch file
  • patch-config: ckey of the patch config file

CDN Configuration

A text file accessed and encoded the same way as Build Configuration.

  • archives: a list of archive ekeys
  • archive-group: an archive ekey which names the merged index of all archive indexes
  • patch-archives: a list of patch archive ekeys
  • patch-archive-group: same as archive-group, but for patch archives
  • builds: a list of build config ckeys, ordered new to old

Patch Configuration

A text file accessed and encoded the same as above. This contains a text representation of data present in the header of the patch file, and so it is not typically used by tact clients.

  • patch-entry: one patch-entry is provided for each file (encoding, install, and download). The value begins with {filename} {ekey} {encodedSize} {ckey} {decodedSize} {frame encoding}, and thereafter contains multiple mappings from old files' ckeys to the patch file ckey that brings it up to date: {old ckey} {old size} {patch ckey} {patch size}
  • patch: ekey of the patch file

Archive Index

A binary file containing a list of {ekey, size, offset} for all files in the corresponding archive. Entries are sorted by ekey so that finding an entry by ekey is a simple bsearch. Size here is the encoded size (and so it is the actual number of bytes the file takes up in the archive).

// big-endian, pack(1)
struct ArchiveIndexEntry {
	uint8_t ekey[16];
	uint32_t size;
	uint32_t offset;
};

struct ArchiveIndexFooter {
	uint8_t kind; // must be 1
	uint8_t unk0[2]; // always zero
	uint8_t unk1; // 4
	uint8_t unk2; // 4
	uint8_t unk3; // 0x10
	uint8_t checksumLength; // 8, legal range is [0, 0x10]
	uint32_t numEntries; // little-endian
	uint8_t checksum[8];
};

// used in the .index files for archive-group and patch-group
// big-endian, pack(1)
struct MultiArchiveIndexEntry {
	uint8_t ekey[16];
	uint32_t size;
	uint8_t archive; // index in the sorted list of archives
	uint32_t offset;
};

This file is intended to be loaded one page (4096 bytes) at a time, so the remainder of each page divided by the length of an entry is padding. Entries are 0x18 bytes for single archive indexes and 0x19 bytes for multi-archive indexes, leaving a remainder 0x10 bytes and 0x15 bytes in each page, respectively.

The last 0x14 bytes of the file are the footer (ArchiveIndexFooter). Before these 0x14 bytes are three pieces of data:

  • the ekey of the last entry in the page, for each page (0x10 bytes * numPages); this should be used as to determine which page to bsearch when querying by ekey.
  • the first 8 bytes of the md5 hash of the page, for each page (8 bytes * numPages)
  • the first 8 bytes of the md5 hash of the previous (0x18 bytes * numPages) of data

Checksum verification:

  • checksum is the first 8 bytes of the md5 of the footer (checksum is zeroed when the hash is performed)
  • the archive ekey is the md5 of the last 0x14 + hashLength bytes of the index file

Block Table

A Block Table file represents an individual file that has been split into frames (frame size usually maxes at 64k) which are individually encoded to compress and/or encrypt the file data.

// All Block Table header fields are serialized big-endian.
//
//  0        4               8         9           C
//  +--+--+--+---+---+---+---+---------+---+---+---+
//  | BLTE   | headerSize    | version | numFrames |
//  +--+--+--+---+---+---+---+---------+---+---+---+
struct BlockTableHeader {
	uint32_t magic; // 'BLTE'
	uint32_t headerSize;
	uint8_t version;
	uint32_t numFrames; // 24-bit
};

struct BlockTableFrameHeader {
	uint32_t encodedSize;
	uint32_t decodedSize;
	uint8_t hash[16]; // md5 hash of encoded bytes
};

The file begins with a BlockTableHeader, and following the header is an array of numFrames BlockTableFrameHeader structs. Following the header data are the encoded frame bytes. The first byte of the frame determines the type of the frame:

  • N: None
  • Z: Zlib
  • E: Encrypted
  • F: Frame

None frame

The frame contains unencoded data.

memcpy(dst, src + 1, decodedSize);

Zlib frame

The frame contains zlib-compressed data. This is normal zlib, as specified by RFC 1950.

Using miniz.c:

size_t result = tinfl_decompress_mem_to_mem(
	dst, decodedSize, src + 1,
	encodedSize - 1,
	TINFL_FLAG_PARSE_ZLIB_HEADER);

Encrypted frame

The frame contains Salsa20-encrypted data. The decrypted data is another frame so that data may be compressed and encrypted. decodedSize is the size of the data after both frames are decoded.

In the encrypted frame, following the 'E' is data required to setup the Salsa20 state:

uint8_t keyHashLen;
uint8_t keyHash[keyHashLen];
uint8_t nonceLen;
uint8_t nonce[nonceLen];
char endMark; // 'S' or 'A'

The keyHash is used to find the encryption key for Salsa20. These keys are distributed external to the tact system. In WoW, TactKey.db2 contains the keys themselves, and TactKeyLookup.db2 maps a keyHash to a TactKey row. In Overwatch, two keys are static initialized, and all other keys are sent by the game server at login.

The nonce value is xor'd with the current frame index and then used as a nonce for Salsa20. Here is an overview of the decryption process:

uint32_t keyHashLen = src[1];
keyHash = src + 2;
uint32_t nonceLen = keyHash[keyHashLen];
uint8_t key[16];
LookupEncryptionKey(key, keyHash);
nonce = keyHash + keyHashLen;
nonce[0] ^= (frameI & 0xff);
nonce[1] ^= (frameI & 0xff00) >> 8;
nonce[2] ^= (frameI & 0xff0000) >> 16;

// using reference implementation:
ECRYPT_keysetup(ctx, key, 128, nonceLen * 8);
ECRYPT_ivsetup(ctx, nonce);
uint8_t *buf = nonce + nonceLen + 1; // after endMark
uint32_t bufSize = encodedSize - (buf - src);
ECRYPT_decrypt_bytes(ctx, buf, buf, bufSize);

// Process buf as another encoded frame (buf[0] should be 'Z' or 'N')

Frame frame

A frame containing another frame. Yo dawg. In reality, this is never used.

Single-frame Block Table

If the headerSize field of the Block Table file header is zero, then the file is a single-frame block table. The frame starts 8 bytes into the file--the last 4 bytes of the BlockTableHeader struct are not present. The decodedSize necessary to decode the file is taken from the encoding file.

Root

This file is used directly only by the game, and the format of this file is specific to the game. It is used to lookup filename => ckey.

WoW root

SC2/Heroes root

D3 root

Download

This file is used by the installer to determine the order by which files are made local in the archives and to determine which stages require which files (i.e. not playable, playable, complete).

// Read in order, all big-endian:
char magic[2]; // 'DL'
uint8_t unk0[3]; // \x01\x10\x01
uint32_t numFiles;
uint16_t numOptions;

for (int i = 0; i < numFiles; i++) {
	uint8_t ekey[16];
	uint8_t unk0;
	uint32_t filesize;
	uint8_t stage;
	uint32_t unk1;
}

for (int i = 0; i < numOptions; i++) {
	char name[]; // null-terminated
	uint16_t group;
	uint8_t bitmask[(numFiles + 7) / 8];
}

Install

This file is used by the installer to place files on disk directly, such as executables and shared libraries.

// Read in order, all big-endian:
char magic[2]; // 'IN'
uint8_t unk0[2]; // \x01\x10
uint16_t numOptions;
uint32_t numFiles;

for (int i = 0; i < numOptions; i++) {
	char name[]; // null-terminated
	uint16_t group;
	uint8_t bitmask[(numFiles + 7) / 8];
}

for (int i = 0; i < numFiles; i++) {
	char name[]; // null-terminated
	uint8_t ckey[16];
	uint32_t size;
}

Options include operating systems and locales. One option from each group must be selected. Selected option bitmasks are ANDed together to determine which files are installed.

Encoding

This file is used to map from ckey to ekey.

The header of the file:

// Read in order, all big-endian:
char magic[2]; // 'EN'
uint8_t unk0[7]; // \x01\x10\x10\0\x04\0\x04
uint32_t numPages;
uint32_t numSpecPages;
uint8_t unk1; // must be 0
uint32_t encodingStringsSize;
char encodingStrings[encodingStringsSize];

// Create an array from the null-terminated strings in encodingStrings:
std::vector<char *> encodingSpecs; {
	char *start = encodingStrings;
	char *end = start + encodingStringsSize;
	for (char *c = start; c < end; c++) {
		if (!(*c)) {
			encodingSpecs.push_back(start);
			start = c + 1;
		}
	}
}

Following the encodingStrings is an array of numPages 0x20-byte structs:

struct EncodingPageHeader {
	uint8_t ckey[16]; // ckey of the first entry in the page
	uint8_t hash[16]; // md5 of the entire page
};

Following the PageHeaders are numPages pages each filled with 0x26-byte structs:

struct EncodingEntry {
	uint8_t unk0[2]; // \x01\0
	uint32_t decodedSize; // Still big-endian
	uint8_t ckey[16];
	uint8_t ekey[16];
};

The encoding entries are all sorted by ckey, and so can be bsearched in the same way as the archive indices. Note, however, that the encoding page headers use the first ckey of the page, whereas archive index page headers use the last ekey of the page.

After the ckey => ekey map is the encoding spec for each encoded file. First, an array of numSpecPages 0x20-byte structs:

struct EncodingSpecPageHeader {
	uint8_t ekey[16]; // ekey of the first entry
	uint8_t hash[16]; // md5 of the entire page
};

Similar to before, numSpecPages pages each filled with 0x19-byte structs:

// big-endian, pack(1)
struct EncodingSpecEntry {
	uint8_t ekey[16];
	uint32_t spec; // an index in to the array created from encodingStrings
	uint8_t unk0;
	uint32_t encodedSize;
};

Following the spec data is the encoding spec string for the encoding file itself.

Patch

0xa byte header

data/%02x%06x.idx

Name is (subset, revision). What defines subset? Revision increases whenever the patcher feels like it.

uint8_t ekey[9]; // truncated
uint8_t offset[5]; // 40-bit big-endian integer; top 10 bits indicate archive
uint32_t size; // little-endian

data/data.%02d

uint8_t ekey[16]; // bytes are in reverse order of everywhere else
uint32_t size;
uint16_t unk0;
uint64_t unk1;
char filedata[size - 0x1e];
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment