filestore-design.md

Storage

Filestore should implement the blockstore interface and contain a normal Blockstore for non-filestore blocks, as well as the filestore datastore for writing filestore metainfo to.

'Put' to the filestore will check the type of its input. If the input is a special filestoreNode type, it will store it specially via the filestore. otherwise it will store the block normally via the standard blockstore. If a normal block is being put and it exists already in the filestore, it does not need to be put to the blockstore. There should be a mechanism to override this behaviour (perhaps a flag that signals to use the normal blockstore for a given operation)

'Get' will check both the filestore and the blockstore, returning the block from whichever is found.

'AllKeysChan' should return all keys from both the standard blockstore and the filestore.

'Delete' should remove the block from both the normal blockstore and the filestore.

Add Flow

When adding files to the filestore, it will be easiest to just use raw-leaves. Data nodes (raw nodes containing file data) chunked during a filestore add will be wrapped in a filestoreNode object that allows the blockstore to detect how they should be stored.

Node Construction

When creating an ipfs node, if the filestore is marked 'enabled' in the config, A Filestore object (as discussed in 'Storage') should be created and assigned to the 'Blockstore' field in the core.IpfsNode so that all operations over the blockstore now use the filestore.

GC

The behaviour of the garbage collector should not change. Given the interface discussed in 'Storage', files stored in the filestore and those stored in the blockstore should be treated exactly the same in regards to pinning and gc. If a file in the filestore is unpinned and gc is run, the filestore references to those blocks should be removed, and the intermediate nodes stored in the blockstore should be removed as usual.

Please note, that I have not discussed this with @whyrusleeping. Some of it based on what we discussed, but others of it where never discussed with me. In particular the filestore supports more than raw leaves. The code to support non raw-leaves is very well isolated and I think it will be a mistake to remove it.

Review Notes:

• Add by default uses the normal blockstore, implying that adding data that exists in the filestore already duplicates it to the blockstore by default. We will then add a flag to add that does respect the filestores pre-existing blocks. (basically, default the flag i mentioned to true)
• config option to enable filestore should be under Experimental until we finalize this.