Encbox.md

Build your own private, encrypted, open-source Dropbox-esque sync folder

Prerequisites:

• One or more clients running a UNIX-like OS. Examples are given for Ubuntu 12.04 LTS, although all software components are available for other platforms as well (e.g. OS X). YMMV
• A cheap Ubuntu 12.04 VPS with storage. I recommend Backupsy, they offer 250GB storage for $5/month. Ask Google for coupon codes.

Software components used:

• Unison for file synchronization
• EncFS for folder encryption

Set up file synchronization with unison

On your server:

• Install Unison:

  $ sudo apt-get install unison

• Add a user for our project and give him a decent password:

  $ sudo adduser encbox

On your client(s):

• Install Unison:

  $ sudo apt-get install unison

• Enable key-based authentication for SSH (replace your.vps.com with your VPS' hostname):

  $ ssh-keygen
  $ cat ~/.ssh/id_rsa.pub | ssh [email protected] "mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys"

• Create the folder holding our encrypted files:

  $ mkdir ~/.encbox

• Create a unison profile in ~/.unison/encbox.prf (replace your.vps.com with your VPS' hostname):

  $ mkdir ~/.unison; tee ~/.unison/encbox.prf <<__EOF
  root = /home/$USER/.encbox
  root = ssh://[email protected]/encbox
  prefer = ssh://[email protected]/encbox
  
  backups = true
  times = true
  terse = true
  
  repeat = 2
  __EOF

• Create an upstart file to start synchronization automatically (in /etc/init/encbox.conf):

  $ sudo tee /etc/init/encbox.conf <<__EOF
  description "encbox"
  
  start on desktop-session-start
  stop on desktop-shutdown
  
  setuid $USER
  setgid $USER
  env HOME=$HOME
  
  respawn
  
  script
      bash -l -c "unison encbox; sleep 2"
  end script
  __EOF

• Start Unison:

  # either run in foreground:
  $ unison encbox
  
  # or run it as a system service:
  $ sudo service encbox start # starts your Encbox
  $ sudo service encbox stop  # stops your Encbox

  If unison complains about archive files on your client, run it once with the -ignorearchives flag:

  $ unison -ignorearchives encbox

Create an encrypted folder using EncFS

On your client(s):

• Install EncFS:

  $ sudo apt-get install encfs

• Mount encrypted directory:

  $ encfs ~/.encbox ~/Encbox

  Agree to have your target directory created, then choose p to use EncFS' preconfigured paranoia mode. Give your folder a decent password.

• That's it! All files in ~/Encbox will be encrypted and synced securely between your VPS and all clients.

Tips

• Polling:

  Unfortunately, Unison doesn't support file system notifications (i.e. inotify, libnotify), so it polls your file system for changes every 2 seconds (if you set repeat = 2 in your profile as we did above). This isn't nice, but I did not observe any negative effects on my notebook's battery life. Feel free to set the repeat parameter to some higher value if you observe negative effects. Bandwith usage for polling is negligible as well (~0.9 kB/poll, ~3.5 kbit/s).

• Unison version

  Rumor has it you should use the same Unison version on your server as well as on all your clients, or you will run into problems. I didn't test it, though.

• Android / iOS app:

  This setup is compatible with Boxcryptor Classic's smartphone apps. You just have to create an EncFS folder in expert mode (x) and follow the instructions at Boxcryptor's Support Desk. Then install a WebDAV server of your choice and expose /home/encbox/encbox via WebDAV (not in this document's scope).

I believe it could also be improved greatly to an extent where one could serve a small local group of people as a service.

While this might be a good advice, the problem with this setup as far as i can tell is that the EncFS volume will always stay to be visible. So if under some strange circumstances someone might gain access to the system (at this point you already have lost the game) he'll be able to see all the files regardless of encryption used or not.

What about http://labs.bittorrent.com/experiments/sync.html instead Unison?

@thinkxl Sync isn't open source.

@jeregrine my bad, you're right

Unison is nice, I've been using it for syncing (altho manually) for years. I would suggest also looking at lsyncd.

As you're talking about a Dropbox replacement, adding a web client would probably be interesting too. I've been using h5ai myself and found it to be a pretty good one. It doesn't have authentication, but HTTP auth could fix that.

EncFS needs to be done at the origin before you synchronize them to the remote storage.

ownCloud is an open source file sync solution that may be worth checking out:

http://owncloud.org

Thanks for the how-to. I've had good experiences with http://sparkleshare.org/ so probably a mashup between them would be great.

Perhaps is http://labs.bittorrent.com/experiments/sync.html an alternative protocol for this. There are clients for android, windows, mac and iphone(soon) available. And btsync seems to be at least open concept. I had a quick look on the data which is generated by it's network protocol. It syncs the files nearly in real time, but to do this, there is a need for much network traffic. The battery of your mobile phone will go down within hours. I do not know if this is caused by the protocol it self and cannot be changed or if this is a kind of bug, because the project is not that old.

What is realy nice about this idea is that there is no real master. But this leads to the problem of conflicts, if two devices lose the sync to the network and start to modify the same file at the same time. How to decide which change will survive.

My guess would be to have something like git oder mercurial to handle conflicts. But as a matter of fact, this is not compatible with non programming people. I'd like to have a solution which is able to handle the sync of a shopping list other at least 4 devices.

Are there libraries or tools which are able to merge changes on images, word documents and mp3s for example?

Multiple changes to images may be stored as layers, so a human can handle those conflicts. Word may be as simple as a source code. Mp3s are a little bit complicated to merge. In most use cases the sync of the meta data would be the solution, but if the file is cutted or normalized this change must be synced to all other devices in the sync network. If a mp3 get's corrupted, because the sd card in the mobile phone has an error, is an other problem. Because how can a simple sync tool can decide that this change has not to be synced.

@akbar501 owncloud is nice for contacts and calendar sync, but webdav is not a realy good protocol to sync files over multiple devices. There is no good way to handle conflicts.

Just a quick tip.

$ cat ~/.ssh/id_rsa.pub | ssh [email protected] "mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys"

You can do the same with

$ ssh-copy-id [email protected]

...and it's actually more reliable. For one, you won't wreck your authorized_keys file if you make a typo.

Moreover, latest SSH versions support signed keys. This may be the case where a separate key is warranted, but you might want to configure your box to accept any key signed by your CA rather than manage them individually.

It seems unison is "is no longer under active development". http://en.wikipedia.org/wiki/Unison_(file_synchronizer)

I'm using Seafile in production as a replacement for Dropbox. Supports repository-level encryption of data at rest, open source and a user-friendly version management GUI. Clients for Windows / Mac / Linux and Android / iOS.

There is also git annex :)
http://git-annex.branchable.com/

Isn't this what git-annex was designed to do? With the "new" git-annex assistant, it's pretty user-friendly, too.

I created a DropBox clone not long ago, used to have the instructions online but not anymore. I mostly relied on lsyncd which supports inotify and is open source;

https://code.google.com/p/lsyncd/

My goal was to make it cross-platform without writing any Windows services, so I created a 'mini vm' which mounted a windows directory via SMB, then that VM ran lsyncd and vola, inotify against a Windows directory. It actually worked but was way too crude, lol.

Unison is a bad way to go, wasteful and inelegant.

There just will not be a good DropBox clone until someone decides to write cross-platform services. Until that day you probably won't do better than a solution built on lsyncd.

There's also incrond, probably not a great option because it'd require a lot of scripting.