git annex - this is a separate bit of software that sits "atop" git and lets you (in it's words):
allows managing files with git, without checking the file contents into git.
Say what!? It basically stores your directory contents in git via symlinks, so really you are just checking in a bunch of symlinks - not content. This keeps the repo very small.