Mercurial 2.0 Released with LargeFiles extension

Posted by Alex Blewitt on Nov 03, 2011

Sections

Development

Topics

Versioning ,

Version Control ,

Source Control ,

Programming

Share |

RELATED Research

Selenic have released Mercurial 2.0, their trianual upgrade to their namesake DVCS tool.

This release brings the Largefiles Extension, a way of bringing centralised data storage to an otherwise distributed version control system. As the Mercurial Revlog storage mechanism is not well suited to the storage of large binary files, the Largefiles Extension provides a centralised server location where such files can be downloaded on an as-needed basis.

Storing large binary files in any DVCS can sometimes be a problem if those files are updated frequently; even if the binary file can be stored using delta compression (i.e. just storing the changed parts) large files can quickly inflate the size of the repository. In addition, if these resources form part of the repository's history then they can't be pruned from the repository without affecting the content hashes, and therefore the repository's version.

Depending on the version system used, whether you store large assets in different branches (i.e. ones that you don't fetch regularly) can affect the size of the cloned repository (but not the source). Mercurial largefiles takes a different route, and provides (in effect) a symlink to the large file in the cloned repository.

Providing that the file is not needed in the currently checked out version, it is not downloaded as part of the clone or in fetch/push operations. However, if a revision is checked out which contains one or more of these largefile links, it will perform a separate connection to download the content from the server. Clearly this implies that there needs to be connectivity to check out some versions of the repository, but the local Mercurial clone will keep a cache of previously downloaded largefiles and can supply them if they've been downloaded already.

A new option to the hg add command has been added, --large, which allows a file to be specified as large (and therefore taking this alternative route to centralised-but-cached-on-demand storage). Alternatively, a fixed size can be set (those over 10Mb by default) or a naming pattern (e.g. *.zip) can be used to automatically flag large files. Note that the repository format must be upgraded with the hg lfconvert command to take advantage of this new format. Note that the dirstate command, which tracks the state of largefiles, is currently limited to 2Gb in Mercurial 2.0, although there are hopes for a fix post the 2.0 release.

Mercurial 2.0 also brings in the graft command, which implements cherry picking for Mercurial. It uses the merge logic to determine what changes should be present, and then applies them on a piecemeal basis but without creating a merge node. As with other implementations of cherry-picking, if the change has already been copied in the past then it will not be duplicated.