[PATCH 00/16] LMDB refs backend atop pre-vtable

[PATCH 00/16] LMDB refs backend atop pre-vtable

I'm starting the patchset numbering over from 1 here, because this
version of the patchset is a subset of the last version.

This version of the patch set applies on top of
dt/refs-backend-pre-vtable. This required moving a bunch of stuff
that was in refs.h in previous versions into refs/refs-internal.h.

Since the last patchset, I added support for symlink HEAD refs, and
broke out the initdb stuff into a separate commit.

I also rearranged the order of the backend functions to make the vtable
easier to read.

I removed for_each_reftype_fullpath, which was at one point in next
but is not anymore. And I did a bit more code cleanup/rearrangement
on the LMDB stuff: I removed memory leaks, improved style, and just
generally spruced things up.

As usual, the normal tests and the same set of hacked tests pass.

I've read over each of these patches a few times, but I've probably
still managed to miss things. I look forward to your review.

[PATCH 04/16] refs: add do_for_each_per_worktree_ref

Alternate refs backends might still use files to store per-worktree
refs. So the files backend's ref-loading infrastructure should be
available to those backends, just for use on per-worktree refs. Add
do_for_each_per_worktree_ref, which iterates over per-worktree refs.

[PATCH 07/16] refs: add method for delete_refs

In the file-based backend, delete_refs has some special optimization
to deal with packed refs. In other backends, we might be able to make
ref deletion faster by putting all deletions into a single
transaction. So we need a special backend function for this.

[PATCH 08/16] refs: add methods to init refs backend and db

Alternate refs backends might not need the refs/heads directory and so
on, so we make ref db initialization part of the backend. We also
might need to initialize ref backends themselves, so we'll add a
method for that as well.

[PATCH 12/16] refs: always handle non-normal refs in files backend

Sometimes a ref transaction will update both a per-worktree ref and a
normal ref. For instance, an ordinary commit might update
refs/heads/master and HEAD (or at least HEAD's reflog).

We handle three cases here:

1. updates to normal refs continue to go through the chosen backend

2. updates to non-normal refs with REF_NODEREF or to non-symbolic refs
are moved to a separate files backend transaction.

3. updates to symbolic refs are dereferenced to their base ref. The
update to the base ref then goes through the ordinary backend, while
the files backend is directly called to update the symref's reflog.

+const char split_transaction_fail_warning[] =
+ "A ref transaction was split across two refs backends. Part of the "
+ "transaction succeeded, but then the update to the per-worktree refs "
+ "failed. Your repository may be in an inconsistent state.";
+
/*
* We always have a files backend and it is the default.
*/
@@ -784,6 +789,13 @@ void ref_transaction_free(struct ref_transaction *transaction)
free(transaction);
}

DESCRIPTION
-----------
@@ -113,6 +113,10 @@ does not exist, it will be created.

--

+--refs-backend-type=<name>::
+Type of refs backend. Default is to use the original "files" backend,
+which stores ref data in files in .git/refs and .git/packed-refs.
+
TEMPLATE DIRECTORY
------------------

[PATCH 15/16] refs: add LMDB refs backend

Add a database backend for refs using LMDB. This backend runs git
for-each-ref about 30% faster than the files backend with fully-packed
refs on a repo with ~120k refs. It's also about 4x faster than using
fully-unpacked refs. In addition, and perhaps more importantly, it
avoids case-conflict issues on OS X.

LMDB has a few features that make it suitable for usage in git:

1. It is relatively lightweight; it requires only one header file, and
the library code takes under 64k at runtime.

2. It is well-tested: it's been used in OpenLDAP for years.

3. It's very fast. LMDB's benchmarks show that it is among
the fastest key-value stores.

Ronnie Sahlberg's original version of this patchset used tdb. The
major disadvantage of tdb is that tdb is hard to build on OS X. It's
also not in homebrew. So lmdb seemed simpler.

To test this backend's correctness, I hacked test-lib.sh and
test-lib-functions.sh to run all tests under the refs backend. Dozens
of tests use manual ref/reflog reading/writing, or create submodules
without passing --refs-backend-type to git init. If those tests are
changed to use the update-ref machinery or test-refs-lmdb-backend (or,
in the case of packed-refs, corrupt refs, and dumb fetch tests, are
skipped), the only remaining failing tests are the git-new-workdir
tests and the gitweb tests.

+core.refsBackendType::
+ Type of refs backend. Default is to use the original files
+ based backend. Set to 'lmdb' to activate the lmdb database
+ backend. If you use the lmdb backend,
+ core.repositoryFormatVersion must be set to 1, and
+ extensions.refBackend must be set to 'lmdb'.
+
core.sharedRepository::
When 'group' (or 'true'), the repository is made shareable between
several users in a group (making sure all the files and objects are
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 431575b..739c116 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -224,7 +224,7 @@ objects from the source repository into a pack in the cloned repository.

--refs-backend-type=<name>::
Type of refs backend. Default is to use the original files based
- backend.
+ backend. Set to "lmdb" to activate the lmdb database backend.

--refs-backend-type=<name>::
Type of refs backend. Default is to use the original "files" backend,
-which stores ref data in files in .git/refs and .git/packed-refs.
+which stores ref data in files in .git/refs and .git/packed-refs. Set
+to "lmdb" to activate the lmdb database backend.

TEMPLATE DIRECTORY
------------------
diff --git a/Documentation/technical/refs-lmdb-backend.txt b/Documentation/technical/refs-lmdb-backend.txt
new file mode 100644
index 0000000..c497ffc
--- /dev/null
+++ b/Documentation/technical/refs-lmdb-backend.txt
@@ -0,0 +1,50 @@
+Notes on the LMDB refs backend
+==============================
+
+Design:
+------
+
+Refs and reflogs are stored in a lmdb database in .git/refdb. All
+keys and values are \0-terminated.
+
+Keys for refs are the name of the ref (e.g. refs/heads/master).
+Values are the value of the ref, in hex
+(e.g. 61f23eb0f81357c19fa91e2b8c6f3906c3a8f9b0).
+
+All per-worktree refs (refs/bisect/* and HEAD) are store using
+the traditional files-based backend.
+
+Reflogs are stored as a series of database entries.
+
+For non-empty reflogs, there is one entry per logged ref
+update. The key format is logs/[refname]\0[timestamp]. The timestamp
+is a 64-bit unsigned integer number of nanoseconds since 1/1/1970.
+This means that reflog entries are chronologically ordered. Because
+LMDB is a btree database, we can efficiently iterate over these keys.
+
+For an empty reflog, there is a "header" entry to show that a reflog
+exists. The header has the same format as an ordinary reflog, but with
+a timeztamp of all zeros and an empty value.
+
+Reflog values are in the same format as the original files-based
+reflog.
+
+Weaknesses:
+-----------
+
+The reflog format is somewhat inefficient: a binary format could store
+reflog date/time information in somewhat less space.
+
+The rsync and file:// transports don't work yet, because they
+don't use the refs API.
+
+git new-workdir is incompatible with the lmdb backend. Fortunately,
+git new-workdir is deprecated, and worktrees work fine.
+
+LMDB locks the entire database for write. Any other writer waits
+until the first writer is done before beginning. Readers do not wait
+for writers, and writers do not wait for readers. The underlying
+scheme is approximately MVCC; each reader's queries see the state of
+the database as-of the time that the reader acquired its read lock.
+This is not too far off from the files backend, which loads all refs
+into memory when one is requested.
diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt
index 00ad379..04c085d 100644
--- a/Documentation/technical/repository-version.txt
+++ b/Documentation/technical/repository-version.txt
@@ -86,3 +86,8 @@ for testing format-1 compatibility.
When the config key `extensions.preciousObjects` is set to `true`,
objects in the repository MUST NOT be deleted (e.g., by `git-prune` or
`git repack -d`).
+
+`refBackend`
+~~~~~~~~~~~~
+This extension allows the user of alternate ref backends. The only
+defined value is `lmdb`.
diff --git a/Makefile b/Makefile
index 5bd68e0..77b96d9 100644
--- a/Makefile
+++ b/Makefile
@@ -1037,6 +1037,17 @@ ifdef USE_LIBPCRE
EXTLIBS += -lpcre
endif