Having the search index stored on the normal file system gives us
great performance and avoids having to tweak Lucene too much to suit
our needs. However, the extra storage mechanism adds all sorts of
consistency, backup and transaction complexities.

Especially with things like the data store it should be feasible to
achieve good enough search performance even if the index is stored
inside the repository either as normal (hidden/protected) JCR nodes or
as special lower level constructs that still leverage our standard
persistence mechanism.

What would we need to do to achieve this, and would the result be
worth the effort?