Kind of surprised it all came together so well today, especially because I
noticed another big problem with the design, but I was able to work around
that and import/export with preferred content works great.

I did end up limiting import to supported a subset of preferred
content expressions. Downloading content that it doesn't yet know if it
wants to import seemed too surprising and potentially very expensive.

Made git annex export --to remote honor the preferred content of the
remote. In a nice bit of code reuse, adjustTree was just what I needed to
filter unwanted content out of the exported tree.

Then a hard problem: When a tree is exported with some non-preferred
content filtered out, importing from the remote generates a tree that is
lacking those files, but merging that tree would delete the files
from the working tree. Solving that took the rest of the day.

I've developed a plan for how to handle export preferred content.
And today I'm working on making git annex import --from remote honor
the preferred content of the remote. It doesn't make sense to support it
for one and not the other, so this is on the preferred git branch for now.

One use case for this is to configure an import to exclude certain file
extensions or directories. Such unwanted content will be left as-is
in the remote's data store, but won't be imported, so from git-annex's
POV, it won't be present on the remote.

The tricky thing is, when importing, the key is not known until the file
is downloaded, but you don't want git-annex downloading content that is not
preferred. I'm finessing that problem by checking the subset of preferred
content expressions that are not dependent on the file's content, which will
avoid downloads of unwanted content in probably most cases.

What should it do when the preferred content expression is dependent on
the file's content? I'm undecided if it's better to warn and not import,
or to download the content once in order to check the preferred content
expression, and then throw unwanted content away.

Finally got the remote tracking branch for import from S3 into a good shape.

Rather than coaxing git into generating the same commits each time for
imports (which would have needed commit dates to be stable somehow),
I made git-annex always preserve a reference to the last import commit.

Here's how it looks when a rename was exported to S3, which resulted in a
history in S3 that diverged and reconverged with the git history.

And here's how the history develops as more changes get exported and
imported.

With adb and S3 import done, the first phase of import tree is
complete.

I could not find a good solution to the S3 history matching problem, so I
think that was the wrong approach. Now I have what seems to be a better
approach implemented: When an import of history from S3 contains some trees
that differ from the trees that were exported to S3, all git-annex needs to
do is make git aware of that, and it can do so by making the remote
tracking branch contain a merge between what was exported to S3 and what
was imported from it.

That does mean that there can be some extra commits generated form an
import, with the same trees as commits that the user made, but a different
message. That seems acceptable. Less so is that repeated imports generate
different commits each time; I need to make it generate stable commits. I
should also add back detection of the simple fast-forward case which was
working but got broken today.

I've been working on matching up the git history with the history from a
versioned S3 export. Got sidetracked for quite a while building an
efficient way to get the git history up to a certian depth (including all
sides of merge commits) without reading the entire git log output.

The history matching is mostly working now, but there's a problem when a
rename is exported to S3, because it's non-atomic on S3 and atomic in git,
and so the histories stop matching up. This is not fatal, just results
in an ugly git history with the right tree at the top of it. It's
not entirely wrong; the git repo and the S3 bucket did legitimately diverge
for a while, so shouldn't the merged history reflect that? The problem is
just that the divergence is not represented in the opimal way.

I hate giving up at the final hurdle, but I feel I need to think about this
some more, so merging import-from-s3 is postponed for another day, or
likely until Monday.

Got S3 import and export working fully for both versioned and unversioned
buckets. This included developing a patch to the aws library; only
versioned buckets are fully supported until that gets merged.

I'm left with one blocking problem before merging import-from-s3: The commit
history when importing from a versioned bucket is too long. It needs to
find the point in the versioned import that has already been committed and
avoid committing it again. Have started on that, but didn't get all the way
today.

Also, this S3 import feature should be able to be used with anonymous S3
access to a bucket, and indeed that might be more common than wanting to
import from a bucket you own or have credentials to allow access to.
But the S3 remote does not currently try to use anonymous S3 access,
so supporting that will need some more changes.

Despite struggling with a keyboard controller that's increasingly prone to
flaking out and not registering some key presses while doubling others, I
managed to finis implementing import from versioned S3 buckets. It's quite
nice to see it download past versions of files and construct a git history.

Still enough unimplemented stuff and bugs to need to work on this for
probably one more day.

Started today on git annex import from S3, in the "import-from-s3"
branch.

It looks like I'm going to support both versioned and unversioned buckets;
the latter will need --force to initialize since it can lose data.

One thought I had about that is: It's probably better for git-annex to be
able to import data from an unversioned S3 bucket with caveats about
avoiding unsafe operations (export) that could lose data, than it is for
git-annex to not be able to import from the bucket at all, guaranteeing
that past versions of modified files will be lost. (Rationalization is a
powerful drug.)

To support unversioned buckets, some kind of stable content identifier is
needed other than the S3 version id. Luckily, S3 has etags, which are
md5sum of the content, so will work great. But, the aws haskell library
needs one small change to return an etag, so this will be
blocked on that change.

I've gotten listing importable contents from S3 working for unversioned
buckets, including dealing with S3's 1000 item limit by paging.
Listing importable contents from versioned buckets is harder, because
it needs to synthesize a git version history from the information that S3
provides. I think I have a method for doing this that will generate the
trees that users will expect to see, and also will generate the same past
trees every time, avoiding a proliferation of git trees. Next step:
Converting my prose description of how to do that into haskell.

It was not very hard to get git annex import working with adb special
remotes. This is a nice alternative to installing git-annex on an Android
device for syncing with it. See android sync with adb.

I'm still thinking about supporting import from special remotes that can't
avoid most race conditions. But for adb, the only race conditions that
I couldn't avoid are reasonably narrow, nearly as narrow as git
checkout's own race conditions, with only the added overhead of
adb. So I let them slide.

Whew, I got the finishing touches on the import tree feature, and it's
merged into master! Still work to do on that, particularly supporting more
interesting special remotes than directory. It may be two or three weeks
until I get back to working on it.

Even with only directory special remotes, some nice things can be done with
this. Like bi-directional syncing between a removable drive with no
git-annex repository on it, and a subdirectory of the git-annex repository:

The feature is close to being mergeable to master now, but still needs some
work on the progress display of git annex import, and on supporting
imports from the same special remote to different git repos.

Still working on import tree today. I decided to
make the imported commit be on an unrelated history from the main branch,
which avoids some very susprising merge behavior, but means
users will need to pass --allow-unrelated-histories to git merge.

Also got export and sync commands updating the remote tracking branch.
It was surprisingly complicated to do.

With that done, I've tested exporting to a directory remote,
then making changes to the directory manually, and importing, and it all
works together.

Yesterday I implemented the command-line interface for
git annex import branch --from remote and today I got a prototype of it
it working with the directory special remote. Still a whole lot to do
before this feature is ready for release, but it's good to have the command
line interface working to play with it. The workflow feels pretty good:

Yesterday, got the git tree generation done and working. The main import
tree code is also implemented, though it may need some fine tuning.

Today I've been working on firming up user interface design and documentation.
Turns out that import tree is going to lead to some changes to export tree.
A remote tracking branch will be updated by both export tree and import
tree, since those operations are similar to git push and git fetch.
And git annex export --tracking will be deprecated in favor
of a git config setting that configures both import and export.

Started off by adding a QuickCheck test of the content
identifier log, which did find one bug in that code.

Then started roughing out the core of the importing operation, which involves
building up git trees for the files that are imported. But that needs a
way to graft an imported tree into a subdirectory of another tree,
and the only way I had available to do it needed to read in the entire
recursive tree of the current branch, which would be slower and use
more memory than I like.

So, got sidetracked building a git tree grafter. It turns out that
the export tree code also needs to graft a tree (into the git-annex
branch), and did so using the same innefficient method that I want to
avoid, so it will also be able to be improved using the grafter.

Unfortunately, I had to stop for the day with the grafter not quite working
properly.

Started building import tree (in the importtree branch). So far
the content identifier storage in the git-annex branch is done. Since the
API tells me it will need to both map from a key to content identifiers,
and from content identifier to the key, I also added a sqlite database to
handle the latter.

While implementing that, I happened to notice a bug in storage of metadata
that contains newlines; internals said that would be base64'd, but it
was not. That bug turns out to have been introduced by the ByteString
conversion in January, and it's the second bug caused by that conversion.
The other one broke git-annex on Windows, which was fixed by a release
yesterday.

I've been working through the design for the import tree feature,
and I think I finally have a design that I'm happy with. There were some
very challenging race conditions, and so import tree may only be safely
able to be implemented for a few remotes; S3 (with versioning enabled),
directory, maybe webdav and I hope adb. Work on this included finding
equivilant race conditions in git's update of the worktree, which do
turn out to exist if you go looking, but have much narrower time
windows there.

And I'll be running a tutorial for people who want to
learn about git-annex internals at the code level, to start development or
be better able to design their own features. That's in Montreal, March
26th-27th (8 hours total), hosted at McGill university. There may be one or
two seats left, so if you are interested in attending, please get in touch
with me by email. Haskell is not a prerequisite.

The 2018 user's survey
is closed, time for a look at the results. Several of the questions were
also on the two past surveys, so we can start to look at historical
trends as well.

Very similar numbers of people responded in 2018 as in 2015.
The 2013 survey remains a high water mark in participation.
My thoughts on the 2015 survey
participation level mostly still stand, although there has been a
consistent downwards trend in Debian popcon
since 2015.

Also interesting that several people skipped the first question on the
survey, perhaps because it was a fairly challenging question? And later
questions saw much higher response rates this time than in either of the
previous surveys, thanks to improvements in the survey interface.

v7

v7 unlocked files are being used by 7% of users, pretty impressive uptake
for a feature that has only been really finished for a couple of months.
Direct mode is still used by 7% of users, while its v7 replacement of
adjusted unlocked branches is only used by 1% so far. That's still some
decent progress toward eliminating the need for direct mode.

command line vs assistant

Well that's plain enough isn't it? Although note that I myself have
the assistant running in some repos all the time, but would of course
vote "command line" since I interact with that much more.

Also notice that people who apparently don't use git-annex
but wanted to fill out the survey anyway was the same for 2013-2015,
but has now declined.

operating system

Android users have more or less gone away since I deprecated the app.
I hope the termux integration brings some back.

how git-annex is installed

Good to see the increase in using git-annex packages from the OS
or a third-party package manager.

missing/incomplete ports

Good improvement here since 2015 with 60% now satisfied with available
ports.

Worth noting that in 2013, 6% wanted a way to use git-annex on
Synology NAS. That is possible now via the standalone linux tarball.
This year, 2% wanted "Synology NAS (app store package)".

Also honorable mention to the anonymous person who rewrote
git-annex in another language. You should release the code!

number of repositories

Increasingly users seem to have just a couple repositories or a large
number, with the middle ground shrinking. A few percent have 200+
repositories now. The sense is of a split between causual users who
perhaps clone one repository to a few places, and power users who
are adding new repositories over time.

data stored in git-annex

Increasing growth in the high end with many users storing dozens of terabytes
of data in git-annex and a couple storing more than 64 terabytes.
And a bit of growth in the low end storing under 100 gb.

The total data stored in git-annex looks to be around 650-1300 terabytes
now. It was around 150-300 terabytes in 2013. That doesn't count redundant
data. And it could be off slightly if shared repositories were reported by
multiple users.

(Compare with the Internet Archive, which was 15000 terabytes in 2016
but I think they keep two copies of everything, so call it 7000
terabytes of unique data.)

git level

The same question was asked in the git surveys
so I have included those in the graph for comparison.

git-annex users trend more experienced than git users,
which is not surprising. You have to know some stuff about
git to understand why you'd want to use git-annex.

Notice that git knowledge level is generally going up over time in both
surveys.

happyness with the software

A similar question on the git survey included for comparison.

There's a bimodal distribution to git-annex user's happyness,
with more unhappy with it than with git, but also more so happy
they gravitate toward extreme praise.

There seem to be more unhappy users in 2018 than in 2015 though.
The 2018 results are very close to the 2013 results.

blocking problems

Notably 15% of users now find git-annex too hard to use, up from 5% in
2015. Which seems to correlate with some users being more unhappy with it.
I don't think git-annex has gotten any harder to use, so this must
reflect a change in expectations and/or demographics. (2013 had similar
numbers to 2018.)

Very few complain about the documentation now, down to 3% from 13% in 2015,
but 12% want to see more tutorials showing how to tie the features
together.

And a staggering 21% picked a write-in, "no issues personally, but people
don't see (or realize they need) the immense benefits it provides".
Need to find better ways to market git-annex, essentially.

size of group using git-annex together

A similar distribution to 2015. One person said they're using git-annex in
a group of 50+, and 5 reported groups larger than 10 people.

scientific data

A new high of 11% of respondants are using git-annex to store scientific data.
(Other kinds of data it's used for seem more or less the same.)

Part of that growth is because of the companion
2018 git-annex scientific data survey
which was promoted in some scientific communities, and
so brought more scientists to the main survey.

The use for neuroscience is no surprise, but so much use for astronomy and
physics is. And "other" in that pie chart includes statistics,
social sciences, mathematics, education, linguistics, biomedical
engineering, EE, and physiology -- wow!

survey reach

All participants in the science survey did go on to answer at least part of
the main survey. So 37% of respondants to the main survey are scientists.

A full 27% of survey respondants have their name on the thanks
page, many for financial support. Which is really great, but also speaks to
the fraction of the git-annex user base who saw the survey,
because I really doubt that a quarter of the users of any free software
are financially supporting it.

As with any online survey, the results are skewed by who bothers to answer it.
Still, a lot of useful information to mull over.

Started off the day with some more improvements and bug fixes for export
remotes.

Then I noticed that there is no progress displayed for transfers to export
remotes; it seems I forgot to wire that up. That really ought to be handled
by the special remote setup code, the same way it is for non-export
remotes. But it was not possible to do it there the way that export actions
are structured.

I got sidetracked with how S3 prepares a handle to the server. That didn't
work as well as it might have; most of the time each request to the remote
actually prepared a new handle, rather than reusing a single handle. Though
the http connection to the server did get reused, that still caused a lot
of unncessary work. I fixed that, and the fix also allowed me to
restructure export actions in the way I need for progress bars.

I've ran out of time to finish adding the missing progress bars today, so
I'll do it tomorrow.

Today's release is to fix a
data loss bug,
that affects S3 remotes configured with exporttree=yes that got versioning=yes
turned on after some unversioned data is stored in them. If you use the new
versioning=yes feature with S3, please upgrade.

After a long struggle with the test suite, the new
git-annex release is finally out today.

A few last-minute changes in the release include removing
box.com from the webapp since their webdav gateway is EOL at the end of the
month, supporting armv71 in the android installation script, allowing
installation with 64 bit git on windows, and shortening the estimated time
to completion display.

Offline today due to weather, but there's lots of nice backlog to work
on...

I've written down a external remote querying transition plan.
If you maintain an external special remote that implements WHEREIS or
GETINFO, please take a look as your code would need to be updated if this
is done.

Ilya suggested making git annex testremote be able to test readonly
remotes, and I implemented that.

There was a discussion in the forum about .git/annex/misctmp/
containing cruft left by an interrupted git-annex process.
I was surprised to find half a gigabyte of old files on my own laptop due
to this problem. I've put in a fix, so git-annex will clean up such temp
files that were left behind by a previous interrupted git-annex process.

I said I was going to stop with the ByteString conversion, but then I
looked at profiling, and I knew I couldn't stop there --
conversion between String and ByteString had became a major cost center.

So today, converted all the code that reads and parses symlinks and pointer files
to ByteString, now ByteString is used all the way from disk to Key. Also
put in some caching, so git-annex does not need to re-serialize a Key
that it's just deserialized from a ByteString.

There's still some ByteString to String conversion when generating
FilePaths; to avoid that will need an equivilant of System.FilePath that
operates on RawFilePath, and I don't think there is one yet? But the
profiling does show improvement, it's more and more dominated by IO
operations that can't be sped up, and less by slow code.

Microbenchmarks of Keys improved, especially parsing them got 700% faster.
But key parsing is not enough of an overhead in any commands I benchmarked
to be a real improvement.

The new key parser is much stricter than the old one, which helps the
speed. Hopefully the oddly formatted edge cases that the old parser allowed
are not really in use; they include keys with fields out of the usual order,
and keys with multiple values for the same field.

The next step would probably be to convert the git interface to use
ByteStrings, and that plus the current groundwork is likely to lead to some
real performance improvements. But I'm going to stop here with the
ByteString conversion for now.

Spent two days converting all code that deal with git-annex branch log
files to use attoparsec and bytestring builders.

For most of them, I'm not expecting much if any speed improvements, since
often git-annex only ever parses a given log file once, and writes to many
log files are only done rarely. The main candidates for speedup are chunk
logs and remote state logs. Also Group was converted to a ByteString,
which may speed up queries that involve groups. I have not benchmarked.
It was still worth doing the conversion, for consistency and better code
if not speed.

I found a few bugs in the old parsers for log files along the way.
The uuid.log parser was not preserving whitespace in repositiory
descriptions; the new one will. And the activity.log parser filtered out
unknown values, not leaving room for expansion.

Continuing with the ByteString conversion marathon, today worked on
converting the metadata types. Actually, metadata field names made more
sense to change to Text, since they're limited to a subset of utf-8.

I lost 2 hours to a puzzling quickcheck failure of metadata serialization.
It turned out to involve unicode non-breaking spaces. Aaargh.
Otherwise, fairly straightforward changes, but metadata is used all over
git-annex, so the final patch was nearly 1000 lines.

Benchmark time:

setting metadata in 1000 files...... 1% speedup
getting metadata from 1000 files.... 0.5% speedup
finding a single file out of 1000 that has a given metadata value... 5% speedup

I've been benchmarking whole calls to eg git annex whereis, and that's
not ideal because git-annex has some startup overhead that I'm not
interested in benchmarking (right now), and often that overhead swamped
the things I wanted to benchmark, making it difficult to trust my results.

So, I've built a git annex benchmark command, that can benchmark any other
git-annex commands, without starting a new git-annex process. It uses
criterion to get statistically meaningful benchmark results. And operations
as fast as 10 ms can be benchmarked now, without needing to write any
special purpose benchmark code.

Converted git-annex branch access to use ByteStrings, with support also for
writing to it using bytestring-builder, which is supposed to be faster.
Finished both an attoparsec parser and a builder for the location logs.
All the other logs just convert to and from String for now, so there is
still a lot of work to do.

The git annex whereis benchmark looks to be around 6% total speedup now,
so this only improved it by a few percent, but these little speedups are adding up.

Writing to the git-annex branch may also have sped up significantly; the
builder is probably able to stream out to git without doing any internal
copies. But there are not many cases where git-annex does a lot of writes to
the branch without some other operation that is much more expensive, so I don't
anticipate much speed improvement on that side.

Built an attoparsec parser for timestamps, and unsurprisingly it's 15 times
faster parsing ByteStrings with it than the old String parser. The
surprising thing to me was that converting a String to a ByteString and
using the new parser is 10 times as fast as the old parser despite the
conversion overhead. A nice immediate speedup for many parts of git-annex!

Of course timestamp parsing is not a major cost center in git-annex, but
benchmarking git annex whereis run on 1000 files, there is a real
speedup already, approximately 4%.

Today was spent converting the UUID data type to use a ByteString, rather
than a String, and also converting repo descriptions to ByteString.
That's groundwork for reading and writing log files on the git-annex
branch using attoparsec and ByteString builders, which will hopefully
improve performance.

Until that's complete, it will often convert a String to a ByteString and
then back to a String, which could actually make performance slightly
worse. Benchmarking git annex whereis doesn't find much of a change. It
may have gotten a slightly faster overall, due to the faster Eq and
Ord instances making the map of repositories faster.

Fixed several bugs involving upgrade to v7 when the git repository
already v7 contained unlocked files. The worst of those involved direct
mode and caused the whole file content to get checked into git. While
that's a fairly unusual case, it's an ugly enough bug that I rushed out a
release to fix it.

Snowed in and without internet until now, I've been working on the backlog.
This included adding git annex find --branch and adding support
for combining options like --include, --largerthan etc with
--branch.

I had said I was going to clear votes to the draft survey, but for
technical reasons I decided not to. So if you already voted you don't need
to vote again, but if you made any joke votes please go change your vote.
(I removed the proposed port of git-annex to the TOPS-20.)

I fixed two reversions yesterday (neither related to v7 repos) during a day
of triage in preparation for the release of git-annex 7.

One of the reversions broke adding remotes in the webapp, and was filed all
the way back in January with lots of confirmations. I feel bad I didn't get
around to even looking at that bug report until now.

My backlog is kind of large, it hovers around 400 messages most of the time
now, there needs to be a better way to make sure I notice such bad bugs. Would
someone like to help with git-annex bug triage, picking out bugs that
multiple users have confirmed, or that have good intructions to reproduce
them, and helping me prioritize them? No coding required, massive
contribution to git-annex. Please get in touch.

Anyway, after that full day's work, I took a look at the autobuilders, and
it was bad; the test suite was failing everywhere testing v7. For quite a
while I've been seeing intermittent test suite failures involving the new
repo version, that mostly only happened on the autobuilders. But now they
were more reproducible; a recent change made them happen much more
frequently. That was good; it made it easier to track down the problem.

Which was that git-annex was getting mtime information with 1 second
granularity. So when the test suite modified a file several times in the
same second, git-annex could fail to notice some of the modifications. I
think when I origianlly developed the inode cache module in 2013, for
direct mode, there was no easy way to access high-precision mtimes from
haskell, but there is now, and git-annex will use them.

That left one other failure in the test suite, an intermittent crash of
sqlite with ErrorIO on Linux. May be related to the known sqlite crashes in WSL.
I've been trying various things today to try to fix it, but have to run the
test suite in a loop for several hours to reproduce it reliably.

In the delaysmudge branch, I've implemented the delayed worktree update in
the post-merge/post-checkout hooks for v6. It works very well!

In particular, with annex.thin set, checking out a branch
containing a huge unlocked file does a fast hard link to the file.

Remaining problem before merging that is, how to get the new hooks installed? Of
course git annex init and git annex upgrade install them, but I know plenty
of people have v6 repositories already, without those hooks.

So, would it be better to bump up to v7 and install the hooks on that upgrade,
or stay on v6 and say that it was, after all, experimental up until now, and
so the minor bother of needing to run git annex init in existing v6 repositories
is acceptable? If the version is bumped to v7, that will cause some pain
for users of older versions of git-annex that won't support it, but those old
versions also have pretty big gaps in their support for v6. I'm undecided,
but leaning toward v7, even though it will also mean a lot of work to update
all the documentation, as well as needing changes to projects like datalad that use
git-annex. Feedback on this decision is welcomed below...

Dreadfully early this morning I developed a plan for a way to finish the
last v6 blocker, that works around most of the problems with git's smudge
interface. The only problem with the plan is that it would make both git
stash and git reset --hard leave unlocked annexed files in an
unpopulated state when their content is available. The user would have to
run git-annex afterwards to fix up after them. All other git checkout,
merge, etc commands would work though.

Not sure how I feel about this plan, but it seems to be the best one so
far, other than going off and trying to improve git's smudge interface again.
I also wrote up ?git smudge clean interface suboptiomal which explains
the problems with git's interface in detail.

Goal for today was to make git annex sync --content operate on files hidden by
git annex adjust --hide-missing. However, this got into the weeds pretty
quickly due to the problem of how to handle --content-of=path when
either the whole path or some files within it may be hidden.

Eventually I discovered that git ls-files --with-tree can be used to
get a combined list of files in the index plus files in another tree,
which in git-annex's case is the original branch that got adjusted.
It's not documented to work the way I'm using it (worrying), but it's
perfect, because git-annex already uses git ls-files extensively and this
could let lots of commands get support for operating on hidden files.

That said, I'm going to limit it to git annex sync for now,
because it would be a lot of work to make lots of commands support them,
and there could easily be commands where supporting them adds lots of
complexity or room for confusion.

At long last there's a way to hide annexed files whose content
is missing from the working tree: git-annex adjust --hide-missing

And once you've run that command, git annex sync will update the tree
to hide/unhide files whose content availability has changed.
(So will running git annex adjust again with the same options.)

You can also combine --hide-missing with --unlock, which should prove
useful in a lot of situations.

My implementation today is as simple as possible, which means that every
time it updates the adjusted branch it does a full traversal of the
original branch, checks content availability, and generates a new branch.
So it may not be super fast in a large repo, but I was able to implement
it in one day's work. It should be possible later to speed it up a
lot, by maintaining more state.

No time to blog yesterday, but I somehow found the time to fix the second
to last known major issue with v6 mode, a database inconsistency problem
involving touching annexed files.

The only remaining blocker for v6 not being experimental is that git
checkout of large unlocked files can use a lot of memory (and doesn't
honor annex.thin).

Also I finally have a rought plan for how to hide missing files:
Have git annex sync update the working tree to only show visible files.
Still details to work out, but it would be great to finally get this
often-requested feature.

Pulled the trigger on the old Android builds, and made a
massive commit
removing all the cruft that had built up to enable them. Running in Termux
is just better. It's important to note this does not mean I've given up on
more native git-annex Android stuff, indeed there are promising
developments in ghc Android support that I'm keeping an eye on.

I'll kind of miss the EvilSplicer, that was 750 lines of crazy code to be
proud of. But really, it's going to be great to not have hanging over me
the prospect that any change could break the Android build and end up
needing tons of work to resolve.

I've improved the termux installation, adding an installer script to make
it easier, and fixing some issues that have been reported. And it supports
arm64 and also should work on Intel android devices.
This feels very close to being able to remove the old deprecated Android
apps.

I'm temporarily running the arm64 builds on my phone, in a Debian chroot. But
it overheats, so this is a stopgap and it won't autobuild daily, only manually
at release time.

Been making some improvements to git-annex export over the past couple
days, but took time off this afternoon to set up a new phone, and try
git-annex in termux
on it. Luckily, I was able to reproduce the signal 11
on arm64 problem that several users have reported earlier, and also found a fix,
which is simply to build git-annex for arm64.

So I want to set up a arm64 autobuilder, and if someone has an
arm64 server that could host it, that would be great. Otherwise, I could
use Scaleway, but I'd rather avoid that ongoing expense.

Also fixed a recent reversion in the linux standalone
runshell that broke git-annex in termux, and probably on some other
systems.

Started work on http://git-annex.branchable.com/todo/to_and_from_multiple_remotes/.
It's going slow, I had to start with a large refactoring. So far, option
parsing is working, and a few commands are almost working, but concurrency
is not working right, and concurrency is the main reason to want to support
this (along with remote groups).

Unix would be better if filenames could not contain newlines. But they can,
and so today was spent dealing with some technical debt.

The main problem with using git-annex with filenames with newlines is that
git cat-file --batch uses a line-based protocol. It would be nice if that
were extended to support -z like most of the rest of git does, but I
realized I could work around this by not using batch mode for the rare
filename with a newline. Handling such files will be slower than other
files, but at least it will work.

Then I realized that git-annex has its own problems with its --batch
option and files with newlines. So I added support for -z to every batchable
command in git-annex, including a couple of commands that did batch input
without a --batch option.

Now git-annex should fully support filenames containing newlines, as well
as anything else. The best thing to do if you have such a file is to
commit it and then git mv it to a better name.

Well it took the whole day to finish the release. Including fixing a
deadlock when the new v6 code runs with an older git, and some build
errors. Ant there's still an intermittent test suite failure involving v6 on
one autobuilder, which will need to be dealt with later.

This is a big release, lots of bug fixes, lots of v6 improvements, and
significant S3 improvements.

I'm in release prep mode now, fixing build problems and a few bugs, but
the v6 sprint is well over, though v6 still has its issues.
Might as well release all the last month's work.

Yesterday was taken up with dealing with some very ugly git interface
stuff that changed between versions. An July workaround for a bug in
git turns out to have caused reversions with older versions, and was not
a complete fix either. Tuned it to hopefully work better.

Got git-annex downloading versioned files from S3, without needing S3
credentials. This makes a S3 special remote be equally capable as a
git-annex repository exported over http, other than of course not including
the git objects.

This is nice; I only wish it were supported by other special remotes.
It seems that any special remote could be made to support it, but ones not
supporting some kind of versioning would need to store each file twice,
and many would also need each file to be uploaded to them twice. But
perhaps there are others that do have a form of versioning.
WebDAV for one has a versioning extension in RFC 3253.

Also did a final review of a patch Antoine Beaupré is working on to
backport the recent git-annex security fixes to debian oldstable,
git-annex 5.20141125. He described the backport in his blog:

This time again, Haskell was nice to work with: by changing type
configurations and APIs, the compiler makes sure that everything works out and
there are no inconsistencies. This logic is somewhat backwards to what we are
used to: normally, in security updates, we avoid breaking APIs at all costs.
But in Haskell, it's a fundamental way to make sure the system is still
coherent.

Several little things today, including a git-annex.cabal patch from
fftehnik that fixed building without the assistant, and supporting
AWS_SESSION_TOKEN. The main work was on making git annex drop --dead
prune obsolete per-remote metadata, and on fixing a bug in v6 mode that
left git-annex object files writable.

Today's work was sponsored by Paul Walmsley in honor of Mark Phillips.

The storage of S3 version IDs got rethought -- I was not comfortable with
using per-remote state in the git-annex branch which would have caused
problems if dropping from these remotes later gets supported.

So, I added per-remote metadata in about 1 hour! It's like git-annex's regular
metadata, but scoped so only the remote that owns it can see it. This is
perfect for storing things like S3 version IDs. It probably ought to be added
to the external special remote interface since it could be used for lots of
stuff.

Most of the way done with implementing support for export to S3 buckets
with versioning enabled. This will make the files from the most recent
git annex export be visible to users browsing the bucket, while letting
git-annex download any of the content from previous exports too.

Still need to test it. And, deletion of old content from such a bucket is
not supported, and my initial thoughts are that it might not be possible
in a multi-writer situation. I need to think about it more.

Looked over bugs filed about v6 mode and did some triage and analysis.
smudge has the details.

This led to changing what's done by git add and git commit file when
annex.largefiles is not configured. Rather than behaving like git annex
add and always storing the file in the annex, it will store it in the
annex if the old version was annexed, and in git if the old version was
stored in git. This avoids accidental conversions.

It might make sense to have git annex add also do this, even in v5
repositories, but I want to concentrate on v6 for now, and also don't
think that git add and git annex add necessarily need to behave
identically in v6 mode. While using git commit -a doesn't imply
anything about whether you want the file in git or the annex,
using git-annex add seems to imply they you want it in
the annex, unless you've gone out of your way to configure otherwise.

Also did some design work on supporting versioned S3 buckets with
git-annex export.

That's a lot of races! Well, 4 of them were all related, but the fixes to
them had to be made in two different places.

Hopefully that's all the v6 races fixed. I've been finding these races by
inspection, who knows if I missed some. Anyway, I'm now down to one todo
item left on the v6 sprint. Gonna take a break for a couple days before
tackling it.

More v6 work. Got most of the way to a solution to the problem of updating
the associated files database for staged changes to unlocked files, eg a
git mv.

While writing the test case, I was surprised to find that the problem
is timing dependent. If a git mv is run less than a second after git
add, git runs the smudge filter for whatever reason, which avoids the
problem. With a longer delay, it doesn't run the smudge filter. Seems this
could be the cause of intermittent glitches with v6 mode, and I've seen a
few such glitches before.

Anyway, I developed an inexpensive way to find the relevant staged changes,
using git diff with a full page of options to tweak its behavior just
right. Still need to make that only run when the index has changed, not
every time git-annex runs.

There's still a race between a command like git mv and git annex
drop/get, that can result in the unlocked file's content not being
updated. Don't have a solution to that yet.

Sleeping on that race from yesterday, I realized there is a way to fix it,
and have implemented the fix. It doubled the overhead of updating the
index, but that's worth it to not have a race condition to worry about.

Found a better way to update the index after get/drop in v6 repositories.
I was able to close all the todos around that.

Only problem is there is a race where a modification that happens to a file
soon after get/drop gets unexpectedly staged by the index update. I made
this race's window as small as I reasonably can. Fully fixing it would
involve improvements to the git update-index interface, or another way to
update the index.

Only two todos remain in smudge that
I want to fix in the remainder of this v6 sprint.

I've now fixed the worst problem with v6 mode, which was that get/drop of
unlocked files would cause git to think that the files were modified.

Since the clean filter now runs quite fast, I was able to fix that by,
after git-annex updates the worktree, restaging the not-really-modified
file in the index.

This approach is not optimal; index file updates have overhead; and only
one process can update the index file at one time. smudge has a
bunch of new todo items for cases where this change causes problems. Still,
it seems a lot better than the old behavior, which made v6 mode nearly
unusable IMHO.

Working on a "filterdriver" branch, I've implemented support for the
long-running smudge/clean process interface.

It works, but not really any better than the old smudge/clean interface.
Unfortunately git leaks memory just as badly in the new interface as it did
in the old interface when sending large data to the smudge filter. Also,
the new interface requires that the clean filter read all the content of the
file from git, even when it's just going to look at the file on disk, so
that's worse performance.

So, I don't think I'll be merging that branch yet, but git's interface does
support adding capabilities, and perhaps a capability could be added that
avoids it schlepping the file content over the pipe. Same as my old git
patches tried to do with the old smudge/clean interface.

Spent today implementing the git pkt-line protocol. Git uses it for a bunch
of internal stuff, but also to talk to long-running filter processes.

This was my first time using attoparsec, which I quite enjoyed aside from
some difficulty in parsing a 4 byte hex number. Even though parsing to a
Word16 should naturally only consume 4 bytes, attoparsec will actually
consume subsequent bytes that look like hex. And it may parse fewer than 4
bytes too. So my parser had to take 4 bytes and feed them back into a call
to attoparsec. Which seemed weird, but works. I also used
bytestring-builder, and between the two libraries, this should be quite a
fast implementation of the protocol.

With that 300 lines of code written, it should be easy to implement support
for the rest of the long-running filter process protocol. Which will surely
speed up v6 a bit, since at least git won't be running git-annex over and
over again for each file in the worktree. I hope it will also avoid a memory
leak in git. That'll be the rest of the low-hanging fruit, before v6
improvements get really interesting.

Plan is to take some time this August and revisit v6, hoping to move it
toward being production ready.

Today I studied the "Long Running Filter Process" documentation in
gitattributes(5), as well as the supplimental documentation in git about
the protocol they use. This interface was added to git after v6 mode was
implemented, and hopefully some of v6's issues can be fixed by using it in
some way. But I don't know how yet, it's not as simple as using this
interface as-is (it was designed for something different), but
finding a creative trick using it.

So far I have this idea
to explore. It's promising, might fix the worst of the problems.

Also, reading over all the notes in smudge, I finally
checked and yes, git doesn't require filters to consume all stdin anymore,
and when they don't consume stdin, git doesn't leak memory anymore either.
Which let me massively speed up git add in v6 repos. While before git
add of a gigabyte file made git grow to a gigabyte in memory and copied a
gigabyte through a pipe, it's now just as fast as git annex add in v5
mode is.

After the big security fix push, I've had a bit of a vacation. Several new
features have also landed in git-annex though.

git-worktree support is a feature I'm fairly excited by.
It turned out to be possible to make git-annex just work in working trees
set up by git worktree, and they share the same object files. So,
if you need several checkouts of a repository for whatever reason,
this makes it really efficient to do. It's much better than the old
method of using git clone --shared.

A new --accessedwithin option matches files whose content was accessed
within a given amount of time. (Using the atime.) Of course it can
be combined with other options, for example
git annex move --to archive --not --accessedwithin=30d
There are a few open requests for other new file matching options that I
hope to get to soon.

A small configuration addition of remote.name.annex-speculate-present
to make git-annex try to get content from a remote even if its records
don't indicate the remote contains the content allows setting up an interesting
kind of local cache of annexed files
which can even be shared between unrelated git-annex repositories, with
inter-repository deduplication.

I suspect that remote.name.annex-speculate-present may also have other
uses. It warps git-annex's behavior in a small but fundamental way which
could let it fit into new places. Will be interesting to see.

There's also a annex.commitmessage config, which I am much less excited by,
but enough people have asked for it over the years.

Also fixed a howler of a bug today: In -J mode, remotes were sorted not
by cost, but by UUID! How did that not get noticed for 2 years?

Much of this work was sponsored by NSF-funded DataLad project at Dartmouth
Colledge, as has been the case for the past 4 years. All told they've
funded over 1000 hours of work on git-annex. This is the last month of that
funding.

Please go upgrade now, read the release notes
for details about some necessary behavior changes,
and if you're curious about the details of the security holes,
see the advisory.

I've been dealing with these security holes for the past week and a half,
and decided to use a security embargo while fixes were being developed
due to the complexity of addressing security holes that impact both
git-annex and external special remote programs. For the full story
see past 5 posts in this devblog, which are being published all together
now that the embargo is lifted.

Was getting dangerously close to burnt out, or exhaustion leading to
mistakes, so yesterday I took the day off, aside from spending the morning
babysitting the android build every half hour. (It did finally succeed.)

Today, got back into it, and implemented a fix for CVE-2018-10859 and also
the one case of CVE-2018-10857 that had not been dealt with before.
This fix was really a lot easier than the previous fixes for
CVE-2018-10857.
Unfortunately this did mean not letting URL and WORM keys be downloaded
from many special remotes by default, which is going to be painful for some.

Started testing that the security fix will build everywhere on
release day. This is being particularly painful for the android build,
which has very old libraries and needed http-client updated, with many
follow-on changes, and is not successfully building yet after 5 hours.
I really need to finish deprecating the android build.

Pretty exhausted from all this, and thinking what to do about
external special remotes, I elaborated on an idea that Daniel Dent had
raised in discussions about vulnerability, and realized that git-annex
has a second, worse vulnerability. This new one could be used to trick a
git-annex user into decrypting gpg encrypted data that they had
never stored in git-annex. The attacker needs to have control of both an
encrypted special remote and a git remote, so it's not an easy exploit to
pull off, but it's still super bad.

This week is going to be a lot longer than I thought, and it's already
feeling kind of endless..

Spent several hours dealing with the problem of http proxies, which
bypassed the IP address checks added to prevent the security hole.
Eventually got it filtering out http proxies located on private IP
addresses.

Other than the question of what to do about external special remotes
that may be vulerable to related problems, it looks like the security
hole is all closed off in git-annex now.

Added a new page security with details of this and past security holes
in git-annex.

Several people I reached out to for help with special remotes have gotten
back to me, and we're discussing how the security hole may affect them and
what to do. Thanks especially to Robie Basak and Daniel Dent for their
work on security analysis.

Also prepared a minimal backport of the security fixes for the git-annex in
Debian stable, which will probably be more palatable to their security team
than the full 2000+ lines of patches I've developed so far.
The minimal fix is secure, but suboptimal; it prevents even safe urls from
being downloaded from the web special remote by default.

Unforunately as part of this, had to make youtube-dl and curl not be used
by default. The annex.security.allowed-http-addresses config has to be
opened up by the user in order to use those external commands, since they
can follow arbitrary redirects.

Also thought some more about how external special remotes might be
affected, and sent their authors' a heads-up.

Most of the day was spent staring at the http-client source code and trying
to find a way to add the IP address checks to it that I need to fully close
the security hole.

In the end, I did find a way, with the duplication of a couple dozen lines
of code from http-client. It will let the security fix be used with
libraries like aws and DAV that build on top of http-client, too.

While the code is in git-annex for now, it's fully disconnected and
would also be useful if a web browser were implemented in Haskell,
to implement same-origin restrictions while avoiding DNS rebinding attacks.

Looks like http proxies and curl will need to be disabled by default,
since this fix can't support either of them securely. I wonder how web
browsers deal with http proxies, DNS rebinding attacks and same-origin?
I can't think of a secure way.

Next I need a function that checks if an IP address is a link-local address
or a private network address. For both ipv4 and ipv6. Could not find
anything handy on hackage, so I'm gonna have to stare at some RFCs. Perhaps
this evening, for now, it's time to swim in the river.

I'm writing this on a private branch, it won't be posted until a week from
now when the security hole is disclosed.

Security is not compositional. You can have one good feature, and add
another good feature, and the result is not two good features, but a new
security hole. In this case
security hole private data exposure via addurl (CVE-2018-10857).
And it can be hard to spot this kind of security hole, but then once it's
known it seems blindly obvious.

It came to me last night and by this morning I had decided the potential
impact was large enough to do a coordinated disclosure. Spent the first
half of the day thinking through ways to fix it that don't involve writing
my own http library. Then started getting in touch with all the
distributions' security teams. And then coded up a fairly complete fix for
the worst part of the security hole, although a secondary part is going to
need considerably more work.

It looks like the external special remotes are going to need at least some
security review too, and I'm still thinking that part of the problem over.

Took a while to find the necessary serial cables and SD cards to test
propellor's ARM disk image generation capabilies.

Ended up adding support for the SheevaPlug because it was the first board I
found the hardware to test. And after fixing a couple oversights, it worked
on the second boot!

Then after a trip to buy a microSD card,
Olimex Lime worked on the first boot! So did CubieTruck and Banana Pi.
I went ahead and added a dozen other sunxi boards that Debian supports,
which will probably all work.

(Unfortunately I accidentially corrupted the disk of my home server
(router/solar power monitor/git-annex build box) while testing the
CubieTruck. Luckily, that server was the first ARM board I want to rebuild
cleanly with propellor, and its configuration was almost entirely in
propellor already, so rebuilding it now.)

I started by adding support for generating non-native chroots.
qemu-debootstrap makes that fairly simple. Propellor is able to run
inside a non-native chroot too, to ensure properties in there;
the way it injects itself into a chroot wins again as that just worked.

Then, added support for flash-kernel, and for u-boot installation.
Installing u-boot to the boot sector of SD cards used by common ARM boards
does not seem to be automated anywhere in Debian, just README.Debian files
document dd commands. It may be that's not needed for a lot of boards
(my CubieTruck boots without it), but I implemented it anyway.

And, Propellor.Property.Machine is a new module with properties for
different ARM boards, to get the right kernel/flash-kernel/u-boot/etc
configuration easily.

This all works, and propellor can update a bootable disk image for an ARM
system in 30 seconds. I have not checked yet if it's really bootable.

Tomorrow, I'm going to dust off my ARM boards and try to get at least 3
boards tested, and if that goes well, will probably add untested properties
for all the rest of the sunxi boards.

I've been using Secure Scuttlebutt for 6 months
or so, and think it's a rather good peer-to-peer social network. But it's very
Javascript centric, and I want to be able to play with its data from Haskell.

The scuttlebutt-types library is based on some
earlier work
by Peter Hajdu. I expanded it to be have all the core Scuttlebutt data
types, and got it parsing most of the corpus of Scuttlebutt messages.
That took most of yesterday and all of today. The next thing to tackle
would be generating JSON for messages formatted so the network accepts it.

I don't know what scuttlebutt-types will be used for. Maybe looking up
stats, or building bots, or perhaps even a Scuttlebutt client? We'll
see..

After a rather interesting morning, the secret-project is doing exactly
what I set out to accomplish! It's alllive!

(I found a way to segfault the ghc runtime first.
And then I also managed to crash firefox, and finally managed to crash and
hard-hang rsync.)

All that remains to be done now is to clean up the user interface.
I made propellor's output be displayed in the web browser, but currently
it contains ansi escape sequences which don't look good.

This would probably be a bad time to implement a in-browser terminal
emulator in haskell, and perhaps a good time to give propellor customizable
output backends. I have 3 days left to completely finish this project.

One last detour: Had to do more work than I really want to at this stage,
to make the secret-project pick a good disk to use. Hardcoding a disk
device was not working reliably enough even for a demo. Ended up with some
fairly sophisticated heuristics to pick the right disk, taking disk size
and media into account.

Then finally got on with grub installation to the target disk. Installing
grub to a disk from a chroot is a fiddley process hard to get right. But, I
remembered writing similar code before; propellor installs grub to a disk
image from a chroot. So I generalized that special-purpose code to something
the secret-project can also use.

It was a very rainy day, and rain fade on the satellite internet prevented
me from testing it quickly. There were some dumb bugs. But at 11:30 pm,
it Just Worked! Well, at least the target booted. /etc/fstab is not 100% right.

Late Friday evening, I realized that the secret-project's user interface
should be a specialized propellor config file editor. Eureka! With that in mind,
I reworked how the UserInput value is passed from the UI to propellor;
now there's a UserInput.hs module that starts out with an unconfigured
value, and the UI rewrites that file. This has a lot of benefits, including
being able to test it without going through the UI, and letting the UI be
decoupled into a separate program.

Also, sped up propellor's disk image building a lot. It already had some
efficiency hacks, to reuse disk image files and rsync files to the disk
image, but it was still repartitioning the disk image every time, and the
whole raw disk image was being copied to create the vmdk file for
VirtualBox. Now it only repartitions when something has changed, and the
vmdk file references the disk image, which sped up the secret-project's 5
gigabyte image build time from around 30 minutes to 30 seconds.

With that procrastinationWgroundwork complete, I was finally at the point of
testing the secret-project running on the disk image. There were a few minor
problems, but within an hour it was successfully partitioning, mounting,
and populating the target disk.

Still have to deal with boot loader installation and progress
display, but the end of the secret-project is in sight.

It would probably be good to extend that with a combinator that sets
a minimum allows size, so eg / can be made no smaller than 4 GB.
The implementation should make it simple enough to add such combinators.

I thought about reusing the disk image auto-fitting code, so the
target's / would start at the size of the installer's /.
May yet do that; it would make some sense when the target and
installer are fairly similar systems.

After building the neato Host versioning described in
Functional Reactive Propellor this weekend,
I was able to clean up secret-project's config file significantly
It's feeling close to the final version now.

At this point, the disk image it builds is almost capable of installing the
target system, and will try to do so when the user tells it to. But,
choosing and partitioning of the target system disk is not implemented yet,
so it installs to a chroot which will fail due to lack of disk space there.
So, I have not actually tested it yet.

Integrating propellor and secret-project stalled out last week.
The problem was that secret-project can only be built with stack right now
(until my patch to threepenny-gui to fix drag and drop handling gets
accepted), but propellor did not support building itself with stack.

I didn't want to get sucked into yak shaving on that, and tried to find
another approach, but finally gave in, and taught propellor how to build
itself with stack. There was an open todo about that, with a hacky
implementation by Arnaud Bailly, which I cleaned up.

Then the yak shaving continued as I revisited a tricky intermittent
propellor bug. Think I've finally squashed that.

Finally, got back to where I left off last week, and at last here's a
result! This is a disk image that was built entirely from a propellor
config file, that contains a working propellor installation, and that
starts up secret-project on boot.

Now to make this secret-project actually do something more than looking
pretty..

This was a one step forward, one step back kind of day, as I moved the
working stuff from yesterday out of my personal propellor config file and
into the secret-project repository, and stumbled over some issues while
doing so.

But, I had a crucial idea last night. When propellor is used to build an
installer image, the installer does not need to bootstrap the target system
from scratch. It can just copy the installer to the target system, and then
run propellor there, with a configuration that reverts any properties that
the installer had but the installed system should not. This will be a
lot faster and avoid duplicate downloads.

That's similar to how d-i's live-installer works, but simpler, since
with propellor there's a short list of all the properties that the
installer has, and propellor knows if a property can be reverted or not.

Indcidentially, I have power and bandwidth to work on this kind of
propellor stuff from home all day, for the first time!
Up until now, all propellor features involving disk images were
either tested remotely, or developed while I was away from home.
It's a cloudy, rainy day; the
solar upgrade
and satellite internet paid off.

What is this strange thing? It's a prototype, thrown together with
open clip art in a long weekend. It's an exploration how far an interface
can diverge from the traditional and still work. And it's a mini-game.

Watch the video, and at the end, try to answer these questions:

What will you do then?

What happens next?

Spoilers below...

What I hope you might have answered to those questions, after watching the
video, is something like this:

What will you do then?
I'll move the egg into the brain-tree.

What happens next?
It will throw away the junk I had installed and replace it with what's in the egg.

The interface I'm diverging from is this kind of thing:

My key design points are these:

Avoid words entirely

One of my takeaways from the Debian installer project is that it's
important to make the interface support non-English users, but
maintaining translations massively slows down development.
I want an interface that can be quickly iterated on or thrown away.

Massively simplify

In the Debian installer, we never managed to get rid of as many questions
as we wanted to. I'm starting from the other end, and only putting in the
absolute most essential questions.

Do you want to delete everything that is on this computer?

What's the hostname?

What account to make?

What password to use?

I hope to stop at the first that I've implemented so far. It should be
possible to make the hostname easy to change after installation, and for an
end user installation, the username doesh't matter much, and the password
generally adds little or no security (and desktop environments should make
it easy to add a password later).

Don't target all the users

Trying to target all users constrained the Debian installer in weird ways
while complicating it massively.

This if not for installing a server or an embedded system.
This interface is targeting end users who want a working desktop system
with minimum fuss, and are capable of seeing and dragging.
There can be other interfaces for other users.

Now it will look for the gpg key of the developer in keyring
files in /usr/share/debug-me/keyring/, and tell the user which project(s)
the developer is known to be a member of. So, for example, Debian
developers who connect to debug-me sessions of Debian users will be
identified as a Debian developer. And when I connect to a debug-me user's
session, debug-me will tell them that I'm the developer of debug-me.

This should help the user decide when to trust a developer to connect to
their debug-me session. If they develop software that you're using, you
already necessarily trust them and letting them debug your machine is not
a stretch, as long as it's securely logged.

Thanks to Sean Whitton for the idea of checking the Debian keyring, which
I've generalized to this.

Also, debug-me is now just an apt-get away for Debian unstable users,
and I think it may end up in the next Debian release.

I made one last protocol compatibility breaking change before the release.
Realized that while the websocket framing protocol has a version number,
the higher-level protocol does not, which would made extending it very hard
later. So, added in a protocol version number.

The release includes a tarball that should let debug-me run on most linux
systems. I adapted the code from git-annex for
completely linux distribution-independent packaging. That
added 300 lines of code to debug-me's source tree, which is suboptimal. But
all the options in the Linux app packaging space are suboptimal. Flatpak
and snappy would both be ok -- if the other one didn't exist and they
were were installed everywhere. Appimage needs the program to be built against
an older version of libraries.

Recorded a 7 minute screencast demoing debug-me, and another 3 minute
screencast talking about its log files. That took around 5 hours of work
actually, between finding a screencast program that works well (I used
kazam), writing the "script", and setting the scenes for the user and
developer desktops shown in the screencast.

While recording, I found a bug in debug-me. The gpg key was not available
when it tried to verify it. I thought that I had seen gpg --verify
--keyserver download a key from a keyserver and use it to verify but seems
I was mistaken. So I changed the debug-me protocol to include the gpg
public key, so it does not need to rely on accessing a keyserver.

Also deployed a debug-me server, http://debug-me.joeyh.name:8081/. It's
not ideal for me to be running the debug-me server, because when me and a
user are using that server with debug-me, I could use my control of the
server to prevent the debug-me log being emailed to them, and also delete
their local copy of the log.

Added an additional hash chain of entered values to the debug-me data
types. This fixes the known problem with debug-me's proof chains, that
the order of two entered values could be ambiguous.

And also, it makes for nicer graphs! In this one, I typed "qwertyuiop" with
high network lag, and you can see that the "r" didn't echo
back until "t" "y" "u" had been entered. Then the two
diverged states merged back together when "i" was entered
chaining back to the last seen "r" and last entered "u".

Having debug-me generate these graphs has turned out to be a very good
idea. Makes analyzing problems much easier.

Also made /quit in the control window quit the debug-me session.
I want to also let the user pause and resume entry in the session, but
it seems that could lead to more ambiguity problems in the proof chain,
so I'll have to think that over carefully.

debug-me seems ready for release now, except it needs some servers,
and some packages, and a screencast showing how it works.

Fixed a tricky race condition that I think was responsible for some
recent instability seen in debug-me when connecting to a session.
It's a bit hard to tell because it caused at least 3 different types
of crashes, and it was a race condition.

Made debug-me prompt for the user's email address when it starts up, and
then the server will email the session log to the user at the end. There
are two reasons to do this. First, it guards against the developer
connecting to the session, doing something bad, and deleting the user's
local log file to cover their tracks. Second, it means the server doesn't
have to retain old log files, which could be abused to store other data on
servers.

** debug-me session control and chat window
Someone wants to connect to this debug-me session.
Checking their Gnupg signature ...
gpg: Signature made Sat Apr 29 14:47:29 2017 JEST
gpg: using RSA key
gpg: Good signature from "John Doe" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: B2CF F6EF 2F01 96B1 CD2C 5A03 16A1 2F05 4447 4791
Checking the Gnupg web of trust ...
Their identity cannot be verified!
Let them connect to the debug-me session and run commands? [y/n]

The debug-me user is likely not be connected to the gpg web of trust,
so debug-me will download the developer's key from a keyserver,
and uses the https://pgp.cs.uu.nl/ service to check if the developer's
key is in the strong set of the web of trust. It prints out the best-connected
people who have signed the developer's key, since the user might recognise some
of those names.

While relying on a server to determine if the developer is in the strong set is
not ideal, it would be no better to have debug-me depend on wotsap,
because wotsap still has to download the WoT database. (Also,
the version of wotsap in debian is outdated and insecure)
The decentralized way is for the user do some key signing, get into the WoT,
and then gpg can tell them if the key is trusted itself.

debug-me is now nearly feature-complete!

It has some bugs, and a known problem with the evidence chain that
needs to be fixed. And, I want to make debug-me servers email logs
back to users, which will change the websockets protocol, so ought to be
done before making any release.

I've been trying to write a good description of debug-me, and here's
what I've got so far:

Short description: "secure remote debugging"

Long description:

Debugging a problem over email is slow, tedious, and hard. The developer
needs to see the your problem to understand it. Debug-me aims to make
debugging fast, fun, and easy, by letting the developer access your
computer remotely, so they can immediately see and interact with the
problem. Making your problem their problem gets it fixed fast.

A debug-me session is logged and signed with the developer's Gnupg
key, producing a chain of evidence of what they saw and what they did.
So the developer's good reputation is leveraged to make debug-me secure.

When you start debug-me without any options, it will connect to a debug-me
server, and print out an url that you can give to the developer to get
them connected to you. Then debug-me will show you their Gnupg key and who
has signed it. If the developer has a good reputation, you can proceed
to let them type into your console in a debug-me session. Once the
session is done, the debug-me server will email you the signed
evidence of what the developer did in the session.

If the developer did do something bad, you'd have proof that they cannot
be trusted, which you can share with the world. Knowing that is the case
will keep most developers honest.

I think that's pretty good, and would like to know your thoughts, reader,
as a prospective debug-me user.

Most of today was spent making debug-me --control communicate over a socket
with the main debug-me process. That is run in a separate window,
which is the debug-me session control and chat window. Things typed there
can be seen by the other people involved in a debug-me session. And, the gnupg
key prompting stuff will be done in that window eventually.

Screenshot of the first time that worked. The "user" is on the left and the
"developer" is on the right.

Went ahead and made debug-me use protocol buffers for its wire protocol.
There's a nice haskell library for this that doesn't depend on anything
else, and can generate them directly from data types, but I had to write a
shim between the protobuf style data types and debug-me's internal
data types. Took 250 lines of quite tedious code.

Then I finally implemented the trick I thought of to leave out the previous
hash from debug-me messages on the wire, while still including
cryptograhically secure proof of what the previous hash was. That reduced
the overhead of a debug-me message from 168 bytes to 74 bytes!

I doubt debug-me's wire format will see any more major changes.

How does debug-me compare with ssh? I tried some experiments, and
typing a character in ssh sends 180 bytes over the wire, while doing the
same in debug-me sends 326 bytes. The extra overhead must be due to using
websockets, I guess. At least debug-me is in the same ballpark as ssh.

Working on polishing up debug-me's features to not just work, but work
well.

On Monday, I spent much longer than expected on the problem that when a
debug-me session ended, clients attached to it did not shut down.
The session shutdown turned out to have worked by accident in one case,
but it was lacking a proper implementation, and juggling all the threads
and channels and websockets to get everything to shut down cleanly was
nontrivial.

Today, I fixed a bug that made debug-me --download fail while downloading a
session, because the server didn't relay messages from developers, and so
the proof chain was invalid. After making the server relay those messages,
and handling them, that was fixed -- and I got a great feature for free:
Multiple developers can connect to a debug-me session and all interact with
it at the same time!

Also, added timing information to debug-me messages. While time is relative
and so it can't be proved how long the time was between messages in the
debug-me proof chain, including that information lets debug-me --download
download a session and then debug-me --replay can replay the log file
with realistic pauses.

Started on gpg signing and signature verification, but that has a user
interface problem. If the developer connects after the debug-me session has
started, prompting the user on the same terminal that is displaying the
session would not be good. This is where it'd be good to have a library for
multi-terminal applications. Perhaps I should go build a
prototype of that. Of perhaps I'll make debug-me wait for one developer to
connect and prompt the user before starting the session.

Hardest thing today was when a developer connects and the server needs to
send them the backlog of the session before they start seeing current
activity. Potentially full of races. My implementation avoids race
conditions, but might cause other connected developers to see a stall in
activity at that point. A stall-free version is certianly doable, but this
is good enough for now.

There are quite a few bugs to fix. Including a security hole in the proof
chain design, that I realized it had when thinking about what happens with
multiple people are connected to a debug-me session who are all typing at
once.

(There have actually been 3 security holes spotted over the past day; the
one above, a lacking sanitization of session IDs, and a bug in the server
that let a developer truncate logs.)

So I need to spend several mode days bugfixing, and also make it only allow
connections signed by trusted gpg keys. Still, an initial release does not
seem far off now.

Worked today on making debug-me run as a client/server, communicating using
websockets.

I decided to use the "binary" library to get an efficient serialization of
debug-me's messages to send over the websockets, rather than using JSON.
A typically JSON message was 341 bytes, and this only uses 165 bytes, which
is fairly close to the actual data size of ~129 bytes. I may later use
protocol buffers to make it less of a haskell-specific wire format.

Currently, the client and server basically work; the client can negotiate
a protocol version with the server and send messages to it, which the
server logs.

Also, added two additional modes to debug-me. debug-me --download url will
download a debug-me log file. If that session is still running, it keeps
downloading until it's gotten the whole session. debug-me --watch url
connects to a debug-me session, and displays it in non-interactive mode.
These were really easy to implement, reusing existing code.

Ed25519 signatures add 64 bytes overhead to each message, on top of the
64 bytes for the hash pointer to the previous message. But, last night I
thought of a cunning plan to remove that hash pointer from the wire
protocol, while still generating a provable hash chain. Just leave it out
of the serialized message, but include it in the data that's signed.
debug-me will then just need to try the hashes of recent messages until
it finds one for which the signature verifies, and then it will know
what the hash pointer is supposed to point to, without it ever having been
sent over the wire! Will implement this trick eventually.

Solved that bug I was stuck on yesterday. I had been looking in the code
for the developer side for a bug, but that side was fine; the bug was
excessive backlog trimming on the user side.

Now I'm fairly happy with how debug-me's activity chains look,
and the first stage of developing debug-me is complete. It still doesn't do
anything more than the script command, but all the groundwork for the
actual networked debug-me is done now. I only have to add signing,
verification of gpg key trust, and http client-server to finish debug-me.

(Also, I made debug-me --replay debug-me.log replay the log with
realistic delays, like scriptreplay or ttyplay. Only took a page
of code to add that feature.)

I'm only "fairly happy" with the activity chains because there is a
weird edge case.. At high latency, when typing "qwertyuiop", this
happens:

That looks weird, and is somewhat hard to follow in graph form, but
it's "correct" as far as debug-me's rules for activity chains go.
Due to the lag, the chain forks:

It sends "wer" before the "q" echos back

It replies to the "q" echo with tyuio" before
the "w" echos back.

It replies to the "w" echo with "p"

Finally, all the delayed echos come in, and it sends
a carriage return, resulting in the command being run.

I'd be happier if the forked chain explicitly merged back together,
but to do that and add any provable information, the developer would have
to wait for all the echos to arrive before sending the carriage return,
or something like that, which would make type-ahead worse. So I think I'll
leave it like this. Most of the time, latency is not so high, and so this
kind of forking doesn't happen much or is much simpler to understand when
it does happen.

Working on getting the debug-me proof chain to be the right shape,
and be checked at all points for valididity. This graph of a session
shows today'ss progress, but also a bug.

At the top, everything is synchronous while "ls" is entered and echoed back.
Then, things go asynchronous when " -la" is entered, and
the expected echos (in brackets) match up with what really gets
echoed, so that input is also accepted.

Finally, the bit in red where "|" is entered is a bug
on the developer side, and it gets (correctly) rejected
on the user side due to having forked the proof chain. Currently stuck on
this bug.

The code for this, especially on the developer side, is rather hairy,
I wonder if I am missing a way to simplify it.

Yesterday a few small improvements, but mostly I discovered the
posix-pty library, and
converted debug-me to use it rather than wrangling ptys itself. Which
was nice because it let me fix resizing. However, the library had a bug
with how it initializes the terminal, and investigating and working around
that bug used up too much time. Oh well, probably still worth it.

That's a pretty verbose way of saying: I typed "ls" and saw the list of
files. But it compresses well. Each packet for a single keystroke will
take only 37 bytes to transmit as part of a compressed stream of JSON,
and 32 of those bytes are needed for the SHA256 hash. So, this is probably
good enough to use as debug-me's wire format.

(Some more bytes will be needed once the signature field is not empty..)

It's also a good logging format, and can be easily analized to eg, prove
when a person used debug-me to do something bad.

Wrote a quick visualizor for debug-me logs using graphviz. This will be
super useful for debug-me development if nothing else.

Proceeding as planned, I wrote 170 lines of code to make debug-me
have separate threads for the user and developer sides, which send
one-another updates to the activity chain, and check them for validity.
This was fun to implement! And it's lacking only signing to be a full
implementation of the debug-me proof chain.

Then I added a network latency simulation to it and tried different
latencies up to the latency I measure on my satellite internet link
(800 ms or so)

That helped me find two bugs, where it was not handling echo simulation
correctly. Something is still not handled quite right, because when
I put a network latency delay before sending output from the user
side to the developer side, it causes some developer input to get rejected.
So I'm for now only inserting latency when the developer is sending input
to the user side. Good enough for proof-of-concept.

Result is that, even with a high latency, it feels "natural" to type
commands into debug-me. The echo emulation works, so it accepts
typeahead.

Using backspace to delete several letters in a row feels "wrong";
the synchronousness requirements prevent that working when latency
is high. Same problem for moving around with the arrow keys.
Down around 200 ms latency, these problems are not apparent, unless
you mash down the backspace or arrow key.

How about using an editor? It seemed reasonably non-annoying at 200 ms
latency, although here I do tend to mash down arrow keys and then it
moves too fast for debug-me to keep up, and so the cursor movement stalls.

At higher latencies, using an editor was pretty annoying. Where I might
normally press the down arrow key N distinct times to get to the line
I wanted, that doesn't work in debug-me at 800 ms latency. Of course,
over such a slow connection, using an editor is the last thing you want to
do anyway, and vi key combos like 9j start to become necessary (and work
in debug-me).

Based on these experiements, the synchronousness requirements are not as
utterly annoying as I'd feared, especially at typical latencies.

And, it seems worth making debug-me detect when several keys are pressed
close together, and send a single packet over the network combining those.
That should make it behave better when mashing down a key.

First, wrote down some data types for debug-me's proof of developer activity.

Then, some terminal wrangling, to get debug-me to allocate a
pseudo-terminal, run an interactive shell in it, and pass stdin
and stdout back and forth to the terminal it was started in.
At this point, debug-me is very similar to script, except
it doesn't log the data it intercepts to a typescript file.

Terminals are complicated, so this took a while, and it's still not perfect,
but good enough for now. Needs to have resize handling added, and for some
reason when the program exits, the terminal is left in a raw state, despite
the program apparently resetting its attributes.

Next goal is to check how annoying debug-me's insistence on a
synchronous activity proof chain will be when using debug-me across a
network link with some latency. If that's too annoying, the design
will need to be changed, or perhaps won't work.

To do that, I plan to make debug-me simulate a network between the user and
developer's processes, using threads inside a single process for now. The
user thread will builds up an activity chain, and only accepts inputs from the
developer thread when they meet the synchronicity requirements. Ran out of
time to finish that today, so next time.

The hard part of that turned out to be that inside the chroot it's
building, /usr/local/propellor is bind mounted to the one outside the
chroot. But this new property needs to populate that directory in
the chroot. Simply unmounting the bind mount would break later properties,
so some way to temporarily expose the underlying directory was called for.

At first, I thought unshare -m could be used to do this, but for some
reason that does not work in a chroot. Pity. Ended up going with a
complicated dance, where the bind mount is bind mounted to a temp dir,
then unmounted to expose the underlying directory, and once it's set up,
the temp dir is re-bind-mounted back over it. Ugh.

I was able to reuse Propellor.Bootstrap to bootstrap propellor inside the
chroot, which was nice.

Also nice that I'm able to work on this kind of thing at home despite it
involving building chroots -- yay for satellite internet!

Propellor was recently ported to FreeBSD, by Evan Cofsky. This new feature
led me down a two week long rabbit hole to make it type safe. In particular,
Propellor needed to be taught that some properties work on Debian, others on
FreeBSD, and others on both.

The user shouldn't need to worry about making a mistake like this;
the type checker should tell them they're asking for something that can't
fly.

-- Is this a Debian or a FreeBSD host? I can't remember, let's use both package managers!
host "example.com"$ props
& aptUpgraded
& pkgUpgraded

As of propellor 3.0.0 (in git now;
to be released soon), the type checker will catch such mistakes.

Also, it's really easy to combine two OS-specific properties into a
property that supports both OS's:

upgraded = aptUpgraded `pickOS` pkgUpgraded

type level lists and functions

The magick making this work is type-level lists. A property has a
metatypes list as part of its type. (So called because it's additional
types describing the type, and I couldn't find a better name.) This list
can contain one or more OS's targeted by the property:

In Haskell type-level lists and other DataKinds are indicated by the
' if you have not seen that before. There are some convenience
aliases and type operators, which let the same types be expressed
more cleanly:

Whenever two properties are combined, their metatypes are combined
using a type-level function. Combining aptUpgraded and pkgUpgraded
will yield a metatypes that targets no OS's, since they have none in
common. So will fail to type check.

My implementation of the metatypes lists is hundreds of lines of
code, consisting entirely of types and type families. It includes a basic
implementation of singletons, and is portable back to ghc 7.6 to support
Debian stable. While it takes some contortions to support such an old
version of ghc, it's pretty awesome that the ghc in Debian stable supports
this stuff.

But, it seemed unnecessary verbose to have types like Property NoInfo Debian.
Especially if I want to add even more information to Property
types later. Property NoInfo Debian NoPortsOpen would be a real mouthful to
need to write for every property.

Luckily I now have this handy type-level list. So, I can shove more
types into it, so Property (HasInfo + Debian) is used where necessary,
and Property Debian can be used everywhere else.

Since I can add more types to the type-level list, without affecting
other properties, I expect to be able to implement type-level port
conflict detection next. Should be fairly easy to do without changing
the API except for properties that use ports.

singletons

As shown here, pickOS makes a property that
decides which of two properties to use based on the host's OS.

Any number of OS's can be chained this way, to build a property that
is super-portable out of simple little non-portable properties.
This is a sweet combinator!

Singletons are types that are inhabited by a single value.
This lets the value be inferred from the type, which came in handy
in building the pickOS property combinator.

Its implementation needs to be able to look at each of the properties at
runtime, to compare the OS's they target with the actial OS of the host.
That's done by stashing a target list value inside a property. The target
list value is inferred from the type of the property, thanks to singletons,
and so does not need to be passed in to property. That saves
keyboard time and avoids mistakes.

is it worth it?

It's important to consider whether more complicated types are a net
benefit. Of course, opinions vary widely on that question in general!
But let's consider it in light of my main goals for Propellor:

Help save the user from pushing a broken configuration to their machines
at a time when they're down in the trenches dealing with some urgent
problem at 3 am.

Advance the state of the art in configuration management by
taking advantage of the state of the art in strongly typed haskell.

This change definitely meets both criteria. But there is a tradeoff;
it got a little bit harder to write new propellor properties. Not only do
new properties need to have their type set to target appropriate systems,
but the more polymorphic code is, the more likely the type checker can't
figure out all the types without some help.

The type checker will complain that "The type variable ‘metatypes1’ is
ambiguous". Problem is that it can't infer the type of p because many
different types could be combined with the bar property and all would
yield a Property UnixLike. The solution is simply to add a type signature
like p :: Property UnixLike

Since this only affects creating new properties, and not combining existing
properties (which have known types), it seems like a reasonable tradeoff.

That witness is used to type check that the inner property targets
every OS that the outer property targets. I think it might be possible
to store the witness in the monad, and have ensureProperty read it, but
it might complicate the type of the monad too much, since it would
have to be parameterized on the type of the witness.

Oh no, I mentioned monads. While type level lists and type functions and
generally bending the type checker to my will is all well and good,
I know most readers stop reading at "monad". So, I'll stop writing. ;)

thanks

Thanks to David Miani who answered my first tentative question with
a big hunk of example code that got me on the right track.

Also to many other people who answered increasingly esoteric Haskell type
system questions.

Also thanks to the Shuttleworth foundation, which funded this work
by way of a Flash Grant.

I'm using the reference letsencrypt client. While I've seen complaints that
it has a lot of dependencies and is too complicated, it seemed to only need
to pull in a few packages, and use only a few megabytes of disk space, and
it has fewer options than ls does. So seems fine. (Although it would be
nice to have some alternatives packaged in Debian.)

I ended up implementing this:

letsEncrypt :: AgreeTOS -> Domain -> WebRoot -> Property NoInfo

This property just makes the certificate available, it does not configure
the web server to use it. This avoids relying on the letsencrypt client's
apache config munging, which is probably useful for many people, but not
those of us using configuration management systems. And so avoids most of
the complicated magic that the letsencrypt client has a reputation for.

Instead, any property that wants to use the certificate can just
use leteencrypt to get it and set up the server when it makes a change to
the certificate:

That's about as simple a configuration as I can imagine for such a website!

The two parts of letsencrypt that are complicated are not the fault of
the client really. Those are renewal and rate limiting.

I'm currently rate limited for the next week because I asked letsencrypt
for several certificates for a domain, as I was learning how to use it and
integrating it into propellor. So I've not quite managed to fully test
everything. That's annoying. I also worry that rate limiting could hit at
an inopportune time once I'm relying on letsencrypt. It's especially
problimatic that it only allows 5 certs for subdomains of a given domain
per week. What if I use a lot of subdomains?

Renewal is complicated mostly because there's no good way to test it. You
set up your cron job, or whatever, and wait three months, and hopefully it
worked. Just as likely, you got something wrong, and your website breaks.
Maybe letsencrypt could offer certificates that will only last an hour,
or a day, for use when testing renewal.

Also, what if something goes wrong with renewal? Perhaps letsencrypt.org
is not available when your certificate needs to be renewed.

What I've done in propellor to handle renewal is, it runs letsencrypt every
time, with the --keep-until-expiring option. If this fails, propellor will
report a failure. As long as propellor is run periodically by a cron job,
this should result in multiple failure reports being sent (for 30 days I
think) before a cert expires without getting renewed. But, I have not been
able to test this.

In quiet moments at ICFP last August, I finished teaching
Propellor to generate disk images. With an emphasis on doing a
whole lot with very little new code and extreme amount of code reuse.

For example, let's make a disk image with nethack on it. First,
we need to define a chroot. Disk image creation reuses propellor's chroot
support, described back in propelling containers. Any propellor
properties can be assigned to the chroot, so it's easy to describe
the system we want.

The disk image partitions default to being sized to fit exactly the files
from the chroot that go into each partition, so, the disk image is as small
as possible by default. There's a little DSL to configure the partitions.
To give control over the partition size, it has some functions, like
addFreeSpace and setSize. Other functions like setFlag and
extended can further adjust the partitions. I think that worked out
rather well; the partition specification is compact and avoids unecessary
hardcoded sizes, while providing plenty of control.

By the end of ICFP, I had Propellor building complete disk images,
but no boot loader installed on them.

Fast forward to today. After stuggling with some strange grub behavior,
I found a working method to install grub onto a disk image.

Which is about half the size of
vmdebootstrap
1/4th the size of partman-base
(probably 1/100th the size of total partman), and 1/13th the size of
live-build. All of which do
similar things, in ways that seem to me to be much less flexible than
Propellor.

One thing I'm considering doing is extending this so Propellor can
use qemu-user-static to create disk images for eg, arm. Add some u-boot
setup, and this could create bootable images for arm boards. A library of
configs for various arm boards could then be included in Propellor.
This would be a lot easier than running the Debian Installer on an arm
board.

Oh! I only just now realized that if you have a propellor host configured,
like this example for my dialup gateway, leech --

This also means you can start with a manually built system,
write down the properties it has, and iteratively run Propellor against it
until you think you have a full specification of it, and then
use that to generate a new, clean disk image. Nice way to transition
from sysadmin days of yore to a clean declaratively specified system.

With the disclamer that I don't really know much about
orchestration,
I have added support for something resembling it to Propellor.

Until now, when using propellor to manage a bunch of hosts, you
updated them one at a time by running propellor --spin $somehost,
or maybe you set up a central git repository, and a cron job to run
propellor on each host, pulling changes from git.

I like both of these ways to use propellor, but they only go so far...

Perhaps you have a lot of hosts, and would like to run propellor on them
all concurrently.

Removes any cycles that might have snuck in by accident, before they
cause foot shooting.

Arranges for the ssh keys to be accepted as necessary.
Note that you you need to add ssh key properties to all relevant hosts
so it knows what keys to trust.

Arranges for the private data of a host to be provided to
the hosts that conduct it, so they can pass it along.

I've very pleased that I was able to add the Propellor.Property.Conductor
module implementing this with only a tiny change to the rest of propellor.
Almost everything needed to implement it was there in propellor's
infrastructure already.

Also kind of cool that it only needed 13 lines of imperative code, the
other several hundred lines of the
implementation being all pure code.

I've been doing a little bit of dynamically typed programming in Haskell,
to improve Propellor's Info type. The result is kind of
interesting in a scary way.

Info started out as a big record type, containing all the different sorts
of metadata that Propellor needed to keep track of. Host IP addresses, DNS
entries, ssh public keys, docker image configuration parameters... This got
quite out of hand. Info needed to have its hands in everything,
even types that should have been private to their module.

To fix that, recent versions of Propellor let a single
Info contain many different types of values. Look at it one way and
it contains DNS entries; look at it another way and it contains ssh public
keys, etc.

As an émigré from lands where you can never know what type of value is in
a $foo until you look, this was a scary prospect at first, but I found
it's possible to have the benefits of dynamic types and the safety of
static types too.

The key to doing it is Data.Dynamic. Thanks to Joachim Breitner for
suggesting I could use it here. What I arrived at is this type (slightly
simplified):

newtype Info = Info [Dynamic]deriving(Monoid)

So Info is a monoid, and it holds of a bunch of dynamic values, which could
each be of any type at all. Eep!

So far, this is utterly scary to me. To tame it, the Info constructor is not
exported, and so the only way to create an Info is to start with mempty
and use this function:

The important part of that is that only allows adding values that are in
the IsInfo type class. That prevents the foot shooting associated with
dynamic types, by only allowing use of types that make sense as Info.
Otherwise arbitrary Strings etc could be passed to addInfo by accident, and
all get concated together, and that would be a total dynamic programming
mess.

Only monoids can be stored in Info, so if you ask for a type that an Info
doesn't contain, you'll get back mempty.

Crucially, IsInfo is an open type class. Any module in Propellor
can make a new data type and make it an instance of IsInfo, and then that
new data type can be stored in the Info of a Property, and any Host that
uses the Property will have that added to its Info, available for later
introspection.

For example, this weekend I'm extending Propellor to have controllers:
Hosts that are responsible for running Propellor on some other hosts.
Useful if you want to run propellor once and have it update the
configuration of an entire network of hosts.

There can be whole chains of controllers controlling other controllers etc.
The problem is, what if host foo has the property controllerFor bar
and host bar has the property controllerFor foo? I want to avoid
a loop of foo running Propellor on bar, running Propellor on foo, ...

To detect such loops, each Host's Info should contain a list of the
Hosts it's controlling. Which is not hard to accomplish:

This is all internal to the module that needs it; the rest of
propellor doesn't need to know that the Info is using used for this.
And yet, the necessary information about Hosts is gathered as
propellor runs.

So, that's a useful technique. I do wonder if I could somehow make
addInfo combine together values in the list that have the same type;
as it is the list can get long. And, to show Info, the best I could do was
this:

instanceShow Info whereshow(Info l) ="Info "++show(map dynTypeRep l)

The resulting long list of the types of vales stored in a host's info is not
a useful as it could be. Of course, getInfo can be used to get any
particular type of value:

It's 2004 and I'm in Oldenburg DE, working on the Debian Installer. Colin and I
pair program on partman, its new partitioner, to get it into shape. We've
somewhat reluctantly decided to use it. Partman is in some ways a beautful
piece of work, a mass of semi-object-oriented, super extensible shell code that
sprang fully formed from the brow of Anton. And in many ways, it's mad, full of
sector alignment twiddling math implemented in tens of thousands of lines of
shell script scattered amoung hundreds of tiny files that are impossible to keep
straight. In the tiny Oldenburg Developers Meeting, full of obscure hardware
and crazy intensity of ideas like porting Debian to VAXen, we hack late into
the night, night after night, and crash on the floor.

It's 2015 and I'm at a Chinese bakery, then at the Berkeley pier, then in a SF
food truck lot, catching half an hour here and there in my vacation to add some
features to Propellor. Mostly writing down data types for things like
filesystem formats, partition layouts, and then some small amount of haskell
code to use them in generic ways. Putting these peices together and reusing
stuff already in Propellor (like chroot creation).

Before long I have this, which is only 2 undefined functions away from
(probably) working:

This is at least a replication of vmdebootstrap, generating a bootable disk
image from that config and 400 lines of code, with enormous customizability
of the disk image contents, using all the abilities of Propellor. But is also,
effectively, a replication of everything partman is used for (aside from UI and
RAID/LVM).

What a difference a decade and better choices of architecture make! In many
ways, this is the loosely coupled, extensible, highly configurable system
partman aspired to be. Plus elegance. And I'm writing it on a lark, because I
have some spare half hours in my vacation.

Past Debian Installer team lead Tollef stops by for lunch, I show him
the code, and we have the conversation old d-i developers always have about
partman.

I can't say that partman was a failure, because it's been used by millions to
install Debian and Ubuntu and etc for a decade. Anything that deletes that many
Windows partitions is a success. But it's been an unhappy success. Nobody has
ever had a good time writing partman recipes; the code has grown duplication
and unmaintainability.

I can't say that these extensions to Propellor will be a success; there's no
plan here to replace Debian Installer (although with a few hundred more lines
of code, propellor is d-i 2.0); indeed I'm just adding generic useful stuff
and building further stuff out of it without any particular end goal. Perhaps
that's the real difference.

Since July, I have been aware of an ugly problem with
propellor. Certain
propellor configurations could have a bug. I've tried to solve the problem at
least a half-dozen times without success; it's eaten several weekends.

Today I finally managed to fix propellor so it's impossible to write code that
has the bug, bending the Haskell type checker to my will with the power of
GADTs and type-level functions.

The problem comes about because some properties in propellor have Info
associated with them. This is used by propellor to introspect over the
properties of a host, and do things like set up DNS, or decrypt
private data used by the property.

At the same time, it's useful to let a Property internally decide to
run some other Property. In the example above, that's the ensureProperty
line, and the setupFoo Property is run only sometimes, and is
passed data that is read from the filesystem.

This makes it very hard, indeed probably impossible for Propellor to
look inside the monad, realize that setupFoo is being used, and add
its Info to the host.

Probably, setupFoo doesn't have Info associated with it -- most
properties do not. But, it's hard to tell, when writing such a Property
if it's safe to use ensureProperty. And worse, setupFoo could later
be changed to have Info.

Now, in most languages, once this problem was noticed, the solution would
probably be to make ensureProperty notice when it's called on a Property
that has Info, and print a warning message. That's Good Enough in a sense.

But it also really stinks as a solution. It means that building propellor
isn't good enough to know you have a working system; you have to let it
run on each host, and watch out for warnings. Ugh, no!

the solution

This screams for GADTs. (Well, it did once I learned how what GADTs are
and what they can do.)

With GADTs, Property NoInfo and Property HasInfo can be separate data
types. Most functions will work on either type (Property i) but
ensureProperty can be limited to only accept a Property NoInfo.

Property combinators

There are a lot of Property combinators in propellor. These combine
two or more properties in various ways. The most basic one is requires,
which only runs the first Property after the second one has successfully
been met.

So, what's it's type when used with GADT Property?

requires :: Property i1 -> Property i2 -> Property ???

It seemed I needed some kind of type class, to vary the return type.

class Combine x y r where
requires :: x -> y -> r

Now I was able to write 4 instances of Combines, for each combination
of 2 Properties with HasInfo or NoInfo.

It type checked. But, type inference was busted. A simple expression
like

To avoid that, it needed ":: Property HasInfo" appended --
I didn't want the user to need to write that.

I got stuck here for an long time, well over a month.

type level programming

Finally today I realized that I could fix this with a little type-level
programming.

class Combine x y where
requires :: x -> y -> CombinedType x y

Here CombinedType is a type-level function, that calculates the type that
should be used for a combination of types x and y. This turns out to be really
easy to do, once you get your head around type level functions.

(Bonus: I added some more intances of CombinedType for combining
things like RevertableProperties, so propellor's property
combinators got more powerful too.)

Then I just had to make a massive pass over all of Propellor,
fixing the types of each Property to be Property NoInfo or Property HasInfo.
I frequently picked the wrong one, but the type checker was able to detect
and tell me when I did.

A few of the type signatures got slightly complicated, to provide the type
checker with sufficient proof to do its thing...

.. This mostly happened in property combinators, which is an acceptable
tradeoff, when you consider that the type checker is now being used to prove
that propellor can't have this bug.

Mostly, things went just fine. The only other annoying thing was that some
things use a [Property], and since a haskell list can only contain a
single type, while Property Info and Property NoInfo are two different
types, that needed to be dealt with. Happily, I was able to extend
propellor's existing (&) and (!) operators to work in this situation,
so a list can be constructed of properties of several different types:

propertyList "foos"$ props
& foo
& foobar
! oldfoo

conclusion

The resulting 4000 lines of changes will be in the next release of
propellor. Just as soon as I test that it always generates the same Info
as before, and perhaps works when I run it. (eep)

These uses of GADTs and type families are not new; this is merely the first
time I used them. It's another Haskell leveling up for me.

Anytime you can identify a class of bugs that can impact a complicated code
base, and rework the code base to completely avoid that class of bugs, is a
time to celebrate!

You have a machine someplace, probably in The Cloud, and it has Linux
installed, but not to your liking. You want to do a clean reinstall,
maybe switching the distribution, or getting rid of the cruft. But
this requires running an installer, and it's too difficult to run d-i on
remote machines.

Wouldn't it be nice if you could point a program at that machine and have
it do a reinstall, on the fly, while the machine was running?

This is what I've now taught propellor to do! Here's a working
configuration which will make propellor convert a system running Fedora
(or probably many other Linux distros) to Debian:

It was surprisingly easy to build this. Propellor already knew how to
create a chroot, so from there it basically
just has to move files around until the chroot takes over from the old OS.

After the cleanInstallOnce property does its thing, propellor is running
inside a freshly debootstrapped Debian system. Then we just need a few more
Propertites to get from there to a bootable, usable system: Install grub
and the kernel, turn on shadow passwords, preserve a few config files
from the old OS, etc.

It's really astounding to me how much easier this was to build than it
was to build d-i. It took years to get d-i to the point of being able to
install a working system. It took me a few part days to add this capability
to propellor (It's 200 lines of code), and I've probably spent a total of
less than 30 days total developing propellor in its entirity.

So, what gives? Why is this so much easier? There are a lot of reasons:

Technology is so much better now. I can spin up cloud VMs for testing
in seconds; I use VirtualBox to restore a system from a snapshot. So
testing is much much easier. The first work on d-i was done by booting
real machines, and for a while I was booting them using floppies.

Propellor doesn't have a user interface. The best part of d-i is preseeding,
but that was mostly an accident; when I started developing d-i the first
thing I wrote was main-menu (which is invisible 99.9% of the time)
and we had to develop cdebconf, and tons of other UI. Probably 90% of
d-i work involves the UI. Jettisoning the UI entirely thus speeds up
development enormously. And propellor's configuration file blows d-i
preseeding out of the water in expressiveness and flexability.

Propellor has a much more principled design and implementation.
Separating things into Properties, which are composable and reusable
gives enormous leverage. Strong type checking and a powerful programming
language make it much easier to develop than d-i's mess of shell scripts
calling underpowered busybox commands etc. Properties often Just Work
the first time they're tested.

No separate runtime. d-i runs in its own environment, which is really
a little custom linux distribution. Developing linux distributions is
hard. Propellor drops into a live system and runs there. So I don't need
to worry about booting up the system, getting it on the network, etc etc.
This probably removes another order of magnitude of complexity from
propellor as compared with d-i.

This seems like the opposite of the Second System effect to me.
So perhaps d-i was the second system all along?

I don't know if I'm going to take this all the way to
propellor is d-i 2.0. But in theory, all that's needed now is:

Teaching propellor how to build a bootable image, containing a
live Debian system and propellor. (Yes, this would mean reimplementing
debian-live, but I estimate 100 lines of code to do it in propellor;
most of the Properties needed already exist.)
That image would then be booted up and perform the installation.

Some kind of UI that generates the propellor config file.

Adding Properties to partition the disk.

cleanInstallOnce and associated Properties will be included in
propellor's upcoming 1.1.0 release, and are available in git now.

Oh BTW, you could parameterize a few Properties by OS, and Propellor
could be used to install not just Debian or Ubuntu, but whatever Linux
distribution you want. Patches welcomed...

That makes example.com have a web server in a docker container, as you'd
expect, and when propellor is used to deploy the DNS server it'll
automatically make www.example.com point to the host (or hosts!) where
this container is docked.

I use docker a lot, but I have drank little of the Docker KoolAid.
I'm not keen on using random blobs created by random third parties
using either unreproducible methods, or the weirdly underpowered dockerfiles.
(As for vast complicated collections of containers that each run one
program and talk to one another etc ... I'll wait and see.)

That's why propellor runs inside the docker container and deploys whatever
configuration I tell it to, in a way that's both replicatable later
and lets me use the full power of Haskell.

Which turns out to be useful when moving on from docker containers to
something else...

systemd-nspawn containers

Propellor now supports containers using systemd-nspawn. It looks a lot
like the docker example.

Notice how I specified the Debian Unstable chroot that forms the basis of
this container. Propellor sets up the container by running debootstrap,
boots it up using systemd-nspawn, and then runs inside the container
to provision it.

Unlike docker containers, systemd-nspawn containers use systemd as their
init, and it all integrates rather beautifully. You can see the container
listed in systemctl status, including the services running inside it,
use journalctl to examine its logs, etc.

But no, systemd is the devil, and docker is too trendy...

chroots

Propellor now also supports deploying good old chroots. It looks a lot
like the other containers. Rather than repeat myself a third time,
and because we don't really run webservers inside chroots much,
here's a slightly different example.

In fact, the systemd-nspawn container code reuses the chroot code,
and so turns out to be really rather simple. 132 lines for the chroot
support, and 167 lines for the systemd support (which goes somewhat beyond
the nspawn containers shown above).

Which leads to the hardest part of all this...

debootstrap

Making a propellor property for debootstrap should be easy. And it was,
for Debian systems. However, I have crazy plans that involve running
propellor on non-Debian systems, to debootstrap something, and installing
debootstrap on an arbitrary linux system is ... too hard.

In the end, I needed 253 lines of code to do it, which is barely
one magnitude less code than the size of debootstrap itself.
I won't go into the ugly details, but this could be made a lot easier
if debootstrap catered more to being used outside of Debian.

closing

Docker and systemd-nspawn have different strengths and weaknesses,
and there are sure to be more container systems to come. I'm pleased
that Propellor can add support for a new container system in a few hundred
lines of code, and that it abstracts away all the unimportant differences
between these systems.

PS

Seems likely that systemd-nspawn containers can be nested to any depth.
So, here's a new kind of fork bomb!

Strongly typed purely functional container deployment can only protect us
against a certian subset of all badly thought out systems. ;)

Note that the above was written in 2014 and some syntatix details have
changed. See the documentation for Propellor.Property.Chroot,
Propellor.Property.Debootstrap, Propellor.Property.Docker,
Propellor.Property.Systemd for current examples.

Today I did something interesting with the Debian packaging for
propellor, which seems like it could
be a useful technique for other Debian packages as well.

Propellor is configured by a directory, which is maintained as a local git
repository. In propellor's case, it's ~/.propellor/. This contains a lot
of haskell files, in fact the entire source code of propellor! That's
really unusual, but I think this can be generalized to any package whose
configuration is maintained in its own git repository on the user's
system. For now on, I'll refer to this as the config repo.

The config repo is set up the first time a user runs
propellor. But, until now, I didn't provide an easy way to update
the config repo when the propellor package was updated. Nothing would
break, but the old version would be used until the user updated it
themselves somehow (probably by pulling from a git repository over the
network, bypassing apt's signature validation).

So, what I wanted was a way to update the config repo, merging in any
changes from the new version of the Debian package, while preserving the
user's local modifications. Ideally, the user could just run git merge
upstream/master, where the upstream repo was included in the
Debian package.

But, that can't work! The Debian package can't reasonably include the
full git repository of propellor with all its history. So, any git repository
included in the Debian binary package would need to be a synthetic one, that
only contains probably one commit that is not connected to anything else. Which
means that if the config repo was cloned from that repo in version 1.0, then
when version 1.1 came around, git would see no common parent when merging 1.1
into the config repo, and the merge would fail horribly.

To solve this, let's assume that the config repo's master branch has
a parent commit that can be identified, somehow, as coming from a past
version of the Debian package. It doesn't matter which version, although
the last one merged with will be best. (The easy way to do this is to set
refs/heads/upstream/master to point to it when creating the config repo.)

Once we have that parent commit, we have three things:

The current content of the config repo.

The content from some old version of the Debian package.

The new content of the Debian package.

Now git can be used to merge #3 onto #2, with -Xtheirs, so the result
is a git commit with parents of #3 and #2, and content of #3. (This can be
done using a temporary clone of the config repo to avoid touching its contents.)

Such a git commit can be merged into the config repo, without any conflicts
other than those the user might have caused with their own edits.

So, propellor will tell the user when updates are available, and they can
simply run git merge upstream/master to get them. The resulting history
looks like this:

So, generalizing this, if a package has a lot of config files, and
creates a git repository containing them when the user uses it
(or automatically when it's installed), this method can be used to provide
an easily mergable branch that tracks the files as distributed with the package.

It would perhaps not be hard to get from here to a full git-backed version of
ucf. Note that the Debian binary package doesn't have to ship a git
repisitory, it can just as easily ship the current version of the config files
somewhere in /usr, and check them into a new empty repository as part of the
generation of the upstream/master branch.

I wrote my own init. I didn't mean to, and in the end, it took 2 lines of
code. Here's how.

Propellor has the nice feature of
supporting provisioning of Docker containers. Since Docker normally runs
just one command inside the container, I made the command that docker
runs be propellor, which runs inside the container and takes care of
provisioning it according to its configuration.

Propellor-as-init also starts up a simple equalivilant of rsh on a named pipe
(for communication between the propellor inside and outside the container),
and also runs a root login shell (so the user can attach to the container and
administer it). Also, running a compiled program from the host system
inside a container, which might use a different distribution or
architecture was an interesting challenge
(solved using the method described in completely linux distribution-independent packaging).
So it wasn't entirely trivial, but as far as init goes, it's probably
one of the simpler implementations out there.

I know that there are various other solutions on the space of an init for
Docker -- personally I'd rather the host's systemd integrated with it
so I could see the status of the container's daemons in systemctl status.
If that does happen, perhaps I'll eventually be able to remove 2 lines of code
from propellor.

When it sees this host, Propellor adds its IP addresses to the example.com
DNS zone file, for both its main hostname ("blue.example.com"), and also its
relevant aliases. (The .museum alias would go into a different zone file.)

Multiple hosts can define the same alias, and then you automaticlly get
round-robin DNS.

The web server part of of the blue.example.com config can be cut and
pasted to another host in order to move its web server to the other host,
including updating the DNS. That's really all there is to is, just cut,
paste, and commit!

I'm quite happy with how that worked out. And curious if Puppet etc
have anything similar.

One tricky part of this was how to ensure that the serial number automtically
updates when changes are made. The way this is handled is Propellor starts
with a base serial number (100 in the example above), and then it adds to it
the number of commits in its git repository. The zone file is only updated
when something in it besides the serial number needs to change.

The result is nice small serial numbers that don't risk overflowing the
(so 90's) 32 bit limit, and will be consistent even if the configuration
had Propellor setting up multiple independent master DNS servers for the
same domain.

Another recent feature in Propellor is that it can use Obnam
to back up a directory. With the awesome feature that if
the backed up directory is empty/missing,
Propellor will automcatically restore it from the backup.

Here's how the backedup property used in the example above
might be implemented:

Which of course ties back into the DNS and gets this hostname set in it.
But also, the ssh public key is available for this host and visible to the
DNS zone file generator, and that could also be set in the DNS, in a SSHFP
record. I haven't gotten around to implementing that, but hope at some
point to make Propellor support DNSSEC, and then this will all combine even
more nicely.

By the way, Propellor is now up to 3 thousand lines of code
(not including Utility library). In 20 days, as a 10% time side project.

In just released Propellor 0.3.0, I've improved improved
Propellor's config file DSL significantly. Now properties can
set attributes of a host, that can be looked up by its other properties,
using a Reader monad.

The eventual plan is that the cname property won't be defined as a
property of the host, but of the container running inside it.
Then I'll be able to cut-n-paste move docker containers between hosts,
or duplicate the same container onto several hosts to deal with load,
and propellor will provision them, and update the zone file appropriately.

Also, Chris Webber had suggested that Propellor be able to separate values
from properties, so that eg, a web wizard could configure the values easily.
I think this gets it much of the way there. All that's left to do is two easy
functions:

With these, propellor's configuration could
be adjusted at run time using JSON from a file or other source. For example,
here's a containerized webserver that publishes a directory from
the external host, as configured by JSON that it exports:

Propellor ensures that a list of properties about a system
are satisfied. But requirements change, and so you might want to revert
a property that had been set up before.

For example, I had a system with a webserver container:

Docker.docked container hostname "webserver"

I don't want a web server there any more. Rather than having a separate
property to stop it, wouldn't it be nice to be able to say:

revert (Docker.docked container hostname "webserver")

I've now gotten this working. The really fun part is, some properies
support reversion, but other properties certianly do not. Maybe the code to
revert them is not worth writing, or maybe the property does something
that cannot be reverted.

For example, Docker.garbageCollected is a property that makes sure there
are no unused docker images wasting disk space. It can't be reverted.
Nor can my personal standardSystem Unstable property, which amoung other
things upgrades the system to unstable and sets up my home directory..

I found a way to make Propellor statically check if a property can be
reverted at compile time. So revert Docker.garbageCollected will fail
to type check!

The tricky part about implementing this is that the user configures
Propellor with a list of properties. But now there are two distinct
types of properties, revertable ones and non-revertable ones.
And Haskell does not support heterogeneous lists..

My solution to this is a typeclass and some syntactic sugar operators.
To build a list of properties, with individual elements that might be
revertable, and others not:

Docker containers are set up using Properties too, just like regular
hosts, but their Properties are run inside the container.

That means that, if I change the web server port above, Propellor will
notice the container config is out of date, and stop the container,
commit an image based on it, and quickly use that to bring up a new
container with the new configuration.

If I change the web server to say, lighttpd, Propellor will run inside
the container, and notice that it needs to install lighttpd to satisfy
the new property, and so will update the container without needing to take
it down.

Adding all this behavior took only 253 lines of code, and none of it
impacts the core of Propellor at all; it's all in
Propellor.Property.Docker. (Well, I did need another hundred lines
to write a daemon that runs inside the container and reads commands
to run over a named pipe... Docker makes running ad-hoc commands inside a
container a PITA.)

So, I think that this vindicates the approach of making the configuration
of Propellor be a list of Properties, which can be constructed by
abitrarily interesting Haskell code. I didn't design Propellor to support
containers, but it was easy to find a way to express them as shown above.

All puppet manages is running the image and a simple static command
inside it. All the complexities that puppet provides for configuring
servers cannot easily be brought to bear inside the container, and
a large reason for that is, I think, that its configuration file is just
not expressive enough.

Whups, I seem to have built a configuration management system this evening!

Propellor has similar goals to chef or puppet or ansible, but with an approach
much more like slaughter.
Except it's configured by writing Haskell code.

The name is because propellor ensures that a system is configured with the
desired PROPerties, and also because it kind of pulls system configuration
along after it. And you may not want to stand too close.

Disclaimer: I'm not really a sysadmin, except for on the scale of "diffuse
administration of every Debian machine on planet earth or nearby", and so I
don't really understand configuration management. (Well, I did write
debconf, which claims to be the "Debian Configuration Management system"..
But I didn't understand configuration management back then either.)

So, propellor makes some perhaps wacky choices. The least of these
is that it's built from a git repository
that any (theoretical) other users will fork and modify; a cron job can
re-make it from time to time and pull down configuration changes, or
something can be run to push changes.

A really simple configuration for a Tor bridge server
using propellor looks something like this:

Since it's just haskell code, it's "easy" to refactor out common
configurations for classes of servers, etc. Or perhaps integrate
reclass? I don't know. I'm happy
with just pure functions and type-safe refactorings of my configs, I
think.

Properties are also written in Haskell of course. This one ensures that
all the packages in a list are installed.

Here's part of a custom one that I use to check out a user's
home directory from git. Shows how to make a property require
that some other property is satisfied first, and how to test
if a property has already been satisfied.

I'm about 37% happy with the overall approach to listing properties
and combining properties into larger properties etc. I think that some
unifying insight is missing -- perhaps there should be a Property monad?
But as long as it yields a list of properties, any smarter thing should
be able to be built on top of this.

Propellor is 564 lines of code, including 25 or so built-in properties like
the examples above. It took around 4 hours to build.

I'm pretty sure it was easier to write it than it would have been to look
into ansible and salt and slaughter (and also liw's human-readable configuration
language whose name I've forgotten) in enough detail to pick one, and learn
how its configuration worked, and warp it into something close to how I
wanted this to work.

I think that's interesting.. It's partly about NIH and
I-want-everything-in-Haskell, but it's also about a complicated system
that is a lot of things to a lot of people --
of the kind I see when I look at ansible -- vs the tools and experience
to build just the thing you want without the cruft. Nice to have the latter!

Posted in the wee hours of Saturday night, March 30th, 2014
Tags:
devbloghaskellpropellor