Technically there was a 1.0 release followed by a 1.0.1 release because
the 1.0 release had issues.

The story of Bleach and html5lib

I work on Bleach which is a Python library
for sanitizing and linkifying text from untrusted sources for safe usage in
HTML. It relies heavily on another library called html5lib-python. Most of the work that I do on
Bleach consists of figuring out how to make html5lib do what I need it to do.

Over the last few years, maintainers of the html5lib library have been working
towards a 1.0. Those well-meaning efforts got them into a versioning model which
had some unenthusing properties. I would often talk to people about how I was
having difficulties with Bleach and html5lib 0.99999999 (8 9s) and I'd have to
mentally count how many 9s I had said. It was goofy [2].

In an attempt to deal with the effects of the versioning, there's a parallel set
of versions that start with 1.0b. Because there are two sets of versions, it was
a total pain in the ass to correctly specify which versions of html5lib that
Bleach worked with.

While working on Bleach 2.0, I bumped into a few bugs and upstreamed a patch for
at least one of them. That patch sat in the PR queue for months. That's what got
me wondering--is this project dead?

I tracked down Geoffrey and talked with him a bit on IRC. He seems to be the
only active maintainer. He was really busy with other things, html5lib doesn't
pay at all, there's a ton of stuff to do, he's burned out, and recently there
have been spats of negative comments in the issues and PRs. Generally the
project had a lot of stop energy.

Some time in August, I offered to step up as an interim maintainer and shepherd
html5lib to 1.0. The goals being:

Thoughts on being an interim maintainer

I see a lot of open source projects that are in trouble in the sense that they
don't have a critical mass of people and energy. When the sole part-time
volunteer maintainer burns out, the project languishes. Then the entitled users
show up, complain, demand changes, and talk about how horrible the situation is
and everyone should be ashamed. It's tough--people are frustrated and then do a
bunch of things that make everything so much worse. How do projects escape the
raging inferno death spiral?

For a while now, I've been thinking about a model for open source projects where
someone else pops in as an interim maintainer for a short period of time with
specific goals and then steps down. Maybe this alleviates users' frustrations?
Maybe this gives the part-time volunteer burned-out maintainer a breather? Maybe
this can get the project moving again? Maybe the temporary interim maintainer
can make some of the hard decisions that a regular long-term maintainer just
can't?

I wondered if I should try that model out here. In the process of convincing
myself that stepping up as an interim maintainer was a good idea [3], I looked
at projects that rely on html5lib [4]:

most web browsers (Firefox, Chrome, servo, etc) have it in their repositories
because web-platform-tests uses
it

I talked with Geoffrey and offered to step up with these goals in mind.

I started with cleaning up the milestones in GitHub. I bumped everything from
the 0.9999999999 (10 9s) milestone which I determined will never happen into a
1.0 milestone. I used this as a bucket for collecting all the issues and PRs
that piqued my interest.

I went through the issue tracker and triaged all the issues. I tried to get
steps to reproduce and any other data that would help resolve the issue. I
closed some issues I didn't think would ever get resolved.

I triaged all the pull requests. Some of them had been open for a long time. I
apologized to people who had spent their time to upstream a fix that sat around
for years. In some cases, the changes had bitrotted severely and had to be
redone [5].

This is what happens on projects that don't have a critical mass of
energy/people. It sucks for everyone involved.

Conclusion and thoughts

I finished up as interim maintainer for html5lib. I don't think I'm going to
continue actively as a maintainer. Yes, Bleach uses it, but I've got other
things I should be doing.

I think this was an interesting experiment. I also think it was a successful
experiment in regards to achieving my stated goals, but I don't know if it gave
the project much momentum to continue forward.

I'd love to see other examples of interim maintainers stepping up, achieving
specific goals, and then stepping down again. Does it bring in new people to the
community? Does it affect the raging inferno death spiral at all? What kinds of
projects would benefit from this the most? What kinds of projects wouldn't
benefit at all?

providing a decoupled architecture making it easier to write code to generate
metrics without having to worry about making sure creating and configuring a
metrics client has been done--similar to the Python logging Python logging
module in this way

I use it at Mozilla in the collector of our crash ingestion pipeline. Peter used
it to build our symbols lookup server, too.

v1.0 released!

This is the v1.0 release. I pushed out v0.2 back in April 2017. We've been using
it in Antenna (the collector of the Firefox crash ingestion pipeline) since
then. At this point, I think the API is sound and it's being used in production,
ergo it's production-ready.

This release also adds Python 2.7 support.

Why you should take a look at Markus

Markus does three things that make generating metrics a lot easier.

First, it separates creating and configuring the metrics backends from
generating metrics.

Let's create a metrics client that sends data nowhere:

importmarkusmarkus.configure()

That's not wildly helpful, but it works and it's 2 lines.

Say we're doing development on a laptop on a speeding train and want to spit out
metrics to the Python logging module so we can see what's being generated. We
can do this:

importmarkusmarkus.configure(backends=[{# Log metrics to the logs'class':'markus.backends.logging.LoggingMetrics',},{# Log metrics to statsd'class':'markus.backends.statsd.StatsdMetrics','options':{'statsd_host':'statsd.example.com','statsd_port':8125,'statsd_prefix':'',}}])

That's it. Tada!

Markus can support any number of backends. You can send data to multiple statsd
servers. You can use the LoggingRollupBackend which will generate statistics
every flush_interval of count, current, min, and max for incr stats and count,
min, average, median, 95%, and max for timing/histogram stats for metrics data.

If Markus doesn't have the backends you need, writing your own metrics backend
is straight-forward.

That's it. No bootstrapping problems, nice handling of metrics key prefixes,
decorators, context managers, and so on. You can use multiple metrics
interfaces in the same file. You can pass them around. You can reconfigure
the metrics client and backends dynamically while your program is running.

importmarkusfrommarkus.testingimportMetricsMockdeftest_something():withMetricsMock()asmm:# ... Do things that might publish metrics# This helps you debug and write your testmm.print_records()# Make assertions on metrics publishedassertmm.has_metric(markus.INCR,'some.key',{'value':1})

I use it with pytest on my projects, but it is testing-system agnostic.

Why not use statsd directly?

You can definitely use statsd/dogstatsd libraries directly, but using Markus is
a lot easier.

With Markus you don't have to worry about the order in which you
create/configure the statsd client versus using the statsd client. You don't
have to pass around the statsd client. It's a lot easier to use in Dango and
Flask where bootstrapping the app and passing things around is tricky sometimes.

With Markus you get to degrade to sending metrics data to the Python logging
library which helps surface issues in development. I've had a few occasions when
I thought I wrote code to send data, but it turns out I hadn't or that I had
messed up the keys or tags.

With Markus you get a testing mock which lets you write tests guaranteeing that
your code is generating metrics the way you're expecting.

If you go with using the statsd/dogstatsd libraries directly, that's fine, but
you'll probably want to write some/most of these things yourself.

Summary

Over the last year, I rewrote the Socorro collector which is the edge of the
Mozilla crash ingestion pipeline. Antenna (the new collector) receives crashes from
Breakpad clients as HTTP POSTs, converts the POST payload into JSON, and then
saves a bunch of stuff to AWS S3. One of the problems with this is that it's a
pain in the ass to do development on my laptop without being connected to the
Internet and using AWS S3.

This post covers the various things I did to have a locally running fake AWS S3
service.

v0.9 released!

This was the last big thing I wanted to do before doing a 1.0 release. I
consider Everett 0.9 to be a solid beta. Next release will be a 1.0.

Why you should take a look at Everett

At Mozilla, I'm using Everett 0.9 for Antenna which is running in our -stage
environment and will go to -prod very soon. Antenna is the edge of the crash
ingestion pipeline for Mozilla Firefox.

When writing Antenna, I started out with python-decouple, but I didn't like the
way python-decouple dealt with configuration errors (it's pretty hands-off) and
I really wanted to automatically generate documentation from my configuration
code. Why write the same stuff twice especially where it's a critical part of
setting Antenna up and the part everyone will trip over first?

python-decouple is a great project and does a good job at what it was
built to do. I don't mean to demean it in any way. I have additional
requirements that python-decouple doesn't do well and that's where I'm
coming from.

What is it?

Bleach is a Python library for sanitizing
and linkifying text from untrusted sources for safe usage in HTML.

Bleach v2.0 released!

Bleach 2.0 is a massive rewrite. Bleach relies on the html5lib library. html5lib 0.99999999 (8
9s) changed the APIs that Bleach was using to sanitize text. As such, in order
to support html5lib >= 0.99999999 (8 9s), I needed to rewrite Bleach.

Before embarking on the rewrite, I improved the tests and added a set of tests
based on XSS example strings from the OWASP site. Spending quality time with
tests before a rewrite or refactor is both illuminating (you get a better
understanding of what the requirements are) and also immensely helpful (you know
when your rewrite/refactor differs from the original). That was time well spent.

Given that I was doing a rewrite anyways, I decided to take this opportunity to
break the Bleach API to make it more flexible and easier to use:

added Cleaner and Linkifier classes that you can create once and reuse to
reduce redundant work--suggested in #125

created BleachSanitizerFilter which is now an html5lib filter that can be used
anywhere you can use an html5lib filter

created LinkifyFilter as an html5lib filter that can be used anywhere you use
an html5lib filter including as part of cleaning allowing you to clean and
linkify in one pass--suggested in #46

changed arguments for attribute callables and linkify callbacks

and so on

During and after the rewrite, I improved the documentation converting all the
examples to doctest format so they're testable and verifiable and adding
examples where there weren't any. This uncovered bugs in the documentation and
pointed out some annoyances with the new API.

As I rewrote and refactored code, I focused on making the code simpler and
easier to maintain going forward and also documented the intentions so I and
others can know what the code should be doing.

I also adjusted the internals to make it easier for users to extend, subclass,
swap out and whatever else to adjust the functionality to meet their needs
without making Bleach harder to maintain for me or less safe because of
additional complexity.

For API-adjustment inspiration, I went through the Bleach issue tracker and
tried to address every possible issue with this update: infinite loops,
unintended behavior, inflexible APIs, suggested refactorings, features, bugs,
etc.

The rewrite took a while. I tried to be meticulous because this is a security
library and it's a complicated problem domain and I was working on my own during
slow times on work projects. When working on one's own, you don't have benefit
of review. Making sure to have good test coverage and waiting a day to
self-review after posting a PR caught a lot of issues. I also go through the PR
and add comments explaining why I did things to give context to future me. Those
habits help a lot, but probably aren't as good as a code review by someone else.

Some off-the-cuff performance benchmarks

I ran some timings between Bleach 1.5 and various uses of Bleach 2.0 on the
Standup corpus.

Here's the results:

what?

time to clean and linkify

Bleach 1.5

1m33s

Bleach 2.0 (no code changes)

41s

Bleach 2.0 (using Cleaner and Linker)

10s

Bleach 2.0 (clean and linkify--one pass)

7s

How'd I compute the timings?

I'm using the Standup corpus which has 42000 status messages in it. Each
status message is like a tweet--it's short, has some links, possibly has HTML
in it, etc.

I wrote a timing harness that goes through all those status messages and
times how long it takes to clean and linkify the status message content,
accumulates those timings and then returns the total time spent cleaning and
linking.

I ran that 10 times and took the median. The timing numbers were remarkably
stable and there was only a few seconds difference between the high and low
for all of the sets.

I wrote the median number down in that table above.

Then I'd adjust the code as specified in the table and run the timings again.

I have several observations/thoughts:

First, holy moly--1m33s to 7s is a HUGE performance improvement.

Second, just switching from Bleach 1.5 to 2.0 and making no code changes (in
other words, keeping your calls as bleach.clean and bleach.linkify
rather than using Cleaner and Linker and LinkifyFilter), gets you a
lot. Depending on whether your have attribute filter callables and linkify
callbacks, you may be able to just upgrade the libs and WIN!

Third, switching to reusing Cleaner and Linker also gets you a lot.

Fourth, your mileage may vary depending on the nature of your corpus. For
example, Standup status messages are short so if your text fragments are larger,
you may see more savings by clean-and-linkify in one pass because HTML parsing
takes more time.

How to upgrade

Upgrading should be straight-forward.

Here's the minimal upgrade path:

Update Bleach to 2.0 and html5lib to >= 0.99999999 (8 9s).

If you're using attribute callables, you'll need to update them.

If you're using linkify callbacks, you'll need to update them.

Read through version 2.0 changes
for any other backwards-incompatible changes that might affect you.

Run your tests and see how it goes.

Note

If you're using html5lib 1.0b8, then you have to explicitly upgrade the
version. 1.0b8 is equivalent to html5lib 0.9999999 (7 9s) and that's not
supported by Bleach 2.0.

You have to explicitly upgrade because pip will think that 1.0b8 comes
after 0.99999999 (8 9s) and it doesn't. So it won't upgrade html5lib for
you.

If you're doing 9s, make sure to upgrade to 0.99999999 (8 9s) or higher.

If you're doing 1.0bs, make sure to upgrade to 1.0b9 or higher.

If you want better performance:

Switch to reusing bleach.sanitizer.Cleaner and
bleach.linkifier.Linker.

If you have large text fragments:

Switch to reusing bleach.sanitizer.Cleaner and set filters to include
LinkifyFilter which lets you clean and linkify in one step.

Many thanks

Many thanks to James Socol (previous maintainer) for walking me through why
things were the way they were.

Many thanks to Geoffrey Sneddon (html5lib maintainer) for answering questions,
helping with problems I encountered and all his efforts on html5lib which is a
huge library that he works on in his spare time for which he doesn't get
anywhere near enough gratitude.

Summary

I work on a lot of different things. Some are applications, are are libraries,
some I started, some other people started, etc. I have way more stuff to do than
I could possibly get done, so I try to spend my time on things "that matter".

For Open Source software that doesn't have an established community, this is
difficult.

This post is a wandering stream of consciousness covering my journey figuring
out who uses Bleach.

v0.8 released!

As I sat down to write this, I discovered I'd never written about Everett
before. I wrote it initially as part of another project and then extracted it
and did a first release in August 2016.

Since then, I've been tinkering with how it works in relation to how it's used
and talking with peers to understand their thoughts on configuration.

At this stage, I like Everett and it's at a point where it's worth telling
others about and probably due for a 1.0 release.

This is v0.8. In this release, I spent some time polishing the autoconfig
Sphinx directive to make it more flexible to use in your project documentation.
Instead of having configuration bits all over your project, you centralize it in
one place and then in your Sphinx docs, you have something like:

Over the weekend, I wanted to implement something that acted as both a class and
function decorator, but could also be used as a context manager. I needed this
flexibility for overriding configuration values making it easier to write tests.
I wanted to use it in the following ways:

as a function decorator:

@config_override(DEBUG='False')deftest_something():...

as a class decorator that would decorate all methods that start with
test_:

What is it?

Dennis is a Python command line
utility (and library) for working with localization. It includes:

a linter for finding problems in strings in .po files like invalid
Python variable syntax which leads to exceptions

a template linter for finding problems in strings in .pot files that
make translator's lives difficult

a statuser for seeing the high-level translation/error status of
your .po files

a translator for strings in your .po files to make development
easier

v0.7 released!

It's been 10 months since the last release. In that time, I:

Added a lot more tests and fixed bugs discovered with those tests.

Added lint rule for bad format characters like %a (#68)

Missing python-format variables is now an error (#57)

Fix notype test to handle more cases (#63)

Implement rule exclusion (#60)

Rewrite --rule spec verification to work correctly (#61)

Add --showfuzzy to status command (#64)

Add untranslated word counts to status command (#55)

Change Var to Format and use gettext names (#48)

Handle the standalone } case (#56)

I thought I was close to 1.0, but now I'm less sure. I want to unify
the .po and .pot linters and generalize them so that we can
handle other l10n file formats. I also want to implement a proper
plugin system so that it's easier to add new rules and it'd allow
other people to create separate Python packages that implement rules,
tokenizers and translaters. Plus I want to continue fleshing out the
tests.

At the (glacial) pace I'm going at, that'll take a year or so.

If you're interested in dennis development, helping out or have things
you wish it did, please let me know. Otherwise I'll just keep on
keepin on at the current pace.