11/19/10

Statistics about program operation are an invaluable monitoring and debugging tool. How many requests are being handled per second, how much of various resources are in use, how long we've been up. Unfortunately, the gathering and reporting of these critical values is usually ad-hoc. It would be nice if we had 1) a centralized place for gathering statistical performance data, 2) a system for extrapolating that data into more useful information, and 3) a method of serving that information to both human investigators and monitoring software. I've got a proposal. Let's examine each of those points in more detail.

Data Gathering

Just as Python's logging module provides a common importable for gathering and sending messages, statistics need a similar mechanism, and one that does not require each package which wishes to collect stats to import a third-party module. Therefore, we choose to re-use the logging module by adding a statistics object to it.

It is not a custom class, because that would 1) require apps to import a third-party module in order to participate, 2) inhibit innovation in extrapolation approaches and in reporting tools, and 3) be slow. There are, however, some specifications regarding the structure of the dict.

The logging.statistics dict has strictly 4 levels. The topmost level is nothing more than a set of names to introduce modularity. If SQLAlchemy wanted to participate, it might populate the item logging.statistics['SQLAlchemy'], whose value would be a second-layer dict we call a "namespace". Namespaces help multiple emitters to avoid collisions over key names, and make reports easier to read, to boot. The maintainers of SQLAlchemy should feel free to use more than one namespace if needed (such as 'SQLAlchemy ORM').

Each namespace, then, is a dict of named statistical values, such as 'Requests/sec' or 'Uptime'. You should choose names which will look good on a report: spaces and capitalization are just fine.

In addition to scalars, values in a namespace MAY be a (third-layer) dict, or a list, called a "collection". For example, the CherryPy StatsTool keeps track of what each worker thread is doing (or has most recently done) in a 'Worker Threads' collection, where each key is a thread ID; each value in the subdict MUST be a fourth dict (whew!) of statistical data about
each thread. We call each subdict in the collection a "record". Similarly, the StatsTool also keeps a list of slow queries, where each record contains data about each slow query, in order.

Values in a namespace or record may also be functions, which brings us to:

The collection of statistical data needs to be fast, as close to unnoticeable as possible to the host program. That requires us to minimize I/O, for example, but in Python it also means we need to minimize function calls. So when you are designing your namespace and record values, try to insert the most basic scalar values you already have on hand.

When it comes time to report on the gathered data, however, we usually have much more freedom in what we can calculate. Therefore, whenever reporting tools fetch the contents of logging.statistics for reporting, they first call extrapolate_statistics (passing the whole statistics dict as the only argument). This makes a deep copy of the statistics dict so that the reporting tool can both iterate over it and even change it without harming the original. But it also expands any functions in the dict by calling them. For example, you might have a 'Current Time' entry in the namespace with the value "lambda scope: time.time()". The "scope" parameter is the current namespace dict (or record, if we're currently expanding one of those instead), allowing you access to existing static entries. If you're truly evil, you can even modify more than one entry at a time.

However, don't try to calculate an entry and then use its value in further extrapolations; the order in which the functions are called is not guaranteed. This can lead to a certain amount of duplicated work (or a redesign of your schema), but that's better than complicating the spec.

After the whole thing has been extrapolated, it's time for:

Reporting

A reporting tool would grab the logging.statistics dict, extrapolate it all, and then transform it to (for example) HTML for easy viewing, or JSON for processing by Nagios etc (and because JSON will be a popular output format, you should seriously consider using Python's time module for datetimes and arithmetic, not the datetime module). Each namespace might get its own header and attribute table, plus an extra table for each collection. This is NOT part of the statistics specification; other tools can format how they like.

Turning Collection Off

It is recommended each namespace have an "Enabled" item which, if False, stops collection (but not reporting) of statistical data. Applications SHOULD provide controls to pause and resume collection by setting these entries to False or True, if present.

09/22/10

I'm tired of sessions. They lock for too long, reducing concurrency, and in my current case, don't fail gracefully when a request takes longer than the session timeout.

Problem: Session locks

Session implementations typically lock very near the beginning of a request, and unlock near the end of a request. They tend to do this even if the current request handler does no writing to the session. Why so aggressive? Because the typical test case trotted out for sessions is that of a page hit counter: session.counter += 1. What if the user opens two tabs pointing at the same page at once? The count might be off by one!

But if you don't do any counting, what's the benefit of such aggressive, synchronous locking? What we could really use is a system that used atomic commits instead of large, pessimistic locks.

Problem: Session timeouts

Sessions are often used for sites with thousands, even millions, of users. When any one of those users walks away from their computer, the servers usually try to free up resources by expiring any such inactive sessions. But lots of my admin-y sites have a few dozen users, not thousands. I'm just not that concerned with expiration of session state. I'm a little bit concerned, still, with cookies, so I still want to expire auth tokens. But there's no need to aggressively expire user data. But I find my current apps are so aggressive at expiring data that we frequently get errors in production where request A locked the session, and while it was processing a large job, request B locked the session because A was taking too long. B finishes normally, but then A chokes because it had the session lock forcibly taken away from it. Not fun.

What we could really use is a system that allows tokens to expire, or be reused concurrently, without forcing user data to expire or other, concurrent processes to choke.

Problem: Session conflation

Sessions are used for more than one kind of data. In my current apps, it's used to store:

Cookie tokens. In fact, the session id is the cookie name.

Common user information, like user id, name, and permissions, and

Workflow state, such as when a user builds up an action over multiple pages using multiple forms.

The problem is that each of these three kinds of data has a different lifecycle. The session id tends to get recreated often as sessions and cookies time out (taking all of the rest of the data with it). The user info tends to change very rarely, being nearly read-only, but is often read on every page request (for example, to display the user's name in a corner, or to apply the user's timezone to time output). Workflow data, in contrast, persists for a few seconds or minutes as the user completes a particular task, and is then discardable at the end of the process; it never needs concurrency isolation, because the user is working synchronously through a single task.

Sessions traditionally lump all of these together into a single bag of attributes, and place the entire bag under a single large lock. What we could really use is a solution that had finer-grained control over locking for each kind of data, even for each kind of info or workflow!

Solution: Slates

We can achieve all of the above by abandoning sessions. Let's face it: sessions were cool when they were invented but they're showing their age. And rather than try to patch them up and keep calling them "sessions", I'm inventing something new: "slates".

I'm implementing slates in MongoDB, but you don't have to in order to get the benefits of slates. All you need is some sort of storage that uses atomic commits, and that allows you to partition such that you have a moderate number of "collections" (one for each user, plus a special "_auth" collection), and a moderate number of "documents" (one for each use case) in each collection. Let's look at an example:

As you can see, there is a collection for the username "admin". It contains 2 documents.

User info

The first returned document is what I called "user info" above: things most pages want to know about the logged-in user. They're read for almost every request but changed hardly ever, and when they're read, it's very near the beginning of the request. Here's the Python code I use to grab the whole document:

request.user = Slate(username).user

...which is API sugar for:

request.user = pool.slates[username].find_one('user') or {}

Most pages perform this quick read and never write it back.

Workflow data

The second document returned above is workflow data for a domain-specific process I called 'new_id_set': the user uploads a large number of id's in a CSV file and gives them a name. But if there are problems with a few of the id's, we want to ask the user whether to discard the conflicts or continue anyway. But we don't want to go making records in our Postgres database tables until the numbers are confirmed, and it's prohibitive to have the client upload the same file again after confirmation. So we need a temporary place to stick this data while the user is in the middle of the activity.

Slates to the rescue! Unlike sessions, which tend to dump all their data into a single big bag, when we use slates we store our data in multiple 'bags'. That means that our user can upload their ids, be prompted for confirmation, go elsewhere to investigate the conflicts further, and come back and confirm the ids. The time they spend investigating incurs no performance penalty, because those pages don't load and re-save the 'new_id_set' slate--only the pages directly concerned with that particular slate do. Once the user has confirmed the upload, the slate is deleted.

Auth tokens

Most of the use cases for slates fit nicely into "user slates"; that is, a collection that is identified by the user's username. But when you receive an auth token in a cookie, how do you match it to a username so you can look up the slate?

The answer is to create a special, global slate which I named "_auth" in my implementation. You can name it whatever you like. This collection contains a map from tokens to usernames:

When a user visits a page, their token is searched for in the "_auth" collection, the username is retrieved, and that value is stored for the request. Typically, their "user info" slate is then retrieved. Finally, if they are visiting a page that participates in a slate-based workflow, that slate is retrieved (and saved if any changes are made).

Conclusion

Slates provide finer-grained locking than sessions in order to meet the varying needs of auth tokens, user info, and workflow data. They lock for much shorter durations, over smaller scopes, and take advantage of the native atomicity of the storage layer (MongoDB, in my case) allowing much more parallelism between requests.

04/10/09

I cringe at a lot of API's these days, because I see designers making the same mistakes again and again. Perhaps the most pervasive mistake is the dreaded NBU design: Namespacing By Underscores. For example, imagine you have a "Thing" class with a "color" attribute:

t = Thing()
t.color = 'red'

One day, you decide to switch from color names to RGB triples. Why, oh, why is this your first thought?

t.color_r = 255
t.color_g = 0
t.color_b = 0

That's your PHP (or Javascript, or SQL, or other) experience poking its ugly head in. Yes, PHP 5.3 finally has namespaces, and you can use objects as namespaces in JS if you're diligent. But chances are, you won't.

In Python, namespaces are easy. Use them. Ask yourself what the clearest syntax is, and you might come up with something like this:

>>> t.color = RGB(255, 0, 0)
>>> t.color.red
255

This is not just a matter of clever delegation (replacing a str attribute with an RGB object)--it covers all manner of interface design decisions. Here's a recent example from python-dev regarding the email package's interface for Python 3:

message.headers['Subject']
message.bytes_headers['Subject']

Please don't do that--it makes it seem as if the "message" object has a set of headers and a distinct set of bytes_headers. At the least, you've elevated the rare case to be a peer of the common case. A new user of the email module shouldn't see anything about bytes in help(message) or dir(message). Instead, write this:

If message.headers[x].encoding is given a sane default, and you expect the vast majority of users to only deal in unicode, they may never see the .encoding and .encode attributes. Good! We've made the common case easy and the rare cases possible.

In addition, we've embellished the Header object with a bytes representation using standard Python conventions: just like Python 3's str object has an encode method, so does our Header object. It's far easier to remember that such a convention applies, than to remember a brand-new name like "bytes_headers" or "decoded_headers".

Namespaces are one honking great idea -- let's do more of those! But please not faked via underscores.

04/07/09

For PyCon 2009, I'm giving two talks! One on extending CherryPy and one on the innards of Dejavu/GeniuSQL. I think I've finally reduced my talks to the required time slots (I could easily have made 4-hour talks for each and posted my presentations:

Use the arrow keys or mouse-click to proceed through them. The images don't load as fast over the network as they will when I present, so be patient if you preview them yourself. Also, try to use 1024 x 768 fullscreen--they're laid out specifically for that resolution.

Update: video is now available thanks to the great people who put on PyCon: