Archive for Programming

People’s coding styles tend to evolve and change over time. One particular habit I seem to have picked up is to sprinkle the code liberally with numerous TODO markers. I wish I could say it’s clear a sign of my ever-present dissatisfaction with imperfect solutions, but I suspect I simply adopted it while working at the current company :)

In any case, TODOs (and FIXMEs &c.) are not actually something to scoff at, not too much at least. They are certainly better than the alternative, which is to commit shady code without explanation, rationale, or ideas for improvement. With TODOs, you are at least making the technical debt apparent and explicit, thus increasing the likelihood you’ll eventually come around to pay it off.

When that glorious day comes, though, it would be nice to get a quick overview of the code’s shortcomings, so that you can decide what to work on first. Getting a list of TODOs scattered over many files sounds like a great task for grep, and a relatively simple one at that. But somehow, every time I wanted to do that I ended up spending some non-negligible time just working out the details of grep‘s syntax and flags.

Thus, the logical course of action would be craft a simple script which would relieve me from doing that ever again. However, when I got around to writing it, I quickly realized the task it’s not actually that simple. In fact, it’s totally impossible to do it with just grep, since it would require matching a regex against multiple subsequent lines of input . Standard GNU grep doesn’t support that at all.

Well, at this point I should’ve probably taken the hint and realize it’s not exactly the best idea to use a shell script here. But hey, not everything has to be written in Python, right? :) So I rolled up my sleeves and after a fair amount of googling (and stack-overflowing), I unleashed a horror that I hereby present:

For best result, it is necessary to have pcregrep installed, which is an extended version of grep that supports the full spectrum of Perl-compatible regular expressions. On most popular Linux distros, pcregrep is just one apt-get install away.

When writing tests, ideally you should verify your code’s behavior not only in the usual, “happy” cases, but also in the erroneous ones. Although you may very well accept that a function blows when feed with incorrect data, it should blow up predictably and consistently. An error, exception or panic is still an output; and it should be possible to capture and examine it in tests.

Raise!

The Python unittest module has a couple of ways to deal with expected error. Probably the most useful among them is the TestCase.assertRaises method. It does pretty much exactly what it names hints to: asserting that a piece of code raises a specific type of exception:

To clarify, assertRaises will execute given block of code (or a callable, like in the first example) and throw AssertionError if an exception of given type was not raised by the code. By calling the tested function with incorrect data, we intend to provoke the exception, affirm the assertion, and ultimately have our test pass.

Be more specific

I mentioned, however, that it doesn’t just matter if your code blows up in response to invalid input or state, but also how it does so. Even though assertRaises will verify that the exception is of correct type, it is often not nearly enough for a robust test. In Python, exception types tend to be awfully broad, insofar that a simple information about throwing TypeError may tell you next to nothing about the error’s true nature.

Ironically, designers of the unittest module seemed to be vaguely aware of the problem. One of their solutions, though, was to introduce a proverbial second problem, taking the form of assertRaisesRegexp method. While it may kinda-sorta work in simple cases, I wouldn’t be very confident relying on regular expressions for anything more complex.

Especially when the other possible approach appears much more sound anyway. Using the feature of with statement, we can capture the exception object and examine it ourselves, in a normal Python code. Not just the type or message (though these are typically the only reliable things), but also whatever other data it may carry:

class Foo(TestCase):

def test_none(self):

withself.assertRaises(TypeError)as r:

__unit__.foo(None)

self.assertIn("NoneType",str(r.exception))

Sometimes, those checks might grow quite sophisticated. For example, in Python 3 you have exception chaining; it allows you to look not only at the immediate exception object, but also its __cause__, which is analogous to Throwable.getCause in Java or Exception.InnerException in C#. If you need to dig this deep, I’d suggest extracting a function with all that code – essentially a specialized version of assertRaises, preferably with all the context manager goodness that would enable us to use it like the original.

Over the course of several past months and years I was coding in Python, I’ve created quite a few Python packages: both open source and for private projects. Even though their most important part was always the code, there are numerous additional files that are necessary for the package to correctly serve its purpose. Rather than part of the Python language, they are more closely related to the Python platform.

But if you look for any definite, systemic info about them, you will at best find some scattered pieces of knowledge in various unrelated places. At worst, the only guidance would come in the form of a multitude of existing Python package sources, available on GitHub and similar sites. Parroting them is certainly an option, although I believe it’s much more advantageous to acquire firm understanding of how those different cogs fit together. Without it, following the modern Python’s best development practices – which are all hugely beneficial – is largely impossible.

So, I want to fill this void by outlining the structure of a Python package, as completely as possible. You can follow it as a step-by-step guide when creating your next project. Or just skim through it to see what you’re missing, and whether it’d be worthwhile to address such gaps. Any additional element or file will usually provide some tangible benefit, but of course not every project requires all bells and whistles.

Without further ado, let’s see what’s necessary for a complete Python software bundle.

If you code in Python, then chances are that at some point, you have written a check similar to this one:

def function(arg):

ifnotisinstance(arg,basestring):

raiseTypeError("%r is not a string" % arg)

# ... rest of the function ...

Some would of course argue against putting such an explicit if in the code, insisting to rely on duck typing instead. But while this is an easy target of critique, it’s nowhere near the biggest problem you can find in the snippet above.

This code has a subtle bug. The bug is not even limited to checks like this one; it can occur in many different situations. It surfaces rarely, too, so it’s all the more surprising when it actually rears its ugly head.

The bug is related to string formatting, which in this case points to this expression:

"%r is not a string" % arg

Most of the time, it is perfectly fine and works flawlessly. But since arg is a value we do not have any control over, sometimes it may not work correctly. Sometimes, it can just blow the whole thing up, likely in a way we have not intended.

Wild tuple appears!

All it takes is for arg to be a tuple – any tuple. Tuples are special, because the string formatting operator (%) expects you’ll use them to pass more than one argument to fill in placeholders in the string:

def print_square(x):

print"%d ^ 2 = %d" % (x, x*x)

The construct of a string followed by percent sign, followed by parenthesis, is very likely familiar to you. Notice, however, that there is nothing exceptional about using a tuple literal: what is important is the tuple type. Indeed, we could rewrite the above in the following manner:

def print_square(x):

args =(x, x*x)

print"%d ^ 2 = %d" % args

and the end result would be exactly the same. The only reason we prefer the first version is its obviously superior readability.

Tuple uses Misformat! It’s super effective!

Comparing that last piece of code with the first one, we can see quite clearly how everything will go horribly wrong should we try to format the TypeError‘s message using arg which happens to be a tuple. Not just one, but three different failure modes are possible here:

empty tuple (too few arguments for string formatting)

tuple with at least 2 elements (too many arguments)

tuple with exactly one element

Last one is particularly jarring. It raises no exceptions on by itself, and can additionally result in confusing messages, along the lines of:

'Alice has a cat'isnot a string

Much head-scratching would probably ensue if you stumbled upon exception that reports something like this.

Tuple was caught!

To avoid these problems, one solution is to engage in some sort of pythonic homeopathy. As it turns out, we can cure the malady of tuples by adding even more tuples:

raiseTypeError("%r is not a string" % (arg,))

Through this weird (arg,) singleton (1-tuple), we are explicitly sidestepping the error-prone feature of % operator, where it allows a single right-hand side argument to be passed directly. Instead, we are always wrapping all the arguments in a tuple – yes, even if it means using the bizarre (1,) syntax. This way, we can fully control how many of arguments we actually give to the formatter, regardless of what they are and where did they come from.

It’s not pretty, I know – it adds some visual clutter. But the total alternative, the format method, is even more verbose and ridden with issues. C’est la vie.

There’s no better way to start a new year than a hearty, poignant rant. To set the bar up high right off the bat, I’m not gonna be picking on some usual, easy target like JavaScript or PHP. To the contrary, I will lash out on everyone-and-their-mother’s favorite language; the one sitting comfortably in the middle between established, mature, boring technologies of the enterprise; and the cutting edge, misbegotten phantasms of the GitHub generation.

That’s, of course, Python. A language so great that nearly all its flaws people talk about are shared evenly by others of its kin: the highly dynamic, interpreted languages. One would be hard-pressed to find anything that’s not simply a rehashed argument about maintainability, runtime safety or efficiency – concerns that apply equally well to Ruby, Perl and the like. What about anything specifically “pythonic”?…

Every computer program expands until it can read e-mail – or so they say. But many applications need not to read, but to send e-mails; web apps or web services are probably the most prominent examples. If you happen to develop them, you may sometimes want a local, dummy SMTP server just for testing this functionality. It doesn’t even have to send anything (it must not, actually), but it should allow you to see what would be sent if the app worked in a production environment.

By far the easiest way to setup such a server involves, quite surprisingly, Python. There is a standard library module called smtpd, which is built exactly for this purpose. Amusingly, you don’t even have to write any code that uses it; you can invoke it straight from the command line:

$ python -m smtpd -n-c DebuggingServer

This will start a server that listens on port 8025 and dumps every message “sent” through it to the standard output. A custom port is chosen because on *nix systems, only the ports above 1024 are accessible to an ordinary user. For the standard SMTP port 25, you need to start the server as root:

$ sudo python -m stmpd -c DebuggingServer localhost:25

While it’s more typing, it frees you from having to change the SMTP port number inside your application’s code.

If you plan to use smtpd more extensively, though, you may want to look at the small runner script I’ve prepared. By default, it tries to listen on port 25, but you can supply a port number as its sole argument.

What is your first association evoked by a mention of the Java programming language? If you said “slow”, I hope all is well back there in the 90s! But if you said “verbose” instead, you would be very much in the right. Its wordiness is almost proverbial by now, with only some small improvements – like lambdas – emerging at the distant horizon.

Thankfully, more modern and higher level languages are much better in this regard. The likes of Python, Ruby or JavaScript, that is. They are more expressive and less bureaucratic, therefore requiring much less code to accomplish the same thing that takes pages upon pages in Java, C# or C++. The amount of “boilerplate” – tedious, repetitive code – is also said to be significantly lower, almost to the point of disappearing completely.

Sounds good? Sure it does. The only problem here is that most of those optimistic claims are patently false. Good real-world code written in high level, dynamic language does not necessarily end up being much shorter. Quite often, it is either on par or even longer than the equivalent source in Java et al.
It happens for several reasons, most of which are not immediately obvious looking only at a Hello World-like samples.

You need copious amount of documentation

It is said Python is pretty much pseudocode that happens to be syntactically strict enough to parse and run. While usually meant as a compliment, this observation has an uglier counterbalance. Being a pseudocode, the raw Python source is woefully incomplete when it comes to providing necessary information for future maintainers. As a result, it gets out of hand rather quickly.

To rein it, we need to put that information back somehow. Here’s where the documentation and commenting part steps in. In case of Python, there is even a syntactical distinction on the language level between both. The former takes the form of docstrings attached to function and class definitions. They are meant to fill in the gaps those definitions always leave open, including such “trivialities” as what kind of arguments the function actually takes, what it returns, and what exceptions it may raise.

Even when you don’t write expansive prose with frequent usage examples and such, an adequate amount of docstrings will still take noticeable space. The reward you’ll get for your efforts won’t be impressive, too: you’ll just add the information you’d include anyway if you were writing the code in a static language.
The catch here is that you won’t even get the automatic assurance that you’ve added the information correctly… which brings us immediately to the next point.

You need a really good test suite

There are two types of programmers: those who write tests, and those who will be writing tests. Undoubtedly, the best way to have someone join the first group is to make them write serious code in any interpreted, dynamic language.

In those languages, a comprehensive set of tests is probably the only automated and unambiguous way to ensure your code is satisfying even some basic invariants. Whole classes of errors that elsewhere would be eliminated by the compiler can slip completely undetected if the particular execution path is not exercised regularly. This includes trying to invoke a non-existent method; to pass an unexpected argument to a function; call a “function” which is not callable (or conversely, not call a function when it should have been); and many others. To my knowledge, none of these are normally detected by the various tools for static analysis (linting).

So, you write tests. You write so many tests, in fact, that they easily outmatch the very code they’re testing – if not necessarily in effort, then very likely in quantity. There’s no middle ground here: either you blanket your code with good tests coverage, or you can expect it to break in all kinds of silly ways.

You need to keep abstractions in check

Documentation, comments and tests take time and space. But at least the code in high level language can be short and sweet. Beautiful, elegant, crisp and concise, without any kind of unnecessary cruft!
Just why are you staring at me like that when I say I used a metaclass here?…

See, the problem with powerful and highly expressive languages is that they are dangerous. The old adage of power and responsibility does very much apply here. Without a proper harness, you will one day find out that:

the clever trick you’ve used half a year ago is not nearly as obvious as it was back then

the programing style which feels natural for one member of your team is nearly incomprehensible to some of the others

new people have serious trouble grasping all the neat abstractions you’ve used in your project so liberally

Unless you can guarantee you’ll always have only real Perl/Python/Ruby rockstars on board, it’s necessary to tame that wild creativity at least a little bit. The inevitable side effect is that your code will sometimes have to be longer, at least compared to what that smart but mystifying technique would yield.