Special Method Names

Diving In

We’ve already covered a few special method names elsewhere in this book — “magic” methods that Python invokes when you use certain syntax. Using special methods, your classes can act like sequences, like dictionaries, like functions, like iterators, or even like numbers! This appendix serves both as a reference for the special methods we’ve seen already and a brief introduction to some of the more esoteric ones.

Basics

If you’ve read the introduction to classes, you’ve already seen the most common special method: the __init__() method. The majority of classes I write end up needing some initialization. There are also a few other basic special methods that are especially useful for debugging your custom classes.

Notes

You Want…

So You Write…

And Python Calls…

①

to initialize an instance

x = MyClass()

x.__init__()

②

the “official” representation as a string

repr(x)

x.__repr__()

③

the “informal” value as a string

str(x)

x.__str__()

④

the “informal” value as a byte array

bytes(x)

x.__bytes__()

⑤

the value as a formatted string

format(x, format_spec)

x.__format__(format_spec)

The __init__() method is called after the instance is created. If you want to control the actual creation process, use the __new__() method.

By convention, the __repr__() method should return a string that is a valid Python expression.

Python 3 will call seq.__iter__() to create an iterator, then call the __next__() method on that iterator to get each value of x. When the __next__() method raises a StopIteration exception, the for loop ends gracefully.

If your class defines a __getattribute__() method, Python will call it on every reference to any attribute or method name (except special method names, since that would cause an unpleasant infinite loop).

If your class defines a __getattr__() method, Python will call it only after looking for the attribute in all the normal places. If an instance x defines an attribute color, x.color will not call x.__getattr__('color'); it will simply return the already-defined value of x.color.

The __setattr__() method is called whenever you assign a value to an attribute.

The __delattr__() method is called whenever you delete an attribute.

The __dir__() method is useful if you define a __getattr__() or __getattribute__() method. Normally, calling dir(x) would only list the regular attributes and methods. If your __getattr()__ method handles a color attribute dynamically, dir(x) would not list color as one of the available attributes. Overriding the __dir__() method allows you to list color as an available attribute, which is helpful for other people who wish to use your class without digging into the internals of it.

The distinction between the __getattr__() and __getattribute__() methods is subtle but important. I can explain it with two examples:

The attribute name is passed into the __getattr()__ method as a string. If the name is 'color', the method returns a value. (In this case, it’s just a hard-coded string, but you would normally do some sort of computation and return the result.)

If the attribute name is unknown, the __getattr()__ method needs to raise an AttributeError exception, otherwise your code will silently fail when accessing undefined attributes. (Technically, if the method doesn’t raise an exception or explicitly return a value, it returns None, the Python null value. This means that all attributes not explicitly defined will be None, which is almost certainly not what you want.)

The dyn instance does not have an attribute named color, so the __getattr__() method is called to provide a computed value.

After explicitly setting dyn.color, the __getattr__() method will no longer be called to provide a value for dyn.color, because dyn.color is already defined on the instance.

On the other hand, the __getattribute__() method is absolute and unconditional.

The __getattribute__() method is called to provide a value for dyn.color.

Even after explicitly setting dyn.color, the __getattribute__() method is still called to provide a value for dyn.color. If present, the __getattribute__() method is called unconditionally for every attribute and method lookup, even for attributes that you explicitly set after creating an instance.

☞If your class defines a __getattribute__() method, you probably also want to define a __setattr__() method and coordinate between them to keep track of attribute values. Otherwise, any attributes you set after creating an instance will disappear into a black hole.

You need to be extra careful with the __getattribute__() method, because it is also called when Python looks up a method name on your class.

This class defines a __getattribute__() method which always raises an AttributeError exception. No attribute or method lookups will succeed.

When you call hero.swim(), Python looks for a swim() method in the Rastan class. This lookup goes through the __getattribute__() method, because all attribute and method lookups go through the __getattribute__() method. In this case, the __getattribute__() method raises an AttributeError exception, so the method lookup fails, so the method call fails.

Classes That Act Like Functions

You can make an instance of a class callable — exactly like a function is callable — by defining the __call__() method.

The zipfile module uses this to define a class that can decrypt an encryptedzip file with a given password. The zip decryption algorithm requires you to store state during decryption. Defining the decryptor as a class allows you to maintain this state within a single instance of the decryptor class. The state is initialized in the __init__() method and updated as the file is decrypted. But since the class is also “callable” like a function, you can pass the instance as the first argument of the map() function, like so:

The _ZipDecryptor class maintains state in the form of three rotating keys, which are later updated in the _UpdateKeys() method (not shown here).

The class defines a __call__() method, which makes class instances callable like functions. In this case, the __call__() method decrypts a single byte of the zip file, then updates the rotating keys based on the byte that was decrypted.

zd is an instance of the _ZipDecryptor class. The pwd variable is passed to the __init__() method, where it is stored and used to update the rotating keys for the first time.

Given the first 12 bytes of a zip file, decrypt them by mapping the bytes to zd, in effect “calling” zd 12 times, which invokes the __call__() method 12 times, which updates its internal state and returns a resulting byte 12 times.

Classes That Act Like Sequences

If your class acts as a container for a set of values — that is, if it makes sense to ask whether your class “contains” a value — then it should probably define the following special methods that make it act like a sequence.

Once you create an instance of the cgi.FieldStorage class, you can use the “in” operator to check whether a particular parameter was included in the query string.

The __contains__() method is the magic that makes this work.

When you say if 'q' in fs, Python looks for the __contains__() method on the fs object, which is defined in cgi.py. The value 'q' is passed into the __contains__() method as the key argument.

The same FieldStorage class also supports returning its length, so you can say len(fs) and it will call the __len__() method on the FieldStorage class to return the number of query parameters that it identified.

The self.keys() method checks whether self.list is None, so the __len__ method doesn’t need to duplicate this error checking.

Classes That Act Like Dictionaries

Extending the previous section a bit, you can define classes that not only respond to the “in” operator and the len() function, but they act like full-blown dictionaries, returning values based on keys.

# A script which responds to http://example.com/search?q=cgi
import cgi
fs = cgi.FieldStorage()
if 'q' in fs:
do_search(fs['q']) ①
# An excerpt from cgi.py that shows how it works
class FieldStorage:
.
.
.
def __getitem__(self, key): ②
if self.list is None:
raise TypeError('not indexable')
found = []
for item in self.list:
if item.name == key: found.append(item)
if not found:
raise KeyError(key)
if len(found) == 1:
return found[0]
else:
return found

The fs object is an instance of cgi.FieldStorage, but you can still evaluate expressions like fs['q'].

fs['q'] invokes the __getitem__() method with the key parameter set to 'q'. It then looks up in its internally maintained list of query parameters (self.list) for an item whose .name matches the given key.

Classes That Act Like Numbers

Using the appropriate special methods, you can define your own classes that act like numbers. That is, you can add them, subtract them, and perform other mathematical operations on them. This is how fractions are implemented — the Fraction class implements these special methods, then you can do things like this:

That’s all well and good if x is an instance of a class that implements those methods. But what if it doesn’t implement one of them? Or worse, what if it implements it, but it can’t handle certain kinds of arguments? For example:

This is not a case of taking a Fraction and dividing it by an integer (as in the previous example). That case was straightforward: x / 3 calls x.__truediv__(3), and the __truediv__() method of the Fraction class handles all the math. But integers don’t “know” how to do arithmetic operations with fractions. So why does this example work?

There is a second set of arithmetic special methods with reflected operands. Given an arithmetic operation that takes two operands (e.g.x / y), there are two ways to go about it:

Tell x to divide itself by y, or

Tell y to divide itself into x

The set of special methods above take the first approach: given x / y, they provide a way for x to say “I know how to divide myself by y.” The following set of special methods tackle the second approach: they provide a way for y to say “I know how to be the denominator and divide myself into x.”

Note: for the most part, the in-place operation methods are not required. If you don’t define an in-place method for a particular operation, Python will try the methods. For example, to execute the expression x /= y, Python will:

Try calling x.__itruediv__(y). If this method is defined and returns a value other than NotImplemented, we’re done.

Try calling x.__truediv__(y). If this method is defined and returns a value other than NotImplemented, the old value of x is discarded and replaced with the return value, just as if you had done x = x / y instead.

Try calling y.__rtruediv__(x). If this method is defined and returns a value other than NotImplemented, the old value of x is discarded and replaced with the return value.

So you only need to define in-place methods like the __itruediv__() method if you want to do some special optimization for in-place operands. Otherwise Python will essentially reformulate the in-place operand to use a regular operand + a variable assignment.

There are also a few “unary” mathematical operations you can perform on number-like objects by themselves.

Classes That Can Be Compared

I broke this section out from the previous one because comparisons are not strictly the purview of numbers. Many datatypes can be compared — strings, lists, even dictionaries. If you’re creating your own class and it makes sense to compare your objects to other objects, you can use the following special methods to implement comparisons.

☞If you define a __lt__() method but no __gt__() method, Python will use the __lt__() method with operands swapped. However, Python will not combine methods. For example, if you define a __lt__() method and a __eq()__ method and try to test whether x <= y, Python will not call __lt__() and __eq()__ in sequence. It will only call the __le__() method.

Classes That Can Be Serialized

Python supports serializing and unserializing arbitrary objects. (Most Python references call this process “pickling” and “unpickling.”) This can be useful for saving state to a file and restoring it later. All of the native datatypes support pickling already. If you create a custom class that you want to be able to pickle, read up on the pickle protocol to see when and how the following special methods are called.

* To recreate a serialized object, Python needs to create a new object that looks like the serialized object, then set the values of all the attributes on the new object. The __getnewargs__() method controls how the object is created, then the __setstate__() method controls how the attribute values are restored.

Classes That Can Be Used in a with Block

Python 3 supports the with statement, which allows you to access an object’s properties and methods without explicitly referencing the object every time. A with block defines a runtime context; you “enter” the context when you execute the with statement, and you “exit” the context after you execute the last statement in the block.

Any class can be used in a with block; no special methods are required. The Python interpreter will automatically set up the runtime context and dispatch all the property and method lookups to your class. However, if you want your class to do something special upon entering or exiting a runtime context, you can define the following special methods.

The file object defines both an __enter__() and an __exit__() method. The __enter__() method checks that the file is open; if it’s not, the _checkClosed() method raises an exception.

The __enter__() method should almost always return self — this is the object that the with block will use to dispatch properties and methods.

After the with block, the file object automatically closes. How? In the __exit__() method, it calls self.close().

☞The __exit__() method will always be called, even if an exception is raised inside the with block. In fact, if an exception is raised, the exception information will be passed to the __exit__() method. See With Statement Context Managers for more details.