Django is easily the most popular Python web framework these days.
For all of its features, and ease of use, though, sometimes it just seems misleading on purpose.
This morning I fixed a mysterious problem, and once again I was reminded of how Django can seem simple
until things go wrong, and then it's weirdly complex.

In particular, how the settings work is just odd. There are two ways that Django does
two things when it would be better to do only one.

For Ibis Reader, our settings machinery is elaborate:
the settings file imports from product_settings.py, then from a host-specific settings file, then
from a local_settings.py which isn't committed to source control:

# Settings.py

#.. lots of settings ..

fromproduct_settingsimport*

# Settings particular to this host.# For a host named xyz01.myapp.com, # create a file host_settings/xyz01_myapp_com.pyimportplatformhost_name=platform.node().replace('.','_').replace('-','_')try:exec"from ibis.host_settings.%s import *"%host_nameexceptImportError:pass

# Last resort (good for dev machines): # import settings that aren't in the repo.try:fromlocal_settingsimport*exceptImportError:pass

This scheme works great: you can put settings in the file that corresponds logically
to why the setting needs the value.

But something odd was happening: if a setting was
in both product_settings.py and the host settings file, then the value in
product_settings won. How could this be? The host settings file is applied after
product_settings!

Part of the answer is the first thing that Django does twice that should only happen
once: the settings file is imported twice. This flies in the face of everything
we know about Python modules, but it happens. So the actual order of imports for
my settings files is:

from product_settings import *

from ibis.host_settings.my_host import *

from local_settings import *

from product_settings import *

from ibis.host_settings.my_host import *

from local_settings import *

I don't know why Django imports twice, but it's long been true, and I've had to
rediscover it the hard way a few times.

But this still doesn't explain the mystery: every time product_settings is applied,
host settings should then be applied over it, so why would a setting in product_settings
take effect over one in host settings? The answer is in the second thing that
Django does twice: adding directories to the Python path.

I don't know if this is really Django's fault, or something about the way people seem
to always configure their Django projects, but it seems to very often be true:
your source files are available through two different import paths, because your
source tree has been added to the Python path twice at two different levels.

A Django project has a top level corresponding to the project ("ibis" in this case),
and then apps beneath that. The Python path is constructed so that you can import
a file as "my_project.my_app", or just as "my_app". Except that for some reason,
this double-view of the source tree isn't always available, and it isn't during
that second series of settings imports!? The path is being modified between
the two import sequences!

So the import march actually looks like this:

from product_settings import *

from ibis.host_settings.my_host import *

from local_settings import *

from product_settings import *

from ibis.host_settings.my_host import *: Import failed!

from local_settings import *

The net result is that settings in both product_settings and host settings will keep the
value from product_settings, even though host settings is imported second.

The fix is really easy: remove "ibis." from the host settings import line,
taking advantage of the fact that either form will work, and in fact, the second form is more
robust since it seems to always be available on the Python path. The settings files still
get imported twice, but at least the same thing happens both times.

I still don't understand why all these things happen.
I hope part of this is my fault, because then I can fix it for real.

Yeah, this is a wart. It's very very slightly your fault (if you pay really close attention to PYTHONPATH this doesn't happen) but it's mostly Django's fault. manage.py puts both . and .. on PYTHONPATH, which is just an accident waiting to happen. Further, the from local_settings import * trick is a bit of an antipattern, but the docs don't do anything to encourage you to do it differently.

* I'd like to kill the double-import thing, but it's a side effect of how Django discovers custom management commands and it's a bit tricky to get rid of.

* I'm starting to try to spread a new pattern of handling multiple settings files, and hopefully I can find some time to get it written up and in the docs. You can check out this slide deck, especially slides 47 - 51.

configglue makes it easy to manage multiple layered configuration files, amongst other things, and the integration with django works fine. An example of the nice things you get is to be able to lookup in which file a setting was lastly defined.

@Jacob, Your slides suggest a simple change to the settings choreography, but I'm not sure it would have solved this problem. I certainly agree with you that using local.py to handle server differences is a bad idea, which is why I use a hostname-based scheme.

@Ricardo: I hadn't seen configglue before, thanks for the links.

Sever Băneșiu 1:10 PM on 6 Dec 2011

While this won't solve your problem, I think an entry-point based solution for extending the settings is better than hardcoding the imports. django-extraconfig can serve as an example.

You can still use a hostname-based scheme with the kind of approach that Jacob lays out. I use settings/__init__.py as a traffic cop; it contains any logic about which settings file to import. I used this approach in epio_skel to good effect, there it toggles based on the presence of an environment variable.

The only key thing (as pointed out by Jacob) is to import specific->general, e.g. import your most specific settings file, and have that import more generic ones at the top of your file.

@Sever: thanks for the pointer to django-extraconfig. Both it and configglue seeem to add extra steps: configglue wants me to make explicit a schema for my settings, and django-extraconfig requires a setup.py for each extra settings file? Solutions will need to be as simple as "one more settings.py file" to succeed, I think.

@Idan: You are right, I can combine the host-based scheme with the idea of importing specific to general. My point was just that specific-to-general seems to be independent of the issues I was having. I would still have had double importing, and I would still have had mysteriously changing python paths.

Nick Coghlan 4:59 PM on 7 Dec 2011

While it's great for app-level configuration and calculation of default settings, I personally consider using Python for production settings a bit of an anti-pattern (since it makes life harder for sysadmins). Accordingly, I have my settings.py set up to read the real settings with ConfigParser:

The Django 1.3 tutorial basically tells you to make the mistake of referring to your modules under two different paths. Imagine you are a new Django programmer and are following the tutorial.

The first thing the tutorial suggests you do is run

django-admin.py startsite mysite

Suppose you do this in the directory ~. Then you get the file ~/mysite/settings.py which contains the line:

ROOT_URLCONF = 'mysite.urls'

So this will only work if ~ is on the path.

Then the tutorial suggests that you cd mysite and run

python manage.py startapp polls

And then it says, "Edit the settings.py file again, and change the INSTALLED_APPS setting to include the string 'polls'."

So this will only work if ~/mysite is on the path.

All goes well as long as you are running the development server via manage.py, because it puts both directories on the path. But when it comes to deploying the site, it won't work unless you put both directories on the path. (And the Apache/mod_wsgi documentation doesn't mention this.)

Is this a bug in the tutorial, the startsite command, or the Apache/mod_wsgi documentation?

What is the 'settings.deploy' referred to in the last line? How are settings.staging or .production intended to be used? How is this local.py different from the antipattern of using local.py you condone in the previous slides?

TBH, I have similar questions about many of the slides in this presentation. In the spirit of Christmas, would you reconsider posting an intact version of the speakers notes for that presentation, even if they were never intended for publication? For the children???