Answering readers and critics on Linux configuration anarchy

Marco Fioretti got some great feedback on his post about Linux configuration anarchy -- so much so that he offers some thanks and a few rebuttals on the topic.

A couple of weeks ago I wrote about the Linux configuration anarchy that bothers me and got a lot of feedback. As promised, here is a follow-up to thank the readers who like the general idea, and even extended it, and also to answer some critics and issues raised by that post.

This is about configuration data, not interfaces. And no LinuxConf, thanks!

A couple of readers said, more or less, "I remember a program called Linuxconfig or something like that, and it sought to be what the author seeks to find: a way to 'fix' configuration anarchy...It was really what it said it was."

Marco: Sorry, but first of all, Linuxconf never was what I am suggesting. Secondly, the implementation was bad! I only suggest that configuration files for all programs of the same class should have the same format. Linuxconf offered a unified interface: when you told Linuxconf what you wanted, it would rewrite all the involved configuration files, each with its own format. Then, at the next reboot, some init scripts would rewrite some parameters, and things wouldn't work. There's a reason why Linuxconf disappeared (even if it wasn't all its own fault) and, personally, I am happy it did. This doesn't mean that a unified configuration interface is intrinsically a bad idea, it's just a separate issue.

Configuration anarchy is either intrinsically right or unavoidable, because every program is different

This is the thesis behind comments like:

[different programs] offer different features thus different settings. By making a standard you cripple diversity which is huge in the Linux world

There's a reason each software follows its own configuration, the developer can implement changes more easily, structure the data the way he sees fit and doesn't rely on compatibility issues...

It's easy to logically separate programs for different tasks. For example: Personal e-mail could all be in Thunderbird. My work e-mail comes in Evolution

Marco: Please (re)read the whole article. I never said that there should be only one program per task or a unified configuration interface. My idea may be wrong, but surely it doesn't cripple any diversity. How does asking that I only have to type some data once and have it saved on disk in only one place limit the developers' freedom to change how their software works, or its user interface?

Besides, I explicitly stated how to manage application-specific settings and, as other readers pointed out (thanks!), that a single program may still have custom settings for common options. For example, a 'KONQUEROR_HTTP_PROXY=' line in the .browser_conf file would tell Konqueror to use that proxy instead of the one specified by the generic 'HTTP_PROXY=' option.

Finally, well written FOSS programs have always had the possibility to use a whole alternative configuration file, even if just for one single session. Such a feature is very convenient for reasons that have nothing to do with my proposal, so if it's missing, that's a problem regardless of my ramblings.

Text and only text, please

Some readers, concerned by data corruption caused by concurrent write access of two or more programs to the same file, suggested to use databases ("especially SQLite, since it's already used by both Firefox and Chrome"). Others said plain text files are okay if one adds some elaborated locking scheme.

Marco: No databases, please! Compared to databases, text files are easier to edit manually or via scripts, much simpler to back up, much more portable (even across multiple OSes!) and much more likely to be adopted. The set of FOSS programs that should avoid configuration anarchy is much bigger than Firefox and Chrome, and most of them do not use SQLite today. Convincing their developers to change the syntax of something they almost always already write to plain text files is immensely easier than making them add any other dependency to their code. Besides, file locking could be much simpler than most readers think. See next point.

The solution is a setting synchronization/conversion tool. And since this is Open Source, do it yourself!

Some people asked questions like:

Why don't you just write an app that synchronizes your settings?

Why don't you be quiet and rattle off some code and really show us how?

Marco: The first reason why I won't write myself a "synchronization app" is that I am not asking for anything of that sort. I am suggesting that certain configuration files should have the same format. The reason why I not only won't write patches to make that happen inside other people's code, but that a I also actively discourage people from doing it, is that it doesn't make sense.

There are much better uses for good programmers than making them learn internal parts of many unrelated applications written by somebody else and then systematically patch each release of all those applications. Synchronization? It is necessary in many cases, but at this level (that is "how should a bookmark or the name of an SMTP server be stored on disk?") it is really silly to create the need for it.

If the developers of some software want to continue to keep all its configuration also in its current custom format, because it's much better for performance, robustness and what not at run time, that's okay for me -- as long as I also get only one plain text file to back up per task, and I get to set ("how" is a different topic) each of its options only once!

It isn't mandatory to cancel what already exists or write a unified interface in order to give end users the benefits of a common configuration file. It would be enough to add to each program one routine that loads that file at start up if it changed after last session, and writes it if needed right before quitting. Only the original developers of each program could do this in a sustainable way, with the smallest possible effort.

Of course, this would work only if those developers agreed on the format of the plain text files. Paraphrasing another reader, this proposal could "help us to see how serious they actually are about standardization."

About Marco Fioretti

Marco Fioretti is a freelance writer and teacher whose work focuses on the impact of open digital technologies on education, ethics, civil rights, and environmental issues.

As a part of my day to day tasks, I develop small programs that convert data and perform various other tasks. I write almost exclusively in Java because you just dont know what system the program might end up on. Anyway, about 6 months after I started work here, I decided I needed to do exactly what you are talking about: define a single way to handle configuration files. I even went as far as developing my own take on a multi-value hash map.
Here's how it works from the programming side of things. First, I instantiate a class with the config file name as an argument. Then I can ask that class for a parameter by typing something like parameters.getString(SOME_PARAMETER). There are other methods that return int or whatever you are looking for. You can also call Parameters.set(key, value). From my point of view, I love being able to just add another parameter to the file and have it so easily accessible in the program. That class handles all I/O to all of my program config files. Location of those files follows a ~/.program_name.conf convention.
I think this is basically what you're talking about. From a user perspective, you will see the same format of config file no mater what program you're looking at. Flat files are what I use just because Ive never needed to handle a lot of parameters. This also allows people in the field to easily change configurations. If you're writing really large and complex programs, then I could understand a database approach. Anyway, that's my two cents.

I do not expect you (or anyone else) to change all programs to use some new standard, but someone has to give the community something to work with before anything gets done. Here's what I think the approximate process is.
Steps:
Write a couple articles about how neat it would be to have standardized configuration on Linux.
Come up with a basic, but specific description of how the files would be structured and where they would be stored.
Submit it to freedesktop.org as daengbo suggested.
Someone writes a library that applications can include to make it easy to follow the new standard.
A bunch of people file bug reports with their favorite applications requesting them to add the standard to upcoming versions.
App maintainers start adding adopting the new standard.
Problems are found with the standard and library.
The problems are fixed in a backwards-compatible way that doesn't corrupt anyone's config files when two of their apps are using different versions of the library.
There is now a universal configuration format followed across nearly all of Linux.
Five years later one of the BSDs does the same thing only with the benefits of hindsight and not having to maintain backwards compatibility so it's way better. Also, it uses the BSD license instead of the LGPL making it less of a legal headache for proprietary programs to include it.
Some Linux programs support the improved BSD standard and others don't; a series of anarchic methods are used to haphazardly attempt to reconcile the two types of programs....
Steps 1 and 2 require relatively minimal time and can be done by nearly anyone who can think logically. Step 3 requires coding skills. Steps 4-8 are optimistic, but with enough highly-motivated and well-connected people it is possible. Steps 9 and 10 are satirical, but should still be seriously considered during step 1 to prevent them from happening. Also, step 1 will probably require a few iterations before the standard is acceptable to everyone. As you've already done step 0, I'll provide an alpha version of step 1.
Shared configuration format:
For each application class make a plain text file called "default" in "~/.config/appclass/". Each file consists of a series of "variable=value" lines for configurations related to the app class, similar to but better coordinated than the .ini files that were widespread on Windows before the mess that's called the registry was conceived.
Application-specific configuration:
Each individual application has its own "appname" file in "~/.config/appclass/" which contains application-specific configurations. App-specific configuration files can also override the defaults for people who want different applications to do the same thing differently (different apps for personal and work stuff, etc). Apps can optionally have multiple config files named "appname.whatever" with "whatever" differentiating between different profiles, app versions, or whatever reason the app has for using multiple configs.
Generic configuration:
Many application types rely on many of the same configurations as other application classes. For example, you probably want the same proxy configuration for your email client and your browser and you probably want similar graphics settings for your different games. Such generic configurations would be stored in "~/.config/generic/" and named according to purpose ("network", "3dgraphics", etc). As with shared configs, these can also be overriden by each application. Also, generic configs can be overriden by application class configs, so if you want a proxy for your browsing but nothing else, you can do that too.
Data:
The described format and library only applies to configuration (settings, etc). Data should be stored in "~/.data/appclass/shared/" with separate files for each type of data the application class uses. Each data file should use a standardized and data-appropriate means of storage won't automatically be plain text. Shared data files should probably be named "datatype.format" so the format isn't locked in stone. If there can be multiple independent instances of the same data type, it all goes in a separate "datatype/" folder and the files are named "dataname.format". Unshared data would be stored in "~/.data/appclass/appname/" using whatever format the application wants.
Example application class data/config file usage:
Doom is a classic first person shooter whose source code has been released under the GPL several years ago and then forked like crazy into zillions of different (some such as Chex Quest being nearly unrecognizable from the original), but similar projects. The similarity means much of the configuration and data is the same, but the different directions of the different forks means that there's a lot of different information to handle separately.
Any configuration that's likely to be useful to more than one fork of Doom would be stored in "~/.config/doom/default" and everything port-specific would be in "~/.config/doom/portname". If one port accidentally puts something port-specific in the default config, that line will just be ignored by the others. If a port puts something general in its port-specific config, it will just not be auto-updated when the user changes the default.
In Doom all game resources (levels, textures, sounds, etc) for a complete game are stored in a single resource file, but add-ons can lack resources and the game will use whatever is in the loaded main resource file. These full game and add-on resource files would be stored in "~/.data/doom/game/" and "~/.data/doom/addon/" respectively. Save game files would be in "~/.data/doom/save/". Finally, any files not compatible with the original would be stored in "~/.data/doom/portname/whatever/" where "portname" is the name of the least-evolved port that the file is fully compatible with and "whatever" is the purpose of the file as described above. If a port knows it can handle data files made for another port, it can search in the other port's folder for additional game data.
Hopefully that's something useful to work off of. Maybe I should write the browser example next....

I have rarely left comments to articles. It has been enough for me to read, learn, and experiment with the ideas and concepts shared in those articles. This authors position on a topic very important to me however, warrants my taking the time to share my thoughts. First, I couldn't agree more! I have been an avid Linux user since I first installed Mandrake Release #1 quite a few years ago. In that time, I have had more than my fair share of frustration with the issue of config files not being compatible and/or written in way too many different ways. I also get the author's point. It wold be a simple matter for all designers to adopt common file names, locations, and formats for common apps. Look at how different distros place the same files in differing locations, i.e. /usr/bin, /usr/share, etc.

The solution is for _you_ to go over to the FreeDesktop.org mailing list, float this proposal, then write the spec for the file format for each application you want to standardize. I think that a good, general set of config options should be present in .config/allmailclients, and application-specific configs should be in a separate config file per application. For instance, there's no reason for Thunderbird configurations to be in the allmailclients file.
Good luck with this. It sounds like a good idea.

I absolutely agree with you. Having a standard format for specific types of application does not prevent individual customizations or diversity. I have been saying the same thing about resume formatting. Many times, filling out the application forms on many sites is so time consuming and frustrating that it in itself is a barrier to applying for more jobs. May times I have spent hours typing in information in addition to uploading the resume to their database. If we had a special format for different types of resumes, then all you need to do when formatting your own resume is to fill in the relevant fields and when it is uploaded, the applcation will extract the data and place them in the correct fields. All databases would be able to pull in the data and fill in its own fields properly. Makes job application a cinch.
What you are saying is how efficient programs are written though. You write a module once and store it in location, then call it when needed and modify it as applicable to you program. Like writing firewall filters. Instead of writing the same filter for every interface at each interface, you write the filters then apply it to the relevant interfaces and modify it if necessary. If you need to modify the main, one, then you only need to do it at one place to be applied universally. Makes sense? Who knows, maybe I was yapping too much.

The Linux system no matter which distro needs to be bullet proof for the new user. Install, GUI, adding software etc. etc. the simple things. Ubuntu is making a stab at it and that is why they are used so much. There needs to be two levels, one for new people to use Linux like a "toaster" and one for the technically inclined.

Sorry Marco, but thumbs down. I don't think you've made an adequate or clear case. I suggest that for products where the config files aren't in a sensible form, you submit suggestions for improvements to the respective projects. In general there isn't enough bad stuff out there to justify your moan.

The "configuration anarchy" you mentioned on your articles is far away from a particular platform. This is not a Linux issue, as you will get the very same situation on Windows, OS X and many other OSs. As long as you have more than one organization making apps for the same purpose {[( a web browser for example )]}. You are suggesting a superior authority telling how configuration should be compatible with a platform.
I can not imagine that happening in any platform near today as it stands over apps and not the OS. You could propose an standard or a RFC for configuration, but wait for a while till the industry hears that.
There are many ways to avoid the issues you showed about configuration and it happens on every platform.

Your idea is brilliant but analogous to implementing world peace.
1. For years browser publishers could not follow their own guidelines (I admit many were not Linux but this illustrates how difficult guidelines are to implement and maintain).
2. GUI config applications for any major Linux application, even within the same product class (OS, DE, Office, Browser, etc) are inconsistent. How often do you find yourself hunting for a section or specific config entry? Why are these be uniform? Word based config files are generally far more robust, harder to organize.
3. Many Linux developers like the lack of guidelines. They can have it their way. A corner stone of the community - - maybe.
4. The biggest obstacle is lack of leadership; no preliminary example or framework for the community to digest, tear apart, rebuild, fine tune and finally offer up for community acceptance. The folks who wanted you to roll up your sleeves and write code may have been snide, but they also have a point. Some one needs to get the ball rolling. You have a good idea, why not get it moving?

Too bad so much work has to go into combating comprehension failure rather than further discussing the actual idea, but this certainly seems to be par for the course. Some authors and commenters have certainly tackled this sort of thing in threads, but I'm not sure I've seen a substantial portion of an article devoted to addressing this. (Well, maybe not recently, anyway. :^0 )
I still think the idea in the original piece is a good one.

I like your idea and when writing software the first thing I do is if I need to write config files is find a way to do it easy and usually that means copying the way it was done before or just using theirs. But not everyone does this, there are people who want to create their own thing and it's their software, they will do it the way they want to.
Standardizing config files won't happen and here is why; foss software is not about replacing windows. It's not about making software that everyone on the planet will use, it's not about taking over the world. foss software is about choice and sharing. Developer writes some software for himself is how most foss software starts and then they decide to share it. If it works for them it's already met it's whole purpose and is successful, and other people using it is gravy. Some software then go on to be major projects like Linux. But I am sure that the first time Linus was able to run his own kernel and it worked it was a success for him. Now years later releasing a new kernel to everyone is also a success but Linux was a complete success before most of us ever heard of it. So while I applaud your efforts and agree with you on the idea and I will try to hold to it in my coding, foss or otherwise, don't expect everyone to do the same.
And the idea of cutting back on duplicate apps is also a non-starter because of the reason for foss in the first place which is choice and sharing. Cutting back on the choices people have is 100% diametrically opposed to everything foss stands for.
Sorry if I'm a downer, don't take it that way, your idea is already a success if only a few developers switch to using more standard settings.
Thanks.

Good follow on to the original. I believe this is the single largest obstacle to wider Linux adoption.
I emphasize what you mention above for clarity: proper standardization could result in compatibility across platforms, the OS could be rendered irrelevant so far as user preferences and data.
In plain English, if there is a standard text format, you could take your settings (and some content) with you easily, from Linux to windows and vice versa, to MAC, what have you. Of course the installer and resulting binaries would be different, but the point is people use apps to generate their own data, and configure the apps to their liking for that purpose.
Plain text is the simplest and best way all around, from backups to rescue. Since the advent of the live CD I can't count the number of times I've booted up, found an errant text file on a hard drive and made a simple edit to render the hard drive system bootable again. (Linux systems, of course)
I suppose it was inevitable that some thought of the multiplicity of applications for given tasks, eg email clients. There would be some positives for Linux in general if the number of 'duplicate' apps were cut back a bit, IF the reason is to apply the developers of the abandoned app to the development of one of the remainders. But that's a whole different topic.

Precisely - the same "anarchy" could be termed creativity. Different vendors make it a point not to follow a standard - or add to it.
Let's get even a bit more involved - SQL - while SQL was supposed to be a universal language, Oracle took advantage of internal developments and created PL-SQL. So did other vendors. The result? Large, enterprise apps (e.g. SAP) have their own translator which optimises the SQL it talks to the db end depending on which db platform is used for the installation.
So... config files a problem? I think it is more an issue of specialization than anything else. After all, once we spent some years in the enterprise arena, we all grow "soft" and acquire knee-jerk reactions pointing to the "best" app for the task. Sadly, at a point we stop learning - at least on the infrequently used apps and keep using the ones familiar to us.
And finally... even in the same class, apps are designed for different purposes. One may be trivially simple, while the other one incredibly complex. Should they all have compatible config files? Would that help in any way?