Ever since I started using computers, I've been baffled by the relative clumsiness of installing applications. Whether we are talking the really old days (launching the Rambo game off a tape), the '90s (running Keen or using installers in Windows 95), or the modern days (still those installers, but now also package management and self-contained applications); it's all relatively cumbersome, and they all have their downsides. I decided to put my money where my mouth is, and come up with my idealistic, utopian method of installing, running, updating, and uninstalling applications.

//Whether we are talking the really old days (launching the Rambo game off a tape)// er ... i thought the "really old days" was when I typed three dozens lines of Pascal into a PDP-11 in order to get a simple tax calculation to run ...tapes? You're high-tech!

//Whether we are talking the really old days (launching the Rambo game off a tape)// er ... i thought the "really old days" was when I typed three dozens lines of Pascal into a PDP-11 in order to get a simple tax calculation to run ...tapes? You're high-tech!

"//Whether we are talking the really old days (launching the Rambo game off a tape)// er ... i thought the "really old days" was when I typed three dozens lines of Pascal into a PDP-11 in order to get a simple tax calculation to run ...tapes? You're high-tech!

It doesn't matter how long it is. A good idea is a good idea in two lines and in ten pages. Just because it can be detailed for hours doesn't make any idea good. Note, with that I don't say this particular idea is no good.

The only thing is, it's hard to satisfy everybody's taste. E.g. I don't like the Mac-style of application installing, I prefer dpkg, and I don't like the folder names with compulsory capitals, and so on. That doesn't mean the concept itself is wrong.

He critisized the guy that said Gobolinux is not a good idea, but that other guy did not give ANY REASON AT ALL.

Noone can act upon FUD

And btw, this suggestion here is different from Gobolinux in a few key areas. I am not saying it is good or bad here, but it is obviously that there are a few people who neither read this, nor know about Gobolinux - or opt to simply not give any reasons at all why something is "good" or "bad".

It is pointless to debate about opinions without people giving any REASONS behind them.

I don't think I misunderstood, but nevermind [I was just trying to point at what I frequently ran into in the past, that someone can actually make a case about an idea in a really long form, still the idea can be completely wrong, which in this particular case isn't].

As about Gobo, I'd say the above ideas are far better than Gobo's, since I always found Gobo to be a bit strained, and I actually found the trials I made with it irritating. It's not a native approach, not a trial to make a better file/dir hierarchy and better package/app management, it feels more like a hack. A system following the above ideas could actually be something better, even if I don't really like some aspects of it.

It looks somewhat similar, but Gobolinux is an attempt to achieve modularity within the typical GNU/Linux paradigm, where there's no distinction between the base system and the applications, that is, any package can depend on other packages that may or may not be present, and it can fullfill the requirements of yet other packages.

On the other hand, Thom's proposal assumes a base system, which makes it more like PC-BSD or Mac OS X. Once you have a system/applications distinction, package management is almost trivial. But then all distros would have to agree on one base system, which would either kill the well-known configurability of GNU/Linux (as in being able to have a very small installation) or be such a small system that lots of duplicated libraries would coexist, among other problems. The LSB guys have been trying to do just that for years, and I don't know how much success the've had, but the fact is that third-party program packages are still distro-specific. TANSTAAFL :p

Well, I meant that the LSB project tries to stablish a core system (including specific libraries), so that third party application packages can just depend on lsb-core. For instance, take this section:

"Packages shall have a dependency that indicates which LSB modules are required. [...] Packages shall not depend on other system-provided dependencies. They shall not depend on non-system-provided dependencies unless the package provider also makes available the LSB conforming packages needed to satisfy such dependencies. "

Well, I meant that the LSB project tries to stablish a core system (including specific libraries), so that third party application packages can just depend on lsb-core. For instance, take this section:

"Packages shall have a dependency that indicates which LSB modules are required. [...] Packages shall not depend on other system-provided dependencies. They shall not depend on non-system-provided dependencies unless the package provider also makes available the LSB conforming packages needed to satisfy such dependencies. "

Yes, LSB tries to establish a core. And most of it is okay, but it still doesn't go quite far enough. Personally, I think they need to get RPM out of LSB and replace it with something like (though not necessarily) Autopackage. It would make the LSB a far better standard, and would really help towards Linux on the Desktop too.

No, it's an elaboration on MacOS X. The system bundle is presumed to provide a generous set of shared libraries and prolonged ABI stability -- no small feat.

As with MacOS, any library excluded from the stable system bundle must be statically linked into all dependent program bundles, each of which may contain a different outdated version of the library.

The main differences from MacOS are the semantic filesystem and the bundle repository, along with the minor hierarchy change. It wouldn't be difficult to prototype this system for OS X using Spotlight or perhaps Nepomuk.

I have one question for Thom: I assume the desktop environment would be part of the system bundle so that its libraries could be shared amongst program bundles. So where on the filesystem would the per-user settings for such system components live? For example, does my wallpaper belong in /Settings/butters/System or /System/Settings/butters?

"Applications in Mac OS X are generally not easy to remove at all, because they leave a trail of files around outside of /Applications that normal users rarely encounter. Over the course of time, this can amount to quite the mess."

The only files that are lying around are:
- plist for preferences (like .dot files in ~/ in Linux)
- cache files
- sometimes a folder in "Application Support"
That's all. You can easily search for the name of the program with Spotlight/Finder to remove those items, though it is NOT necessary because the only downside of those files lying on your disk would be waste MB... you have to run thousands of apps and deleting them to call it a mess.

"In addition, Mac OS X provides no way of updating applications in a central way, resulting in each application in OS X having its own updater application; hardly the user-friendly and consistent image Apple tries to adhere to."

If an application is started it will inform you about a new version, quite simple. Most of these dialogs are pretty consistent to each other too. And there are applications or even widgets that can check all your applicatins for updates, though it isnt really necessary, because.. when u run it it gets updated / will inform you. It would be nice to see Apple implementing this mechanism that those update-applications use into their own software-updater. But as said.. not really necessary.

The big problem with the Mac is that the apps arent consistent. Not all auto update themselves. Just like in Linux some software has to be compiled. Still, it would be nice if the world could agree on one single way to notify and put out updates. RSS maybe? In a corporate setting, repos work well since its pretty easy to make packages of something that needed to be compiled. But for single users adding a lot of repos leads to package poisoning.

I so wish I had time to read this Thom. Like many of us, package management is the first and foremost reason we "advanced" Linux users will choose a distribution. Well, actually since I discovered Debian I've never looked back except for the occasional brush with Yum... and thinking that Red Hat has had APT as the example for YEARS, and Yum is the crap they came up with?!?!?

Anyway, care to make it a podcast or something that I can put on and tune in and out while I'm coding? Mostly kidding, that's a lot of work... would be more efficient in the end if I just read it Anyway, nice article, love the exclusives.

'Like many of us, package management is the first and foremost reason we "advanced" Linux users will choose a distribution.'

Not me. I guess I am not alone though. Personally, I simply refuse to use solutions that do not give me the _OPTION_ of having something like AppDirs.

AppDirs are simply a better way to manage applications than FHS, period. If I want to have something removed, i go kill the dir. I dont _want_ to DEPEND on a package manager in order to uninstall something again.

One of my many pet projects for Haiku has been a new "package format" for applications as well as a consistent updating mechanism and some sort of central application repository.

This might make a good initial design, since you make use of the live queries that we will have in Haiku. Overall it sounds pretty good.

I think the permissions system you describe could also be implemented.

But I think the file system layout will need to remain in the standard BeOS format we are inheriting. But I still think much of your design could still be used for that. For one thing I don't see why you need a separate /Settings hierarchy when you could just have /Users/User 1/Settings or in Haiku /boot/home/user/config. Of course the multi-user aspects of Haiku are still in flux and probably won't be sorted out until R2.

This might make a good initial design, since you make use of the live queries that we will have in Haiku. Overall it sounds pretty good.

You're right Ryan, this whole thing actually started out as a discussion in #haiku a long time ago. I was just musing aloud at how you could use the attributes and live queries in BFS to manage applications, and from there, this whole idea started to grow.

So it makes sense that my ideas fit Haiku so well.

For one thing I don't see why you need a separate /Settings hierarchy when you could just have /Users/User 1/Settings or in Haiku /boot/home/user/config.

The reason I chose for a separate hierarchy is because I want the /User/User 1 directory to be strictly a directory for the user's documents, movies, photos, pr0n, and so on. I'm someone with a strong inclination towards order and cleanliness, so you can imagine why I'd like to not put settings files into the home directory.

And thanks for the compliments . I've spent a lot of time on this proposal, and I believe I'm only scratching the surface of what attributes+live queries+program bundles can equate to. If you want to discuss this in more detail for whenever the package management plans for R1+1 come up, feel free to contact me, I'd love to participate in that discussion .

I don't think that separating the user and its setup configuration files is a good idea: this means that when you want to backup your account, you need to save two directories not one..

As for the cleaness point of view as long as all the configuration files are in the setting directory, I don't think that this is dirty.

That said, I've been convinced recently that purely hierarchical file is impossible to get 'right' so if the user documents and his configuration file are tagged by his login name (automatically) then the backup of the user's data becomes far more easy..

I always see my User folder as "everything that makes this computer unique to me". That includes my settings, so I can backup just one folder, and even move it to another computer and log in with everything there.

Locking would still be necessary because while queries are fast, installing is not.
Let's say you use one query to install program P and start another query and uninstall library L (something P depends on) along with all older programs that need L.

There are two ways out of this:
a) Using a more fine grained locking along the lines of "install anything you like but make sure not to destroy L since P (currently being installed) will need it". What, however, if you said install P from vendor V and while it was installing told the computer to remove all programs of vendor V. I'm almost sure no matter how smart the algorithm is there'll still be situations where it has to say "Encountered Conflict C, do you want to do x or y?"

b) Making every program completely self contained.
This would have HUGE security implications.
A defect was found in a library that 100 of your applications use? Well, that's to bad, you have to reinstall them all.

You can of course draw the line at some arbitrary point and say this lib is used 'a lot', so it's shared.

In short, while your idea sounds good I believe it has tons of details, corner cases, and trade offs that still need to be sorted out.
Furthermore, it can only be realized embedded in a bigger ecosystem. To revisit the example with a defect in a lib, developers could tag their applications like "works with version x of this library" in a way that the program could automatically tell you "vulnerability found, switching to fixed version of lib L" or "vulnerability found, program not yet tested with safe version of L, do you want to upgrade and risk a crash or keep running this unsafe program".

In addition, your system is still centralized.
There needs to be someone or some server to say "No, malwareGuy, you cannot call this 'Paint 6', there's already 'Paint 5' and you didn't write it".

Somebody has to decide what goes on the server and what doesn't.

I could go on and on but I think you get the point: The idea is good but the devil is in the details.

Maybe I just misunderstand what you say but aren't all these attributes together sort of an identifier?

So you could have two files with attributes
program=paint
vendor=ms
version=4.1
patch_level=127
...

and if they contained different binaries it would be a big problem. You could, of course, add mechanisms like checksumming the binaries and so on.
But then again you'd need a central server/group of servers that tell the user that a program with attributes x and y should have checksum z.
Am I missing something?

Or alternatively, the developer/maintainer or whoever does the bundling can simply use some sort of cryptographic key, which the installer uses to confirm the new bundle is indeed from the same source as the already installed bundle.

Basically, the problem you're trying to solve is intrinsic to any form of installation, not only the idea posted in this article, so whatever solutions exist already can be used without any need for centralised administration.

If I recall correctly, zero-install used some crypto-signing for something similar.

Nice try, Thom - if you're just reading it casually you almost miss the hand-waving about binaries that don't belong in any particular 'program bundle', and about the issue of shared libraries, which is of course the big drawback of the OS X system that you *don't* mention (because it persists in your vision). If you use shared libraries, you have a reliance on your vendor (situation with all current Linux distributions). If you don't, you have security issues and ancient bugs that were fixed long ago cropping up all over the place (situation with Windows and OS X).

Nice try, Thom - if you're just reading it casually you almost miss the hand-waving about binaries that don't belong in any particular 'program bundle',

Miss? It's right there in the article:

"I also took a cue from Mac OS X by creating the specific /System/Utilities directory, where the operating system can store utilities such as the Activity Monitor, graphical Bluetooth tools, Network Utilty (graphical ping, whois, etc.), those sorts of things. What does and doesn't go into that directory is fairly arbitrary, and is open for debate. Other binaries that usually reside in /bin on UNIX systems can also go into /System (say, something like /System/Binaries, since this is the 21st century - why use unclear acronyms)."

'If you use shared libraries, you have a reliance on your vendor (situation with all current Linux distributions).'

That is true, and I am one that constantly critisizes the upstream vendors/maintainers as well, but you see - the situation on Mac is as bad as on Windows in that you are dependent on a company just as well.

So all these worlds more or less have similar problems. You depend on someone else.

Now, with similar problems already, I as a user would still like to choose AppDirs instead of FHS.

I dont want that others enforce the FHS upon me. But the big distributions have no inclination to change to AppDirs at all.

We should also have a /Users/Common directory for documents (and a /Settings/Common for settings) that all users on the system need access to. Of course the system needs to support user-controlled file permissions so User 1 can grant write privaleges to User 2 to a directory he owns but not allow User 3 to read it without being a System user and needing create a group.

What about file associations? I know this could be handled by the window manager but I'd like some central database of what applications are associated with what files, even on a user by user basis. On Computer 1 Firefox should be the default browser for all users but Computer 2 User 1 wants to use Firefox while User 2 likes Konqueror.

This new program management is the best idea I have ever heard for managing programs!

Real separability. For example, many people who test distros will want to share /home, but later find themselves with overlapping Settings that do not work with the other distros.

These days I maintain my files in /home/xiaokj/Link and make the link to some central repository (another partition). It would be nice if I need not do that, but rather link specific apps' settings (like IM/Email that is really supposed to be persistant) instead of making sure the dozen other settings that should be separate (like Xauthority, KDE/Gnome settings...)

If each program has an "internal" version number, then perhaps the corresponding /Settings/User/X directory be the version number instead of the program name? So if Garden Designer is v846, then the file is /Settings/User1/846. This way, you'd really be able to run parallel versions of apps.

But there's another problem - there might be another program that is internally 846. We need a unique number. Wait, that already exists!; enter the GUID.

My point is, there are some really cool ideas in here, but ultimately, I think it needs a lot of refinement. Storing the settings in a separate space just means you've improperly used the home directory in the first place (see my above comment). Your permissions *will* be screwy when you put settings in one top level directory and user data in another. This is why there are hidden directories. Also, your arbitrary requirement of the settings directory sharing the name of the software package will need some sort of low-level monitor. It also means that when I release a new version of a package and choose to rename it with a version number, I can't reliably find your user settings from any of the last several versions of my software, I have to try every possible combo or not let you migrate your settings from any version but the last one.

Frankly, I think the current OS X way is near perfect, with plenty of room for small improvements. I don't think that, with disk space as it is, the system is "cluttered" by having lots of settings file - stored properly where settings should be stored! The idea is that it's persistent, it's there the next time you install the app. And I think Leopard's Spotlight is plenty fast enough for 99% of users. Live queries are very cool, but very few people would use them in a mainstream OS, and existing technologies provide most of the end result.

If each program has an "internal" version number, then perhaps the corresponding /Settings/User/X directory be the version number instead of the program name? So if Garden Designer is v846, then the file is /Settings/User1/846. This way, you'd really be able to run parallel versions of apps.

But there's another problem - there might be another program that is internally 846. We need a unique number. Wait, that already exists!; enter the GUID.

And how user friendly is that? The hierarchy I devised is supposed to be 'human readable' - actually, that was one of its primary goals. File systems have been a mess since day one, and it simply needs to be fixed. OS X made tremendous strides in that regard, but in the end it's just a virtual directory structure draped over a traditional UNIX/POSIX structure.

This is why there are hidden directories.

Hidden directories are evil. If a system needs to be secretive in order to not confuse a user, there's a design error somewhere.

Also, your arbitrary requirement of the settings directory sharing the name of the software package will need some sort of low-level monitor.

I addressed that issue with a typical utopian statement in the article:

"To prevent the mess Mac OS X has with separating settings files from the program bundles, the system should somehow 'know' the two belong together - this shouldn't be too hard to realise. One option would be to use BFS-like attributes; the Garden Designer settings file could have an attribute attached to it that links it to the Garden Designer application bundle, for instance. This way, when you want to remove Garden Designer from your system, the system can ask you if you want to delete its settings files too."

I think most of us have thought about such ideas in the past. The devil really is in the details as someone said above.

I tend to like the idea of 'application bundles' but then how do you handle shared libraries. You will end up doing the weird windows DLL handling.
For example you could have a '.lib' file in each program directory that tells the loader which shared libraries to use for the application. The application bundle would include a version of the library that is 'known' to work.

shared libraries not belonging to a particular application can also be installed a generation /system/libs directory.

If a newer version of a '.lib' is available, then somehow the system must decide if it wants to try that version instead of the version in the application bundle. This 'somehow' is undefined. Perhaps the system detects a new version, asks you if you want to try the upgraded version... and goes back if it fails. Maybe we leave it to some online repo... This somehow could get complicated fast.

--------
in terms of updating the system. I don't really know why you're complicating it with all this search and queries... Wouldn't a simple file with the application bundle, pointing to some server location do. The system can check if there is an updated version and if so, prompt for you to download/install it.

One of the most frustrating thing when you live on a linux where you don't have root access is to install programs. you basically need to go the "./configure --prefix=$HOME && make && make install" and do it yourself dependencies.

Not all programs require root privileges. Then, why not allow users to do "apt-get install whatever" and it goes to their directory until the program is superseded by a system wide installation?

Going along this idea would be a first step towards better program management.

Not all programs require root privileges. Then, why not allow users to do "apt-get install whatever" and it goes to their directory until the program is superseded by a system wide installation?

I have wondered about the same thing myself. But I guess it all boils down to the fact that people expect only two kinds of users to install software: system admins or users who own the computer. The truth is however that there are also people who don't have the root password but might still wish to install something additional. Such users could f.ex. be your children.

So yeah, I basically like your idea. All the files and folders should however go under a single folder in the users' home, like f.ex. /home/user/Applications and of course everything should be owned by the user and writable only by that user. Oh, and yes, the system admin should still be allowed to choose which users are allowed to install software this way. In corporate environments for example it's often preferred that users are not allowed to install anything but instead ask the admin for that.

The "ask the admin" approach usually takes some time and gives the admins a bad reputation of being not very responsive.

If not in an environment where installing an application may compromise security and should be prohibited, there is a clear benefit in letting the user do basic administration tasks (installing stuff). Debian-style package management makes this possible because installing something is designed to be a zero hassle operation. However, the neat thing about being superuser to install stuff is that it keeps the applications clean and working. A similar approach should be used by not giving the user actual write rights in his app folder, but using a setuid mechanism so that only the package manager can do that.

If programs are organized as AppDirs, and they are not allowed to rely on being installed in a specific location in the file system hierarchy, this may be achieved simply by unzipping the app wherever you prefer. No need for installers at all.

The trick with Thom's attributes idea, is that you can still query installed applications even if no installer was used to put them there.

Oh, and yes, the system admin should still be allowed to choose which users are allowed to install software this way. In corporate environments for example it's often preferred that users are not allowed to install anything but instead ask the admin for that.

Unix security has a very simple way to do this. Just mount the home partition as noexec. There simply is no way you can stop the user from putting a binary in his home directory, but noexec makes sure that no user installed files can be executed.

I enjoyed reading the article very much and it seems like a good base for building a new packaging scheme. However, there are still some areas that need further work. Some assumptions of the proposed system were also not clear.

The first one is the operating system, is this meant for Windows, OS X (OS XI?), Linux, some other OS, or an entirely new system? For "legacy" systems, the biggest issue of all seems to be providing a migration path from existing software deployment methods to the new method.

The handling of shared libraries, and the necessity of separating user settings from user homes have been mentioned by others. Also, the write-up talks about a single host, but in practice, program and user directories are often shared among several hosts. It is not obvious that the system would work well in such scenarios. More detailed consideration may be needed about what is shared among which hosts and users. Depending on the OS environment, support for multiple architectures may need to be considered as well.

One of the biggest issues is the storing of settings. Supporting multiple versions of a program in parallel, sharing the same set of settings, is a pain in the ass for software vendors. I think that a lot of conflicts would arise from trying to run different versions of Word, Photoshop, or even GNOME2 with the same set of settings.
So if different versions have to share the same set of settings, many software will likely refuse to install without having prior versions removed first. In such case, if you have other software packages that depend on this kind of a package, you're again stuck with dependency hell.

Furthermore, as mentioned in the article, sometimes the user or administrator does not trust a new version of a program. So even when the software supports shared settings, the user may not want to allow it to modify the settings used for the "trusted" version.

One solution may be to store settings for different versions separately, but have the installer generate the new settings automatically based on those for previous version.

Finally, the proposed system also does not seem to support installing programs without system privileges, which is normally possible for most Linux / OSX software and an increasing share of Windows software.

Good luck for working on this new system, and looking forward to hearing about the progress.

is this meant for Windows, OS X (OS XI?), Linux, some other OS, or an entirely new system?

It's not meant for any specific system, but you can see my involvement in Haiku and my love for BeOS shining through.

Supporting multiple versions of a program in parallel, sharing the same set of settings

Except, they don't. The whole idea is that the the directory /Settings/User 1/Garden Designer could contain multiple settings files for multiple versions - just as I explained in the article. They would be differentiated by their internal version numbers. The file manager could show them as follows:

The idea is that the system is clever enough to only display the internal version number when there are actually multiple versions installed. Since installing multiple versions is most likely an expert endeavour only, normal users will never encounter such internal version numbers attached to their program bundle files.

Finally, the proposed system also does not seem to support installing programs without system privileges, which is normally possible for most Linux / OSX software and an increasing share of Windows software.

Installing Linux packages without system privileges? Some exotic Zero Install systems may be, but other, more conventional package systems all require system access (as far as I know).

Anyway, executable code should never be able to be installed without a system password, if you ask me. I'm quite strict in that, I know. Executable code is the basis for A LOT of attacks, so it simply shouldn't be something just any user can dump on the system.

Thanks for the clarifications. And sorry, I obviously somehow missed the paragraph about multiple setting files.

I still have some things to add...

Multiple versions

I don't agree with having multiple versions being an unusual case that "normal users will never encounter". In fact, I think that the ability to have multiple versions of the same software side-by-side might be one of the most sought-after features in program management.

Not being able to have multiple versions side-by-side is what keeps a huge number of users at businesses and many at home stick with older, often outdated software. Many people won't upgrade unless they are absolutely certain that they can do the same things with the new version as they used to do with the previous. And with most existing systems not allowing to use multiple versions side-by-side, there is usually no way to make sure.

So I'm confident that users will install multiple versions when the program management system allows it. This is even more needed for systems with multiple users. In such case, the migration of all users may take longer, and so the time frame when multiple versions are installed side-by-side becomes longer. Note that company and school systems with large numbers of "normal" users fall into this category, where multiple versions would be available for extended periods of time.

Actually, in the utopistic program management system where multiple versions can exist side-by-side without conflicts, there should be no need for the user to explicitly uninstall previous versions. Old versions out of use could be removed automatically in a time-machine-like fashion. (In practice you should consider removing them sooner for security reasons.)

Installing with user privileges

Several other comments have mentioned this as well, so this also seems to be a sought-after feature. (This may also be related to the reasons above, e.g. some user wanting to use Firefox 3 on the company/school etc. system where Firefox 2 or maybe 1.5 is installed system-wide.)

I have to mention that when I said "installing" I was referring to software deployment in general, not one specific deployment framework. For example, I can download an archive ("bundle") of Eclipse for Windows, Linux or Mac OS X. For each OS, I can extract it in my home folder ("install") and run it from there. This does not require administrator privileges ("system password"). (For Linux, installing other applications in home folders may require tweaking with paths, but for most programs it is possible.)

Note that from that user's point of view, after it has been installed, such an application can be used in the same way as a system-wide one, the only difference is that it is not available for other users.

With a clean framework as the one described in the article, it should not be difficult to add support for enabling this kind of installation for all bundles. (As suggested by others, they could be put under "/User/User 1/Programs" instead of "/Programs", for example.) Doing so would reduce the amount of user actions required for such installation, allow automatized upgrading of all software a user has installed, etc. It does not make the system more vulnerable, compared to when the user has to do all these steps manually. Finally, it would also reduce sysadmin load as users who weren't able to install by themselves would not need to ask the admin anymore.

but in the end it seems like it would basically be OSX + a centralized system, which isn't exactly revolutionary. The whole thing with BFS live queries could be handled exactly the same with the centralized system using a database, you're just moving the relation management from the db into the native filesystem.

...but my personal wish is for applications always to be fully self contained with any third party stuff they need contained within the program directory. Sure this would lead to bloat on disk size, but whats taking up more disk space on your hard drive? Those 10Gb of applications or the 100's Gb of MP3s/AVIs?

Im a big game player and i like games which have save games IN the game directory (eg: under \save or \save\profile - for multiple profiles). Saving under My Documents a la Microsofts recommedations gets me really annoyed.

And dont even get me started on saving game/configuration information in the registry. The registry is a bad idea from the start. Whats wrong with .CFG text files?

While i accept the need to consider multi-user systems and privledges im sure self-contained apps could somehow work with this.

The thing is if the application is really 'fully self-contained' that this increase not only the disk usage (which we don't care I agree) but also the memory usage (which we do care about!) because a 'shared' library wouldn't be shared anymore as each application would load the version stored in his directory, the OS wouldn't know that these version are identical..

Currently the way I see to fix this is to have a 'delta' phase in the installation where all the 'shared' library contained in the application would be checked against the already installed one, if a shared library already exist in the same filesystem then it wouldn't be installed but it would be hardlinked to the existing one.
Benefit: no disk or memory bloat and the application are still self-contained.
Inconvenient: an additionnal 'delta' phase

The big downside of the 'self-containement' part is that if a shared library must be upgraded then all the application which use it must be upgraded..

Had this too often with apps with shared components, eg some lib windows\system32. New application breaks old one, or old application breaks new one.

Perhaps this is sometimes caused by bad programming and/or packaging, but it does happen from time to time that two apps require exactly the same named lib that should reside in the exact same location or else it wont work, and those two versions of the lib are not compatible.

Hence my liking of self contained. And besides, sharing only bloats memory of those apps are memory resident/drivers/or simultaneously active. Not going to always be the case.

I wish Ubuntu would use a system like you described in this article. It's much more 'human'. I really have no idea where Ubuntu stores all it's program data, and usually config files and log files etc are all stored in different places, for each different program.

It's too confusing, and when i want to do something myself, instead of using Synaptic, i usually end up confused..

1: Why do you need to know that?
2: Need aside, all that info is stored in the .deb. Kpackage (and Synaptic I think) have a way to show where it installs all the files in the package.

and usually config files and log files etc are all stored in different places

If by "different places" you mean various .directories in ~, then sure. What would you propose? All setting stored in one file (or hive, hello registry)? Or perhaps settings from different files presented in one unified interface (gconf might be what you are looking for)?

Or perhaps you want settings to be stored alongside the binaries in the self contained program folder, ala older Windows stuff? Thing is, that doesn't even work in Windows anymore. Permissions and privileges means user settings go in the user's folder. Are your settings in Application Data? Or Local Settings\Application Data? (or the registry?) By contrast, I find locating ~/.appName to be slightly easier (or ~/.kde/share/config/appName for that other DE).

I've said it before, but your complaints look a lot like what I am now going to dub WindowsPoweruserSyndrome. People who, instead of installing everything to the default location (c:\program files) discover that they can install games in e:\games and little apps to E:\myApps. It seems more organized, and you are in control! Linux then by contrast seems to take some of that control away when apt-get install puts things where it wants. It's disorienting. "Where is the program? What do you mean they are all in /usr/bin? With the data in /usr/share and heaven knows where else?" Where is programX installed? Synaptic knows.

when i want to do something myself, instead of using Synaptic, i usually end up confused..

Compiling and installing a program for system wide use, and doing so outside the confines of the package manager can be a little disorganized, for sure. It's easy enough to make install, but then uninstalling is a crapshoot. For this instance, I'd recommend getting used to checkinstall.

For the rest though, unless a program is not in the package database, it's a lot easier to just let the package manager deal with it.

Linux is different. If one don't want different, maybe one really does in fact want to be using Windows instead. It just seems to me to be a little lopsided to leave Windows for Linux to try to make Linux more like Windows.

Now I could obviously be reading way too much into your post. Maybe you don't like Windows either. Maybe you don't like how any OS does it. In that case, Linux, due to its open nature, is the perfect candidate for creating a system more to your liking, or hoping someone else does. Perhaps GoboLinux is your thing. But don't expect every distro to do that.

Im confused.... is Linux now the choice for freedom or is that Windows?

I see Linux as the freedom lovers choice and i would like the choice of dumping my games in /home/loki/games/... which i do, as my wife has no need to Quake 3 anyway. If she wanted she could have her own install as well with her own mods.

Or if i want a common install why not drop it under /usr/games or whatever, or for that matter /games (eeek!!)... its my computer, its my OS, its my choice.

Package management certainly should be able to (and always presumed it did if you provide the right switches) allow you to dump where you want (providing you have the rights to the directory where you want to install).

Quite a few constructive comments already, thanks guys and lady - that's exactly what I was aiming for. It's one thing to come up with an idea, it's another to be flexible and open enough to take in any criticism (valid or not) and deal with it. I'll address the two repeating issues here, and will address any specific questions in replies to their respective comments.

First I'd like to point out to everyone that this is supposed to be the utopia of program management. Utopia kind of implies that it is impossible to achieve fully. Take any utopia, and you'll see there's massive overlooking of details, or simply not answering certain pressing questions as to ignore certain weaknesses (Earth in Star Trek, anyone?). So, certain things are most likely never fixable or addressable.

What about shared libraries?

This is the most valid of criticisms, and also the hardest to properly address. A typical utopian solution is this: common libraries go into /System, and any special or uncommon libraries go into the program bundle they belong to. Nice utopia, of course, but this raises the question: what is common, and what isn't? I can imagine the operating system vendor/programmers making that decision - they can provide a base set of libraries from the get go, and then listen to developer feedback as to which libraries they like and want in the basic set of libraries.

But you all have a very valid point: this is one of those things that requires balancing on a very thin line. For a company like Apple, something like this is easy to solve (Apple is god in their world), for Microsoft it's a lot more difficult, and for the open source world it's nigh on impossible.

Why not put /Settings in /Users/User 1?

Because that goes against my personal (it's MY utopia, after all ) idea about what the home directory should contain: a user's home directory is for files like photos, documents, pr0n - not for settings or config files. I always find it extremely clumsy to see stuff in my home directory that I would rarely manually touch anyway. I created the separate /Settings hierarchy for this purpose. Settings files != user docs, simply put.

Like I said, this is my utopia, and it might very well be that just about anyone else would like /Settings in their home directory - I don't know. A few of you proposed something like /Users/User 1/Settings - I could probably live with that .

Having had a personal conversation with Thom (more than a few times) and also having had thoughts about this sort of thing, I'm convinced that part of the problem with the article is Thom has unintentionally assumed people fully understand how BeOS/Haiku attributes work, and has left some things unconnected between what he wrote in this article and in his mind, compared to what's going through the mind of the readers.

When used correctly, and with a copy utility that's attribute-aware, the attributes of BeOS/Haiku make it feasible and practical to not get your panties all in a twist as to the exact human-readable directory structure, because when used correctly, you can think of a combination of attributes as an explicitly set directory path that can be set in any particular order. Indeed, attributes are stored in separate indexes which are a form of filesystem directory to start with, that you don't typically see, at least... not without the native index listing tools.

Where am I going with this? Well, think of it like this: each added attribute makes it more readily found for any particular file attached to an application and/or user, and they aren't mutually exclusive. Here is an example of how things could be tied between applications and users:

1. All applications have a human-friendly name: for all applications and their settings files, add the attribute to the files called "ApplicationName" and fill it accordingly.

2. All applications *should* have a version: for all applications and their settings files, add the attribute "ApplicationVersion" and fill it accordingly.

3. All applications have users, even if it is only the System: for all applications and their settings files, add the attribute "User" and fill it accordingly.

4. Someone above mentioned different hosts: for grins, we do exactly the same thing! Ditto for filling that accordingly.

5. Sometimes you want to experiment with different settings: explicitly have a separate date attribute for this purpose, and by default, use the latest dated files: this provides for versioning of settings at will. Ditto for filling that accordingly.

6. For external library dependencies, we do something similar: the library has its "ApplicationName" filled with itself, and lists all applications with their versions as their "users" and we thus get a nice symmetrical solution for handling all that the same way. As soon as there are no users of that particular library, it's safe to blow it away. If you want to blow it away, by running the query, you can tell exactly which applications/libs also depend on it, and which users depend on those, and deal with it accordingly.

By doing a query along the lines of:

finding application name A
With latest version
With user name B
With Host name C
With latest version of settings D
With Libs defined by attributes E

You can explicitly choose to copy or execute any version of any application on the system, assuming you have permissions. Because permissions also exist on the attributes itself, you can hide details from the user where they don't have the rights, and inform those running a query, "One or more dependencies exist for this" that they can't change. The nice thing is because how attributes work, they effectively provide a symbolic link for each time you add one, to their respective information, but it's location-independent: as long as the attribute indexes are built on each volume, the location of the file that has all these attributes (and it'd have to match for each one: because of all the differences in attributes, if you have two different users with otherwise identical attributes, they get two separate files) can be anywhere on the filesystem, so Thom can have his organizational method, and you can have yours, if you so desire: perhaps Thom hasn't convinced you of the wisdom of having a Settings directory outside your home directory, so you have a Settings directory as a subdirectory of your home directory. Regardless of which one you do, you *still* need to have an attribute-aware copy/backup program, and it still requires the filesystem properly index and work with attributes.

An added note: for those libs which are badly abused for names where the names stay the same, but different actual versions end up getting put over them, each application should have a set of "Uses" libs files for each one in its directory that lists the MD5 checksum for each one, so the application (or the application loader) can detect that there's been a change, and at least warn the user of that.

While all these attributes do indeed provide a heck of a lot of power to make things more manageable without an explicit package manager, here are the costs:

1. The user must always have a rather intelligent copy facility that's aware of this information and how it is used.

2. Application developers need to use this internally: this is clearly not POSIX-compliant (for the current state of POSIX) and thus ported applications aren't going to have this if they're simply recompiles.

3. This requires fully indexed attribute storing filesystems: this isn't the problem for the attributes, as Windows NT-based NTFS has attributes, though I don't recall whether they've got their own indexes outside the standard file metadata.

4. Working with applications and files with all these attributes has a time overhead for all the disk seeks: once we move away from rotating media, this likely will be a non-issue, but it exists for now for most non-Flash-based filesystems.

It's called utopia for a reason ... It can never exist in reality , the H factor always screw up things.

H = Humans.

-----

I will pass on commenting on it , I wrote a 500 page in 10 minute about all it's flaws and unrealistic behaviors.

But then who am I to stop progress just upload it to a server so that we can download it and test it .. What's that it's just a theory ?

Also in Utopia there is no need to install , upgrade manually , uninstal software. Software are Free and gratis and you only pay for development or to buy a new computer and they come with 10 X 1 million terrabyte hard drive.

First sorry for the poor english (its not my native language). And I am not shure how this ascii written directory "diagram works" on osnews forum... this is my first post here althou I have been reading osnews for years

/programs/ (For system and all users, programs here)
nodep/programsname-version (must hav no dep! and respect home/settings saving! Otherwice free to custom at will)
defsettings/ (default settings that are copied to users settings when first run by user)
programsname-version/
bin/ (programs binary)
lib/ (programs personal libaries ONLY (no general other use))
defsettings/ (default settings that are copied to users settings when first run by user)
other/ (pictures, sounds etc related to the program)
deps/
strict/ (must have with strick version)
lib/
libname-versision/libfile-version (LINK or FILE see note 1a)
other/
programsname-version/ (LINK or copy of the entire program see note 1b)
musthave/ (must have)
lib/
libname-versision/libfile-version (LINK or FILE see note 2a)
other/
programsname-version/ (LINK or copy of the entire program see note 2b)
optionaly/ (that good to have(more functions), but not mandatory)
lib/
libname-versision/libfile-version (LINK or FILE see note 3a)
other/
programsname-version/ (LINK or copy of the entire program see note 3b)

User and Group directories
/home/username/
programs/ (for the user only own instable programs)
programsname-version LINK#
settings
programsname-version LINK# (settigs for the program that its linked for)

Virtual links from users home and group directories!
/shared/groups/ (shared groups that can be made by the system and run by the groups privigles and rules see note 4)
wingames/ (exsample of "wingames" group that is shared by some users)
programs/sharesave/programsname-version (programs that have/save bin/settings in one directory)
newgames/programs/programsname-version LINK% (program that is shared, but has settings in settings folder)
settings/programsname-version LINK%

Packet manager takes care of all programs excluding /programs/nodep and /shared/grops/programs(basicly just links)

/home/user/programs (if not othervice setup)

(note 1a/2b)
(if the system has it, its linked! otherwice its copied here in the install prosess or if the system changes/deletes the its copied here automaticly)

(note 2a/2b)
(if the system has it, its linked! Otherwice its copied here in the install prosess or if the system changes(question is made: "Do a copy here or relink?"). If Deleted then its copied automaticly here.

(LINK or FILE see note 3a/3b)
((if the system has it it linked! otherwice its asked what to do) and if changed only version numbering LINK is updated and if deleted System asks shall it be copied here or not?(default is no).

The idea of having so many program folder is to have real choise how software can be installed and still keep the system in order.

Perhaps you should check out Plan 9 and OpenVMS for more related ideas. VMS has some advanced FS characteristics inc. versioning and the ability to treat a file as a series of records, so you can choose to refer to a particular field or not within the FS. This includes copying parts of files over LAN's, etc. Incidentally, you will notice MS has picked up quite a few ideas from VMS; search for both to read about their shared heritage, if you're interested. I suppose MS will support versioning soon, since it's been around for 20+ years! Maybe they'll just leapfrog it with WinFS (or whatever they'll end up calling it).

Plan 9 gives each user their own virtualised FS: each user can mount (remote or local) resources as they see fit, creating their own personal FS namespace. This extends beyond files and folders to almost every network- or locally-accessible object. The Plan 9 site is down for me, but you can read a bit about it here: http://www.faqs.org/faqs/comp-os/plan9-faq/

In the medium-long term, I see more FS' dropping the hierarchical `root -> folder 1 -> ... -> folder n' construction (it is only a design choice, after all, not a divine mandate!) and moving to a database-driven FS complete with rich attributes etc.

(Oh, and what about ZeroInstall? That achieves most of what you want, on a system of today. http://0install.net/ )
Thoughts?

I like self contained bundles. Shared libraries are a mess that has never really been sorted out properly. I don't think the security implications of having 15 copies of the same exploitable library in 15 different application bundles is all that big a deal.

I would like the program bundle specification and layout to be platform independent, and "fat" - for example a single bundle could contain Linux, OSX and Windows binaries. Yes, this will make for bigger bundles, but it will also make things simpler. Fundamentally people shouldn't have to know what operating system they are running. They should be able to download a program and have it "just work". Vendors would always have the option of creating single platform bundles if size is an issue.

I am not sure I get your file layout though. /Settings/UserX? How about /Users/UserX/Settings. And why the single global /Programs? Fundamentally any program should be installable by a single user without requiring any privilege escalation. We should have something like:

For shared programs, /Settings would be populated at install time with a default settings file provided in the program bundle. When a user first runs the program the settings file for that bundle will be copied to their local /Users/UserX/Settings. This allows admins to provide global default settings. The bundle could also split it's settings across multiple settings files, and mark some settings files as read only, so that they are never copied as user settings and are not editable by an unprivileged user.

As for program updates, I like the global repository idea, but I'd also like things to be a bit distributed. How about the program bundle provide a URL that always resolves to the latest version of the program bundle? This could be a central repository URL, or a developer's website.

How about making a "configuration" the user interface object the user interacts with, rather than a "program"? So you add, run, and remove a particular configuration of a program, rather than the program itself.

e.g. I add a "configuration" of Firefox ("Firefox 3 + loads of plugins") to my desktop. It's like an AppDir, except it contains the settings, plus a link to the program.

If I want to add another one (e.g. "Firefox 2 with no plugins" for, say, on-line banking) then I add one of those too. I now have two icons / menu entries in my GUI, one for each configuration.

Deleting a configuration loses its settings, which therefore no longer need to be hidden.

Programs themselves can be installed and garbage collected automatically. No need to bother the user about that. Shared libraries, dependency hell, multiple versions, automatic updates, non-root install, etc are easily solved for most software (http://0install.net, as others have mentioned already).