Imagine that I am using a stripped down puppy and want to add a program which needs certain libraries. Those libs are expected to be in, say, usr/lib. However, some other static program may already be using usr/lib as storage for its own libs, and those libs may share the same name as the libs for the new program.

Is it possible to somehow "isolate" the location of certain libs so that a static program is "sandboxed" and unable to be contaminated by an incorrect lib from the host core, or some other program?Last edited by greengeek on Wed 02 Jan 2013, 04:11; edited 1 time in total

Thanks RSH, that looks very interesting. I'd like to see some fork of Puppy use this sort of approach and maybe build it's own repository. Doesn't matter if it is compatible with nothing else. I think it would allow a small core and lots of add-ons that were basically static packages, safe to run without risking conflicts with other packages.

Ok, I was misunderstanding this. I thought a static program simply loaded all of its necessary libs into the filesystem when it ran for the first time. My thinking was that this would potentially cause conflicts with other statics and I was visualising some way that each static could keep its necessary libs completely separate from the filesystem. (and just output data to a "safe" file or location/partition or display window)

Almost like the system core would be capable of running several VMs - each containing some static program (rather than an OS).

This is sort of how I visualise an sfs to be operating (like a VM), except that I think an sfs becomes part of a unified file system - which seems to carry the burden of ensuring that each sfs is compatible with all other programs running in the overall "unified filesystem".

A statically-linked program means that the libraries are built right into the program, so libs aren't even looked for anywhere else.

I'm definitely confused by this. I've been looking at several "static" programs and none of the ones I looked at carry a full load of libs - they may have some of their own libs within their specific directory, but the programs also seem to want to go looking for other libs that are either part of puppy itself, or else have been loaded by other pets or by me manually.

What I was hoping for was a mechanism to have a program running completely standalone (as much as possible anyway...) so that it did not need to access any libs that have been loaded by other programs or pets.

Maybe the "statics" I have been looking at are incorrectly assembled? Pseudo-statics maybe?

Well, I think I know what you are saying, although using the term "static programs" is confusing, since (as amigo has already pointed-out), it makes one think of statically-linked programs, which create no such problems with searching for libraries, since the necessary code is already in the executable binary. But you seem to be looking for something else.

I think you are actually referring to programs that use shared objects (e.g., shared libraries) and have one or more of their own libraries segregated from the general collections of libraries used by other programs. Right?

greengeek wrote:

Those libs are expected to be in, say, usr/lib. However, some other static program may already be using usr/lib as storage for its own libs, and those libs may share the same name as the libs for the new program.

So if I understand you correctly, you are talking about a situation where a program uses a library that has the same name as a library that already exists in the /usr/lib/ directory, but is totally different.

If that is the case, then, yes, there are ways of handling that situation.

The ideal, of course, would be for the developer of the library to come up with a new name. But in a world with thousands of people developing libraries, such name conflicts are inevitable, and convincing one developer that her library should be the one that is renamed, and not the other, may not be easy.

So you get to work around the conflict.

As you have already guessed, you will create a new directory in which to put the library. Let's call it "/usr/local/lib/example_app/". And let's call the library "libexample.so.0", and the program "example_app".

So, in this example, you would have an existing library at /usr/lib/libexample.so.0 and a different library for use by example_app at /usr/local/lib/example_app/libexample.so.0 .

There are various methods that can be used to tell the loader to look in the latter directory at run time.

1. RUNPATH

If you have the source code, you can set the RUNPATH attribute at build time. This attribute is used to define a library search path and lives in the executable file, where the loader will find it at run time. Here is a simple example (which is all one long line):

Unfortunately, although this is perhaps the best way to do what you want, this probably won't work for you in Puppy because many (perhaps all?) Puppies set the LD_LIBRARY_PATH environmental variable, which overrides RUNPATH. (I do not know why Puppy does this. The usual way to define the general library search path is to use the /etc/ld.so.conf file and ldconfig. LD_LIBRARY_PATH is usually just used temporarily for stuff like testing new libraries and debugging.)

2. RPATH

This is an older method of defining a library search path in the executable, using the RPATH attribute. This method is deprecated, since it cannot be overridden by LD_LIBRARY_PATH, which caused many headaches in the past because sometimes it is useful to be able to override the path. So it was replaced by RUNPATH, which is overridable. (Actually, the -rpath combined with the --enable-new-dtags options sets both RUNPATH and RPATH, but if RUNPATH is set, modern loaders ignores RPATH.)

RPATH is set similarly to the way RUNPATH was set, but without the --enable-new-dtags option:

The above code is simplified to illustrate the options that need to be passed to the linker. But in the real world you are unlikely to be calling gcc directly to build a source package. You will probably need to either pass them to a configuration script, or set them manually in a Makefile (for instance, using LDFLAGS). Of course, the exact details of doing so will depend on the source package.

Because it is deprecated, I would not recommend that anyone use this method on a large-scale basis, but to solve an occasional problem like your library name conflict on Puppy, it could be useful.

By the way, the following command will show you if RPATH or RUNPATH are set in a program:

Code:

readelf --dynamic example_app | grep -E "RPATH|RUNPATH"

3. LD_LIBRARY_PATH

This might be what you need if you have only the executable binary, not the source code, and therefore cannot set the RUNPATH or RPATH variables at build time.

As previously mentioned, LD_LIBRARY_PATH is usually just used temporarily for stuff like testing new libraries and debugging, but if the RUNPATH and RPATH methods are not available to you, this could do the job. For example, on the command line:

Note that the the new LD_LIBRARY_PATH also includes the old LD_LIBRARY_PATH, but the new directory is first in the path so that it is searched first. Of course, if all the needed libraries are in the new directory, or in /lib/ or /usr/lib/, or in /etc/ld.so.cache, you don't need the old value of LD_LIBRARY_PATH, and can simply use "LD_LIBRARY_PATH=/usr/local/lib/example_app".

Note also that the above examples cause LD_LIBRARY_PATH to be set to its new value only for running example_app. You would not want to set it permanently for a couple of reasons: first, because doing so would cause the loader to search the new directory first for libraries when running any program, which would be inefficient, and second, because the other library with the same name at /usr/lib/libexample would never be found when it was needed, since the one in the new directory would always be found first.

Be aware that this method will not work with programs that have their set-uid or set-gid permission bits set (unless, respectively, the user ID matches the file's owner or the user is a member of the file's group). (Allowing users to cause their own libraries to be loaded in such programs would, obviously, be a very large security hole.)

4. LD_PRELOAD

If you are just dealing with one or a few libraries, you could use this method to list those specific libraries so that the loader will load them first. It works similarly to LD_LIBRARY_PATH except that you list the absolute name of each library, not just the directory. (Normally this is used for stuff like testing a new library.)

As was the case with LD_LIBRARY_PATH, this LD_PRELOAD method won't work with programs that have their set-uid or set-gid permission bits set (unless, respectively, the user ID matches the file's owner or the user is a member of the file's group).

Further reading - - -

I've tried to give a brief overview of some of your options, but I've probably not explained it well enough to make it really clear. Here are a few suggestions that may help to clear things up:

Let's get the definitions here. There is no such thing as a 'static program'. The correct term is statically-linked, although sometimes statcially-compiled is used.
It is possible to statically link only parts of the libs which any program needs. A completely statically-linked program will contain *all* the libs it uses whatsoever -meaning that the linker will not look for any libs outside the program. Obviously, if you used only statically-linked programs, there would be much duplication of code in your programs -each of them would be larger than the same program dynamically-linked (which means the linker looks for and uses *shared* libs found on your system (*.so libs).
LD_PRELOAD and LD_LIBRARY_PATH are used by the linker for searching for *shared* libs.
The 'linker' is the file /lib/ld-linux.so.2. The kernel does not directly execute programs -instead it calls the linker to do that. The linker looks at the first few bytes of the 'program' (or library) and figures out whether it needs any shared libs or not. If so, then it loads these external shared libs (if it finds them) before loading the actual program.

It's helpful to think of static-linking as 'including' -anything so linked will be included right into the executable.

I've heard the terms ROX-app and application-in-a-directory thrown around. ( file:///usr/local/apps/ROX-Filer/Help/Manual.html#appdirs ) I believe these are applications that contain all the necessary stuff in one directory so they don't depend on shared libraries and are easy to use and delete without affecting anything else.

Sharing libraries sounds like a good idea until you run into applications that require different versions of a library. I didn't even know there were different libraries that have the same name.

Thank you all for the very helpful and indepth info. There is a lot for me to digest and research further.

npierce wrote:

Well, I think I know what you are saying, although using the term "static programs" is confusing, since (as amigo has already pointed-out), it makes one think of statically-linked programs, which create no such problems with searching for libraries, since the necessary code is already in the executable binary. But you seem to be looking for something else.
I think you are actually referring to programs that use shared objects (e.g., shared libraries) and have one or more of their own libraries segregated from the general collections of libraries used by other programs. Right?

I think what I was confused about is there seem to be plenty of “statically-linked” programs that DO in fact require shared libraries, even though it seems that by strict definition that should not be the case - they should already be isolated. Just because someone has labelled a program as xxxx_static.gz doesn’t mean they have done the job as fully as maybe should be the case. Several programs that I have tried recently have been labelled as statics but ended up needed quite a number of other shared libraries being grafted in (because I’m trialling pupngo which has almost no shared libs left...)

Quote:

So if I understand you correctly, you are talking about a situation where a program uses a library that has the same name as a library that already exists in the /usr/lib/ directory, but is totally different.

Yes, that is the problem I am trying to avoid. Heres an example: Lets say I want to start with a stripped down Puppy (pupngo) and add only two other programs - a word processor and a browser: Imagine that both of those programs are labelled as being “statically linked”, yet both initially fail to run because each is looking for something called libXt.so.6 (which suggests to me that they are not fully statically linked...). I can grab a similar named lib from another puppy, but I might find one thats not exactly the right name - like libXt.so.6.1.2, - so I might symlink the missing file to the “appropriate” lib and the programs might run.

However, I have no way of knowing if those programs were intended to run with libXt.so.6.1.2 (which is what they are going to actually be running with...) or whether the person who turned them into “statics” was using other slightly different libs.

There is no such thing as 'glibc2'. libc is a generic name for any of the c-lib alternatives. libc5 was the old original (circa Slackware 8.0 and older). When the glibc(GNU libc) project was started, they used the alternate name 'libc6' since it was not binary-compatrible with libc5.

I'm wondering if someone hasn't been confusing glibc2 with glib2??

Anyway, the whole idea of upgrading glibc without doing a complete upgrade is bound to fail -at some point. Nerly everything on your system uses glibc -fully-statically-linked programs being the exception (like /sbin/init -the real one, which Puppy does not have).
If you simply upgrade glibc and not all the rest, then you run the risk of having old programs not be compatible with the new glibc -resulting in hard-to-diagnose failures.

If you need a newer glibc for a certain program, then place the newer glibc in some out-of-the-way path and write a wrapper for the program which adds LD_LIBRARY_PATH to point to the newer library. This avoids all conflicts.

What I really want to aim for is to insulate each program from the others (despite the potential duplications and wastage of space), so that there is minimal risk of a program grabbing a lib that may have been incorrectly symlinked etc.

I feel I can probably avoid issues if I do the following:
a) restrict myself to a very small puppy with very few programs
b) use only truly statically linked programs.

Not so easy for someone as inexperienced as me.

Of course, different people seem to have different views of the merits of statically linking programs:
(I’m just adding some links here that I want to come back to as I look into this)

In fact, even two libraries of the same version can be incompatible -according to the compile-time options chosen. One version may include a capability (implemented as extra code or inclusion of other externals) which the second lib does not have. If the program you want to run was compiled against lib #1, it will be expecting that specific function or functions. lib #2, failing that extra function will not always satisfy the programs real needs.

This is why picking and choosing libraries from just anywhere you find them will not always work and is a Bad Thing (TM). Properly upgrading a system means re-building everything which depends on something else -each time the something else gets upgraded (or re-compiled with different options which affect compatibility.) Since there are hundreds of programs which all depend on at least glibc, upgrading to a newer glibc means recompiling all those programs -whether or not upgrading those programs at the same time.

Also, about RoxApp's and other bundle/app/single-dir "packages". They commonly will depend on some exterior shared libraries being available on the system -including, most commonly, glibc.

The problem with having every program being completely independent from external libs, is that many, many programs will include a copy of some libs. If every program that needs glibc included the static library of those functions, then any system would be hugely bigger. If every GUI app included all of glibc, xlibs&Co. and GTK-plus-hangers, you'd have lots of apps being really huge. On the other hand, statically-compiled programs usually start faster -but use lots more RAM than shared-lib counterparts.

Anyway, apps usually *do* depend on some expected libs/progs -possibly with a specific version or range of versions.

OP, your best way to package nice, (mostly) self-contained apps, while avoiding duplication, would be to package any needed external libs which you do *not* normally expect to be installed as a separate library AppDir. This would also allow you to install and use alternate versions of already-installed libraries which you want to to 'over-ride', without over-writing. The AppRun script for the program which needs these alternate libs would simply set the LD_LIBRARY_PATH to point to the alternate location.

Puppy`s devx file is a good example of add-on libraries and apps.
There could be a few more for specific purposes, like media, etc.

But as amigo points out, all apps. that use them need to be compiled with them.
You can`t just install a programming language package to Puppy.
Expecting it to work as is,... Maybe yes, and maybe no.
If it uses libs. in the devx file or Puppy, they may not play well together.

That`s why I`ve had very good luck with Lucid Puppy.
It`s based on Ubuntu Lucid and using Ubuntu Lucid apps. works great!
I hope Precise Puppy gets it`s bugs worked out so there`s an upgrade path.

On the other hand, statically-compiled programs usually start faster -but use lots more RAM than shared-lib counterparts.

Why would a statically compiled program use more RAM? I can easily see why you would need more RAM if you had a number of statically compiled programs all running at once (they would take up less ram if they shared common ground...) but wouldn't a single program need pretty much the same amount of ram to accomplish it's functions, regardless of static or no?

What I'm aiming for is a puppy with a small core (eg pupngo) and very little else - in fact I'd be happy with a word processor as the only other add on, if it allowed me to make use of the older PII PCs I'm trying to restore. If a statically compiled word processor is going to require significantly more RAM I'm probably barking (get it?) up the wrong tree.

amigo wrote:

Bad Thing (TM)

<chuckle> so many of the things I've stumbled through with Puppy fit into the "BadThing" category. Much trial and error and lots of swearing.Last edited by greengeek on Mon 31 Dec 2012, 21:06; edited 1 time in total

You cannot post new topics in this forumYou cannot reply to topics in this forumYou cannot edit your posts in this forumYou cannot delete your posts in this forumYou cannot vote in polls in this forumYou cannot attach files in this forumYou can download files in this forum