Relink to shared libs

Relink to shared libs

Why, if I make a small change to my shared library, does cmake relink all of my tests to it? It's a shared library, isn't the point that it doesn't need relinking? Seems like a big waste of time. Can I suppress this in any way?

Re: Relink to shared libs

Hi,

On Fri, Aug 3, 2012 at 8:41 PM, Leif Walsh <[hidden email]> wrote:
> This is sort of the point of shared libs though, you can update them without relinking everything on your system, unless you're doing something really brain-damaged.

Actually no, adding new public API, changing existing API in
compatible ways are things you can do with a shared library which
needs relinking. Changing the implementation is just one of the things
you can do. The point is simply, there's no way for the buildsystem to
find that out and no cmake does not provide ways to disable dependency
tracking, if you want that simply don't use a buildsystem with
dependency tracking - for example a shell script.

Re: Relink to shared libs

On 3 Aug, 2012, at 4:44 PM, Andreas Pakulat <[hidden email]> wrote:
>
> Actually no, adding new public API, changing existing API in
> compatible ways are things you can do with a shared library which
> needs relinking.

Not without changing the header files, which forces recompilation. As the author of a shared library, your header files are your contract with your clients. If you change the library in such a way that requires relinking, it's your responsibility to update the contract. Everyone in the world does this and it works just fine.

My point is that, unless you are deliberately trying to be really sneaky, there's pretty much no way you can introduce a change that requires relinking but not recompiling of binaries. I am not being sneaky so this is a gigantic waste of my time.

> Changing the implementation is just one of the things
> you can do. The point is simply, there's no way for the buildsystem to
> find that out and no cmake does not provide ways to disable dependency
> tracking, if you want that simply don't use a buildsystem with
> dependency tracking - for example a shell script.

The build system could make the reasonable assumption that a binary does not need to be relinked with a changed shared library, if that's the only thing that's changed, or this assumption could be turned on as a feature, maybe with a policy. Most other build systems have this assumption. Tracking dependencies through header files is a much smarter way to do it, and cmake already professes to do this. If cmake cannot turn off this misfeature, it needs to.

Re: Relink to shared libs

For the makefiles I get gcc to output dependency files based on header file usage, using gcc options -MM -MF -MP -MT.

This has worked really well for over a year. If I modify a cpp file that is used by the shared lib, then the executable does not re-link. If I modify a .h file that is used by the shared lib *and* the executable, then it WILL re-link.

Can I confirm that I cannot replicate this functionality with cmake ?

If so then this may be a blocker for CMAKE adoption , since we have long build times, and dont want to make them longer.

Re: Relink to shared libs

David,

I have had some build time success by adopting the ninja generator. It's got some dependency issues but it's workable, and though it doesn't solve the relinking issue, it does make builds generally a lot faster.

> I agree with Leif Walsh....
>
> I am currently looking at replacing my unix makefile (linux+gmake,
> solaris+make/gmake) and Windows VS projects, with cmake.
>
> For the makefiles I get gcc to output dependency files based on header file
> usage, using gcc options -MM -MF -MP -MT.
>
> This has worked really well for over a year. If I modify a cpp file that is
> used by the shared lib, then the executable does not re-link. If I modify a
> .h file that is used by the shared lib *and* the executable, then it WILL
> re-link.
>
> Can I confirm that I cannot replicate this functionality with cmake ?
>
> If so then this may be a blocker for CMAKE adoption , since we have long
> build times, and dont want to make them longer.
>
> regards. David
>
>
>
> --
> View this message in context: http://cmake.3232098.n2.nabble.com/Relink-to-shared-libs-tp7580943p7582156.html> Sent from the CMake mailing list archive at Nabble.com.
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake

Re: Relink to shared libs

> I am currently looking at replacing my unix makefile (linux+gmake,
> solaris+make/gmake) and Windows VS projects, with cmake.
>
> For the makefiles I get gcc to output dependency files based on header file
> usage, using gcc options -MM -MF -MP -MT.
>
> This has worked really well for over a year. If I modify a cpp file that is
> used by the shared lib, then the executable does not re-link. If I modify a
> .h file that is used by the shared lib *and* the executable, then it WILL
> re-link.

I'm not sure that's correct behavior. What if the modification to the
.cpp file was to remove the definition of a function declared in a
header? Now your executable that was using that function will crash when
you try to run it due to a missing symbol. If you had re-linked, you
might instead get a link error letting you know something is wrong.

You could probably modify CMake to generate build rules that do not
treat linked libraries as dependencies for a library/application link
step, but I don't think I would consider that correct behavior.

Re: Relink to shared libs

> I'm not sure that's correct behavior. What if the modification to the
> .cpp file was to remove the definition of a function declared in a
> header? Now your executable that was using that function will crash when
> you try to run it due to a missing symbol. If you had re-linked, you
> might instead get a link error letting you know something is wrong.

Yes, but this is a very rare thing to do. And the "failure" result is
just that the problem would be discovered at test time instead of link
time.

What's being proposed here is a massive optimization for the common case
(changing code in a library and not changing the API or tests) by
sacrificing a little bit of latency (the time between link and test) in
the extremely rare case.

I wouldn't think this would be a hard tradeoff to make. Could it be added
at least as an option?

Re: Relink to shared libs

On 10/25/2012 1:17 PM, Leif Walsh wrote:

> Yes, but this is a very rare thing to do. And the "failure" result is
> just that the problem would be discovered at test time instead of link
> time.
>
> What's being proposed here is a massive optimization for the common case
> (changing code in a library and not changing the API or tests) by
> sacrificing a little bit of latency (the time between link and test) in
> the extremely rare case.
>
> I wouldn't think this would be a hard tradeoff to make. Could it be added
> at least as an option?

CMake errs on the side of being correct. If you do not want to relink,
you can just build the library target directly with the /fast option or
without it.

Let's say you have a project with shared library foo and executable bar.
You have four options:

make foo // build library foo and anything foo depends on

make foo/fast // build just foo and nothing it depends on

make bar // build bar and anything it depends, with a possible relink
when foo changes

make // build all targets in the project, will relink bar when foo changes

Re: Relink to shared libs

Is there anything that would convince kitware that the behavior I describe is worth pursuing, even as a non-default option? I'd implement it myself but your code is foreign to me and I bet someone at kitware knows the exact line that needs to change to enable this.

The reason I'm pushing so hard on this is that my project builds several libraries and each is tested individually, and furthermore, each library builds a shared and a static library and there are a few tests that test the static library but most test the shared library. Obviously the statically linked tests cannot use the optimization I described.

So if I'm developing, I can't just build one library target always, I may need to build different library targets depending on what code I'm modifying, and in most cases I need to build multiple library targets and a few statically linked tests.

Having to do this manually is a big pain, and the natural place to do it is right in the build system, since it already has all the information needed for these decisions.

For me, this is a big regression from Make and Autotools. Not big enough to switch back, certainly, but very big.

> On 10/25/2012 1:17 PM, Leif Walsh wrote:
>> Yes, but this is a very rare thing to do. And the "failure" result is
>> just that the problem would be discovered at test time instead of link
>> time.
>>
>> What's being proposed here is a massive optimization for the common case
>> (changing code in a library and not changing the API or tests) by
>> sacrificing a little bit of latency (the time between link and test) in
>> the extremely rare case.
>>
>> I wouldn't think this would be a hard tradeoff to make. Could it be added
>> at least as an option?
> CMake errs on the side of being correct. If you do not want to relink, you can just build the library target directly with the /fast option or without it.
>
> Let's say you have a project with shared library foo and executable bar. You have four options:
>
> make foo // build library foo and anything foo depends on
>
> make foo/fast // build just foo and nothing it depends on
>
> make bar // build bar and anything it depends, with a possible relink when foo changes
>
> make // build all targets in the project, will relink bar when foo changes
>
>
> --
> Bill Hoffman
> Kitware, Inc.
> 28 Corporate Drive
> Clifton Park, NY 12065
> [hidden email]> http://www.kitware.com> 518 881-4905 (Direct)
> 518 371-3971 x105
> Fax (518) 371-4573
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake

Re: Relink to shared libs

I can't see why you couldn't use set(CMAKE_SKIP_RPATH ON) and post
build copy your library to a path that's on your linker path; have
your tests setup to use the system copy of the library instead of the
local copy.

> This is sort of the point of shared libs though, you can update them without relinking everything on your system, unless you're doing something really brain-damaged.
>
> Is there any way I, as someone who knows my libraries are sane, can prevent this extra work and just resort to a "make clean" if I really screw something up?
>
> Sent from my iPhone
>
> On Jul 20, 2012, at 8:28, Andreas Naumann <[hidden email]> wrote:
>
>> If you are using the Makefile system, then the libraries are newer than your tests, so your tests seems to need an update.
>>
>> And sometimes programs need relinking, where should the build system now, if you really need the relinking?
>>
>>
>> Am 20.07.2012 14:08, schrieb Leif Walsh:
>>> Why, if I make a small change to my shared library, does cmake relink all of my tests to it? It's a shared library, isn't the point that it doesn't need relinking? Seems like a big waste of time. Can I suppress this in any way?
>>>
>>> Sent from my iPhone
>>> --
>>>
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html>>>
>>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.cmake.org/mailman/listinfo/cmake>>>
>>>
>>
>> --
>>
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html>>
>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.cmake.org/mailman/listinfo/cmake> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake

Re: Relink to shared libs

> I can't see why you couldn't use set(CMAKE_SKIP_RPATH ON) and post
> build copy your library to a path that's on your linker path; have
> your tests setup to use the system copy of the library instead of the
> local copy.
>
> ~
> Doug.
>
> On Sat, Aug 4, 2012 at 2:41 AM, Leif Walsh <[hidden email]> wrote:
>> This is sort of the point of shared libs though, you can update them without relinking everything on your system, unless you're doing something really brain-damaged.
>>
>> Is there any way I, as someone who knows my libraries are sane, can prevent this extra work and just resort to a "make clean" if I really screw something up?
>>
>> Sent from my iPhone
>>
>> On Jul 20, 2012, at 8:28, Andreas Naumann <[hidden email]> wrote:
>>
>>> If you are using the Makefile system, then the libraries are newer than your tests, so your tests seems to need an update.
>>>
>>> And sometimes programs need relinking, where should the build system now, if you really need the relinking?
>>>
>>>
>>> Am 20.07.2012 14:08, schrieb Leif Walsh:
>>>> Why, if I make a small change to my shared library, does cmake relink all of my tests to it? It's a shared library, isn't the point that it doesn't need relinking? Seems like a big waste of time. Can I suppress this in any way?
>>>>
>>>> Sent from my iPhone
>>>> --
>>>>
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html>>>>
>>>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.cmake.org/mailman/listinfo/cmake>>>
>>> --
>>>
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html>>>
>>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.cmake.org/mailman/listinfo/cmake>> --
>>
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html>>
>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.cmake.org/mailman/listinfo/cmake

+ // Check whether we should skip dependencies on shared library files.
+ this->LinkDependsNoShared =
+ this->Target->GetPropertyAsBool("LINK_DEPENDS_NO_SHARED");
+
// On platforms without import libraries there may be a special flag
// to use when creating a plugin (module) that obtains symbols from
// the program that will load it.
@@ -650,7 +654,11 @@ void cmComputeLinkInformation::AddItem(std::string const& item, cmTarget* tgt)

Re: [PATCH] Optionally skip link dependencies on shared library files

On 10/26/2012 9:43 AM, Leif Walsh wrote:
> This is exactly what I wanted, and the results were incredible. Thank
> you.
If you want to thank us... Please help out and create a test for this
so we can put it in the next release. The test should build a shared
library and link it to an exe, modify the shared library .c, rebuild and
verify the .exe is still older than the .so or .dll. It would be great
if you could take the time to create a test for this.

Re: [PATCH] Optionally skip link dependencies on shared library files

> On 10/26/2012 9:43 AM, Leif Walsh wrote:
>> This is exactly what I wanted, and the results were incredible. Thank
>> you.
> If you want to thank us... Please help out and create a test for this so we can put it in the next release. The test should build a shared library and link it to an exe, modify the shared library .c, rebuild and verify the .exe is still older than the .so or .dll. It would be great if you could take the time to create a test for this.
>
> Thanks.
>
> -Bill
>
>
> --
> Bill Hoffman
> Kitware, Inc.
> 28 Corporate Drive
> Clifton Park, NY 12065
> [hidden email]> http://www.kitware.com> 518 881-4905 (Direct)
> 518 371-3971 x105
> Fax (518) 371-4573