Project

General

Profile

Bug #570

CMake binaries with shared libs won't run correctly without installing (Ubuntu CMake 2.6.2)

Added by Teemu Murtola about 9 years ago. Updated about 9 years ago.

Status:
Closed
Priority:
Normal
Category:
mdrun
Target version:
Affected version - extra info:
Affected version:
Difficulty:
uncategorized
Close

Description

On my Ubuntu box with CMake 2.6.2 and shared libraries turned on, it appears that CMake sets the RPATH in the built binaries to the installation directory:

$ mkdir cmake-build
$ cd cmake-build
$ cmake [path-to-gromacs-source]
...
$ make -j4
...
$ readelf -d src/kernel/mdrun | grep RPATH
0x000000000000000f (RPATH) Library rpath: [/usr/local/gromacs/lib]
$ src/kernel/mdrun -h
src/kernel/mdrun: error while loading shared libraries: libgmxpreprocess.so.6: cannot open shared object file: No such file or directory

This means that if I run a binary from the build tree without installing, either

1) Gromacs is not yet installed; I get a dynamic linking error.
2) An older version if Gromacs is installed in the same prefix or in the standard search path; the binary links against the old library.

Setting LD_LIBRARY_PATH manually solves the problem (at least in the first case, which I tried). While this is not a problem for users, it can make development very inconvenient. As it is now, I couldn't figure out a way to run tests portably without first installing the libraries (or turning shared libraries off). There has to be some solution to this in CMake, but I couldn't find one with a quick Google search. Or is the solution just to require a newer CMake, or might it even be Ubuntu-specific?


Related issues

Blocked by GROMACS - Bug #549: install-mdrun rule does not install shared libs (CMake)Closed09/06/2010
Blocks GROMACS - Bug #423: no CPack supportClosed06/01/2010

Associated revisions

Revision 36133dbe (diff)
Added by Teemu Murtola about 9 years ago

Use CMake install for install-mdrun (bugs #549, #555)

The custom target install-mdrun now invokes the cmake_install.cmake
script that is generated by CMake to perform the actual installation.
Adds support for DESTDIR and RPATH rewriting (see bug #570).

To implement this, all installation targets are now grouped into
components: "data" for shared data, "development" for headers and
pkgconfig files, "libraries" for libraries needed by mdrun, "mdrun" for
mdrun itself, and "runtime" for other binaries and libraries. Only
"mdrun" and "libraries" are used at the moment (the latter only when
installing with shared libraries).

Haven't tested this with Visual Studio, but if it doesn't work, it
should be easy to adapt.

Revision f7fd40ea (diff)
Added by Szilárd Páll about 9 years ago

now it's possible to run binaries in the build tree (fixes bug #570)

Changed CMAKE_BUILD_WITH_INSTALL_RPATH to FALSE which results in the
rpath being rewritten on while installing binaries. CPack also works.

History

#1 Updated by Szilárd Páll about 9 years ago

What I didn't know is that with autotools even if you set --enable-shared, the binaries in the build tree are static and they get rebuilt/linked at install time. If this is not possible with some magic option in CMake than I think it would be quite a bit difficult to implement.

Mostly because of this is I always turn BUILD_SHARED_LIBS=off and I wonder whether it makes sense to have it on by default - at least in the development tree.

#2 Updated by Teemu Murtola about 9 years ago

(In reply to comment #1)

What I didn't know is that with autotools even if you set --enable-shared, the
binaries in the build tree are static and they get rebuilt/linked at install
time. If this is not possible with some magic option in CMake than I think it
would be quite a bit difficult to implement.

With autotools, the binaries are not static, but there is a lot of libtool magic that may relink the executable when it is executed and/or installed to make it find the correct libraries. From what I gathered from some googling, it seems that CMake should support rewriting the RPATH during installation out-of-the-box from 2.6 upwards, but that the implementation was bad in the beginning and some distributions might have disabled it for that reason. If someone can quickly test whether this works on some other system and/or with a newer version of CMake, we could narrow down the problem. On the machines where I normally do my development, none have a newer CMake, so it would take some additional effort from me, but I can do that if no one has time.

Although disk space during compilation is probably not a problem in most cases, I think getting shared libraries to work properly would still be preferable.

#3 Updated by Szilárd Páll about 9 years ago

(In reply to comment #2)

With autotools, the binaries are not static, but there is a lot of libtool
magic that may relink the executable when it is executed and/or installed to
make it find the correct libraries. From what I gathered from some googling,

$ [path-to-gmx-src]/configure --enable-shared && make mdrun -j6
[...]
$ ldd src/kernel/mdrun
not a dynamic executable
$ du src/kernel/mdrun
8.0K src/kernel/mdrun

So it's obviously not a static binary, but not a dynamic either. Very strange magic...

it seems that CMake should support rewriting the RPATH during installation
out-of-the-box from 2.6 upwards, but that the implementation was bad in the
beginning and some distributions might have disabled it for that reason. If
someone can quickly test whether this works on some other system and/or with a
newer version of CMake, we could narrow down the problem. On the machines
where I normally do my development, none have a newer CMake, so it would take
some additional effort from me, but I can do that if no one has time.

I've just tried with CMake 2.8.0 and it does not seem to work:
$ readelf -d src/kernel/mdrun | grep RPATH
0x000000000000000f (RPATH) Library rpath: [/usr/local/gromacs/lib]

Although disk space during compilation is probably not a problem in most cases,
I think getting shared libraries to work properly would still be preferable.

Sure! This makes me wonder though if I should spend more time on the "make install-mdrun" implementation at all, as it seems to require lots of tedious steps to get everything right with shared libs.

#4 Updated by Teemu Murtola about 9 years ago

(In reply to comment #3)

$ ldd src/kernel/mdrun
not a dynamic executable
$ du src/kernel/mdrun
8.0K src/kernel/mdrun

So it's obviously not a static binary, but not a dynamic either. Very strange
magic...

Yes, it's strange, and quite a bit tricky to run, e.g., through valgrind or gdb if one doesn't know how (the trick is to use "libtool --mode=execute" on the wrapper script), but it does work. The real binary is in src/kernel/mdrun/.libs/mdrun, and the one in src/kernel is just a wrapper shell script that possibly relinks the binary before executing it.

it seems that CMake should support rewriting the RPATH during installation
out-of-the-box from 2.6 upwards, but that the implementation was bad in the
beginning and some distributions might have disabled it for that reason. If
someone can quickly test whether this works on some other system and/or with a
newer version of CMake, we could narrow down the problem. On the machines
where I normally do my development, none have a newer CMake, so it would take
some additional effort from me, but I can do that if no one has time.

I've just tried with CMake 2.8.0 and it does not seem to work:
$ readelf -d src/kernel/mdrun | grep RPATH
0x000000000000000f (RPATH) Library rpath:
[/usr/local/gromacs/lib]

This looks bad. I assume that the binary does not run either without setting LD_LIBRARY_PATH. If there isn't a magic CMake switch that we have omitted, it seems that it's very difficult to do anything reasonable with shared libraries when it comes to, e.g., unit testing. Writing a lot of platform-specific wrapper scripts just for running "make test" isn't a good way (I have now one that I use)... But other big projects like KDE have to have the same problem, so there must be some way around it, I just haven't figured out what it could be.

Sure! This makes me wonder though if I should spend more time on the "make
install-mdrun" implementation at all, as it seems to require lots of tedious
steps to get everything right with shared libs.

I think that would still be useful and separate from the issues here, because as it is, the "make install-mdrun" seems to be a source of a lot of errors for users. In contrast, there are very few cases where users would see the problems discussed in this bug.

#5 Updated by Szilárd Páll about 9 years ago

(In reply to comment #4)

Yes, it's strange, and quite a bit tricky to run, e.g., through valgrind or gdb
if one doesn't know how (the trick is to use "libtool --mode=execute" on the
wrapper script), but it does work. The real binary is in
src/kernel/mdrun/.libs/mdrun, and the one in src/kernel is just a wrapper shell
script that possibly relinks the binary before executing it.

Thanks for enlightening me! :)

This looks bad. I assume that the binary does not run either without setting
LD_LIBRARY_PATH. If there isn't a magic CMake switch that we have omitted, it
seems that it's very difficult to do anything reasonable with shared libraries
when it comes to, e.g., unit testing. Writing a lot of platform-specific
wrapper scripts just for running "make test" isn't a good way (I have now one
that I use)... But other big projects like KDE have to have the same problem,
so there must be some way around it, I just haven't figured out what it could
be.

Sure, it does not. It might be worth a shot writing to the CMake developer list, they seemed to be quite helpful.

I think that would still be useful and separate from the issues here, because
as it is, the "make install-mdrun" seems to be a source of a lot of errors for
users. In contrast, there are very few cases where users would see the
problems discussed in this bug.

Sure, what I was referring to is just that if we fix mdrun to run in the build tree it will get (even) more complicated to install it with "make install-mdrun".

#6 Updated by Teemu Murtola about 9 years ago

Ah, how stupid of me of not looking more carefully at our CMake scripts. Apparently, there is a magic CMake switch (or several, actually) that you explicitly need to set if you don't want to be able to run the binaries from the build tree, which avoids changing the executable at installation time. Erik introduced all these in the end of August, in commit bbc612e5, that just says "Added CPack support". It's possible to override them on a per-target basis, but is there a reason why the global setting should always be the one it is now?

#7 Updated by Teemu Murtola about 9 years ago

If/when the patch in bug #549 gets applied, issues with install-mdrun should no longer influence this bug. This bug would be fixed by simply removing the following lines (or even only the line with CMAKE_BUILD_WITH_INSTALL_RPATH) from the master CMakeLists.txt:
if(NOT CMAKE_SYSTEM_NAME STREQUAL "Darwin")
set(CMAKE_SKIP_BUILD_RPATH FALSE)
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
set(CMAKE_INSTALL_RPATH "${LIB_INSTALL_DIR}")
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
endif()
I can see two possible problems: 1) a minor speed penalty from rewriting the RPATH at installation time, and 2) if CPack doesn't support RPATH rewriting, it won't work. If 2) isn't a problem, I don't understand how the above lines are related to CPack support (see comment #6), and if it is, I would argue that that is a bug in CPack.

Note that the intent of the above lines is not clear: SKIP_BUILD_RPATH is set to FALSE (which by itself would fix this bug), but it is never used because BUILD_WITH_INSTALL_RPATH overrides it.

#8 Updated by Szilárd Páll about 9 years ago

(In reply to comment #7)

If/when the patch in bug #549 gets applied, issues with install-mdrun should no
longer influence this bug. This bug would be fixed by simply removing the
following lines (or even only the line with CMAKE_BUILD_WITH_INSTALL_RPATH)
from the master CMakeLists.txt:
if(NOT CMAKE_SYSTEM_NAME STREQUAL "Darwin")
set(CMAKE_SKIP_BUILD_RPATH FALSE)
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
set(CMAKE_INSTALL_RPATH "${LIB_INSTALL_DIR}")
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
endif()
I can see two possible problems: 1) a minor speed penalty from rewriting the
RPATH at installation time, and 2) if CPack doesn't support RPATH rewriting, it
won't work. If 2) isn't a problem, I don't understand how the above lines are
related to CPack support (see comment #6), and if it is, I would argue that
that is a bug in CPack.

Note that the intent of the above lines is not clear: SKIP_BUILD_RPATH is set
to FALSE (which by itself would fix this bug), but it is never used because
BUILD_WITH_INSTALL_RPATH overrides it.

Works for me with with set(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE), the rpath of binaries are identical in both installed and CPack-ed as well as with the option TRUE and FALSE. ...and the binaries in the build tree run and use the libs from the build tree!

I will commit the fix.

#9 Updated by Szilárd Páll about 9 years ago

(In reply to comment #8)

(In reply to comment #7)

If/when the patch in bug #549 gets applied, issues with install-mdrun should no
longer influence this bug. This bug would be fixed by simply removing the
following lines (or even only the line with CMAKE_BUILD_WITH_INSTALL_RPATH)
from the master CMakeLists.txt:
if(NOT CMAKE_SYSTEM_NAME STREQUAL "Darwin")
set(CMAKE_SKIP_BUILD_RPATH FALSE)
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
set(CMAKE_INSTALL_RPATH "${LIB_INSTALL_DIR}")
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
endif()
I can see two possible problems: 1) a minor speed penalty from rewriting the
RPATH at installation time, and 2) if CPack doesn't support RPATH rewriting, it
won't work. If 2) isn't a problem, I don't understand how the above lines are
related to CPack support (see comment #6), and if it is, I would argue that
that is a bug in CPack.

Note that the intent of the above lines is not clear: SKIP_BUILD_RPATH is set
to FALSE (which by itself would fix this bug), but it is never used because
BUILD_WITH_INSTALL_RPATH overrides it.

Works for me with with set(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE), the rpath of
binaries are identical in both installed and CPack-ed as well as with the
option TRUE and FALSE. ...and the binaries in the build tree run and use the
libs from the build tree!

I will commit the fix.

Fixed in commit f7fd40ea

Also available in: Atom PDF