Fix jenkins build system
Open, WishlistPublic

Description

There is a problem with the build script that we use in Jenkins regarding the handling of build dependencies. Currently, it manifests itself as a build failure of the GPGME project on OpenBSD [0], though it works on the other platforms purely by chance.

0: https://jenkins.gnupg.org/job/gpgme/XTARGET=native,label=openbsd60/781/console

The build script works as follows. The build is parametrized, the two dimensions being 'build host' and 'flavor' (for historical (ha!) reasons called 'XTARGET'). For every XTARGET, there exists a directory $HOME/prefix/$XTARGET where the build artifacts for that flavor are installed into. I will use $PREFIX as an shortcut for $HOME/prefix/$XTARGET. The build now consists of these steps:

  1. the source tree is cleaned
  2. the build directory is configured with PATH=$PREFIX/bin:$PATH
  3. the project is built with PATH=$PREFIX/bin:$PATH
  4. the project is tested with PATH=$PREFIX/bin:$PATH, LD_LIBRARY_PATH=$PREFIX/lib
  5. the project is installed to $PREFIX

There are several problems with that. The most pressing ones afaics are

  1. When building X, the build artifacts from the previous build of X are already installed in $PREFIX. Therefore, the tests may pick up those instead of using the artifacts from the build directory. This has led to problems in the past.
  2. The problem causing [0] is that step 3 is executed with PATH set to $PREFIX/bin, but LD_LIBRARY_PATH *not* being set to $PREFIX/lib. On our OpenBSD box this means that $PREFIX/bin/gpg fails to start because it cannot find a suitable libgcrypt. On the other boxes this works by chance because the one in /usr is compatible.

Discussion.

Problem 1. can be addressed by creating a "cleaner" build environment by some means. This is a challenge because we'd like it to be portable. Maybe stow.

I have no idea what to do about 2. Some ideas are:

  1. Avoid including $PREFIX/bin in PATH. This means that we won't pick up gpg-error-config and friends and have to hard-code the configure options in our build script (or use pkg-config....). Maybe create a trimmed PREFIX that does only contain the *-config scripts. But the hacks never end :(
  2. Include $PREFIX/lib in LD_LIBRARY_PATH. This is very problematic because quite some programs that we use in the build depend on "our" libraries (ghostscript, /bin/ps (yes... thanks systemd)) and they will break when our libraries are compiled with -fsanitize=X.

Using LD_LIBRARY_PATH at all is highly problematic by itself. Quoting myself:

# HACKHACKHACK:
#
# Because newer Debian toolchains prefer RUNPATH over RPATH, and
# RUNPATH has lower precedence than LD_LIBRARY_PATH, we need to
# explicitly add libtool's .libs directory:
case "$JOB_NAME" in
  *gnupg*)
    if [ "${XTEST_GPGME_BUILDDIR}" ]; then
	test_environment="LD_LIBRARY_PATH=${XTEST_GPGME_BUILDDIR}/src/.libs:${XTEST_GPGME_BUILDDIR}/lang/cpp/src/.libs:${XTEST_GPGME_BUILDDIR}/lang/qt/src/.libs:$ORIGINAL_PREFIX/lib"
    fi
    ;;
  *gpgme*)
    test_environment="LD_LIBRARY_PATH=$(pwd)/obj/src/.libs:$(pwd)/obj/lang/cpp/src/.libs:$(pwd)/obj/lang/qt/src/.libs:$ORIGINAL_PREFIX/lib"
    ;;
  *)
    test_environment="LD_LIBRARY_PATH=$(pwd)/obj/src/.libs:$ORIGINAL_PREFIX/lib"
    ;;
esac
#
# If we don't do this, the version tests fail because the runtime
# linker will pick up the library from LD_LIBRARY_PATH.  Also, testing
# the installed version is not what we want ofc.
#
# KCAHKCAHKCAH

Ugh.

Related Objects

StatusAssignedTask
OpenNone
OpenNone
justus created this task.May 4 2017, 10:22 AM

I don't have the full picture here, but maybe this gives you some fresh ideas:

About problem 1: Am I understanding it correctly that all builds share an install tree that is continuiously updated by all builds in all projects (for the same flavor)? Shouldn't each component with dependencies be tested against "known stable" versions of the dependencies? And then maybe add some integration tests for an "all master" build? Isolating builds from each other seems to be a good strategy here, but maybe I am misunderstanding the current system.

About problem 2: In the old days there was a linker option RPATH that makes it possible to put the path of a library into an executable, so LD_LIBRARY_PATH is not needed (I don't recommend RPATH in deployment, but for a test suite it might work).

In T3133#96706, @marcus wrote:

I don't have the full picture here, but maybe this gives you some fresh ideas:

About problem 1: Am I understanding it correctly that all builds share an install tree that is continuiously updated by all builds in all projects (for the same flavor)?

Yes.

Shouldn't each component with dependencies be tested against "known stable" versions of the dependencies? And then maybe add some integration tests for an "all master" build? Isolating builds from each other seems to be a good strategy here, but maybe I am misunderstanding the current system.

We could do that, yes.

About problem 2: In the old days there was a linker option RPATH that makes it possible to put the path of a library into an executable, so LD_LIBRARY_PATH is not needed (I don't recommend RPATH in deployment, but for a test suite it might work).

Yes, that would work, but I have not figured out how to make the linker use RPATH as opposed to using RUNPATH. See the comment in the shell fragment.

werner renamed this task from Fix build system to Fix jenkins build system.May 30 2017, 3:06 PM
justus triaged this task as Wishlist priority.Jun 7 2017, 10:47 AM