diff --git a/misc/jenkins/README.org b/misc/jenkins/README.org index 3cb7b84..4555109 100644 --- a/misc/jenkins/README.org +++ b/misc/jenkins/README.org @@ -1,89 +1,208 @@ * Notes +** Overview +|----------------------+--------+---------+-----------+-----+-----------| +| Configuration Matrix | native | in-tree | sanitizer | w32 | distcheck | +|----------------------+--------+---------+-----------+-----+-----------| +| arch | x | | | | | +| debian | x | | | | | +| macos | x | | | | | +| master | | x | x | x | x | +| openbsd60 | x | | | | | +|----------------------+--------+---------+-----------+-----+-----------| + +There are two dimensions, build host ("label") and build profile +("XTARGET"). The build hosts are described below. The "debian" label +is the same configuration as "master". + +bin/build.bash is the build script. It creates a suitable build +environment, builds, tests, and installs all our packages. The +different build profiles are implemented there. +*** Build profiles +**** native +A straight forward out-of-tree build. +**** in-tree +A straight forward in-tree build. +**** sanitizer +A build with -fsanitize=undefined -fsanitize=address. This catches +many memory errors by instrumenting the code and running the test +suites. +**** w32 +Cross-compile the package for Windows, run the tests using a virtual +machine. +**** distcheck +Executes 'make distcheck'. Makes sure that we can always create +releases. ** Setting up a Jenkins build slave - on soro, create an entry in /etc/hosts - copy root@soro's ssh key to /root/.ssh/authorized_keys - install a jre, make, autoconf, automake, libtool, gcc, git, bison, fig2dev, ghostscript, gnutls, sqlite3, pkg-config, imagemagick, - rngd, python2/3, SWIG, Qt5 base + rngd, python2/3, SWIG, Qt5 base, ccache - setup rngd (test suites will consume quite a bit of entropy) - create a user jenkins - clone gnupg-doc $ git clone git://git.gnupg.org/gnupg-doc.git - link ~/bin $ ln -s gnupg-doc/misc/jenkins/bin - download slave.jar $ wget https://jenkins.gnupg.org/jnlpJars/slave.jar -O bin/slave.jar + Note: + The jar should be updated from time to time, but the documentation + says that the protocol changes rarely. - copy and adapt launcher $ cp bin/jenkins-slave.dist bin/jenkins-slave - make sure that jenkins@soro can ssh to the new node - go to https://jenkins.gnupg.org/computer/new and copy an existing configuration, adapting it as needed - setup 'GPGME tests for GnuPG' as described below - for each project, add the new nodes distinct label to the configuration matrix, and force a rebuild. Start with libgpg-error and walk your way up the dependency chain: - libgpg-error, libnpth, libassuan, libksba, libgcrypt, ntbtls, gnupg, gpgme ** GPGME tests for GnuPG There is a reasonably up-to-date (but this is currently a manual process) GPGME source at $ mkdir $HOME/src $ git clone git://git.gnupg.org/gpgme.git $HOME/src/gpgme-for-gnupgs-tests $ cd $HOME/src/gpgme-for-gnupgs-tests $ ./autogen.sh and a build tree at $ mkdir $HOME/src/gpgme-for-gnupgs-tests/obj $ cd $HOME/src/gpgme-for-gnupgs-tests/obj $ export PATH=$HOME/prefix/native/bin:$PATH $ ../configure --enable-maintainer-mode $ make $ make check and specialized build trees, e.g. for the sanitizer target at $ mkdir $HOME/src/gpgme-for-gnupgs-tests/obj-sanitizer $ cd $HOME/src/gpgme-for-gnupgs-tests/obj-sanitizer $ export PATH=$HOME/prefix/sanitizer/bin:$PATH $ ../configure --enable-maintainer-mode \ --enable-languages="cpp qt" \ CFLAGS="-fsanitize=undefined -fsanitize=address" \ CXXFLAGS="-fsanitize=undefined -fsanitize=address" $ make $ make check and a w32 build tree at $HOME/src/gpgme-for-gnupgs-tests/obj.w32 The tests from there are executed in GnuPG's test suite. -* Virtual machines -** openbsd60 +* Build hosts +** zygalski +Werner manages this box. +** Virtual machines +*** openbsd60 Packages installed: # pkg_add zile zsh git autoconf-2.69p2 automake-1.15p0 gettext-tools \ gmake xfig bison readline libusb-compat ImageMagick makeinfo \ - gcc-4.9.3p9 g++-4.9.3p9 qt5 + gcc-4.9.3p9 g++-4.9.3p9 qt5 python swig ccache Add some compatibility links to PATH: $ mkdir -p ~/compat/{bin,lib,include} $ cd ~/compat/bin $ ln -s /usr/local/bin/gmakeinfo makeinfo $ [ add $HOME/compat/bin to ~/.profile ] $ cd ~/compat/lib $ for F in /usr/local/lib/libbz2* /usr/local/lib/libiconv* /usr/local/lib/libintl* ; do ln -s $F ; done $ cd ~/compat/include $ for F in /usr/local/include/*bz*h /usr/local/include/*iconv*h /usr/local/include/*intl*h ; do ln -s $F ; done Tweak limits: # echo 'jenkins:\ :maxproc-max=1024:\ :maxproc-cur=1024:\ :tc=pbuild: ' >> /etc/login.conf # user mod -L jenkins jenkins +*** win8.1 +Configuration: One user "gpg", disabled Windows update (cpu hog), +disable animations ("make computer easier to see again"). + +Installed applications: https://github.com/PowerShell/Win32-OpenSSH/releases/latest +**** How tests are executed +To run the test suite, the machine is reverted to the snapshot +'tests', and the tests are executed in-place from an ISO image. + +See: + - bin/run-tests-w32.bash + - bin/run-tests.bat + - bin/make-windows-cd.sh +**** Win32-OpenSSH +The implementation seems a bit brittle at the moment. Often, the ssh +server will stop responding to requests, I do not know why. + +To update the ssh server, follow +https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH +first uninstall the old one, then install the new one. + +**** Updating & maintenance +Get a lock on bin/run-tests-w32.bash to avoid it stomping over your +changes: + + jenkins@soro:~$ flock /var/lib/jenkins/bin/run-tests-w32.bash bash + +Start the machine using + + jenkins@soro:~$ virsh -c qemu:///system snapshot-revert --snapshotname tests --force --running win8.1 + +Connect to the machine from your desktop machine: + + you@home $ virt-viewer -c qemu+ssh://jenkins@soro.g10code.com/system win8.1 + +Do whatever maintenance work is necessary. Shutdown the machine. +Create a new snapshot 'test-new': + + jenkins@soro:~$ virsh -c qemu:///system snapshot-create-as win8.1 --name "tests-new" --description "Updated OpenSSH to xxx" + +Archive the current snapshot: + + jenkins@soro:~$ virsh -c qemu:///system snapshot-edit win8.1 --snapshotname "tests" --rename + [... editor pops open, change "tests" + to "tests YYY-MM-DD", save, exit ...] + +Note: The snapshots creation times can be found using: + + jenkins@soro:~$ virsh -c qemu:///system snapshot-list win8.1 | grep tests + tests 2017-03-15 14:21:17 +0100 shutoff + tests 2017-01-31 2017-01-31 11:05:17 +0100 shutoff + +Rename the new snapshot: + + jenkins@soro:~$ virsh -c qemu:///system snapshot-edit win8.1 --snapshotname "tests-new" --rename + [... editor pops open, change "tests-new" + to "tests", save, exit ...] + +Exit the shell to release the lock: + + jenkins@soro:~$ exit + exit + +Voila. +**** Ideas + - Build the installer, put it on the ISO image, and test that as well. +*** openindiana20161030 +So I wanted the most alien UNIX I could get my hands on. I never +configured the build environment though, so this machine lies dormant. +**** Packages installed +pkg install pkg://openindiana.org/runtime/java/openjdk8 top git autoconf automake libtool bison readline +*** archlinux +**** Packages installed +pacman --sync zile bind-tools openssh zsh jre8-openjdk-headless git autoconf automake libtool make wget gcc bison fig2dev ghostscript gnutls sqlite3 pkg-config imagemagick librsvg rng-tools python swig qt5-base +**** Upgrading packages +Note: Some breakage can happen when upgrading the system. Arch users +deal with that by reading the website and following instructions +there. + +pacman -Syu