diff --git a/misc/jenkins/README.org b/misc/jenkins/README.org index 4555109..e484893 100644 --- a/misc/jenkins/README.org +++ b/misc/jenkins/README.org @@ -1,208 +1,216 @@ * Notes ** Overview |----------------------+--------+---------+-----------+-----+-----------| | Configuration Matrix | native | in-tree | sanitizer | w32 | distcheck | |----------------------+--------+---------+-----------+-----+-----------| | arch | x | | | | | | debian | x | | | | | | macos | x | | | | | | master | | x | x | x | x | | openbsd60 | x | | | | | |----------------------+--------+---------+-----------+-----+-----------| There are two dimensions, build host ("label") and build profile ("XTARGET"). The build hosts are described below. The "debian" label is the same configuration as "master". bin/build.bash is the build script. It creates a suitable build environment, builds, tests, and installs all our packages. The different build profiles are implemented there. *** Build profiles **** native A straight forward out-of-tree build. **** in-tree A straight forward in-tree build. **** sanitizer A build with -fsanitize=undefined -fsanitize=address. This catches many memory errors by instrumenting the code and running the test suites. **** w32 Cross-compile the package for Windows, run the tests using a virtual machine. **** distcheck Executes 'make distcheck'. Makes sure that we can always create releases. ** Setting up a Jenkins build slave - on soro, create an entry in /etc/hosts - copy root@soro's ssh key to /root/.ssh/authorized_keys - install a jre, make, autoconf, automake, libtool, gcc, git, bison, fig2dev, ghostscript, gnutls, sqlite3, pkg-config, imagemagick, rngd, python2/3, SWIG, Qt5 base, ccache - setup rngd (test suites will consume quite a bit of entropy) - create a user jenkins - clone gnupg-doc $ git clone git://git.gnupg.org/gnupg-doc.git - link ~/bin $ ln -s gnupg-doc/misc/jenkins/bin - download slave.jar $ wget https://jenkins.gnupg.org/jnlpJars/slave.jar -O bin/slave.jar Note: The jar should be updated from time to time, but the documentation says that the protocol changes rarely. - copy and adapt launcher $ cp bin/jenkins-slave.dist bin/jenkins-slave - make sure that jenkins@soro can ssh to the new node - go to https://jenkins.gnupg.org/computer/new and copy an existing configuration, adapting it as needed - setup 'GPGME tests for GnuPG' as described below - for each project, add the new nodes distinct label to the configuration matrix, and force a rebuild. Start with libgpg-error and walk your way up the dependency chain: - libgpg-error, libnpth, libassuan, libksba, libgcrypt, ntbtls, gnupg, gpgme ** GPGME tests for GnuPG There is a reasonably up-to-date (but this is currently a manual process) GPGME source at $ mkdir $HOME/src $ git clone git://git.gnupg.org/gpgme.git $HOME/src/gpgme-for-gnupgs-tests $ cd $HOME/src/gpgme-for-gnupgs-tests $ ./autogen.sh and a build tree at $ mkdir $HOME/src/gpgme-for-gnupgs-tests/obj $ cd $HOME/src/gpgme-for-gnupgs-tests/obj $ export PATH=$HOME/prefix/native/bin:$PATH $ ../configure --enable-maintainer-mode $ make $ make check and specialized build trees, e.g. for the sanitizer target at $ mkdir $HOME/src/gpgme-for-gnupgs-tests/obj-sanitizer $ cd $HOME/src/gpgme-for-gnupgs-tests/obj-sanitizer $ export PATH=$HOME/prefix/sanitizer/bin:$PATH $ ../configure --enable-maintainer-mode \ --enable-languages="cpp qt" \ CFLAGS="-fsanitize=undefined -fsanitize=address" \ CXXFLAGS="-fsanitize=undefined -fsanitize=address" $ make $ make check and a w32 build tree at $HOME/src/gpgme-for-gnupgs-tests/obj.w32 The tests from there are executed in GnuPG's test suite. * Build hosts ** zygalski Werner manages this box. ** Virtual machines *** openbsd60 Packages installed: # pkg_add zile zsh git autoconf-2.69p2 automake-1.15p0 gettext-tools \ gmake xfig bison readline libusb-compat ImageMagick makeinfo \ gcc-4.9.3p9 g++-4.9.3p9 qt5 python swig ccache Add some compatibility links to PATH: $ mkdir -p ~/compat/{bin,lib,include} $ cd ~/compat/bin $ ln -s /usr/local/bin/gmakeinfo makeinfo $ [ add $HOME/compat/bin to ~/.profile ] $ cd ~/compat/lib $ for F in /usr/local/lib/libbz2* /usr/local/lib/libiconv* /usr/local/lib/libintl* ; do ln -s $F ; done $ cd ~/compat/include $ for F in /usr/local/include/*bz*h /usr/local/include/*iconv*h /usr/local/include/*intl*h ; do ln -s $F ; done Tweak limits: # echo 'jenkins:\ :maxproc-max=1024:\ :maxproc-cur=1024:\ :tc=pbuild: ' >> /etc/login.conf # user mod -L jenkins jenkins *** win8.1 Configuration: One user "gpg", disabled Windows update (cpu hog), disable animations ("make computer easier to see again"). Installed applications: https://github.com/PowerShell/Win32-OpenSSH/releases/latest **** How tests are executed To run the test suite, the machine is reverted to the snapshot 'tests', and the tests are executed in-place from an ISO image. See: - bin/run-tests-w32.bash - bin/run-tests.bat - bin/make-windows-cd.sh **** Win32-OpenSSH The implementation seems a bit brittle at the moment. Often, the ssh server will stop responding to requests, I do not know why. To update the ssh server, follow https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH first uninstall the old one, then install the new one. **** Updating & maintenance Get a lock on bin/run-tests-w32.bash to avoid it stomping over your changes: jenkins@soro:~$ flock /var/lib/jenkins/bin/run-tests-w32.bash bash Start the machine using jenkins@soro:~$ virsh -c qemu:///system snapshot-revert --snapshotname tests --force --running win8.1 Connect to the machine from your desktop machine: you@home $ virt-viewer -c qemu+ssh://jenkins@soro.g10code.com/system win8.1 -Do whatever maintenance work is necessary. Shutdown the machine. -Create a new snapshot 'test-new': +Do whatever maintenance work is necessary. + +Hints: + - Start a powershell: + Open the file explorer, enter 'powershell' into the location bar (ctrl-l). + - Right mouse button pastes in powershell. + - Elevate privileges in powershell: + > Start-Process powershell -verb runAs + +Shutdown the machine. Create a new snapshot 'test-new': jenkins@soro:~$ virsh -c qemu:///system snapshot-create-as win8.1 --name "tests-new" --description "Updated OpenSSH to xxx" Archive the current snapshot: jenkins@soro:~$ virsh -c qemu:///system snapshot-edit win8.1 --snapshotname "tests" --rename [... editor pops open, change "tests" to "tests YYY-MM-DD", save, exit ...] Note: The snapshots creation times can be found using: jenkins@soro:~$ virsh -c qemu:///system snapshot-list win8.1 | grep tests tests 2017-03-15 14:21:17 +0100 shutoff tests 2017-01-31 2017-01-31 11:05:17 +0100 shutoff Rename the new snapshot: jenkins@soro:~$ virsh -c qemu:///system snapshot-edit win8.1 --snapshotname "tests-new" --rename [... editor pops open, change "tests-new" to "tests", save, exit ...] Exit the shell to release the lock: jenkins@soro:~$ exit exit Voila. **** Ideas - Build the installer, put it on the ISO image, and test that as well. *** openindiana20161030 So I wanted the most alien UNIX I could get my hands on. I never configured the build environment though, so this machine lies dormant. **** Packages installed pkg install pkg://openindiana.org/runtime/java/openjdk8 top git autoconf automake libtool bison readline *** archlinux **** Packages installed pacman --sync zile bind-tools openssh zsh jre8-openjdk-headless git autoconf automake libtool make wget gcc bison fig2dev ghostscript gnutls sqlite3 pkg-config imagemagick librsvg rng-tools python swig qt5-base **** Upgrading packages Note: Some breakage can happen when upgrading the system. Arch users deal with that by reading the website and following instructions there. pacman -Syu