Scute tests fail in Jenkins "native" configuration
Closed, ResolvedPublic

Description

Since May 14, 2017, most Scute tests systematically FAIL when Scute is build by Jenkins in the "native" configuration. Here is an excerpt from the build output for May 14:

03:23:15 PASS: t-link
03:23:16 PASS: t-getfunctionlist
03:23:16 PASS: t-initialize
03:23:16 PASS: t-getinfo
03:23:16 FAIL: t-getslotlist
03:23:16 FAIL: t-getslotinfo
03:23:16 FAIL: t-gettokeninfo
03:23:16 FAIL: t-getmechanismlist
03:23:16 FAIL: t-getmechanisminfo
03:23:16 FAIL: t-opensession
03:23:16 FAIL: t-closeallsessions
03:23:16 FAIL: t-getsessioninfo
03:23:16 FAIL: t-findobjects
03:23:16 FAIL: t-getattribute
03:23:16 FAIL: t-auth
03:23:16 FAIL: t-generaterandom

According to some (quick) tests I have done, the only situation where I could get the t-getslotlist test to FAIL is if Scdaemon is not available on the system.

With Scdaemon present, and without a card reader, we should expect the t-getslotlist and t-getslotinfo tests to PASS and all the subsequent tests to SKIP. That's what I observe on my own test system, and that's also we used to have on Jenkins in all builds prior to May 14, such as this May 7 build:

03:21:47 PASS: t-link
03:21:47 PASS: t-getfunctionlist
03:21:48 PASS: t-initialize
03:21:48 PASS: t-getinfo
03:21:48 PASS: t-getslotlist
03:21:50 PASS: t-getslotinfo
03:21:50 SKIP: t-gettokeninfo
03:21:50 SKIP: t-getmechanismlist
03:21:50 SKIP: t-getmechanisminfo
03:21:50 SKIP: t-opensession
03:21:50 SKIP: t-closeallsessions
03:21:50 SKIP: t-getsessioninfo
03:21:50 SKIP: t-findobjects
03:21:50 SKIP: t-getattribute
03:21:50 SKIP: t-auth
03:21:50 SKIP: t-generaterandom

So I am guessing that something must have changed on the system where Jenkins performs its builds, sometimes between May 7 and May 14.

Could someone investigate that? Has Scdaemon been removed from the Jenkins system?

Is it possible to get the tests/test-suite.log file (generated by the tests) from a Jenkins build, in order to understand what caused the tests to fail (if it's not, as a I suspect, due to the absence of Scdaemon)?

gouttegd created this task.Jul 11 2017, 5:03 PM
gouttegd created this object in space S1 Public.
justus triaged this task as High priority.Jul 11 2017, 5:19 PM
justus claimed this task.

I see several problems here:

  1. The tests do not set GNUPGHOME, hence ~/.gnupg is used, and there may or may not be already an agent listening on it. (And indeed, there was, and there is no scdaemon installed in /usr/lib/gnupg/scdaemon).
  2. Scute now hard-codes the path to gpg-connect-agent, determined at configure-time. At configure-time, in our build environment, the only gpg-connect-agent that is available is the one in /usr/bin, but when the tests are run, the correct one is in PATH. (I changed the PATHs in May, so this may be the cause of the current trouble, but I haven't looked at the previous logic). In any case, obeying PATH would be nice.

All build artifacts are accessible

I obviously missed that, thanks!

The tests do not set GNUPGHOME

OK, I will fix that. I will do the same as what is done in GnuPG, i.e. set GNUPGHOME to /bin/pwd (current directory where the tests are run). Is that OK?

I changed the PATHs in May, so this may be the cause of the current trouble, but I haven't looked at the previous logic

The previous logic did not use gpg-connect-agent at all. It attempted to connect to the GPG Agent indicated by the GPG_AGENT_INFO variable, and spawned an agent in pipe mode if it could not find the socket (also using a hard-coded path to gpg-agent). Then it was (and still is, under the current logic) up to the agent to start Scdaemon if needed.

In any case, obeying PATH would be nice.

OK, I will do that. The user will still be allowed to specify a hard-coded path at configure time if he wants to, but the default behavior will be to search for gpg-connect-agent in PATH.

gouttegd closed this task as Resolved.Jul 12 2017, 12:53 PM

I've just pushed the two fixes. GNUPGHOME is now set to the tests directory when running the tests and gpg-connect-agent is now looked for in PATH at runtime.

When the tests are run, Scute now contacts the agent intended for the tests instead of any agent running on behalf of the Jenkins user. And so the tests pass or skip appropriately.

Thanks Justus for pointing out the cause of the problem.

"gouttegd (Damien Goutte-Gattat)" <noreply@dev.gnupg.org> writes:

I've just pushed the two fixes. `GNUPGHOME` is now set to the tests directory when running the tests and `gpg-connect-agent` is now looked for in `PATH` at runtime.

When the tests are run, Scute now contacts the agent intended for the tests instead of any agent running on behalf of the Jenkins user. And so the tests pass or skip appropriately.

Cool, thanks :)