make check fails for libgcrypt on Apple Silicon / ARM Mac
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	mbirth
	Nov 28 2020, 11:43 PM

Description

On a Macbook Pro with the new Apple Silicon M1 chip, compiling libgcrypt finishes successfully, however, make check fails the random test (all other tests PASS) with the following error:

dyld: Library not loaded: /usr/local/lib/libgcrypt.20.dylib
  Referenced from: /Users/mbirth/Development/libgcrypt/libgcrypt-1.8.7/tests/.libs/random
  Reason: image not found
random: running '/Users/mbirth/Development/libgcrypt/libgcrypt-1.8.7/tests/.libs/random --in-recursion --early-rng-check' failed
FAIL: random

This also fails the test when installing the Homebrew package, so the package is never actually installed. See https://github.com/Homebrew/brew/issues/7857#issuecomment-735269479 .

Details

External Link: https://github.com/Homebrew/brew/issues/7857#issuecomment-735269479
Version: 1.8.7

Revisions and Commits

rC libgcrypt
	rC16e855d49ca2 tests: Put a work around to tests/random for macOS.
	rCfc901e978a0c build: Check spawn.h for MacOS X Tiger.
	rC9769b40b54cf tests: Put a work around to tests/random for macOS.

Related Objects
Search...

Status	Assigned	Task
Resolved	• gniibe	T5159 make check fails for libgcrypt on Apple Silicon / ARM Mac
Resolved	• gniibe	T5157 libgcrypt: ARM64 Builds on macOS fail
Resolved	• werner	T5259 Release Libgcrypt 1.9.1

Event Timeline

mbirth created this task.Nov 28 2020, 11:43 PM

mbirth updated the task description. (Show Details)Nov 29 2020, 1:14 PM

• gniibe added a subscriber: • gniibe.Nov 30 2020, 2:51 AM

This comment was removed by • gniibe.

See T2056: libgcrypt: make check fails "random" test on OS X 10.11 with link error, if test with 'random' fails.

Please have a look at T5024: libtool problem for some platforms for 'make check' (program built with -no-install won't work without installation), if make check works after the installation of libgcrypt.

IIUC, for the build of Homebrew, it is the issue of in: https://github.com/Homebrew/homebrew-core/commit/e7da1e2157b2e8373c3b39ea6398f51588ea537c

It changes the library path by the script, so that random can run correctly. I think that for some reason (of the upgrade), lib/"libgcrypt.20.dylib" perhaps would be changed to different place.

Please give us information for your own manual build of libgcrypt (not by the Homebrew).

• werner added a subtask: T5157: libgcrypt: ARM64 Builds on macOS fail.Nov 30 2020, 8:47 AM

After doing:

sudo ln -s /Users/mbirth/Development/libgcrypt/libgcrypt-1.8.7/src/.libs/libgcrypt.20.dylib /usr/local/lib/libgcrypt.20.dylib

All tests pass now.

Just to be clear: I didn't use any parameters when compiling. All I did after untarring was:

./configure
make

After disabling SIP, now all checks pass without having the library symlinked to /usr/local/lib. So it might be T2056: libgcrypt: make check fails "random" test on OS X 10.11 with link error after all.

See my comment in: https://dev.gnupg.org/T5024#139701

BTW, I'm not sure if the claim in T5009#136688 is correct.

If it is the system interpreter invocation (/bin/sh) which sanitizes the environment variable, following patch would work:

diff --git a/tests/random.c b/tests/random.c
index 22927a61..a1244664 100644
--- a/tests/random.c
+++ b/tests/random.c
@@ -505,6 +505,7 @@ check_drbg_reinit (void)
 }
 
 
+#ifdef HAVE_W32_SYSTEM
 /* Because we want to check initialization behaviour, we need to
    fork/exec this program with several command line arguments.  We use
    system, so that these tests work also on Windows.  */
@@ -559,7 +560,64 @@ run_all_rng_tests (const char *program)
 
   free (cmdline);
 }
+#else
+#include <spawn.h>
+
+static void
+run_all_rng_tests (const char *program)
+{
+  static const char *options[][2] = {
+    { "--early-rng-check",     NULL },
+    { "--early-rng-check",     "--prefer-standard-rng" },
+    { "--early-rng-check",     "--prefer-fips-rng" },
+    { "--early-rng-check",     "--prefer-system-rng" },
+    { "--prefer-standard-rng", NULL },
+    { "--prefer-fips-rng",     NULL },
+    { "--prefer-system-rng",   NULL },
+    { NULL, NULL }
+  };
+  int idx;
+  char *argv[8];
 
+  for (idx=0; options[idx][0]; idx++)
+    {
+      int i;
+      pid_t pid;
+      int status;
+
+      if (verbose)
+        info ("now running with options '%s%s%s'\n",
+              options[idx][0],
+              options[idx][1] ? " " : "",
+              options[idx][1] ? options[idx][1] : "");
+
+      i = 0;
+      argv[i++] = xstrdup (program);
+      argv[i++] = xstrdup ("--in-recursion");
+      argv[i++] = xstrdup ("--verbose");
+      argv[i++] = xstrdup ("--debug");
+      argv[i++] = xstrdup ("--progress");
+      argv[i++] = xstrdup (options[idx][0]);
+      if (options[idx][1])
+        argv[i++] = xstrdup (options[idx][1]);
+      argv[i++] = NULL;
+
+      if (posix_spawn (&pid, program, NULL, NULL, argv, environ))
+        die ("spawning '%s' failed\n", program);
+
+      if (waitpid (pid, &status, 0) < 0)
+        die ("waitpid for '%s' failed\n", program);
+
+      if (WIFEXITED (status) && WEXITSTATUS (status))
+        die ("running '%s' failed with %d\n", program, WEXITSTATUS (status));
+      else if (!WIFEXITED (status))
+        die ("running '%s' failed\n", program);
+
+      while (i)
+        xfree (argv[--i]);
+    }
+}
+#endif
 
 static void
 run_benchmark (void)

The problem is that posix_spawn is not portable enough for libgcrypt. It is really time that we move the spawn functions from gnupg to gpgrt so that we can use them also in Libgcrypt.

After applying @gniibe 's patch:

gcc -DHAVE_CONFIG_H -I. -I..  -I../src -I../src  -I/opt/homebrew/opt/libgpg-error/include -g -O2 -fno-delete-null-pointer-checks -Wall -MT random.o -MD -MP -MF .deps/random.Tpo -c -o random.o random.c
random.c:605:57: error: use of undeclared identifier 'environ'
      if (posix_spawn (&pid, program, NULL, NULL, argv, environ))
                                                        ^
1 error generated.
make[1]: *** [random.o] Error 1

Put

extern char **environ;

after the the include directives.

In T5159#139739, @werner wrote:
Put
extern char **environ;
after the the include directives.

That did it!

====================
All 27 tests passed
(1 test was not run)
====================

I created D513: Support macOS build with SIP by using posix_spawn in tests/random, which is more conservative; It only affects build under macOS.

I agree that no posix_spawn in libgcrypt but having spawning a process in libgpg-error is good.

Note that the patch doesn't use use of posix_spawn in the libgcrypt library proper, but only in the test program for macOS.
If it would be difficult for libgcrypt upstream, macOS can use D513 for their build.

So, I'm going to push D513 to both of 1.8 and master (to be 1.9).

• gniibe changed the task status from Open to Testing.Dec 3 2020, 3:11 AM

• gniibe triaged this task as Normal priority.

• gniibe added a commit: rC9769b40b54cf: tests: Put a work around to tests/random for macOS..Dec 3 2020, 3:54 AM

• gniibe added a commit: rCfc901e978a0c: build: Check spawn.h for MacOS X Tiger..Jan 27 2021, 1:11 AM

Also support older MacOS X, which has no posix_spawn.

https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/posix_spawn.2.html dated August 9, 2007.
So, I guess that posix_spawn became available in MacOS X Leopard (10.5).

• werner mentioned this in T5259: Release Libgcrypt 1.9.1.Jan 29 2021, 11:24 AM

• werner changed the status of subtask T5259: Release Libgcrypt 1.9.1 from Open to Testing.

• werner closed subtask T5259: Release Libgcrypt 1.9.1 as Resolved.Feb 12 2021, 12:34 PM

Sorry to dig up an old report...

This looks like System Integrity Protection (SIP) scrubbing certain environmental variables. The trick to the workaround is, have the Makefile set DYLD_LIBRARY_PATH before it calls the test runner.

This patch clears the issue so you don't have to disable SIP: libgcrypt.patch. Here's the interesting part:

--- tests/Makefile.in
+++ tests/Makefile.in
@@ -942,6 +942,10 @@
 check-TESTS: $(TESTS)
 	@failed=0; all=0; xfail=0; xpass=0; skip=0; \
 	srcdir=$(srcdir); export srcdir; \
+	gcrypt_libdir=`dirname $$PWD`/src/.libs; \
+	LD_LIBRARY_PATH=`echo "$$gcrypt_libdir:$$LD_LIBRARY_PATH" | $(SED) 's/:*$$//g'`; \
+	DYLD_LIBRARY_PATH=`echo "$$gcrypt_libdir:$$DYLD_LIBRARY_PATH" | $(SED) 's/:*$$//g'`; \
+	export LD_LIBRARY_PATH; export DYLD_LIBRARY_PATH; \
 	list=' $(TESTS) '; \
 	$(am__tty_colors); \
 	if test -n "$$list"; then \

The commands just prepend the proper directory to LD_LIBRARY_PATH and DYLD_LIBRARY_PATH. The sed removes any trailing : in case the variable was previously empty. Empty paths are removed for one of the BSDs (I think NetBSD is the one that acts funny with empty paths).

dirname is used because you are actually in the tests/ directory, and you want to trim that off. dirname returns a full path like /Users/jdoe/libgcrypt-1.9.2. Then you add src/.libs to it to get a path of /Users/jdoe/libgcrypt-1.9.2/src/.libs.

The Makefile trick does not always work when SIP is in effect because SIP will scrub the variables if you call a system program, like /bin/sh. You have to call your program, and not a system program.

Here's an example where the makefile trick does not work:

check:
    DYLD_LIBRARY_PATH=`echo "$$gcrypt_libdir:$$DYLD_LIBRARY_PATH" | $(SED) 's/:*$$//g'`; \
    export DYLD_LIBRARY_PATH; \
    /bin/sh gcrypt_test_runner.sh --all_tests

The trick would not work in this instance because SIP would clear DYLD_LIBRARY_PATH when /bin/sh is called. To fix it in this instance, gcrypt_test_runner.sh would have to set DYLD_LIBRARY_PATH.

• gniibe added a commit: rC16e855d49ca2: tests: Put a work around to tests/random for macOS..Jun 3 2021, 7:08 PM

• gniibe closed subtask T5157: libgcrypt: ARM64 Builds on macOS fail as Resolved.Mar 16 2022, 3:09 PM

make check fails for libgcrypt on Apple Silicon / ARM MacClosed, ResolvedPublicActions