macOS getentropy
Testing, NormalPublic

Description

D522 is a patch for macOS.

longlong part was fixed already in master (to be 1.9.1).

getentropy part should be handled.

Note that we can't directly apply the patch.

See: https://www.gnu.org/software/gnulib/manual/html_node/sys_002frandom_002eh.html
https://bugs.python.org/issue29057

I realized that it's a bit difficult for macOS to use getentropy.

On some versions, even though it is available in the sys/random.h, it is actually not implemented (just a weak pointer).
See:

https://github.com/bitcoin/bitcoin/pull/10301
https://github.com/bitcoin/bitcoin/pull/15100
https://github.com/bitcoin/bitcoin/pull/15103

gniibe added a comment.EditedJan 26 2021, 5:33 AM

https://www.unix.com/man-page/mojave/2/getentropy/ says getentropy is available in 10.12 or later.

Just for the information, this library of Rust checks if the symbol definition is available (not NULL), and use getentropy in that case.
fall back to /dev/u?random

https://github.com/rust-random/getrandom/blob/master/src/macos.rs

gniibe added a project: libgcrypt.
gniibe moved this task from Backlog to For 1.9 on the libgcrypt board.
gniibe updated the task description. (Show Details)Jan 26 2021, 6:00 AM

To support old macOS (< 10.12), I think that code should be something like this:

diff --git a/random/rndlinux.c b/random/rndlinux.c
index 04e2a464..f378a549 100644
--- a/random/rndlinux.c
+++ b/random/rndlinux.c
@@ -32,6 +32,10 @@
 #include <string.h>
 #include <unistd.h>
 #include <fcntl.h>
+#if defined(__APPLE__) && defined(__MACH__)
+extern int getentropy (void *buf, size_t buflen) __attribute__ ((weak_import));
+#define HAVE_GETENTROPY
+#endif
 #if defined(__linux__) || !defined(HAVE_GETENTROPY)
 #ifdef HAVE_SYSCALL
 # include <sys/syscall.h>
@@ -260,6 +264,9 @@ _gcry_rndlinux_gather_random (void (*add)(const void*, size_t,
        * not been properly seeded.  And it differs from /dev/random by never
        * blocking once the kernel is seeded.  */
 #if defined(HAVE_GETENTROPY) || defined(__NR_getrandom)
+#if defined(__APPLE__) && defined(__MACH__)
+      if (&getentropy != NULL)
+#endif
         {
           long ret;
           size_t nbytes;

We should not use system defined getentropy (on a build system >= 10.12), because it is defined with no 'weak_import' attribute.

I don't know when the symbol of getentropy was available on macOS.
For the old systems with no getentropy symbol, it won't work at all.

To support old macOS (< 10.12), I think that code should be something like this:

diff --git a/random/rndlinux.c b/random/rndlinux.c
index 04e2a464..f378a549 100644
We should not use system defined getentropy (on a build system >= 10.12), because it is defined with no 'weak_import' attribute.

Right now I cannot compile libgcrypt. According to theory I'd say your patch is not correct. GCC 4.2, which is used on Tiger and Leopard (Mac OS X 10.4.11 resp. 10.5.8), has these defines:

#define __MACH__ 1
#define __APPLE__ 1

and so does GCC 7.5, the alternative, when the source follows standards from this century. The version of Mac OS X or macOS can be checked by

#define __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ 10125

The number used expresses Mac OS X 10.12.5. Another option might be to use

if MAC_OS_X_VERSION_MIN_REQUIRED < 101200

The real problem with Tiger is that it misses a few C header files, and of them is required by recent versions of libgcrypt:

/opt/local/bin/gcc-apple-4.2 -DHAVE_CONFIG_H -I. -I..  -I../src -I../src -I/opt/local/include 
-I/opt/local/include -pipe -Os -std=gnu89 -arch ppc -fno-delete-null-pointer-checks -Wall -MT random.o -MD -MP -MF .deps/random.Tpo -c -o random.o random.c
random.c:509:19: error: spawn.h: No such file or directory
random.c: In function 'run_all_rng_tests':
random.c:551: warning: implicit declaration of function 'posix_spawn'
make[2]: *** [random.o] Error 1

sys/spawn.h has been provided as legacy support by MacPorts team, but it seems to be inadequate.

I don't know when the symbol of getentropy was available on macOS.

I think it was macOS Sierra, Version 10.12.x.

@ballapete Thank you for testing.

For the failure of compiling random.c, a patch has just been pushed to master:
rCfc901e978a0c: build: Check spawn.h for MacOS X Tiger.

I wrote:

I don't know when the symbol of getentropy was available on macOS.

You answered:

I think it was macOS Sierra, Version 10.12.x.

I know that the function implementation is available in 10.12 or later.
The problem here is that, in older version of 10.x (x < 12), the symbol itself is available, but there are no implementation actually.
If possible, I'd like to know when the symbol was available.

gniibe changed the task status from Open to Testing.Jan 27 2021, 5:04 AM

Push the change.
For older versions of MacOS X, I'll handle it later.

To support old macOS (< 10.12), I think that code should be something like this:

After libgcrypt 1.9.0 first built on Tiger and 'make check' succeeded I ran 'make clean' and applied the changes to random/rndlinux.c. A simple make succeeded without problem. 'make check' then ran into problems, spitting out many times messages messages like these:

make  check-TESTS
dyld: Symbol not found: _getentropy
  Referenced from: /opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/libgcrypt-1.9.0/src/.libs/libgcrypt.20.dylib
  Expected in: dynamic lookup

FAIL: version
dyld: Symbol not found: _getentropy
  Referenced from: /opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/libgcrypt-1.9.0/src/.libs/libgcrypt.20.dylib
  Expected in: dynamic lookup

It finished with:

FAIL: bench-slope
SKIP: hashtest-256g
=======================================
29 of 29 tests failed
(1 test was not run)
Please report to https://bugs.gnupg.org
=======================================

Thank you for your testing.

Suppose that getentropy is available in MacOS X 10.5 (which conforms to Single Unix Specification). If so, something like following should work:

diff --git a/random/rndlinux.c b/random/rndlinux.c
index a22db177..256f7403 100644
--- a/random/rndlinux.c
+++ b/random/rndlinux.c
@@ -33,9 +33,12 @@
 #include <unistd.h>
 #include <fcntl.h>
 #if defined(__APPLE__) && defined(__MACH__)
+#include <Availability.h>
+#ifdef __MAC_10_5
 extern int getentropy (void *buf, size_t buflen) __attribute__ ((weak_import));
 #define HAVE_GETENTROPY
 #endif
+#endif
 #if defined(__linux__) || !defined(HAVE_GETENTROPY)
 #ifdef HAVE_SYSCALL
 # include <sys/syscall.h>

It only enables use of getentropy for MacOS 10.5 or later.

In the next few days I'll be able to boot into Leopard, Mac OS X 10.5.8, to test this. Right now it seems that I need to fix a few problems with updated software sources…

werner changed the status of subtask T5259: Release Libgcrypt 1.9.1 from Open to Testing.Jan 29 2021, 11:25 AM

I applied the two patches on Mac OS X 10.5.8, Leopard, to random/rndlinux.c, resulting in this unified diff:

  • random/rndlinux.c~ 2021-01-28 23:14:39.000000000 +0100

+++ random/rndlinux.c 2021-01-29 10:56:26.000000000 +0100
@@ -32,6 +32,13 @@
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
+#if defined(APPLE) && defined(MACH)
+#include <Availability.h>
+#ifdef MAC_10_5
+extern int getentropy (void *buf, size_t buflen)
attribute__ ((weak_import));
+#define HAVE_GETENTROPY
+#endif
+#endif
#if defined(linux) || !defined(HAVE_GETENTROPY)
#ifdef HAVE_SYSCALL

  1. include <sys/syscall.h>

@@ -260,6 +267,9 @@

  • not been properly seeded. And it differs from /dev/random by never
  • blocking once the kernel is seeded. */ #if defined(HAVE_GETENTROPY) || defined(__NR_getrandom)

+#if defined(APPLE) && defined(MACH)
+ if (&getentropy != NULL)
+#endif

{
  long ret;
  size_t nbytes;

At this state random/rndlinux.c compiled fine. I am letting run 'make check' – again it fails à la:

make check-TESTS
make[2]: Verzeichnis „/opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/libgcrypt-1.9.0/tests“ wird betreten
dyld: Symbol not found: _getentropy

Referenced from: /opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/libgcrypt-1.9.0/src/.libs/libgcrypt.20.dylib
Expected in: dynamic lookup

/bin/sh: line 5: 78501 Trace/BPT trap (core dumped) GCRYPT_IN_REGRESSION_TEST=1 ${dir}$tst
FAIL: version
dyld: Symbol not found: _getentropy

Referenced from: /opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/libgcrypt-1.9.0/src/.libs/libgcrypt.20.dylib
Expected in: dynamic lookup

/bin/sh: line 5: 78520 Trace/BPT trap (core dumped) GCRYPT_IN_REGRESSION_TEST=1 ${dir}$tst

Wouldn't it be better to move these failures as a single one into the configure script that it definitely can tell "This Mac has getentropy()"?

gniibe added a comment.Feb 1 2021, 1:53 AM

Wouldn't it be better to move these failures as a single one into the configure script that it definitely can tell "This Mac has getentropy()"?

For build from source, this works. Well, that will be the last option, if it will be found too difficult.

I have been trying to support the situation where a binary built by some versions of macOS can be used on different version (like Python and bitcoin, I referred).
To do that, I'd like to know, when the symbol getentropy was added.

To do that, I'd like to know, when the symbol getentropy was added.

So it would be sufficient to find all dylibs and check with nm whether they contain the symbol? On PPC Tiger and Leopard (Mac OS X 10.4.11/10.5.8) this can be performed by me. For later Mac OS X versions I would need to install these versions in a VM… (which might not work because the install programme is possibly checking whether it is guide by a hypervisor)

There is some (partly) good news: The function getentropy() is available in the packet manager MacPorts. It has a legacy support:

/opt/local/lib/libMacportsLegacySupport.dylib:single module: 00002a88 T __getentropy

This legacy support serves as a fall-back for modern functions not found in old systems. I don't whether guards exist for MacPorts…

gniibe added a comment.Feb 2 2021, 7:09 AM

I got hit of search by "$ld$weak$os10.11$_getentropy".
So, I guess that it's 10.11 which has _getentropy as weak symbol, and 10.12 or later has implementation.

gniibe added a comment.Feb 2 2021, 7:27 AM

So, the change against libgcrypt 1.9.1 will be:

diff --git a/random/rndlinux.c b/random/rndlinux.c
index a22db177..a7a78906 100644
--- a/random/rndlinux.c
+++ b/random/rndlinux.c
@@ -33,9 +33,12 @@
 #include <unistd.h>
 #include <fcntl.h>
 #if defined(__APPLE__) && defined(__MACH__)
+#include <Availability.h>
+#ifdef __MAC_10_11
 extern int getentropy (void *buf, size_t buflen) __attribute__ ((weak_import));
 #define HAVE_GETENTROPY
 #endif
+#endif
 #if defined(__linux__) || !defined(HAVE_GETENTROPY)
 #ifdef HAVE_SYSCALL
 # include <sys/syscall.h>

This approach is too simplistic. See Ryan Schmidt's and Joshua Root's comments in https://trac.macports.org/ticket/62278