libgcrypt: build "error: invalid operand for instruction" when compiling with Clang & LTO
Open, NormalPublic

Description

Compiling with Clang works without -flto{,=thin}, compiling with GCC with LTO also works. It is purely -flto & Clang which introduces this issue.

Build version is 1.8.7. Build system is Gentoo ~amd64.

The errors are the following, repeated:

/var/tmp/portage/dev-libs/libgcrypt-1.8.7/work/libgcrypt-1.8.7/cipher/sha256-avx-amd64.S:433:2: note: while in macro instantiation
 FOUR_ROUNDS_AND_SCHED
 ^
<instantiation>:53:2: error: invalid operand for instruction
 and y2, e
 ^

as well as:

/var/tmp/portage/dev-libs/libgcrypt-1.8.7/work/libgcrypt-1.8.7/cipher/sha512-avx-amd64.S:341:2: error: invalid operand for instruction
 add [digest + 8*(7)], h_64
 ^
/var/tmp/portage/dev-libs/libgcrypt-1.8.7/work/libgcrypt-1.8.7/cipher/sha512-avx-amd64.S:344:6: error: ambiguous operand size for instruction 'add'
 add msg, 16*8
     ^~~
/var/tmp/portage/dev-libs/libgcrypt-1.8.7/work/libgcrypt-1.8.7/cipher/sha512-avx-amd64.S:345:6: error: ambiguous operand size for instruction 'dec'
 dec msglen

CFLAGS:
CFLAGS="-march=znver2 -O3 -flto=thin -pipe"

gcc:

Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.0 (Gentoo 10.2.0-r5 p6)

clang:

clang version 11.0.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm/11/bin
Selected GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0

Build log:

telans created this task.Jan 19 2021, 9:16 PM
telans renamed this task from ligcrypt: build "error: invalid operand for instruction" when compiling with Clang & LTO to libgcrypt: build "error: invalid operand for instruction" when compiling with Clang & LTO.
telans changed Version from 1.8.7 to 1.9.0.Jan 19 2021, 9:21 PM

Yes, clang + LTO is broken. Maybe there is issue in clang bug tracker for this already?

werner added a subscriber: werner.Jan 20 2021, 9:21 AM

So is this about 1.8.7 or 1.9.0 (as shown in the Version field)?

werner triaged this task as Normal priority.

Both are affected. I updated to reflect that I tested the newer version

Breakage appears to happen in configure.ac. When building with clang without LTO following check gives "no":

checking whether GCC assembler is compatible for Intel syntax assembly implementations... no

But with "-flto" included in CFLAGS, same test gives "yes":

checking whether GCC assembler is compatible for Intel syntax assembly implementations... yes

There is few x86_64 assembly files in libgcrypt that use Intel syntax and those are not supported by clang (or clang does not support assembler macros used in those same assembly files). Now with "clang -flto" those get included to build and build fails. It would make sense to convert those Intel assembly files to AT&T syntax so that clang builds would benefit from optimized SHA256 & SHA512 implementations.

Clang support Intel syntax after all, but not assembler macros that were used. Here's two patches that fix the configure.ac issue and removes use of assembly macros in Intel syntax assembly files:

Unfortunately these are not enough to enable building with clang LTO. Build now fails when linking test executables (libgcrypt.so gets build however):

...
libtool: compile:  clang -DHAVE_CONFIG_H -I. -I../../src -I.. -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -MT libgcrypt_la-hwf-x86.lo -MD -MP -MF .deps/libgcrypt_la-hwf-x86.Tpo -c ../../src/hwf-x86.c  -fPIC -DPIC -o .libs/libgcrypt_la-hwf-x86.o
mv -f .deps/libgcrypt_la-hwf-x86.Tpo .deps/libgcrypt_la-hwf-x86.Plo
/bin/bash ../libtool  --tag=CC   --mode=link clang  -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith    -Wl,--version-script=../../src/libgcrypt.vers -version-info 23:0:3  -o libgcrypt.la -rpath /usr/local/lib libgcrypt_la-visibility.lo libgcrypt_la-misc.lo libgcrypt_la-global.lo libgcrypt_la-sexp.lo libgcrypt_la-hwfeatures.lo libgcrypt_la-stdmem.lo libgcrypt_la-secmem.lo libgcrypt_la-missing-string.lo libgcrypt_la-fips.lo libgcrypt_la-hmac256.lo libgcrypt_la-context.lo  libgcrypt_la-hwf-x86.lo ../cipher/libcipher.la ../random/librandom.la ../mpi/libmpi.la ../compat/libcompat.la  -L/usr/lib/x86_64-linux-gnu -lgpg-error 
libtool: link: rm -fr  .libs/libgcrypt.la .libs/libgcrypt.lai .libs/libgcrypt.so .libs/libgcrypt.so.20 .libs/libgcrypt.so.20.3.0
libtool: link: clang -shared  -fPIC -DPIC  .libs/libgcrypt_la-visibility.o .libs/libgcrypt_la-misc.o .libs/libgcrypt_la-global.o .libs/libgcrypt_la-sexp.o .libs/libgcrypt_la-hwfeatures.o .libs/libgcrypt_la-stdmem.o .libs/libgcrypt_la-secmem.o .libs/libgcrypt_la-missing-string.o .libs/libgcrypt_la-fips.o .libs/libgcrypt_la-hmac256.o .libs/libgcrypt_la-context.o .libs/libgcrypt_la-hwf-x86.o  -Wl,--whole-archive ../cipher/.libs/libcipher.a ../random/.libs/librandom.a ../mpi/.libs/libmpi.a ../compat/.libs/libcompat.a -Wl,--no-whole-archive  -L/usr/lib/x86_64-linux-gnu -lgpg-error  -O2 -flto -Wl,--version-script=../../src/libgcrypt.vers   -Wl,-soname -Wl,libgcrypt.so.20 -o .libs/libgcrypt.so.20.3.0
libtool: link: (cd ".libs" && rm -f "libgcrypt.so.20" && ln -s "libgcrypt.so.20.3.0" "libgcrypt.so.20")
libtool: link: (cd ".libs" && rm -f "libgcrypt.so" && ln -s "libgcrypt.so.20.3.0" "libgcrypt.so")
libtool: link: ( cd ".libs" && rm -f "libgcrypt.la" && ln -s "../libgcrypt.la" "libgcrypt.la" )
/bin/bash ../libtool  --tag=CC   --mode=link clang  -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith   -o mpicalc mpicalc-mpicalc.o libgcrypt.la -L/usr/lib/x86_64-linux-gnu -lgpg-error 
libtool: link: clang -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -o .libs/mpicalc mpicalc-mpicalc.o  ./.libs/libgcrypt.so -L/usr/lib/x86_64-linux-gnu -lgpg-error
make[2]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/src'
Making all in doc
make[2]: Entering directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/doc'
make  all-am
make[3]: Entering directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/doc'
make[3]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/doc'
make[2]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/doc'
Making all in tests
make[2]: Entering directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/tests'
clang -DHAVE_CONFIG_H -I. -I../../tests -I..  -I../src -I../../src   -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -MT version.o -MD -MP -MF .deps/version.Tpo -c -o version.o ../../tests/version.c
mv -f .deps/version.Tpo .deps/version.Po
/bin/bash ../libtool  --tag=CC   --mode=link clang  -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -no-install  -o version version.o ../src/libgcrypt.la ../compat/libcompat.la -L/usr/lib/x86_64-linux-gnu -lgpg-error  
libtool: link: clang -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -o version version.o  ../src/.libs/libgcrypt.so ../compat/.libs/libcompat.a -L/usr/lib/x86_64-linux-gnu -lgpg-error -Wl,-rpath -Wl,/home/jussi/kernel/libgcrypt/build-clang-overopt/src/.libs
/usr/bin/ld: ../compat/.libs/libcompat.a: error adding symbols: archive has no index; run ranlib to add one
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [Makefile:776: version] Error 1
make[2]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/tests'
make[1]: *** [Makefile:500: all-recursive] Error 1
make[1]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt'
make: *** [Makefile:432: all] Error 2

Any ideas what might be causing this?

Configure output has still has some differences LTO vs non-LTO:

--- non-lto.log 2021-01-21 22:25:14.966099577 +0200
+++ lto.log     2021-01-21 22:25:23.174086100 +0200
@@ -63,7 +63,7 @@
 checking for archiver @FILE support... @
 checking for strip... strip
 checking for ranlib... ranlib
-checking command to parse /usr/bin/nm -B output from clang object... ok
+checking command to parse /usr/bin/nm -B output from clang object... failed
 checking for sysroot... no
 checking for mt... mt
 checking if mt is a manifest tool... no
@@ -75,7 +75,7 @@
 checking if clang static flag -static works... yes
 checking if clang supports -c -o file.o... yes
 checking if clang supports -c -o file.o... (cached) yes
-checking whether the clang linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
+checking whether the clang linker (/usr/bin/ld) supports shared libraries... yes
 checking whether -lc should be explicitly linked in... no
 checking dynamic linker characteristics... GNU/Linux ld.so
 checking how to hardcode library paths into programs... immediate
@@ -168,8 +168,8 @@
 checking whether 'asm' assembler keyword is supported... yes
 checking whether '__asm__' assembler keyword is supported... yes
 checking whether inline assembly memory barrier is supported... yes
-checking whether GCC assembler is compatible for ARM assembly implementations... no
-checking whether GCC assembler is compatible for ARMv8/Aarch64 assembly implementations... no
+checking whether GCC assembler is compatible for ARM assembly implementations... yes
+checking whether GCC assembler is compatible for ARMv8/Aarch64 assembly implementations... yes
 checking whether GCC assembler supports for CFI directives... yes
 checking whether GCC assembler supports for ELF directives... yes
 checking for _ prefix in compiled symbols... no
@@ -240,7 +240,7 @@
 checking if gcc supports -Wno-missing-field-initializers... yes
 checking if gcc supports -Wpointer-arith... yes
 checking whether non excutable stack support is requested... yes
-checking whether assembler supports --noexecstack option... yes
+checking whether assembler supports --noexecstack option... no
 checking that generated files are newer than configure... done
 configure: creating ./config.status
 config.status: creating Makefile
telans added a comment.EditedJan 21 2021, 9:47 PM

Unfortunately these are not enough to enable building with clang LTO. Build now fails when linking test executables (libgcrypt.so gets build however):

Any ideas what might be causing this?

Thanks for the patches, with them I can't replicate your linking issues strangely. It builds & links fine for me, thanks.

I've tried to reproduce it with -flto=thin, -flto, ld.bfd and ld.lld (my default linker) but they all link fine (version 1.9.0). My configure output looks like the same as yours.

Problem was that my build system was selecting "ar" and "ranlib", where as your build system selects "llvm-ar" and "llvm-ranlib".

I got it building with:

AR="llvm-ar" RANLIB="llvm-ranlib" CC=clang CFLAGS="-O2 -flto" ../configure --enable-maintainer-mode

Should we add this to the hints in the README? After all this does not seem to be the standard system compiler or it has not been properly setup as replacement.

That might be helpful. But, on the other hand, if I had just googled the problem I was seeing I would have gotten answer quite fast.

werner changed the status of subtask T5259: Release Libgcrypt 1.9.1 from Open to Testing.