libgcrypt: build "error: invalid operand for instruction" when compiling with Clang & LTO
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	telans
	Jan 19 2021, 9:16 PM

Description

Compiling with Clang works without -flto{,=thin}, compiling with GCC with LTO also works. It is purely -flto & Clang which introduces this issue.

Build version is 1.8.7. Build system is Gentoo ~amd64.

The errors are the following, repeated:

/var/tmp/portage/dev-libs/libgcrypt-1.8.7/work/libgcrypt-1.8.7/cipher/sha256-avx-amd64.S:433:2: note: while in macro instantiation
 FOUR_ROUNDS_AND_SCHED
 ^
<instantiation>:53:2: error: invalid operand for instruction
 and y2, e
 ^

as well as:

/var/tmp/portage/dev-libs/libgcrypt-1.8.7/work/libgcrypt-1.8.7/cipher/sha512-avx-amd64.S:341:2: error: invalid operand for instruction
 add [digest + 8*(7)], h_64
 ^
/var/tmp/portage/dev-libs/libgcrypt-1.8.7/work/libgcrypt-1.8.7/cipher/sha512-avx-amd64.S:344:6: error: ambiguous operand size for instruction 'add'
 add msg, 16*8
     ^~~
/var/tmp/portage/dev-libs/libgcrypt-1.8.7/work/libgcrypt-1.8.7/cipher/sha512-avx-amd64.S:345:6: error: ambiguous operand size for instruction 'dec'
 dec msglen

CFLAGS:
CFLAGS="-march=znver2 -O3 -flto=thin -pipe"

gcc:

Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.0 (Gentoo 10.2.0-r5 p6)

clang:

clang version 11.0.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm/11/bin
Selected GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0

Build log:

libgcrypt-1.8.7-clang-lto.log2 MBDownload

Details

External Link: https://bugs.gentoo.org/765841
Version: 1.9.0

Revisions and Commits

rC libgcrypt
	rC9f49e806f950 sha512/sha256: remove assembler macros from AMD64 implementations
	rC393bd6c3d1aa configure.ac: run assembler checks through linker for better LTO support

Related Objects
Search...

Status	Assigned	Task
Resolved	jukivili	T5255 libgcrypt: build "error: invalid operand for instruction" when compiling with Clang & LTO
Resolved	jukivili	T5256 libgcrypt, convert Intel syntax x86_64 assembly files to AT&T syntax
Resolved	• werner	T5259 Release Libgcrypt 1.9.1

Event Timeline

telans renamed this task from ligcrypt: build "error: invalid operand for instruction" when compiling with Clang & LTO to libgcrypt: build "error: invalid operand for instruction" when compiling with Clang & LTO.Jan 19 2021, 9:16 PM

telans created this task.

telans changed Version from 1.8.7 to 1.9.0.Jan 19 2021, 9:21 PM

Yes, clang + LTO is broken. Maybe there is issue in clang bug tracker for this already?

So is this about 1.8.7 or 1.9.0 (as shown in the Version field)?

• werner triaged this task as Normal priority.Jan 20 2021, 9:33 AM

• werner added projects: clang, libgcrypt, asm.

Both are affected. I updated to reflect that I tested the newer version

Breakage appears to happen in configure.ac. When building with clang without LTO following check gives "no":

checking whether GCC assembler is compatible for Intel syntax assembly implementations... no

But with "-flto" included in CFLAGS, same test gives "yes":

checking whether GCC assembler is compatible for Intel syntax assembly implementations... yes

There is few x86_64 assembly files in libgcrypt that use Intel syntax and those are not supported by clang (or clang does not support assembler macros used in those same assembly files). Now with "clang -flto" those get included to build and build fails. It would make sense to convert those Intel assembly files to AT&T syntax so that clang builds would benefit from optimized SHA256 & SHA512 implementations.

jukivili added a subtask: T5256: libgcrypt, convert Intel syntax x86_64 assembly files to AT&T syntax.Jan 20 2021, 1:24 PM

Clang support Intel syntax after all, but not assembler macros that were used. Here's two patches that fix the configure.ac issue and removes use of assembly macros in Intel syntax assembly files:

0001-configure.ac-fix-Intel-syntax-check-with-clang-flto.patch1 KBDownload

0002-sha512-sha256-remove-assembler-macros-from-AMD64-imp.patch109 KBDownload

Unfortunately these are not enough to enable building with clang LTO. Build now fails when linking test executables (libgcrypt.so gets build however):

...
libtool: compile:  clang -DHAVE_CONFIG_H -I. -I../../src -I.. -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -MT libgcrypt_la-hwf-x86.lo -MD -MP -MF .deps/libgcrypt_la-hwf-x86.Tpo -c ../../src/hwf-x86.c  -fPIC -DPIC -o .libs/libgcrypt_la-hwf-x86.o
mv -f .deps/libgcrypt_la-hwf-x86.Tpo .deps/libgcrypt_la-hwf-x86.Plo
/bin/bash ../libtool  --tag=CC   --mode=link clang  -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith    -Wl,--version-script=../../src/libgcrypt.vers -version-info 23:0:3  -o libgcrypt.la -rpath /usr/local/lib libgcrypt_la-visibility.lo libgcrypt_la-misc.lo libgcrypt_la-global.lo libgcrypt_la-sexp.lo libgcrypt_la-hwfeatures.lo libgcrypt_la-stdmem.lo libgcrypt_la-secmem.lo libgcrypt_la-missing-string.lo libgcrypt_la-fips.lo libgcrypt_la-hmac256.lo libgcrypt_la-context.lo  libgcrypt_la-hwf-x86.lo ../cipher/libcipher.la ../random/librandom.la ../mpi/libmpi.la ../compat/libcompat.la  -L/usr/lib/x86_64-linux-gnu -lgpg-error 
libtool: link: rm -fr  .libs/libgcrypt.la .libs/libgcrypt.lai .libs/libgcrypt.so .libs/libgcrypt.so.20 .libs/libgcrypt.so.20.3.0
libtool: link: clang -shared  -fPIC -DPIC  .libs/libgcrypt_la-visibility.o .libs/libgcrypt_la-misc.o .libs/libgcrypt_la-global.o .libs/libgcrypt_la-sexp.o .libs/libgcrypt_la-hwfeatures.o .libs/libgcrypt_la-stdmem.o .libs/libgcrypt_la-secmem.o .libs/libgcrypt_la-missing-string.o .libs/libgcrypt_la-fips.o .libs/libgcrypt_la-hmac256.o .libs/libgcrypt_la-context.o .libs/libgcrypt_la-hwf-x86.o  -Wl,--whole-archive ../cipher/.libs/libcipher.a ../random/.libs/librandom.a ../mpi/.libs/libmpi.a ../compat/.libs/libcompat.a -Wl,--no-whole-archive  -L/usr/lib/x86_64-linux-gnu -lgpg-error  -O2 -flto -Wl,--version-script=../../src/libgcrypt.vers   -Wl,-soname -Wl,libgcrypt.so.20 -o .libs/libgcrypt.so.20.3.0
libtool: link: (cd ".libs" && rm -f "libgcrypt.so.20" && ln -s "libgcrypt.so.20.3.0" "libgcrypt.so.20")
libtool: link: (cd ".libs" && rm -f "libgcrypt.so" && ln -s "libgcrypt.so.20.3.0" "libgcrypt.so")
libtool: link: ( cd ".libs" && rm -f "libgcrypt.la" && ln -s "../libgcrypt.la" "libgcrypt.la" )
/bin/bash ../libtool  --tag=CC   --mode=link clang  -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith   -o mpicalc mpicalc-mpicalc.o libgcrypt.la -L/usr/lib/x86_64-linux-gnu -lgpg-error 
libtool: link: clang -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -o .libs/mpicalc mpicalc-mpicalc.o  ./.libs/libgcrypt.so -L/usr/lib/x86_64-linux-gnu -lgpg-error
make[2]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/src'
Making all in doc
make[2]: Entering directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/doc'
make  all-am
make[3]: Entering directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/doc'
make[3]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/doc'
make[2]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/doc'
Making all in tests
make[2]: Entering directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/tests'
clang -DHAVE_CONFIG_H -I. -I../../tests -I..  -I../src -I../../src   -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -MT version.o -MD -MP -MF .deps/version.Tpo -c -o version.o ../../tests/version.c
mv -f .deps/version.Tpo .deps/version.Po
/bin/bash ../libtool  --tag=CC   --mode=link clang  -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -no-install  -o version version.o ../src/libgcrypt.la ../compat/libcompat.la -L/usr/lib/x86_64-linux-gnu -lgpg-error  
libtool: link: clang -O2 -flto -fvisibility=hidden -fno-delete-null-pointer-checks -Wall -Wcast-align -Wshadow -Wstrict-prototypes -Wformat -Wno-format-y2k -Wformat-security -W -Wextra -Wbad-function-cast -Wwrite-strings -Wdeclaration-after-statement -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -o version version.o  ../src/.libs/libgcrypt.so ../compat/.libs/libcompat.a -L/usr/lib/x86_64-linux-gnu -lgpg-error -Wl,-rpath -Wl,/home/jussi/kernel/libgcrypt/build-clang-overopt/src/.libs
/usr/bin/ld: ../compat/.libs/libcompat.a: error adding symbols: archive has no index; run ranlib to add one
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [Makefile:776: version] Error 1
make[2]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt/tests'
make[1]: *** [Makefile:500: all-recursive] Error 1
make[1]: Leaving directory '/home/jussi/kernel/libgcrypt/build-clang-overopt'
make: *** [Makefile:432: all] Error 2

Any ideas what might be causing this?

Configure output has still has some differences LTO vs non-LTO:

--- non-lto.log 2021-01-21 22:25:14.966099577 +0200
+++ lto.log     2021-01-21 22:25:23.174086100 +0200
@@ -63,7 +63,7 @@
 checking for archiver @FILE support... @
 checking for strip... strip
 checking for ranlib... ranlib
-checking command to parse /usr/bin/nm -B output from clang object... ok
+checking command to parse /usr/bin/nm -B output from clang object... failed
 checking for sysroot... no
 checking for mt... mt
 checking if mt is a manifest tool... no
@@ -75,7 +75,7 @@
 checking if clang static flag -static works... yes
 checking if clang supports -c -o file.o... yes
 checking if clang supports -c -o file.o... (cached) yes
-checking whether the clang linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
+checking whether the clang linker (/usr/bin/ld) supports shared libraries... yes
 checking whether -lc should be explicitly linked in... no
 checking dynamic linker characteristics... GNU/Linux ld.so
 checking how to hardcode library paths into programs... immediate
@@ -168,8 +168,8 @@
 checking whether 'asm' assembler keyword is supported... yes
 checking whether '__asm__' assembler keyword is supported... yes
 checking whether inline assembly memory barrier is supported... yes
-checking whether GCC assembler is compatible for ARM assembly implementations... no
-checking whether GCC assembler is compatible for ARMv8/Aarch64 assembly implementations... no
+checking whether GCC assembler is compatible for ARM assembly implementations... yes
+checking whether GCC assembler is compatible for ARMv8/Aarch64 assembly implementations... yes
 checking whether GCC assembler supports for CFI directives... yes
 checking whether GCC assembler supports for ELF directives... yes
 checking for _ prefix in compiled symbols... no
@@ -240,7 +240,7 @@
 checking if gcc supports -Wno-missing-field-initializers... yes
 checking if gcc supports -Wpointer-arith... yes
 checking whether non excutable stack support is requested... yes
-checking whether assembler supports --noexecstack option... yes
+checking whether assembler supports --noexecstack option... no
 checking that generated files are newer than configure... done
 configure: creating ./config.status
 config.status: creating Makefile

In T5255#141976, @jukivili wrote:

Unfortunately these are not enough to enable building with clang LTO. Build now fails when linking test executables (libgcrypt.so gets build however):

Any ideas what might be causing this?

Thanks for the patches, with them I can't replicate your linking issues strangely. It builds & links fine for me, thanks.

I've tried to reproduce it with -flto=thin, -flto, ld.bfd and ld.lld (my default linker) but they all link fine (version 1.9.0). My configure output looks like the same as yours.

libgcrypt-1.9.0-clang-lto.build.txt215 KBDownload

Problem was that my build system was selecting "ar" and "ranlib", where as your build system selects "llvm-ar" and "llvm-ranlib".

I got it building with:

AR="llvm-ar" RANLIB="llvm-ranlib" CC=clang CFLAGS="-O2 -flto" ../configure --enable-maintainer-mode

Should we add this to the hints in the README? After all this does not seem to be the standard system compiler or it has not been properly setup as replacement.

That might be helpful. But, on the other hand, if I had just googled the problem I was seeing I would have gotten answer quite fast.

jukivili added a commit: rC393bd6c3d1aa: configure.ac: run assembler checks through linker for better LTO support.Jan 26 2021, 6:54 PM

jukivili added a commit: rC9f49e806f950: sha512/sha256: remove assembler macros from AMD64 implementations.

• Jab added a commit: rGb08418d22cc8: gpg: Fix ugly error message for an unknown symkey algorithm..Jan 27 2021, 12:11 PM

aheinecke removed a commit: rGb08418d22cc8: gpg: Fix ugly error message for an unknown symkey algorithm..Jan 27 2021, 12:21 PM

• gniibe added a subtask: T5259: Release Libgcrypt 1.9.1.Jan 29 2021, 1:11 AM

• werner mentioned this in T5259: Release Libgcrypt 1.9.1.Jan 29 2021, 11:24 AM

• werner changed the status of subtask T5259: Release Libgcrypt 1.9.1 from Open to Testing.

jukivili closed subtask T5256: libgcrypt, convert Intel syntax x86_64 assembly files to AT&T syntax as Resolved.Feb 6 2021, 12:47 PM

• werner closed subtask T5259: Release Libgcrypt 1.9.1 as Resolved.Feb 12 2021, 12:34 PM

jukivili closed this task as Resolved.Apr 26 2021, 5:43 PM

jukivili claimed this task.

libgcrypt: build "error: invalid operand for instruction" when compiling with Clang & LTOClosed, ResolvedPublicActions