libgpg-error: cross-compiling fails with "src/syscfg/lock-obj-pub.linux-gnueabi.h': No such file or directory"
Closed, ResolvedPublic

Description

This bug has been discussed before (eg,
https://lists.gnupg.org/pipermail/gnupg-devel/2015-January/029403.html and
T1744) but no adequate solution has been
proposed. It still exists for various host triplets such as, in my case,
"armv7a-hardfloat-linux-gnueabi".

The problem as I understand it:

  1. The configure script uses build-aux/config.sub to validate the given host

triplet. AFAIK it's not expected to map names to the most common subarch
triplet, just to validate, invalidate, or adjust such triplets for GNU conformance.

  1. The host name triplet is then fed to /src/mkheader.

For "x86_64-pc-linux-gnu" for instance, './mkheader linux-gnu
x86_64-pc-linux-gnu
/var/tmp/portage/dev-libs/libgpg-error-1.22/work/libgpg-error-1.22/src/gpg-error.h.in
../config.h 1.22 0x011600 >gpg-error.h'.
src/mkheader uses the canonical host name for the header file in "src/syscfg",
i.e., "src/syscfg/lock-obj-pub.x86_64-pc-linux-gnu.h". If the name isn't in the
name of one of the now 31 header files, the build ends in an error. Though
there exists a mapping table in src/mkheader.c it is quite small and only offers
a one-to-one correspondence between host triplets.

Assumptions shouldn't be made about the subarch component of the host name.
libgpg-error shouldn't rely on a one-to-one correspondence between valid GNU
host triplets and header files. Though there is no perfect solution there
should at least some minimal glob pattern matching for host triplets and header
files.

This patch is to src/mkheader.c, to enable some host name glob pattern matching.
For instance "arm*linux-gnueabi" to "arm-unknown-linux-gnueabi" and
"i?86*linux-gnu" to "i686-pc-linux-gnu".

Details

Nsane457 set External Link to https://bugs.gentoo.org/show_bug.cgi?id=584052.May 27 2016, 4:58 AM
Nsane457 set Version to 1.22.
Nsane457 added a subscriber: Nsane457.
werner added a subscriber: werner.May 27 2016, 11:53 AM

config.sub is indeed intended to canonicalize triplets. Thus a an up-to-date
config.sub should fix this. In any case you can always override the guessed
value like this:

  -/configure --build=$(build-aux/config.guess) --host=arm-unknown-linux-gnueabi

I fear that a single r.e. is not precise enough; config.sub has more complicated
rules.

Whether or not config.sub is up to date should be irrelevant as to whether
libgpg-error should be able to handle CPU variants and the second field of a
HOST. It is supposed to be treated as a freeform field (see
http://airs.com/ian/configure/configure_4.html). As to the precision of the r.e,
it is not supposed to fix or mimic the logic of config.sub. It is to
effectively pigeonhole some HOST strings to a valid header file. Can you give me
a scenario where arm*linux-gnueabi shouldn't map to
lock-obj-pub.arm-unknown-linux-gnueabi.h?

Also, what is the source of your assertion that armv7a-hardfloat-linux-gnueabi
is not a valid canonical triplet? Everything that I have ever read about HOST
strings such as "armv7a-hardfloat-linux-gnueabi" is that configure scripts treat
them as valid and parse them with case statements (as demonstrated here
http://airs.com/ian/configure/configure_4.html). They don't demand that the
end-user pass a generic version of HOST, like "arm-unknown-linux-gnueabi". The
only reason I chose to patch mkheader.c instead or configure.ac was to build on
the mapping logic already there.

werner added a comment.Jun 1 2016, 2:22 PM

Ian may have a different opinion on that (now) but the GNU build system defines
it in the way I described it.

werner renamed this task from lib-gpgerror: cross-compiling fails with "src/syscfg/lock-obj-pub.linux-gnueabi.h': No such file or directory" to libgpg-error: cross-compiling fails with "src/syscfg/lock-obj-pub.linux-gnueabi.h': No such file or directory".Jun 1 2016, 2:22 PM

The host naming scheme accepted by this project is really just Debian oriented.
There is nothing saying you have to aid other distro's hostnames but I have yet to
witness a project more unfriendly to non-Debianesque distros that libgpg-error.
You don't have to accept the patch. Understandable, since mkheader would not
ideally be the way to go. But you should at least be open to the idea of
supporting the more liberal host naming schemes that other distros employ.

If you decide to change your mind, this patch places the logic in configure.ac.

werner added a comment.Jun 8 2016, 6:46 PM

No, that is not acceptable for libgpg-error. Please report that to the GNU config
folks instead. Their repo is at git://git.sv.gnu.org/config.git

I would except that config.sub already validates such names as 'armv7a-hardfloat-
linux-gnueabi' so there is nothing to fix. It is libgpg-error that doesn't. If
you feel it should invalidate such names please report that to the GNU config
folks instead. Their repo is at git://git.sv.gnu.org/config.git. Otherwise
pleases fix libgpg-error.

Sorry, if _you_ want support for your _new target_ you should make sure that it
is supported by the GNU autotools which is used by a lot of software. This will
the soon be used by GnuPG etc.

It is entirely fine to point us new config versions with support for your
target. We will the update them in our packages - this is how we have done it
for close to 2 decades.

I'm sorry if your understanding of valid hostnames, acceptable by GNU projects, is
two decades old but this project happens to be the only one that assumes there
exists a finite list of valid hostnames without using pattern matching.

Using https://wiki.gentoo.org/wiki/Raspberry_Pi and a hostname of armv7a-
hardfloat-linux-gnu, with the notable exception of libgpg-error, I have been able
to compile and install all other GNU utilities included the core set of Gentoo
Linux, whether via the package's giving configure script or by use of autoreconf.

Hare are just a few of such GNU packages that I have personally been able to build
and install using the hostname "armv7a-hardfloat-linux-gnu":

coreutils-8.25
bash-4.3_p42
diffutils-3.3
findutils-4.6.0
grep-2.25
groff-1.22.3
gzip-1.8
tar-1.29
glibc-2.23
less-483
gawk-4.1.3
which-2.21
nettle-3.2
glibc-2.23
gcc-5.3.0
readline-6.3_p8
nano-2.5.3

The only other GNU package that doesn't compile for me is autogen. But that's due
to a lack of cross-compile support. It otherwise builds just fine natively on
armv7a-hardfloat-linux-gnu.

I would appreciate it if you could provide a specific GNU package using autotools
that you assert should fail to support such a hostname, other than this one, so
that I may provide a build log demonstrating that it indeed does.

Is armv7a-hardfloat-linux-gnu guaranteed to be ABI compatible to some other arm
triplet? If that is the case, I suggest to either drop your(?) invention of
-hardfloat- or, better, to work with the config mainatiners to make sure it is
viewed as an alias.

How does binutils handle this triplet?

If you can describe the user base for that triplet, I may add an exception to
mkheader to get things done faster.

Again, the host is not my invention. I linked it before and I'll do it again:
https://wiki.gentoo.org/wiki/Raspberry_Pi.

Gentoo's cross-compile tool, crossdev, suggests using "-hardfloat-" and "-
softfloat-" in the vendor field.

And here is how binutils handles this (they don't shy away from asterisks):
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=bfd/config.bfd

I can't find an explanation why gentoo inserts "-hardfloat". I doubt that this
is willy-nilly and as long as this has not been figured out, there is a
possibility of a different ABI and thus we can't simply alias it. Can you
please work with Kristian or someone else from gentoo to figure this out?

Thanks for binutils link.

There was a long drawn out discussion as to the validity of "-hardfloat" in the
triplet name. You can peruse at https://bugs.gentoo.org/show_bug.cgi?id=584052 .

I am not a dev and have pretty much given up. The Gentoo devs are adamant that this
is an upstream problem.

Yes, I have seen that URL but what I like to get an answer to my question here
or on gnupg-devel. I do not want to follow a possible long thread of some Linux
distribution.

ping (see T2370 (wk on Nov 18 2016, 08:44 AM / Roundup))

No reply to my question, thus it seems not to be important. Closing.
Note that replying to this will re-open the bug.

werner closed this task as Resolved.Jan 17 2017, 12:36 PM
werner claimed this task.

Hello @wener, I want to say that libgpg-error is the only one (!) application that fails to cross compile using valid toolchains: "armeb-unknown-linux-gnueabi" and "aarch64_be-unknown-linux-gnu". It compiles and runs perfectly using "arm-unknown-linux-gnueabi" and "aarch64-unknown-linux-gnu", but fails with big endian. I see project are actually using "hton/ntoh" so we shouldn't see this error. What this problem is about?

The host naming scheme accepted by this project is really just Debian oriented.

"armeb-unknown-linux-gnueabi" and "aarch64_be-unknown-linux-gnueabi" are 100% valid hosts for Debian, but they are not popular.

The problem is that GNU autotools maps:

"aarch64-unknown-linux-gnu" -> "lock-obj-pub.aarch64-unknown-linux-gnu.h"
"aarch64_be-unknown-linux-gnu" -> "lock-obj-pub.aarch64_be-unknown-linux-gnu.h"

Build tool can't find file "lock-obj-pub.aarch64_be-unknown-linux-gnu.h" and fails.

It looks like libgpg-error developers are using GNU config in a wrong way, it is not designed to handle "aarch64_be" -> "aarch64" cpu mapping. Developers just want more that GNU config can provide.

I see 3 ways:

  1. GNU config won't provide cpu mapping in existing api, it will break compatibility. Ask them to provide a new api with such mapping.
  2. Add cpu mapping like Peter said inside libgpg-error project.
  3. Move away from GNU autotools to something like cmake. It requires user to provide proper toolchain with all fields like "CMAKE_SYSTEM_PROCESSOR".

If I were the author of this project I will definitely move to cmake.

werner added a comment.Jan 2 2020, 9:43 AM

Please read libgpg-error's README. For each architecture we need to have a dedicated config file - this has nothing to do with autotools. Big and little endian variants are obviously different architectures. Here is an excerpt from the README

Libgpg-error needs to figure out some platform specific properties.
These are used to build the platform specific gpg-error.h file. The
detection is done during build time but can't be done when
cross-compiling. Thus if you run into an error during building you
need to figure out these values. You may use these commands:

build="$(build-aux/config.guess)"
./configure --prefix=TARGETDIR --host=TARGET --build=$build
cd src
make gen-posix-lock-obj
scp gen-posix-lock-obj TARGET:
ssh TARGET ./gen-posix-lock-obj >tmp.h
mv tmp.h "syscfg/$(awk 'NR==1 {print $2}' tmp.h)"

If you are using a VPATH build adjust accordingly. If this all works
for you (make sure to run the test programs on the target platform),
please send the generated file to the gnupg-devel mailing list so that
we can include it in the next release. Note that in addition to the
aliasing done by config.sub the src/mkheader build tool does some
extra aliasing to avoid having too much identical syscfg files.

If you have an identical architecture and you only require an alias, the way to go is to add this
to add a line to src/mkheader.c in the canon_host_triplet table.

I totally disagree.

Libgpg-error needs to figure out some platform specific properties.

Yes, project requires something like cmake toolchain. This is just file with target platform description in a declarative way. User defines cpu arch, compiler, linker, asm, special cflags etc. User knows nothing about software that will use it. Project developers uses values from internal cmake toolchain abstraction.

Thus if you run into an error during building you need to figure out these values. You may use these commands

We want user to use special commands for building project. I think this is wrong from architecture perspective. This is a job for advanced layer of abstraction (toolchain). Yesterday I've built complete image with Peter's patch in the following way:

aarch64_be-unknown-linux-gnu-emerge -v1 sys-apps/portage

You can find buildah script here for example.

sys-apps/portage depends on app-crypt/gnupg depends on libgpg-error. Everything just works perfect.

I am sure that syscfg/lock-obj-pub.* related commands need to be implemented internally based on variable (like CMAKE_SYSTEM_PROCESSOR) from toolchain. User don't want to take part in this commands. Toolchain abstraction needs to be provided by build system. User job will be to provide single toolchain for all applications.

If you have an identical architecture and you only require an alias, the way to go is to add this to add a line to src/mkheader.c in the canon_host_triplet table.

Yes, this was my previous patch built in 2016. I am watching on this issue more than 4 years. This is just workaround. Project wants right fix.

PS I forgot to say why movement to cmake will be the best way.

Please see the following github repo. This is just an example of cmake toolchains, There are more than 100 000 cmake toolchains available across internet for different platforms. All these guys won't be able to build libgpg-error. So I think it is too late to improve GNU autotools and add toolchain functionality. It will be better to just remove GNU autotools and use cmake. This is super hard work and can be done by core developers only, of course.