Setup was inserting a yubikey and a "signature card v2.0" and then the loading bar continued for more than an hour (over lunch):
There where no gpg or gpgsm processes running when I noticed the situation.
It stopped after executing:
gpgconf --kill scdaemon
This seems to be not easily reproducible, but maybe you can see something from the log.
After removing the signature card and inserting it again, everything worked as expected and the certificates where loaded (but meanwhile I had activated debugging for gpgsm, if this is relevant):