the passphrase that is read from the --passphrase-fd is read at program startup, way before gnupg can decide which key to use (and hence, which passphrase). Is it possible to read the passphrase from the passphrase-fd later on?
No,,this is not possible for various reasons. Either use gpg-agent or make used of the --status-fd/--command-fd feature to pass the passphrase when requested.
Also consider using gpgme which makes using gpg from other programs much easier.