One can configure gpg-agent to stop caching of passphrases after some time (max-chache-ttl etc.).
As this does not apply for smartcards yet, it would be nice to add this feature. The smartcard is caching the PIN by itself until it is powered down. Therefore a possible solution would be to power down the card after the time specified in max-cache-ttl to make sure, that the PIN is not cached anymore. At least this is the behaviour a user would probably expect when setting this configuration.
See https://lists.gt.net/gnupg/users/81257 for problem description.