gnupg-agent when run with --enable-ssh-support doesn't seem to have any easy way to 'forget' ssh keys and smartcard pin's.
From the man page you would think sending a SIGHUP would do it, but it doesn't seem to have any effect.
Additionally, 'ssh-add -D' says it deleted all identities, but they just reappear when you next use ssh.
You can do:
gpg-connect-agent "SCD KILLSCD" "SCD BYE" /bye
to get it to re-ask your pin for the key, but thats hardly user friendly.