I managed to create (artificial) reproducible case where scdaemon hangs up.
I have a Gnuk Token inserted, and insert/remove Yubikey for this experiment.
(1) first terminal
$ gpg --card-edit ... gpg/card>
This is needed, for another terminal can do DEVINFO --watch for scdaemon by socket connection.
(2) second terminal
Invoke DEVINFO --watch but not consuming its output, by stopping Ctrl-Z
$ gpg-connect-agent > SCD DEVINFO --watch ... ^Z
(3) third terminal
Observing card statuses.
$ while sleep 0.5; do gpg --card-status all; done
And inserting/removing a Yubikey many times.
Eventually (for my case, 45 times of inserting/removing), the gpg process stops proceeding (because scdaemon hangs up).
(4) fourth terminal
After scdaemon hangs up, listing secret keys hangs.
$ gpg -K