This bug is forwarded from http://bugs.debian.org/574672:
"gpg-agent seems to wake up every few seconds, due to select timing out.
gpg-agent should not wake up until either it has something to do, or it
needs to time out a key from memory. The former occurs when select says
it has data to read from its socket, and the latter should sleep until
the target time rather than waking up every couple of seconds."