Improve whitelisting gpg.conf files

As gpgconf creates temporary gpg.conf files with .bak or
random temp suffixes the glob for everything triggers too often.
Not really a big issue as it mainly would not trigger additional
keylists but this way it's cleaner.