Fetch key for every feature

Previously the key was fetched only when the signature is valid. This
also remove some nesting for the if