Fetch key for every signature

Previously the key was fetched only when the signature is valid. This
also remove some nesting for the if