Prevent sign-only keys from being added to a key group
(cherry picked from commit e831a50721ef4f93f39cc8a05fbbaf7b2f258980)