prime: Avoid rare assertion failure in gcry_prime_check.

Authored by werner on Apr 28 2018, 6:30 PM.

Description

prime: Avoid rare assertion failure in gcry_prime_check.

* cipher/primegen.c (is_prime): Don't fail on the assert X > 1.

When using gcry_prime_check the function is_prime can be called with
quite small candidates so there is a real chance that the random X
values is indeed 0 or 1. This would trigger the assert. To avoid
this we now retry in this case.

(cherry picked from commit f3362f10f6f671246c38115ed12b0047966c200e)

  • Reported-by: Heiko Stamer
  • Signed-off-by: Werner Koch <wk@gnupg.org>

Details

Committed
wernerApr 29 2018, 6:05 PM
Parents
rC22db6237de00: Make BMI2 inline assembly check more robust
Branches
Unknown
Tags
Unknown
stm added a subscriber: stm.Sep 9 2018, 8:33 AM

By the attached test program I can confirm that the issue is solved.

Thanks.