Prevent tail call optimization with _gcry_burn_stack
150c0313f971Unpublished
Actions

Unpublished Commit · Learn More

Not On Permanent Ref: This commit is not an ancestor of any permanent ref.

Description

Prevent tail call optimization with _gcry_burn_stack

* configure.ac: New check, HAVE_GCC_ASM_VOLATILE_MEMORY.
* src/g10lib.h (_gcry_burn_stack): Rename to __gcry_burn_stack.
(__gcry_burn_stack_dummy): New.
(_gcry_burn_stack): New macro.
* src/misc.c (_gcry_burn_stack): Rename to __gcry_burn_stack.
(__gcry_burn_stack_dummy): New.

Tail call optimization can turn _gcry_burn_stack call in to tail jump. When
this happens, stack pointer is restored to initial state of current function.
This causes problem for _gcry_burn_stack because its callers do not count in
current function stack depth.

One solution is to prevent gcry_burn_stack being tail optimized by inserting
dummy function call behind it. Another would be to add memory barrier 'asm
volatile("":::"memory")' behind every _gcry_burn_stack call. This however
requires GCC asm support from compiler.

Patch adds detection for memory barrier support and when available uses
memory barrier to prevent when tail call optimization. If not available
dummy function call is used instead.