KeyCache: use weak/shared_ptr without exceptions

We can do this easier, faster and better without exceptions, so let's do
it that way.