Make key fetching async and move it to the model

This also now handle missing recipient keys properly