Fix rerun selftests

Now that we used the cached config we have to clear it
before a rerun.