Fix the same bug as the last commit, but better. Deleting from a set doesn't invalid...