Lord Heseltine said it is time to stop ‘hiding from the damage’ caused by Brexit