There are signing keys involved, so if someone puts up a new server but uses different keys then all sorts of federation trouble will await them.

That said it shouldn’t affect the general network, just that individual server (both the communities and the users of it)

Edit: As for switching domains on an existing server, that would be equally troublesome as ActivityPub kinda relies on domains for all sorts of IDs.