But wouldn’t that invalidate the usage of that word in the circles that use it wrong, and not for those who use it properly.

Like if there was a hypothetical town where the word “good” was used to describe bad things, would that town invalidate the word “good” for every single town? Of course it wouldn’t, it would only invalidate the usage of that word by the ones who use the word in question wrong.