If there is no explicit rule that downvotes are not for agreement, (and there’s pretty universal usage of downvotes in that fashion), then there should never be consequences for using the tool provided. Public and popularized tracking of downvotes will result in consequences. (Bans, harassment)

It would be better for instances to remove downvotes entirely, if they believe downvoting is something to be policed.

Increasing pressure on users to follow rules that are not on the sidebar, and not baked into the tool fundamentally (as in making downvotes unavailable) will only lead to user distaste and reduce engagement.

Users must be able to transparently understand the rules they are subject to