Copyright law in general needs changing though that’s the real problem. I don’t see the advantage of legally mandating that a hacky workaround solution becomes a legally mandated requirement.

Especially because there are many many legitimate reasons to ignore robots.txt including it being misconfigured or it just been set up for search engines when your bot isn’t a search engine crawler.