I’m not sure it would. If my instance knows who up and downvoted a post, then they can work it out on the fly per post without even needing to tell me who did what.

The problem is the sorting of the posts, i.e. what sits at the top of your feed. That can definitely not be calculated on the fly for every single user.