FreeNAS can do it (either via AFS or SMB). Or you can just set up a Samba server with the right magic incantations and use that network share to do Time Machine. I use FreeNAS since I already have it on my network. I gave each Mac its own ZFS volume so I can limit the size independently.
I’ve used IPSec to connect to my home (and office when I had my own company) networks. It has never failed me and the client is built right into my iphone and mac.