You aren’t giving us enough information to even speculate the answer. Are these Enterprise grade servers in a datacenter? Are these home made servers with consumer or low grade hardware you’re calling servers? Are they in the same datacenter or do they go out to the Internet? What exists between the hops on the network? Is the latency consistent? What is the quality of both sides of the connection? Fiber? Wi-Fi? Mobile? Satellite?
Does it drop too nothing or just settle into a constant slower speed? What have you tried to trouble shoot? Is it only rsync or do other tests between the hosts show the same behavior?
Give us more and you might get some help. If these hosts are Linux I would start with iperf to do a more scientific test. And report to us some more info.
There are total bypass options now to completely remove their hardware from your network using an ONT that lets you clone the att device serial number. Just a heads up.