This was a case of "if something seems to be the case, keep digging even if people say it isn't".
Turned out to be an out-of-control process on the remote side. The process was intended to move files out of the destination folder to be processed. It was SUPPOSED to only move a particular filename (ignore all others), and also to only even look for files every five minutes or so.
The reality was it would move any file it found, and it would start trying to move the file about 19 seconds or so after it appeared! So what was happening was this:
1. We start transmitting the file, causing the file to appear in the destination folder.
2. The remote side immediately (within 19 seconds or so) moves the file fragment to another location.
3. Our session, seeing the target file suddenly gone, assumes the connection was broken, and it attempts to reconnect.
4. That was successful, so file transmission continues.
5. The file disappears again
6. and so on.
After we got the remote site to leave the stupid file alone and let us transmit it, it started working perfectly!