I've the same issue.
Seems to be related to the server response. Can't verify but I think it happens when the server doesn't allow any new connections.
I get this "handshake" error messages when I'm connected to one server with multiple WinSCP instances and all of them use multiple connections in the background.
The messages vary sometimes
Received invalid elliptic curve point in ECDH reply
Registered server host key is invalid
(if I remember correctly)
Would be great if this error could be catched and forking new background connections could be stopped for this upload or session.