Same problem with RSA
Very good, thanks so much for your help.
Before posting, please read how to report bug or request support effectively.
Bug reports without an attached log file are usually useless.
Please try 5.8.3 RC. It will provide more details in the error message.
I did some debugging. WinSCP agrees on DH key exchange with the ftp.box.com. But the ftp.box.com offers 768 bits key only.
Since OpenSSL 1.0.1r/1.0.2f, the OpenSSL requires 1024 bit key unconditionally:
https://github.com/openssl/openssl/commit/a4530ce0398d673b25a15b72a18dee1d8b14c07d
Please advice the box.com to upgrade their key.
The next version will include "dh key too small" message in the error.
Thanks for your report. What cipher does WinSCP 5.7.6 use against the server?
Please attach a full log file both from 5.7.6 and 5.7.7.
To generate log file, enable logging, log in to your server and do the operation and only the operation that causes the error. Submit the log with your post as an attachment. Note that passwords and passphrases not stored in the log. You may want to remove other data you consider sensitive though, such as host names, IP addresses, account names or file names (unless they are relevant to the problem). If you do not want to post the log publicly, you can mark the attachment as private.
SSL3 alert write: fatal: handshake failure
TLS connect: error in error
Can't establish TLS connection
Disconnected from server
Connection failed.
2016-03-10 17:35:56.640 --------------------------------------------------------------------------
. 2016-03-10 17:35:56.640 Session upkeep
. 2016-03-10 17:35:56.702 Connecting to ftp.box.com ...
. 2016-03-10 17:35:56.733 Connected with ftp.box.com, negotiating TLS connection...
< 2016-03-10 17:35:56.807 220 Service ready for new user.
> 2016-03-10 17:35:56.807 AUTH SSL
< 2016-03-10 17:35:56.900 234 Command AUTH okay; starting SSL connection.
. 2016-03-10 17:35:58.211 SSL3 alert write: fatal: handshake failure
. 2016-03-10 17:35:58.211 TLS connect: error in error
. 2016-03-10 17:35:58.211 Can't establish TLS connection
. 2016-03-10 17:35:58.211 Disconnected from server
. 2016-03-10 17:35:58.211 Connection failed.
. 2016-03-10 17:35:58.226 Got reply 1004 to the command 1
* 2016-03-10 17:35:58.242 (EFatal) Connection failed.
* 2016-03-10 17:35:58.242 SSL3 alert write: fatal: handshake failure
"Apologies for the delay here. We were able to reproduce your issue with WinSCP, but are having trouble recreating the errors in other FTP clients (eg. Filezilla) with the same setup (Explicit, port 21).
With the new version of WinSCP, are you able to connect with a non-encrypted connection without error?"