It's some decisions from my company's sysadmin to use passwords that long. That's why I can't really do anything about it.

I wonder if this limit can be up, but I understand that it must have some sort of limit, and maybe one day another guy would come in complaining because he has an even-longer password. A warning for the dialog would be nice, then.

WinSCP login dialog indeed limits length of the password to 100 characters.

What's your use case? Why do you use 256-long passwords? Why don't you use private key instead?

By enabling log with sensitive information I was able to see that.

WinSCP either stores, or uses from stored value, only the first 100 characters when trying to log into the server. When it fails and ask for the password again, it can use the full-length password, hence results in successful connection. Here I have a password of 256-character length.

Please see the attached log file for more information.