This error occurs when WinSCP decrypts an SSH packet and the decrypted data makes no sense. This probably means something has gone wrong in the encryption or decryption process. It’s difficult to tell from this error message whether the problem is in the client or in the server, or in between.
Sometimes the error is a result of the connection fatally failing for whatever reason even completely unrelated to the encryption. For example, when the server runs out of disk space.