SFTP protocol specification requires that client and server uses UTF-8 encoding (Unicode) for file names.
WinSCP by default uses UTF-8 encoding. You can force non-default behavior using session option UTF-8 encoding for filenames, particularly when your server does not use UTF-8.
Please be aware that if your server does not support UTF-8 encoding, but uses its local legacy encoding instead, it is its fault. The problem is not on WinSCP side. You should push your server provider to add support for UTF-8, and not ask for support of legacy encoding in WinSCP.
By default WinSCP will automatically fallback to legacy Windows encoding, when a server returns a directory listing with file names not in UTF-8 encoding. For this reasons, you might get an inconsistent behavior during operations with files, depending on whether you visited a folder with non-ASCII file names before or not (or did not visit any directory at all, such as in scripting). To get a consistent behavior with a server that does not use UTF-8, disable UTF-8 encoding explicitly in session settings.
With WebDAV, a client and a server are required to use UTF-8. No configuration is necessary.
When not using UTF-8 encoding, WinSCP assumes the server is using the same legacy ANSI encoding as configured for local machine in Control Panel > Clock and Region > Region > Administrative > Language for non-Unicode programs.