SCP and auto detection of UTF-8 encoding for filenames :: Support Forum

bjmi
Joined:: 2020-02-09
Posts:: 2

SCP and auto detection of UTF-8 encoding for filenames

2020-02-09 10:24

I haven't encounting problems with directory and filename encoding so far.
But recently it was necessary to change UTF-8 encoding for filenames to "On" instead of the default "Auto" to retain right filenames while copying files to SSH server running Unraid OS using SCP method.
Filenames result in GebÃ¼hr or BeitrittserklÃ¤rung but should be Gebühr and Beitrittserklärung.

According to

With SCP protocol this means that UTF-8 will be used when a LANG environment variable is set to “UTF-8” encoding.

https://winscp.net/eng/docs/ui_login_environment#utf
utf8 should be detected.
LANG is set to en_US.UTF-8.

root@unraid:~# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE=C
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

root@unraid:~# printenv LANG
en_US.UTF-8

Any ideas why utf8 auto detection doesn't work or do other checks exist that aren't mentioned in FAQ?

unraid-root.log (29.67 KB)

Description: Debug 1 log to Unraid OS

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,517
Location:: Prague, Czechia

Re: SCP and auto detection of UTF-8 encoding for filenames

2020-02-11

Your LANG variable is probably set for interactive sessions only. You need to fix your startup scripts to set it for all (even non-interactive) sessions.

Typically you need to move the relevant commands from .bash_profile to .bashrc.

This is actually the opposite of this problem:
https://winscp.net/eng/docs/message_large_packet

Reply with quote

bjmi
Joined:: 2020-02-09
Posts:: 2

2020-02-11 16:29

I tried following locations

/root/.bash_profile
/root/.bashrc
/etc/environment
/root/.ssh/rc

but without success. It doesn't detect UTF-8 on Unraid 6.8.2 (Linux 4.19.98-Unraid).

I end up with the global setting to On in winscp.ini

[Sessions\Default%20Settings]
Utf=1

Reply with quote

martin◆ Site Admin

2020-02-14

Well, the point is that in WinSCP session the variable is not set. So it's not a problem on WinSCP side. I do not know your system, nor I'm a *nix expert, so I cannot advice you how to properly configure your system to set the variable for all sessions. Sorry.

Reply with quote

tomftp Guest

FTP detection of UTF8

2021-01-27 15:43

Here https://winscp.net/eng/docs/ui_login_environment#utf it is stated that "With FTP protocol this means that UTF-8 will be used when the server announces support for it."

But on our VSFTPD Linux server the WinSCP app does not detect UTF8 from this FTP server when set to Auto. Everyone needs to remember to change the Environment option to On.

What is expected from the server by WinSCP to detect it? Or perhaps WinSCP detection mechanism from FTP is not good enough solution and WinSCP should be changed?

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,517
Location:: Prague, Czechia

Re: FTP detection of UTF8

2021-01-28

tomftp wrote:

What is expected from the server by WinSCP to detect it? Or perhaps WinSCP detection mechanism from FTP is not good enough solution and WinSCP should be changed?

WinSCP looks for UTF8 in the FEAT response. What the latest version of vsftpd does.

If it does not work for your, please attach a full session log file showing the problem (using the latest version of WinSCP).

To generate the session log file, enable logging. Submit the log with your post as an attachment. Note that passwords and passphrases not stored in the log. You may want to remove other data you consider sensitive though, such as host names, IP addresses, account names or file names (unless they are relevant to the problem). If you do not want to post the log publicly, you can mark the attachment as private.

Reply with quote

SCP and auto detection of UTF-8 encoding for filenames