Topic "File transfer creates 3 lead characters"

Author Message
appleton98
[View user's profile]

Joined: 2014-03-19
Posts: 6
Location: Nottingham
On revision 5.1 build 3799 this did not occur but on version 5.5.1 build 3970 it does and its frustrating.
We copy a file that Business Objects creates from one server to a SFTP server in ASCII and all appears fine. If you then open the text file within WINSCP and select ENCODING from the menu and change it from UTF-8 to 1252 ANSI latin 1 the file suddenly has these three symbols at the start of the file. 
Files transferred in the old version do not have this issue.

Unfortunately due to the nature of the file content i can not post it on here.
Advertisements
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24512
Location: Prague, Czechia
The three characters are UTF-8 BOM mark:
https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8

I suppose the file has the BOM included on its own. When you open the file in WinSCP internal editor, it autodetects the UTF-8 encoding by its presence (and does not show the mark as it's not part of a contents). If you force WinSCP to interpret the file in ANSI encoding, it displays these characters as if they were part of the contents. But this has not changed between 5.1 and 5.5.1. Can you please test this once again using 5.1?

If the above does not explain what you experience, I have further questions:

What encoding do you have configured as default in Preferences?
https://winscp.net/eng/docs/ui_pref_editor

Am I right assuming you are using WinSCP internal editor, right?

What protocol? FTP or SFTP?

Is the BOM present in the file before upload? (the file is 3 bytes longer than its contents if BOM is included).
appleton98
[View user's profile]

Joined: 2014-03-19
Posts: 6
Location: Nottingham
Having checked the original file before WINSCP sees it appears it does indeed have the BOM included on its own. To do this i had to use the internal editor of WINSCP from version 5.5.1
I am using SFTP protocol in both cases however i had made a mistake in the revisions. The new version is indeed 5.5.1 however the old version is not 5.1 but is in fact version 4.3.5 build 1463, appologies
If i transfer the same file from the same location using version 4.3.5 and then open it with the internal editior of 5.5.1 then the three characters do not appear when the encoding is changed between UTF-8 to 1252 ANSI latin 1 and back again. However do exactly the same with version 5.5.1 and they do.

I have enclosed two images from both versions showing the Editor page.
In version 5.5.1 i have tried the transfer with both 1252 ANSI chosen as default and then UTF-8 with the same result.

Is there anything else you could think of i could try?
We upgraded from version 4.3.5 to 5.5.1 in order to fix an authentication problem and the new verion allowed us to use the switch -hostkey="*" after the login process to solve that.
editors 5.5.1.jpg (53.14 KB)

Description: (none)

editors 5.5.1.jpg

editors 4.3.5 1463.JPG (48.76 KB)

Description: (none)

editors 4.3.5 1463.JPG

martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24512
Location: Prague, Czechia
OK, this makes sense.
It's because 4.3.5 stripped the BOM on upload (not download). This has been changed:
http://winscp.net/tracker/show_bug.cgi?id=842
But you can configure the previous behavior. See Remove BOM and EOF marks:
http://winscp.net/eng/docs/ui_transfer_custom
_________________
Martin Prikryl
appleton98
[View user's profile]

Joined: 2014-03-19
Posts: 6
Location: Nottingham
I had previously set winscp to remove the BOM remarks but to no avail. The system basically picks up a file from a server and places it into a folder on a sftp server. IS there something i'm doing wrong?
transfer settings 5.5.1.JPG (53.45 KB)

Description: (none)

transfer settings 5.5.1.JPG

martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24512
Location: Prague, Czechia
It's difficult to debug this remotely.
So when you have the BOM removing enabled, does the BOM disappear from the file on upload or not?

Also, what is the actual problem? Why are you even trying to change the encoding in WinSCP from UTF-8 to Ansi, when the file is (probably) in UTF-8 (providing it does have UTF-8 BOM)?
_________________
Martin Prikryl
appleton98
[View user's profile]

Joined: 2014-03-19
Posts: 6
Location: Nottingham
Hello,
With remove BOM enables it does not appear to remove the BOM characters from the file.
I am trying to change the encoding from UTF-8 to Ansi to see whether the three control characters have been removed.
The customer who i am trying to help requires that the characters be removed however uploading their exported file is automated and it sees the three characters and rejects the file accordingly. I have changed the encoding on files from version 4.3.5 and they do not have this issue. Currently we have moved this customer to our old server running 4.3.5 but this is not realistically a permanent solution.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24512
Location: Prague, Czechia
Please attach a full log file showing upload of a file using 5.5.2 with BOM removing enabled (but not happening).
_________________
Martin Prikryl
appleton98
[View user's profile]

Joined: 2014-03-19
Posts: 6
Location: Nottingham
Hi,
unfortunately this issue was resolved but has occurred once again. I have included a screen shot of the transfer settings. When i manually copy one file from the source to the SFTP server from within WINSCP GUI interface these settings are utilised and i am happy, however when i use a script it ignores these settings and the file contains the three characters once again (as the original also does) Is there a better way to transfer files though a script that will force it to use the settings listed? I have not only added a new settings config but also changed all the others that come with WINSCP to match thinking it must use one of them, but it appears not.
My current transfer script is below

open f***r:q******r@sftp**.********.co.uk
cd /incoming/BO
put -delete "C:\SFTP\*****\Business Objects\*.*"

due to the nature of the company i work for i have had to * out some information.

Is there a way to force the sript to use the transfer settings or should i be using a different command to complete file transfer?
Many thanks
winscp transfer settings.JPG (71.99 KB) [Download]

Description: screen shot showing transfer settings

appleton98
[View user's profile]

Joined: 2014-03-19
Posts: 6
Location: Nottingham
forgot to say i am using WINSCP version 5.5.1 build 3970
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24512
Location: Prague, Czechia
appleton98 wrote:
Is there a better way to transfer files though a script that will force it to use the settings listed?

Please read FAQ:
http://winscp.net/eng/docs/faq_script_ini
Advertisements

You can post new topics in this forum






Search Site

What is WinSCP?

It is award-winning SFTP client, SCP client, FTPS client and FTP client integrated into one software program for file transfer to FTP server or secure SFTP server. [More]

And it's free!

Donate

About donations

$9   $19   $49   $99

About donations

Recommend

WinSCP Privacy Policy

WinSCP License