Topic "FTPS transfers very slow for many small files"

Author Message
Sharken
[View user's profile]

Joined: 2013-07-15
Posts: 11
Hi,
I am transferring a lot of small files (20.000) using Implicit FTPS on port 990.
Transfer is extremely slow, there is a visible delay between each individual file transfer.

If i compare with FileZilla the difference is massive. Apparently WinSCP is doing something that
is hurting performance when copying lots of small files.

Hopefully this behavior can be improved, as it is unusable for a large amount of small files as it is.
This was tested on version 5.2.2 (Build 3365).
Advertisements
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24552
Location: Prague, Czechia
I believe you are a victim of this bug:
http://winscp.net/tracker/show_bug.cgi?id=1021

It is fixed in 5.1.6 and will be fixed in 5.2.3 beta.
_________________
Martin Prikryl
Sharken
[View user's profile]

Joined: 2013-07-15
Posts: 11
Great news, will be testing 5.2.3 beta then
Sharken
[View user's profile]

Joined: 2013-07-15
Posts: 11
Just finished testing 5.2.3 beta and it hasn't improved the situation.

It seems that even the WinSCP GUI is slow when copying many small files. No more than 3-4 files
is copied each second, which is way too slow when you have 20k+ files to copy.

I cant really give any further info, there seems to be some kind of overhead for each file, that
prevents the filecopy processs from running fast :/
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24552
Location: Prague, Czechia
Please attach a full log file showing the problem (using the latest version of WinSCP).

To generate log file, enable logging, log in to your server and do the operation and only the operation that causes the error. Submit the log with your post as an attachment. Note that passwords and passphrases not stored in the log. You may want to remove other data you consider sensitive though, such as host names, IP addresses, account names or file names (unless they are relevant to the problem). If you do not want to post the log publicly, you may email it to me. You will find my address (if you log in) in my forum profile. Please include link back to this topic in your email. Also note in this topic that you have emailed the log.
_________________
Martin Prikryl
Sharken
[View user's profile]

Joined: 2013-07-15
Posts: 11
Email with logfile have been sent to you now, i hope that it will help you locate the issue.
Sharken
[View user's profile]

Joined: 2013-07-15
Posts: 11
Tracefile with additional debugging info sent.
tomb
[View user's profile]

Joined: 2013-07-05
Posts: 19
I was the first victim of this bug 1021

It was indeed solved in 5.1.6. With this bug solved I transferred 2800 small files in 17 min: about 3 files/sec. This is fast enough for me. With the bug it took several hours.

It seems that WINSCP has quite a lot of overhead for transferring one file. I hope it is identified as a new bug and that the overhead can be reduced.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24552
Location: Prague, Czechia
Sharken wrote:
Just finished testing 5.2.3 beta and it hasn't improved the situation.

It seems that even the WinSCP GUI is slow when copying many small files. No more than 3-4 files
is copied each second, which is way too slow when you have 20k+ files to copy.

I cant really give any further info, there seems to be some kind of overhead for each file, that
prevents the filecopy processs from running fast :/

This bug has been added to the tracker:
http://winscp.net/tracker/show_bug.cgi?id=1058
Sharken
[View user's profile]

Joined: 2013-07-15
Posts: 11
Can confirm the bug has been fixed, very pleased with how the bug was handled and ultimately fixed.

Thanks !
tomb
[View user's profile]

Joined: 2013-07-05
Posts: 19
So it seems the bug 1058 is solved in 5.1.8. I have the same performance issue in FTP and SFTP with a transfer speed of at most 3-4 files/sec. I wonder if this bug and fix is specifically for FTPS. Does anyone know this? I am waiting for the release of 5.1.8 to test it myself
Sharken
[View user's profile]

Joined: 2013-07-15
Posts: 11
According to http://winscp.net/eng/docs/history#5.2.4 the fix will be included in 5.1.8/5.2.5 beta. Note that neither of these has been released yet.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24552
Location: Prague, Czechia
tomb wrote:
So it seems the bug 1058 is solved in 5.1.8. I have the same performance issue in FTP and SFTP with a transfer speed of at most 3-4 files/sec. I wonder if this bug and fix is specifically for FTPS. Does anyone know this? I am waiting for the release of 5.1.8 to test it myself

Please start a new topic and attach a complete session log file showing your problem. Separately for FTP and SFTP. Thanks.
_________________
Martin Prikryl
tomb
[View user's profile]

Joined: 2013-07-05
Posts: 19
I just did a short test with SFTP and FTP and found an SFTP speed of 20-25 files/sec for tiny files. That is much faster than FTP and fully adequate for me.

I will start a new topic for the slow FTP.
Advertisements

You can post new topics in this forum






Search Site

What is WinSCP?

It is award-winning SFTP client, SCP client, FTPS client and FTP client integrated into one software program for file transfer to FTP server or secure SFTP server. [More]

And it's free!

Donate

About donations

$9   $19   $49   $99

About donations

Recommend

WinSCP Privacy Policy

WinSCP License