FTPS transfers very slow for many small files

Advertisement

Sharken
Joined:
Posts:
11

FTPS transfers very slow for many small files

Hi,
I am transferring a lot of small files (20.000) using Implicit FTPS on port 990.
Transfer is extremely slow, there is a visible delay between each individual file transfer.

If i compare with FileZilla the difference is massive. Apparently WinSCP is doing something that
is hurting performance when copying lots of small files.

Hopefully this behavior can be improved, as it is unusable for a large amount of small files as it is.
This was tested on version 5.2.2 (Build 3365).

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
28,317
Location:
Prague, Czechia

Re: FTPS transfers very slow for many small files

I believe you are a victim of this bug:
https://winscp.net/tracker/1021

It is fixed in 5.1.6 and will be fixed in 5.2.3 beta.
_________________
Martin Prikryl

Reply with quote

Sharken
Joined:
Posts:
11

Just finished testing 5.2.3 beta and it hasn't improved the situation.

It seems that even the WinSCP GUI is slow when copying many small files. No more than 3-4 files
is copied each second, which is way too slow when you have 20k+ files to copy.

I cant really give any further info, there seems to be some kind of overhead for each file, that
prevents the filecopy processs from running fast :/

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
28,317
Location:
Prague, Czechia

Please attach a full log file showing the problem (using the latest version of WinSCP).

To generate log file, enable logging, log in to your server and do the operation and only the operation that causes the error. Submit the log with your post as an attachment. Note that passwords and passphrases not stored in the log. You may want to remove other data you consider sensitive though, such as host names, IP addresses, account names or file names (unless they are relevant to the problem). If you do not want to post the log publicly, you may email it to me. You will find my address (if you log in) in my forum profile. Please include link back to this topic in your email. Also note in this topic that you have emailed the log.

_________________
Martin Prikryl

Reply with quote

Advertisement

tomb
Joined:
Posts:
19

I was the first victim of this bug 1021

It was indeed solved in 5.1.6. With this bug solved I transferred 2800 small files in 17 min: about 3 files/sec. This is fast enough for me. With the bug it took several hours.

It seems that WINSCP has quite a lot of overhead for transferring one file. I hope it is identified as a new bug and that the overhead can be reduced.

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
28,317
Location:
Prague, Czechia

Sharken wrote:

Just finished testing 5.2.3 beta and it hasn't improved the situation.

It seems that even the WinSCP GUI is slow when copying many small files. No more than 3-4 files
is copied each second, which is way too slow when you have 20k+ files to copy.

I cant really give any further info, there seems to be some kind of overhead for each file, that
prevents the filecopy processs from running fast :/
This bug has been added to the tracker:
https://winscp.net/tracker/1058

Reply with quote

Advertisement

tomb
Joined:
Posts:
19

So it seems the bug 1058 is solved in 5.1.8. I have the same performance issue in FTP and SFTP with a transfer speed of at most 3-4 files/sec. I wonder if this bug and fix is specifically for FTPS. Does anyone know this? I am waiting for the release of 5.1.8 to test it myself

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
28,317
Location:
Prague, Czechia

tomb wrote:

So it seems the bug 1058 is solved in 5.1.8. I have the same performance issue in FTP and SFTP with a transfer speed of at most 3-4 files/sec. I wonder if this bug and fix is specifically for FTPS. Does anyone know this? I am waiting for the release of 5.1.8 to test it myself
Please start a new topic and attach a complete session log file showing your problem. Separately for FTP and SFTP. Thanks.
_________________
Martin Prikryl

Reply with quote

Advertisement

tomb
Joined:
Posts:
19

I just did a short test with SFTP and FTP and found an SFTP speed of 20-25 files/sec for tiny files. That is much faster than FTP and fully adequate for me.

I will start a new topic for the slow FTP.

Reply with quote

Advertisement

You can post new topics in this forum