Slow download using .GetFileToDirectory if directory has about 130.000 files

Advertisement

LukasKarner
Joined:
Posts:
1

Slow download using .GetFileToDirectory if directory has about 130.000 files

Hello,

I am currently having trouble downloading a small file from an FTP server using the WinSCP .NET assembly (C#).
I tried using the .GetFileToDirectory and .GetFiles method.
The WinSCP version I am using is 5.19.6 on Windows 10.

The problem is: It takes a long time to download a small file if the folder it is in has a lot of files in it.
In our case, it takes between 19 and 50 seconds to download a 21 KB file from a folder that has about 130.000 files in it.

From what I have read in other posts and could see in the log files, the method sends a "LIST" command upfront, which appears to be the cause of this delay.
This behavior is present both on a client FTP server and our own FTP server.
I have attached the log files we got from our FTP server using the .GetFileToDirectory method.

I have also tried adding the RawSetting "FtpUseMlsd". "0" resulted in "Could not retrieve file information" and "1" had no change.

Can you tell me if I can disable this list command? Or is there a better solution, how to download a specific file from a folder with thousands of files in it?

Regards Lukas
  • test06-02-2022_remote.7z (251.79 KB, Private file)

Reply with quote

Advertisement

tuliomarostegon
Joined:
Posts:
1
Location:
brazil

Same Scenario - Slow download using .GetFileToDirectory

Hi martin, I have the same scenario here, can you help me about it? i'm trying to download about 2.6k files.

I'm using the method GetFilesToDirectory to, here is my code attached

I tried to recover the log, which even Lukas did, but I was not successful, please let me know case it's necessary considering that it's probably the same case.

Regards!
  • DownloadFTPFilesUsingWINScp.txt (1.11 KB, Private file)
Description: Code

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
40,587
Location:
Prague, Czechia

Re: Same Scenario - Slow download using .GetFileToDirectory

@tuliomarostegon: Please attach a full session log file showing the problem (using the latest version of WinSCP).

To generate the session log file, set Session.SessionLogPath. Submit the log with your post as an attachment. Note that passwords and passphrases not stored in the log. You may want to remove other data you consider sensitive though, such as host names, IP addresses, account names or file names (unless they are relevant to the problem). If you do not want to post the log publicly, you can mark the attachment as private.

Btw, this is pointless:
option.TlsHostCertificateFingerprint = session.ScanFingerprint(option, "SHA-256");
It's an ineffective way to do the same as
option.GiveUpSecurityAndAcceptAnyTlsHostCertificate = true;
As the property name suggests, it's not best thing to do.

Another thing is that you are not calling TransferOperationResult.Check.
See https://winscp.net/eng/docs/library_session#results

Reply with quote

Advertisement

You can post new topics in this forum