Skipping existing files is too slow for two reasons

Advertisement

Guest

Skipping existing files is too slow for two reasons

Hi, I'm trying to download a directory with about a million files using WinSCP 5.19.1. This takes longer than a day. So on the second day, I want to quickly skip existing files and continue downloading the other files.

However, WinSCP apparently is skipping existing files one by one extremely slowly. It skips them more slowly than it downloads them (because downloading can be done for several files in parallel). So on the second day, none of the remaining files are downloaded.

Please let WinSCP get the full file lists from both computers and compare them once with an efficient algorithm (sort both and go through them in parallel), then download the missing files. Don't compare the presence of files one by one.

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
36,052
Location:
Prague, Czechia

Re: Skipping existing files is too slow for two reasons

What exactly did you do to "skip existing files". That can be done at least three different ways in WinSCP.
Did you use "New and updated files" transfer settings?
https://winscp.net/eng/docs/ui_transfer_custom
Or synchronization?
https://winscp.net/eng/docs/task_synchronize_full
Why do you think you cannot transfer the files in parallel when skipping existing files?

Reply with quote

Advertisement

You can post new topics in this forum