Disable size check for background transfers

Advertisement

kklh
Guest

Disable size check for background transfers

Hello.
I have folders on server around 1Tb with about a million small files. I'm trying to download it, but size check takes a lot of time (tens of minutes). I can download it in foreground mode without size check, but in foreground mode it's going with only one connection (~2–3Mb/s). To use few connections (four for 8-10Mb/s) I need to put transfer in background mode, but in the background mode is not possible to disable size check.
Is it possible to add option to disable size check in background mode or option to use >1 connection in the foreground mode?

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
41,852
Location:
Prague, Czechia

Re: disable size check for background transfers

There's no "size check". WinSCP just retrieves the list of files to download. As documented here, it does not add any time to the overall transfer duration:
https://winscp.net/eng/docs/ui_transfer_custom#calculate
The only difference is that the order of operations is different. So it takes longer before the first transfer actually starts. But in the end it does not matter.

Reply with quote

saibot
Guest

Re: disable size check for background transfers

I don't get this (cancelling the calculation)...would be great.
I am looking for the option to use background transfers but at the same time avoid the size calculation.

Currently, even if you disable multiple connections and set parallel connection only to 1, and you have only one background transfer, AND you have disabled the "Calculate total size checkbox" in transfer settings: still the "size calculation" takes place and takes ages.

This is so frustrating.

Even if there is the option to not use the background transfer at all (only then, size calculation does not happen - but you cannot handle multiple projects then at the same time and you don't "see" the current transferred files (like on background connection dialogue or like FileZilla for example handles it)....):

So, Please:
Can you add this as a feature request or provide an option to abort the size calculation - while at the same time using background transfers?

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
41,852
Location:
Prague, Czechia

Re: disable size check for background transfers

@saibot: See https://winscp.net/eng/docs/ui_transfer_custom#calculate:
Note that when using multiple connections, the calculation does not add time to a transfer, as directories are not listed again for the transfer itself.
Please understand that it's not something that can be skipped. WinSCP needs to list the files to download, no matter what. Any client needs to do that. The only difference here is that for background transfers, it lists all folders in advance, while for foreground transfer, it lists one folder, downloads its files, lists another one folder, downloads its files, etc. In the end, it does not matter. The overall operation takes the same time. With background transfers just takes longer before the first transfer happens.
I might change the label to "Listing" instead of "Calculating" to make this more clear:
Issue 2026 – When collecting list of files for background transfer, say "Listing" instead of "Calculating" not to give an impression that it is a superfluous operation

Reply with quote

saibot
Guest

It doesn't take the same amount of time if the download is interrupted. If the download is interrupted, listing up front reduces the amount of time spent actually downloading files.

If you're trying to validate that you're downloading the right thing, you can't start a download, then verify the first few files; you need to verify files, then start a download.

Doing all the listing up front consumes more memory than listing as you go. You don't need a complete file list. You just need one listing for each directory in the stack of directories, up to the directory currently being transferred.

Doing all the listing up front increases the risk of a modification in between the time of listing to time of use of the listing. If, under bandwidth constraints, you spend several hours doing a listing, then several days doing a download, the probability the listing is stale may not be low.

Reply with quote

Advertisement

saibot
Guest

To make this concrete, just now I had a background download. It took about an hour to do the listing part. It was able to do about 20 minutes of file downloading, then it lost the control connection for the main window. It was still downloading files, but acknowledging the dialog broke the background transfer – I guess you can't keep a control connection for the background download after you've lost the control connection for the main window.

So a couple of minutes later, I'm reconnected. And I have to run the listing again. Another hour before anything will get transferred. The vast majority of the last 60 minute listing was wasted work. It never got around to downloading the files it spent ages listing.

Reply with quote

Advertisement

You can post new topics in this forum