Topic "Large numbers of directories"

Author Message
Ron

Guest


Hi,

I've been using WinSCP for about a year and it is GREAT! Thanks for your tremendous contribution.

I am using WinSCP (v. 3.8.2) to keep a backup of my website (on linux) on my home PC (Windows XP) . I have a large number of directories and more than 30,000 files in them, but only a handful change in a given week, and there are perhaps a hundred additions (new files) in a given day.

Even on my 15Mb fiber optic connection where I get a reliable 8Mbit throughput to my server with pretty fast pings, WinSCP is starting to take a long time to investigate all of those directories.

Would it be possible for WinSCP to do a single recursive look into directories on the remote side and process all of that on the local windows side? Seems to me it would be much faster than the back and forth with all of the latency issues.

Perhaps inclusion of an rsync option would help?

I am hoping that something along these lines can be done, because the amount of time to examine the directories is starting to become too much Sad Sad Sad , and I love my WinSCP!!! Mr. Green
Advertisements
Ron

Guest


Actually, the number of files is bigger than I rememebred:

Size: 7.80 GB (8,384,999,253 bytes)
Contains: 44,359 Files, 692 Folders
Ron
[View user's profile]

Joined: 2007-05-06
Posts: 9
Location: Boston, MA USA
I have registered on the forum. Thanks.
_________________
Good luck.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 25025
Location: Prague, Czechia
Ron wrote:
Would it be possible for WinSCP to do a single recursive look into directories on the remote side and process all of that on the local windows side? Seems to me it would be much faster than the back and forth with all of the latency issues.

What do you mean by "single recursive look"? Smile What protocol are you using SCP or SFTP? Do you use synchronization feature of WinSCP?
Ron
[View user's profile]

Joined: 2007-05-06
Posts: 9
Location: Boston, MA USA
prikryl wrote:
What do you mean by "single recursive look"? Smile What protocol are you using SCP or SFTP? Do you use synchronization feature of WinSCP?
Yes, I'm using the sync feature and SCP to keep my PC up to date as a backup of my UNIX website.

What I am suggesting is if you could add the -R to your ls command(s) on the UNIX side and then bring all of that data back, all at once, and process it on the PC side. The examination of the directories is taking more than 6 minutes because of the number of directories and the back-and-forth latency of looking at a directory, comparing it to the PC, look at the next directory, compare it to the PC, etc., etc.

If you asked UNIX to do a recursive look bringing all of the directory information back to the PC in one big grab, then examine all of the PC directories, I would guess this would be a much faster approach.

Thanks for considering it.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 25025
Location: Prague, Czechia
Sure, it would may make it faster.

However synchronization function in WinSCP is implemented in protocol-independent way. Your suggestion is SCP-specific. Also it would list even directories excluded from synchronization or those not present on local side. This in turn may cause the synchronization to be even slower in some cases.

However I'll consider some improvements based on your suggestions.
_________________
Martin Prikryl
Advertisements

You can post new topics in this forum






Search Site

What is WinSCP?

It is award-winning SFTP client, SCP client, FTPS client and FTP client integrated into one software program for file transfer to FTP server or secure SFTP server. [More]

And it's free!

Donate

About donations

$9   $19   $49   $99

About donations

Recommend

WinSCP Privacy Policy

WinSCP License