Topic "Slow to display contents of very large directories"

Author Message
Guest

Guest


When displaying the contents of a remote directory containing a very large number of files (8283) the time taken to display is very long (~125 seconds) and local CPU utilisation is very high (~97%).

The time to generate he same directory listing on the remote host is relatively short (shown below):
bash-2.03$ time ls -l > /tmp/zz
real 0m3.673s
user 0m2.500s
sys 0m0.450s

The time taken to transfer the contents of the directory listing (/tmp/zz) to my local machine is about 7 seconds (using compression during the file transfer).

The time taken to change directories, to say the root dir, when the current dir contains a very large number of files is also quite slow (~25 seconds).

None of these problems are Earth shattering but it would be great if they get resolved (optimised) in the future.

In the meantime I'm forced to sensibly implement an archiving plan to ensure I don't have so many files in any given directory.

Thanks for a great product!
Advertisements
leif

Guest


I'm experiencing the same issue, never happened before, but I recently upgraded to 4.1x and it is rather annoying. Seems to be even worse then the original authors times. Tried upgrading to the newest stable version and the problem is still there.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24555
Location: Prague, Czechia
leif wrote:
I'm experiencing the same issue, never happened before, but I recently upgraded to 4.1x and it is rather annoying. Seems to be even worse then the original authors times. Tried upgrading to the newest stable version and the problem is still there.

The topic you follow up is 6 years old, so it is hardly relevant. Can you please describe on your own, how WinSCP behaves for you?
_________________
Martin Prikryl
rustymonkey

Guest


Hi,

I can describe this issue--it occurs for me using WinSCP (portable) v4.26 running on Windows.

If I browse to a directory containing e.g. 250 files, it takes > 60s for the directory to open--the 'hourglass' icon shows up the entire time and cpu utilization goes to 50%. This occurs whether the directory in question is a local directory or remote directory.

Upon further examination, though, CPU utilization goes up for MsMpEng.exe goes for the duration that the hourglass appears. Apparently, this problem affects other apps as well, and a solution is described at:
http://www.1stbyte.com/2010/02/01/microsoft-security-essentials-msmpeng-exe-using-high-cpu-time/
aforget

Guest


Guest wrote:
When displaying the contents of a remote directory containing a very large number of files (8283) the time taken to display is very long (~125 seconds) and local CPU utilisation is very high (~97%).

The time to generate he same directory listing on the remote host is relatively short (shown below):
bash-2.03$ time ls -l > /tmp/zz
real 0m3.673s
user 0m2.500s
sys 0m0.450s

The time taken to transfer the contents of the directory listing (/tmp/zz) to my local machine is about 7 seconds (using compression during the file transfer).

The time taken to change directories, to say the root dir, when the current dir contains a very large number of files is also quite slow (~25 seconds).

None of these problems are Earth shattering but it would be great if they get resolved (optimised) in the future.

In the meantime I'm forced to sensibly implement an archiving plan to ensure I don't have so many files in any given directory.

Thanks for a great product!


I also have the same problem (ten years later). For directories with many files (hundreds to thousands of text files varying in size from 4KB to several MBs each), it takes a very long time before I can even see the directory's contents, while with PuTTY, a text-based SSH utility, I can see the directory's contents almost instantaneously. I've always had this problem with the Windows-installed version of WinSCP, but it has recently become debilitating because I need to browse the contents of these directories frequently. I'm not sure what more information I could provide, but I hope you can replicate and fix the problem.

Otherwise, I've loved WinSCP for years! I hope I don't need to use another program.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24555
Location: Prague, Czechia
aforget wrote:
I also have the same problem (ten years later). For directories with many files (hundreds to thousands of text files varying in size from 4KB to several MBs each), it takes a very long time before I can even see the directory's contents, while with PuTTY, a text-based SSH utility, I can see the directory's contents almost instantaneously. I've always had this problem with the Windows-installed version of WinSCP, but it has recently become debilitating because I need to browse the contents of these directories frequently. I'm not sure what more information I could provide, but I hope you can replicate and fix the problem.

Otherwise, I've loved WinSCP for years! I hope I don't need to use another program.

Can you send me an email, so I can send you back a debug version of WinSCP to track the problem? Please include link back to this topic in your email. Also note in this topic that you have sent the email. Thanks.

You will find my address (if you log in) in my forum profile.
_________________
Martin Prikryl
aforget
[View user's profile]

Joined: 2013-10-02
Posts: 2
prikryl wrote:
Can you send me an email, so I can send you back a debug version of WinSCP to track the problem? Please include link back to this topic in your email. Also note in this topic that you have sent the email. Thanks.

You will find my address (if you log in) in my forum profile.


Done.
DarkSide

Guest


Any news about this issue?
Directory listing of directories with large amount of files (no symlinks) take ages to load and reloads on almost any activity.
aforget
[View user's profile]

Joined: 2013-10-02
Posts: 2
DarkSide wrote:
Any news about this issue?
Directory listing of directories with large amount of files (no symlinks) take ages to load and reloads on almost any activity.


I provided Martin Prikryl with debug logs, and he seemed to determine it was some kind of UI loading issue, and never replied to my e-mails after that. I assume he gave up on it.

I've since switched to FileZilla, and which at least shows the files in a reasonable amount of time (rather than hanging for many minutes or even hours), and has other nice features that WinSCP doesn't.

If anyone finds an SCP client that can more seamlessly handle large numbers of files (or if this is ever fixed in WinSCP), please let us know!
Advertisements

You can post new topics in this forum






Search Site

What is WinSCP?

It is award-winning SFTP client, SCP client, FTPS client and FTP client integrated into one software program for file transfer to FTP server or secure SFTP server. [More]

And it's free!

Donate

About donations

$9   $19   $49   $99

About donations

Recommend

WinSCP Privacy Policy

WinSCP License