WinSCP Hangs on Very Large Change List

Hi there,

I've been a long time user and supporter of WinSCP and want to say once again, thanks for making the tedious task of remote syncing a simple one.

I know my use case is probably not that common, but I often end up in situations where I am doing comparisons and end up with massive change sets. Because WinSCP will attempt to list every single one as a list item with associated grid columns and checkbox controls, this takes time. When the size of the changeset goes over 10,000, the program will hang for a minute or two before showing the Compare window, and larger changesets will never complete in a reasonable time. This will be applicable to all versions of the program.

I have a couple of suggestions. Clearly, it's not the size of the changeset that causes the issue here, but the impracticality of turning that into a UI. One idea is to implement pagination into the compare window, so only the first, say, 1000 items are shown, with a spinner to flick through the pages, which will swap out the list items 1000 at a time. Alternatively, as an intermediate solution, if the changeset is larger than a certain size, give a warning prompt to the user with choices such as Sync All or Cancel.

So yeah, I might be 'doing it wrong', but adding a failsafe to stop me from having to terminate WinSCP would be a nice bonus. Thanks for reading!

Thanks for your report.
It takes of few seconds for me to display changelist with 10000 items.
Can you attach a screenshot of the change list, in case there's something unusual?

Certainly... I've got some more data for you. I'm using a modestly-powered workstation and did the following tests on version 5.7.1. I have also attached screenshots (with most of the filenames blanked out).

For a changeset of 37,000 items, the window took 2 minutes 30 seconds to display (14800 items per minute)
For a changeset of 294,000 items, the window took 3 hours 10 minutes to display (1547 items per minute)

During this time between analysing the directories and the window showing, the application is unresponsive, and Task Manager shows it's using 100% of the core it's running on, while it continues to allocate a mere 30KB per second.

When the window is finally opened, scrolling the list is very responsive, but as soon as you check or uncheck an item, the application will freeze again for a percentage of the original time taken to load.

Taking all this into consideration, I'm guessing that there might be a potential data lookup issue. when there were 10x as many items, it took roughly 100x the time. Perhaps there is some kind of O(n^2) array scanning happening while building the GUI? That would explain why it's thrashing the CPU while doing very little in the way of allocating resources. If you're able to track that down and replace it with a binary search or hashmap, that should take care of it. :)

Thanks. The times above, do they include "Comparing" phase? Or is this after the comparing finishes, but before the Synchronization checklist window shows?

I didn't include the time taken to scan and compare. That part seems perfectly fine - it's proportional to the number of files and directories. I started the timer as soon as the Comparing window closed.

OK, please try try if you get better performance with WinSCP 5.5.6:
https://sourceforge.net/projects/winscp/files/WinSCP/5.5.6/

I downgraded to 5.5.6 and tested again. It took exactly the same amount of time to load the checklist (just over 3 hours).

It also took a while for the main program to respond again after closing and cancelling the checklist window (around 5 minutes), but I don't remember if that was the case with the latest version because I didn't explicitly test it.

I have sent you an email with a debug version of WinSCP to address you have used to register on this forum.

I've replied to your email with a Dropbox link, since the final trace file wouldn't fit within the 10MB limit here (even after compression).

In the trace, I connected via FTP, browsed to a directory, did a compare, cancelled the checklist, and waited for a little while after the program became responsive again before exiting. This is a representation of the ~37K files use case. The trace file is 365MB uncompressed.

The trace file will contain some company data, so please keep it private. I've tried to redact a few things such as IP addresses and logins, but some may remain. I hope it's still useful to you.

I'm just replying here so others can know the status of this bug report.

You analysed the logs but for now you were unable to spot a specific cause for the quadratic slowdown, and for the time being it's not enough of a concern to warrant rewriting the core code. I perfectly understand and I'm glad that at least I can bring it to your attention.

I wish I had a better understanding of C++ because then I would be able to analyse the source code and perhaps help identify the nested loop in SynchronizeChecklist. I'm more of a Java person myself!