Using background queue parallel transfers via scripting or other options

Advertisement

jwomackgsa
Joined:
Posts:
2

Using background queue parallel transfers via scripting or other options

I am building a script that will go out and pull / delete all the files on the remote endpoint that were posted that day. It usually somewhere around 40,000 files no larger then a couple MB. When I run this in the GUI, I can use the 8 threads queue setting to make the download go much faster, however when I pass this queue setting as a raw setting it doesn't seem to do anything. What are my options here? I have no issues with doing PowerShell if it can be accomplished that way. I would like to have like 8 download threads at a time. If my current in-progress WinSCP script has something like
get -delete *
Can I just run the same winscp.com commandline in parallel 8 times to achieve the same effect?
The data is structure like.
\<timestamped_folder_name>
  - all the files
\<another_folder_name>
  - lots more files

Reply with quote

Advertisement

Guest

I have been testing with the .NET assembly via PowerShell. I am using 5 batches and it seems to be working but at various points in running through a couple of transfers I get this following error.
MethodInvocationException:
Line |
  16 |                              if (!($using:filesEnumerator).MoveNext())
     |                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "MoveNext" with "0" argument(s): "Timeout waiting for WinSCP to respond"
My data is like 30+ folders with each folder roughly having 1000 files in each. I think the fileEnumerator is bogged down and not responding fast enough and eventually times out. I looked at the IEnumerator.MoveNext and it doesn't appear that I can feed any parameters to extend the timeout. Any thoughts?

Reply with quote

Guest

Thanks, I have made that change in the script and so far it appears to not be timing out anymore, but I do often see a transfer/delay lag as it moves between the different folders in the collection. I see that winscp will download across the batch sizes for all the say 1000 files in the folder but then it will lag on I guess the MoveNext in the batch that dives into the next folder. I definitely don't see this delay in the GUI. I am wondering if its something to do with memory management when using powershell and the .net assembly. Any thoughts? Thanks for all the support up to this point!

Reply with quote

Advertisement

martin
Site Admin
martin avatar

The GUI and this code works completely differently. You cannot compare it. Though I'd say that the GUI might be even slower. It just does not have the delays between individual folders, but one huge delay at the very beginning. If you have different experiences, we would need some log files.

Reply with quote

Advertisement

You can post new topics in this forum