More granular parallel transfers / efficient use of threads??
The answer is probably 'no', but I figured I'd ask anyway.
I'm using the powershell code at https://winscp.net/eng/docs/library_example_parallel_transfers#powershell to sync a test folder.
The root of the test folder has 11 folders and 6 files. Deep within one of the folders off the root, there are 2 special test folders.. each with 1000 small files in it.
When I set the number of connections to 4, 3 of the threads finishing in 20 seconds. The last thread finishes in 30 minutes.
If I set the number of connections to 10, only 5 threads receive work and the results are pretty much the same.
My question, is there any way to distribute the threads better? Perhaps to have Powershell/WinSCP start deeper in the directory tree before handing out jobs to each thread?
If curious, below is the breakdown of root folder per thread for the run with 4 connections. My folders with 1000 small files are in the folder aptly named "problem files". I do not understand how PS decides which thread gets which job.
I'm using the powershell code at https://winscp.net/eng/docs/library_example_parallel_transfers#powershell to sync a test folder.
The root of the test folder has 11 folders and 6 files. Deep within one of the folders off the root, there are 2 special test folders.. each with 1000 small files in it.
When I set the number of connections to 4, 3 of the threads finishing in 20 seconds. The last thread finishes in 30 minutes.
If I set the number of connections to 10, only 5 threads receive work and the results are pretty much the same.
My question, is there any way to distribute the threads better? Perhaps to have Powershell/WinSCP start deeper in the directory tree before handing out jobs to each thread?
If curious, below is the breakdown of root folder per thread for the run with 4 connections. My folders with 1000 small files are in the folder aptly named "problem files". I do not understand how PS decides which thread gets which job.
[1] difference = C:\LocalFolder\${}-()[]]+=!@#$%^&~ test.txt ==> /RemoteFolder/1/*
[1] difference = C:\LocalFolder\ExcludeAllFoldersWithThisName\ ==> /RemoteFolder/1/*
[1] difference = C:\LocalFolder\SubfolderTest\ ==> /RemoteFolder/1/*
[1] difference = C:\LocalFolder\SyncThis Folder - Copy\ ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\{0 test.txt ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\New Text Document.txt ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\S16\ ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\t1\ ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\test\ ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\Test File 1.txt ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\testfolder\ ==> /RemoteFolder/1/*
[3] difference = C:\LocalFolder\DoThis\ ==> /RemoteFolder/1/*
[3] difference = C:\LocalFolder\SomeOtherPathForAFileByTheSameName\ ==> /RemoteFolder/1/*
[3] difference = C:\LocalFolder\SyncThis Folder\ ==> /RemoteFolder/1/*
[4] difference = C:\LocalFolder\{ test } .txt ==> /RemoteFolder/1/*
[4] difference = C:\LocalFolder\New Microsoft Excel Worksheet.xlsx ==> /RemoteFolder/1/*
[4] difference = C:\LocalFolder\Problem files\ ==> /RemoteFolder/1/*