Post a reply

Options
Add an Attachment

If you do not want to add an Attachment to your Post, please leave the Fields blank.

(maximum 10 MB; please compress large files; only common media, archive, text and programming file formats are allowed)

Options

Topic review

NottaSmartMan

More granular parallel transfers / efficient use of threads??

The answer is probably 'no', but I figured I'd ask anyway.

I'm using the powershell code at https://winscp.net/eng/docs/library_example_parallel_transfers#powershell to sync a test folder.

The root of the test folder has 11 folders and 6 files. Deep within one of the folders off the root, there are 2 special test folders.. each with 1000 small files in it.

When I set the number of connections to 4, 3 of the threads finishing in 20 seconds. The last thread finishes in 30 minutes.

If I set the number of connections to 10, only 5 threads receive work and the results are pretty much the same.

My question, is there any way to distribute the threads better? Perhaps to have Powershell/WinSCP start deeper in the directory tree before handing out jobs to each thread?

If curious, below is the breakdown of root folder per thread for the run with 4 connections. My folders with 1000 small files are in the folder aptly named "problem files". I do not understand how PS decides which thread gets which job.

[1] difference = C:\LocalFolder\${}-()[]]+=!@#$%^&~ test.txt    ==> /RemoteFolder/1/*

[1] difference = C:\LocalFolder\ExcludeAllFoldersWithThisName\    ==> /RemoteFolder/1/*
[1] difference = C:\LocalFolder\SubfolderTest\          ==> /RemoteFolder/1/*
[1] difference = C:\LocalFolder\SyncThis Folder - Copy\    ==> /RemoteFolder/1/*


[2] difference = C:\LocalFolder\{0 test.txt       ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\New Text Document.txt   ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\S16\       ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\t1\          ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\test\          ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\Test File 1.txt    ==> /RemoteFolder/1/*
[2] difference = C:\LocalFolder\testfolder\       ==> /RemoteFolder/1/*

[3] difference = C:\LocalFolder\DoThis\             ==> /RemoteFolder/1/*
[3] difference = C:\LocalFolder\SomeOtherPathForAFileByTheSameName\    ==> /RemoteFolder/1/*
[3] difference = C:\LocalFolder\SyncThis Folder\          ==> /RemoteFolder/1/*

[4] difference = C:\LocalFolder\{ test } .txt             ==> /RemoteFolder/1/*
[4] difference = C:\LocalFolder\New Microsoft Excel Worksheet.xlsx    ==> /RemoteFolder/1/*
[4] difference = C:\LocalFolder\Problem files\             ==> /RemoteFolder/1/*