Possible issue when using Session.PutFiles with concurrent tasks

Advertisement

wharazim
Guest

Possible issue when using Session.PutFiles with concurrent tasks

Hello-

I am having an issue with multiple concurrent tasks attempting to transfer files from the same localPath to different remotePaths based on file extension.

Given two tasks executing on concurrent isolated processes and using the .NET Assembly:
Task1
.PutFiles("C:\Local\*", "/remote/v1/*", true, new TransferOptions { FileMask = "*.1 | */" })
Task2
.PutFiles("C:\Local\*", "/remote/v2/*", true, new TransferOptions { FileMask = "*.2 | */" })

And given these files in the localPath:
file001.1
file002.1
file003.1
file001.2
file002.2
file003.2

It seems, depending on timing, I get an error on one or the other task
Ex From Task1: File or folder 'file001.2' does not exist.
OR
Ex From Task2: File or folder 'file001.1' does not exist.

Everything always transfers Ok, as intended based on the fileMask. However, periodically, a 'file does not exist' error is raised.

It seems the issue is PutFiles initially selects all files in the localPath as specified by the localPath wildcard, then iterates through them to apply the fileMask and determine which to Put. For files that may have been initially selected by Task1, and concurrently moved by Task2 prior to the Task1 fileMask check & Put, an error is raised that the file cannot be found.

Can this supposition be confirmed?

Thank you,
William

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
40,552
Location:
Prague, Czechia

Re: Possible issue when using Session.PutFiles with concurrent tasks

Yes, your assumption is correct.

You can solve it like this:

.PutFiles("C:\Local\*.1", "/remote/v1/*", true,
    new TransferOptions { FileMask = "*/" })

.PutFiles("C:\Local\*.2", "/remote/v2/*", true,
    new TransferOptions { FileMask = "*/" }) 

Reply with quote

Advertisement

You can post new topics in this forum