Forum » Support and Bug Reports »

Parallel Transfers + IgnorePermErrors

Advertisement

NottaSmartMan
Joined:
Posts:
36

Parallel Transfers + IgnorePermErrors

I'm having an issue where the IgnorePermErrors flag is being set within a thread, but doesn't appear to be taking affect. When a timestamp issue is encountered, the script Aborts rather than Skips.

I've tried setting the flag via xxx.AddRawConfiguration("IgnorePermErrors","1") as well as xxx.AddRawSettings("IgnorePermErrors", "1"). Nothing works.

I will attach my code and logs in question.

The 'thread' log is the one that shows the abort.
  • ThreadedsFTP - Sanatized.txt (6.16 KB, Private file)
  • Thread-1 - Sanatized.log (37.15 KB, Private file)
  • Parent - Sanatized.log (86.8 KB, Private file)

Reply with quote

Advertisement

martin◆
Site Admin
martin avatar
Joined:
Posts:
41,405
Location:
Prague, Czechia

Re: Parallel Transfers + IgnorePermErrors

Why do you think it's IgnorePermErrors issue? Isn't it about TransferOptions.PreserveTimestamp? Though if your server does not support preserving timestamps, then timestamp-based synchronization won't be working.

Reply with quote

NottaSmartMan
Joined:
Posts:
36

The server does support preserving timestamps. 99% of the time it works just fine. However, I have noticed issues at times where setting the timestamp will fail because the destination file is 'in use'. I suspect that there's an anti-virus scanner that sometimes gets in the way.

I use the IgnorePermErrors so that when these errors popup, that the error is skipped rather than aborted. It's worked fine up until I started playing with making my script threaded.

I don't understand why WinSCP is now choosing "Abort" rather than "Skip". Is something different with the $session.CompareDirectories([WinSCP.SynchronizationMode] $difference.Resolve syntaxes? Or Am I barking up the wrong tree and there's actually a different command that instructs WinSCP to Skip rather than Abort?

Example: Here's a sanitized excerpt from a file that succeeded, then then the next file that failed was skipped. This is from a different script that isn't multithread; it just uses the $session.SynchronizeDirectories and that's it. 
. 2024-10-27 16:34:00.035 Preserving timestamp [2024-03-01T13:16:16.000Z]
. 2024-10-27 16:34:00.113 Transfer done: '\\<path removed>\SuccessfulFile.xlsx' => '/<Path Removed>/SuccessfulFile.xlsx' [301102]
. 2024-10-27 16:34:00.113 File: '\\<path removed>\FileThatHadIssues.xlsx' [2024-05-15T14:26:14.282Z] [301198]
. 2024-10-27 16:34:00.457 Copying "\\<path removed>\FileThatHadIssues.xlsx" to remote directory started.
. 2024-10-27 16:34:00.753 Preserving timestamp [2024-05-15T14:26:14.000Z]
* 2024-10-27 16:34:00.753 (ETerminal) General failure (server should provide error description).
* 2024-10-27 16:34:00.753 Error code: 4
* 2024-10-27 16:34:00.753 Error message from server (en-US): 
* 2024-10-27 16:34:00.753 
* 2024-10-27 16:34:00.753 Common reasons for the Error code 4 are:
* 2024-10-27 16:34:00.753 - Renaming a file to a name of already existing file.
* 2024-10-27 16:34:00.753 - Creating a directory that already exists.
* 2024-10-27 16:34:00.753 - Moving a remote file to a different filesystem (HDD).
* 2024-10-27 16:34:00.753 - Uploading a file to a full filesystem (HDD).
* 2024-10-27 16:34:00.753 - Exceeding a user disk quota.
. 2024-10-27 16:34:00.753 Asking user:
. 2024-10-27 16:34:00.753 **Upload of file 'FileThatHadIssues.xlsx' was successful, but error occurred while setting the permissions and/or timestamp.**
. 2024-10-27 16:34:00.753 
. 2024-10-27 16:34:00.753 If the problem persists, turn off setting permissions or preserving timestamp. Alternatively you can turn on 'Ignore permission errors' option. ("General failure (server should provide error description).
. 2024-10-27 16:34:00.753 Error code: 4
. 2024-10-27 16:34:00.753 Error message from server (en-US): 
. 2024-10-27 16:34:00.753 
. 2024-10-27 16:34:00.753 Common reasons for the Error code 4 are:
. 2024-10-27 16:34:00.753 - Renaming a file to a name of already existing file.
. 2024-10-27 16:34:00.753 - Creating a directory that already exists.
. 2024-10-27 16:34:00.753 - Moving a remote file to a different filesystem (HDD).
. 2024-10-27 16:34:00.753 - Uploading a file to a full filesystem (HDD).
. 2024-10-27 16:34:00.753 - Exceeding a user disk quota.")
< 2024-10-27 16:34:00.753 Script: Upload of file 'FileThatHadIssues.xlsx' was successful, but error occurred while setting the permissions and/or timestamp.
< 2024-10-27 16:34:00.753 If the problem persists, turn off setting permissions or preserving timestamp. Alternatively you can turn on 'Ignore permission errors' option.
< 2024-10-27 16:34:00.753 Script: General failure (server should provide error description).
< 2024-10-27 16:34:00.753 Error code: 4
< 2024-10-27 16:34:00.753 Error message from server (en-US): 
< 2024-10-27 16:34:00.753 Common reasons for the Error code 4 are:
< 2024-10-27 16:34:00.753 - Renaming a file to a name of already existing file.
< 2024-10-27 16:34:00.753 - Creating a directory that already exists.
< 2024-10-27 16:34:00.753 - Moving a remote file to a different filesystem (HDD).
< 2024-10-27 16:34:00.753 - Uploading a file to a full filesystem (HDD).
< 2024-10-27 16:34:00.753 - Exceeding a user disk quota.
. 2024-10-27 16:34:00.753 Answer: Skip

Reply with quote

martin◆
Site Admin
martin avatar

You are getting error 4 (failure), not 3 (permission denied). So the IgnorePermErrors does not apply.
If you believe that WinSCP behaviour is different with single-threaded code, please post log file showing that.

Reply with quote

Advertisement

NottaSmartMan
Joined:
Posts:
36

I've done more digging and created a similar and simple single instance and multithreaded version of the code. I tried adding the add_QueryReceived event, for I think that's what I was truly missing.

After some testing, it seems the code is getting hung up around the $syncSession.add_QueryReceived event. The event is triggered, but the script just hangs calling the FileQuery function. I've attach a sanitized copy of the script.

If I remove the QueryReceived event, then the script doesn't hang... rather it just aborts. Neither are what I want... I want it to skip/continue.

The last line of the log are: 
< 2024-11-06 13:28:42.265 Script: Upload of file '<removed>' was successful, but error occurred while setting the permissions and/or timestamp.
< 2024-11-06 13:28:42.265 If the problem persists, turn off setting permissions or preserving timestamp. Alternatively you can turn on 'Ignore permission errors' option.
< 2024-11-06 13:28:42.265 Script: General failure (server should provide error description).
< 2024-11-06 13:28:42.265 Error code: 4
< 2024-11-06 13:28:42.265 Error message from server (en-US): 
< 2024-11-06 13:28:42.265 Common reasons for the Error code 4 are:
< 2024-11-06 13:28:42.265 - Renaming a file to a name of already existing file.
< 2024-11-06 13:28:42.265 - Creating a directory that already exists.
< 2024-11-06 13:28:42.265 - Moving a remote file to a different filesystem (HDD).
< 2024-11-06 13:28:42.265 - Uploading a file to a full filesystem (HDD).
< 2024-11-06 13:28:42.265 - Exceeding a user disk quota.
  • sFTPCopy-Threaded.txt (6.51 KB, Private file)

Reply with quote

NottaSmartMan
Joined:
Posts:
36

I think I may have figured it out. I'm very very new to multithreading and didn't realize that threads couldn't see function defined outside the thread job. I didn't see that demo'd in the examples I ran across. Things worked once I placed the FileQuery function within a code block inside the thread-job.

This is a whole new world.

Reply with quote

Advertisement

You can post new topics in this forum

Documentation

Support

Associations

Follow Us