Topic "How to skip file on error?"

Author Message
dhostetter
[View user's profile]

Joined: 2012-09-11
Posts: 2
Location: Georgia, USA
Hi, I have a powershell script to automate uploading of files. I am simulating upload errors by denying the script access permissions to one file out of a group of test files. The idea being that WinSCP will attempt to upload all files that match the filespec and if one file cannot be uploaded, the others will be. Afterwards the script analyses the transfer results and reports what files failed and why.

Problem is that WinSCP aborts when it encounters an error, the remaining files are not uploaded. My script will then retry uploading the filespec but now the file causing an error is the first to go and therefore WinSCP aborts again and never gets to the remaining files no matter additional retries. I want WinSCP to skip any problem file and continue to upload the remaining files.

I know I can script around this by handling the files individually but I would really like to upload all files in one session, not open and close individual sessions.

Here are my session settings, minus the site-specific information

$FTPSessionOptions = New-Object WinSCP.SessionOptions
$FTPSessionOptions.Protocol = [WinSCP.Protocol]::sftp
$FTPSessionOptions.AddRawSettings("ContinueOnError","1")

$FTPTransferOptions = New-Object WinSCP.TransferOptions
$FTPTransferOptions.TransferMode = [WinSCP.TransferMode]::Binary
$FTPTransferOptions.FilePermissions = $Null
$FTPTransferOptions.PreserveTimestamp = $False

Here is how I upload the files:

try {
$FTPSession = New-Object WinSCP.Session
$FTPSession.Open($FTPSessionOptions)
$TransferResult = $FTPSession.PutFiles("$TargetFolder\$FileFilter", "./", $TRUE, $FTPTransferOptions)
}

catch {
#Log-Event is my own logging function
Log-Event "Error" "Error Transferring Files: [$($FTPSession.output)]" 305 $False
}

finally {
try { $FTPSession.Dispose() }
catch {$Error.clear()} #ignore any error from the above
}

The Putfiles method does not return a trappable error - or for some other reason the catch block doesn't ever fire. Execution skips immediately to finally. The output of the session is:


winscp> option batch on
batch on
winscp> option confirm off
confirm off
winscp> open -hostkey="xxxxxxxxx" -timeout=15 "sftp://user:pw@hostname.com" -rawsettings ContinueOnError="1"
Searching for host...
Connecting to host...
Authenticating...
Using username "user".
Authenticating with pre-entered password.
Authenticated.
Starting the session...
Reading remote directory...
Session started.
Active session: [1] user@hostname.com
winscp> put -delete -nopermissions -nopreservetime -transfer="binary" -- "c:\temp\*.tst" "./"
Can't open file 'c:\temp\test41.tst'.
System Error. Code: 5.
Access is denied
(A)bort, (R)etry, (S)kip, Ski(p) all: Abort


SO How do I make that last "Abort" into a "Skip" so that the rest of the files are uploaded? Evidently the raw setting "ContinueOnError" doesn't do that.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24555
Location: Prague, Czechia
There's no workaround for this atm. Except for the solution you have suggested yourself.
_________________
Martin Prikryl
dhostetter
[View user's profile]

Joined: 2012-09-11
Posts: 2
Location: Georgia, USA
Noticing 5.1.2 has come out, I updated WinSCP files and am testing my script again.

I still have the same issue and I notice one minor bug.

The bug is that if you set the SessionOptions SshHostKey property, you can't 'unset' it. Setting SshHostKey, after it has a value, to "" or null results in this error:

[quote]Exception setting "SshHostKey": "SSH host key fingerprint" does not match pattern /(ssh-rsa |ssh-dss )?\d+ ([0-9a-f]{2}:){15}[0-9a-f]{2}(;(ssh-rsa |ssh-dss )?\d+ ([0-9a-f]{2}:){15}[0-9a-f]{2})*/[/quote]

The situation is that you might first set Protocol to SFTP and set SshHostKey but later you may need to change the protocol to FTP. If you leave SshHostKey set then you cannot make the connection due to another exception that SshHostKey is set but Protocol is FTP.

The work around is to re-create the SessionOptions variable if you want to switch from SFTP to FTP.

It's clearly not a serious problem but you might consider to modify the regex for SshHostKey to allow empty string or ignore SshHostKey if Protocol is FTP instead of throwing an exception.

The second issue is the same as I earlier posted. In a batch-transfer situation, if any error is encountered the WinSCP process terminates and subsequent matching files are not transferred.

A potential situation is that occasionally the process that creates the target files to transfer could hang and a file become perpetually locked. In that case future files are not transferred because WinSCP keeps processing the locked file first and terminating on failure.

The third issue is that the order of file selection does not appear to be modifiable and this prevents a simple workaround for the previous issue.

[code]PutFiles("c:\temp\*.txt", "./", $TRUE, $FTPTransferOptions)[/code]

This appears to operate on an alphabetical list of files that match the source filespec. The problem is that let's say that file "targetfile_5.txt" is locked or has some other issue, "targetfile_6.txt" is never touched by WinSCP. Some time later target files 7, 8, 9 are created, the putfiles method is invoked again, but first up is "targetfile_5.txt" which is still locked. So any subsequent file, ordered alphabetically, will never be processed by WinSCP.

If the file list could be sorted by date from newest to oldest, naturally any problem file will be the oldest file so newer files would be processed first on subsequent passes.

I could find no simple way to re-order the files. I assume that WinSCP uses a .net method to access the file list and the default .net sort order is alphabetical.

Otherwise workarounds could include sending urgent email on error and understanding that transfers may fail until the issue is resolved, renaming any accessible source file so that 'good' files are alphabetically before 'bad' files or just recoding to send files individually instead of batching them.

Any chance of having WinSCP .net assembly modified to continue on errors?
Advertisements

You can post new topics in this forum






Search Site

What is WinSCP?

It is award-winning SFTP client, SCP client, FTPS client and FTP client integrated into one software program for file transfer to FTP server or secure SFTP server. [More]

And it's free!

Donate

About donations

$9   $19   $49   $99

About donations

Recommend

WinSCP Privacy Policy

WinSCP License