.NET PowerShell script - assist with improving upload resilience

Advertisement

dverbern
Joined:
Posts:
3
Location:
Melbourne Australia

.NET PowerShell script - assist with improving upload resilience

Hello,
First time here, although long time WinSCP user.
I've been using PowerShell and the WinSCP .NET functionality to do the bulk of my scripting and automation of SFTP file uploads and downloads, mostly with fantastic results.

I'm currently trying to troubleshoot and improve a situation whereby reasonably large file uploads of zip files (100–300 Megabytes) are occasionally resulting either in those files only being partially uploaded (i.e. file size smaller than the original) or the file seeming to upload fully and the file sizes matching, but the uploaded file is corrupted in some way as I cannot uncompress it.

I tried to make use of the Session.CalculateFileChecksum method, but unfortunately the remote SFTP server I am dealing with must not support it. I have downloaded the remote parties' copy of the offending zip file and used PowerShell to run a file hash comparison and sure enough, the two files are different, despite the file size being identical.

I've seen quite a few examples of PowerShell code using .NET WinSCP and calling the Check result of a transfer object as it iterates through files to upload/download, but I must admit I'm sometimes a bit confused by the overall syntax.

What I want is to improve the resilience of my script so that if for whatever reason a given file does NOT complete uploading, my script can 'tell' that has occurred and can essentially 'try again' until it HAS confirmed the file is properly transferred and then and ONLY then move onto the next file to process.

I feel embarrassed to ask for any code examples, but it would be much appreciated. I want to verify transfers before moving on to the next file, thank you.

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
40,884
Location:
Prague, Czechia

Re: .NET PowerShell script - assist with improving upload resilience

That's rather broad question.
I've posted a rather broad answer at Stack Overflow:
How to perform checksums during a SFTP file transfer for data integrity?
Basically, if your code is correct, and it did not report any errors, the file was transferred correctly. If it was stored incorrectly on the server, there's really no way to know that as something unpredictable and undetectable has happened. Either you need to download each file back to check it or have a way to calculate the remote file checksum. Do you have a shell access to the server?

Reply with quote

dverbern
Joined:
Posts:
3
Location:
Melbourne Australia

Re: .NET PowerShell script - assist with improving upload resilience

Martin, it's an honour, thank you for your post. I'm sorry to have been vague, I can see how frustrating it can be to have users fail to provide specifics so that you can be equally specific in your answer.

I'm not sure if I have SCP access to the 3rd party system, but I can check. I'm assuming that if SCP is available, it opens up command options that aren't part of SFTP?

It's definitely something I'll look into.
Thank you sir.

Reply with quote

Advertisement

You can post new topics in this forum