Delete after Syncing Causing Error (SynchronizeDirectories)
Hi,
I'm using SynchronizeDirectories to copy files from one server to another. After copying, I'm deleting the files from the source using the "Deleting remote files after successful remote to local synchronization" solution (https://winscp.net/eng/docs/library_example_delete_after_successful_download).
Files are available at the end of every month OR they can be generated adhoc. The requirement is that the files be available at the destination, which is a file server, as soon as possible so that users can access them. So I have it scheduled to sync every minute.
The problem I'm having is the first file of each month is very large so while the file is being generated, the process is downloading and then deleting the file before it's completed. This generates an error in the application generating the file as well as a 0 byte file.
I changed the script to compare the size of the file at source with the size of the file at destination before the deleting. If they aren't equal don't delete. Otherwise delete it. I also changed the sync criteria to size so that if the file isn't the same size, the next time the process kicks off, it will check the size and then sync the remainder of the file. That doesn't work because the first file ends up being equal: 0 bytes.
I then added that when after the sync occurs then it sleeps for 20 seconds before performing the delete. This has worked so far with the monthly files but when the year end files were generated as part of a test, the error occurred again.
I'm not sure if there are other options for these type of scenario or a better way of doing this...we're just sooo close with this!
The code is as follows:
Let me know. Thanks!
I'm using SynchronizeDirectories to copy files from one server to another. After copying, I'm deleting the files from the source using the "Deleting remote files after successful remote to local synchronization" solution (https://winscp.net/eng/docs/library_example_delete_after_successful_download).
Files are available at the end of every month OR they can be generated adhoc. The requirement is that the files be available at the destination, which is a file server, as soon as possible so that users can access them. So I have it scheduled to sync every minute.
The problem I'm having is the first file of each month is very large so while the file is being generated, the process is downloading and then deleting the file before it's completed. This generates an error in the application generating the file as well as a 0 byte file.
I changed the script to compare the size of the file at source with the size of the file at destination before the deleting. If they aren't equal don't delete. Otherwise delete it. I also changed the sync criteria to size so that if the file isn't the same size, the next time the process kicks off, it will check the size and then sync the remainder of the file. That doesn't work because the first file ends up being equal: 0 bytes.
I then added that when after the sync occurs then it sleeps for 20 seconds before performing the delete. This has worked so far with the monthly files but when the year end files were generated as part of a test, the error occurred again.
I'm not sure if there are other options for these type of scenario or a better way of doing this...we're just sooo close with this!
The code is as follows:
param ($Source, $Destination, $Protocol, $HostName, $UserName, $Password, $FingerPrint, $SessionLog)
# Load WinSCP .NET assembly
Add-Type -Path "E:\Program Files\WinSCP 5.13\WinSCPnet.dll"
$Destination
# Main script
try
{
$session = New-Object WinSCP.Session
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
Protocol = [WinSCP.Protocol]::$Protocol
HostName = "$($HostName)"
UserName = "$($UserName)"
Password = "$($Password)"
SshHostKeyFingerprint = $FingerPrint
}
try
{
if(!(Test-Path -Path $Destination )){
New-Item -ItemType directory -Path $Destination
}
$session.SessionLogPath = "$($SessionLog)"
Write-Host "Connected... " (Get-Date).ToString()
$session.Open($sessionOptions)
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Automatic
Write-Host "Synchronize files... " (Get-Date).ToString()
$synchronizationResult = $session.SynchronizeDirectories(
[WinSCP.SynchronizationMode]::Local, "$($Destination)", "$($Source)", $False, $False, [WinSCP.SynchronizationCriteria]::Size, $transferOptions)
#$synchronizationResult.Check()
Write-Host "Sleeping for 20... " (Get-Date).ToString()
Start-Sleep -s 20
Write-Host "Continuing processing... " (Get-Date).ToString()
foreach ($download in $synchronizationResult.Downloads)
{
# Success or error?
if ($download.Error -eq $Null){
Write-Host "Download of $($download.FileName) succeeded, removing from source " (Get-Date).ToString()
# Download succeeded, remove file from source. This includes the full path
$fileNameSource = [WinSCP.RemotePath]::EscapeFileMask($download.FileName)
$fileNameDestArray = $download.FileName -split "/"
$sourceLength = $session.GetFileInfo($fileNameSource).Length
Write-Host "Source file size " $sourceLength
$fileNameDest = $Destination + "\" + $fileNameDestArray[6]+ "\"+ $fileNameDestArray[7]+ "\" + $fileNameDestArray[8]
$destLength = (Get-Item $fileNameDest).Length
Write-Host "Destination file size" $destLength
if ($sourceLength -eq $destLength){
Write-Host "Attempting to remove $($download.FileName) " (Get-Date).ToString()
$removalResult = $session.RemoveFiles($fileNameSource)
if ($removalResult.IsSuccess){
Write-Host "Removing of file $($download.FileName) succeeded " (Get-Date).ToString()
}else{
Write-Host "Removing of file $($download.FileName) failed" (Get-Date).ToString()
}
}else{
Write-Host "File $($download.FileName) was not removed. File size of downloaded file is not equal. File is still being written to Oracle." + (Get-Date).ToString()
}
}else{
Write-Host "Download of $($download.FileName) failed: $($download.Error.Message)" (Get-Date).ToString()
}
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}
catch
{
Write-Host "Error: $($_.Exception.Message)"
exit 1
}