PowerShell - sFTP files less than 10 minutes old to remote directory not working; syncs all files

Advertisement

crashIO
Joined:
Posts:
3
Location:
Colorado

PowerShell - sFTP files less than 10 minutes old to remote directory not working; syncs all files

I am trying to only copy files 10 minutes old or less but the synch keeps grabbing all the files. What am I doing wrong? I don't see any errors returned nor in the log.

#$ErrorActionPreference = "SilentlyContinue"

# Load WinSCP .NET assembly
Add-Type -Path "C:\Program Files (x86)\WinSCP\WinSCPnet.dll"

#Current Date-Time minus:
$Age = (Get-Date).AddMinutes(-10)

$Folder_EGDMS_DIU = "F:\EGDMS_DIU"
$Folder_EGDMS_LOGS = "F:\EGDMS_LOGS"
$Folder_EGDMS_PROVISION = "F:\EGDMS_PROVISION"
$Folder_EGDMS_RAW = "F:\EGDMS_RAW"
$Folder_EGDMS_REPORTS = "F:\EGDMS_REPORTS"

$listOfFolders = $Folder_EGDMS_DIU#, $Folder_EGDMS_LOGS, $Folder_EGDMS_PROVISION, $Folder_EGDMS_RAW, $Folder_EGDMS_REPORTS

ForEach ($Folder in $listOfFolders){
$newFiles = Get-ChildItem -Recurse $Folder | Where-Object { $_.LastWriteTime -ge $Age}
$newFiles | Select Name, DirectoryName, LastWriteTime
}

$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "FileServer"
UserName = "User"
Password = "Password"
GiveUpSecurityAndAcceptAnySshHostKey = $true
}

$session = New-Object WinSCP.Session
$session.SessionLogPath = "C:\Temp\sFTP.log"
$session.Open($sessionOptions)

# Synchronize files
ForEach ($file in $newFiles){
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = "*>=$Age"
$newFolder = $file.DirectoryName
$destFolder = $newFolder.Trim("F:\")

$synchronizationResult = $session.SynchronizeDirectories(
[WinSCP.SynchronizationMode]::Remote, $file.DirectoryName, "/home/egdms_user/Miami/" + $destFolder -replace ("\\", "/"), $False, $transferOptions)

# Throw on any error
$synchronizationResult.Check()
}

# Disconnect, clean up
$session.Dispose()

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
29,032
Location:
Prague, Czechia

Re: PowerShell - sFTP files less than 10 minutes old to remote directory not working; syncs all files

Your code seems overcomplicated. What exactly are you trying to do?
Why do you first programmatically find the new files, but then instruct WinSCP to try find the new files again. You already, know what are the new files.

Anyway, the timetamp in FileMask need to have a fixed syntax yyyy-mm-dd hh:mm:ss.
So you need something like:
$transferOptions.FileMask = "*>=" + $Age.ToString("yyyy-MM-dd hh:mm:ss")
See https://winscp.net/eng/docs/script_formatting_timestamp_batch_file

Though actually, this is way easier:
$transferOptions.FileMask = "*>=10N"
See https://winscp.net/eng/docs/file_mask#size_time

Reply with quote

crashIO
Joined:
Posts:
3
Location:
Colorado

Re: PowerShell - sFTP files less than 10 minutes old to remote directory not working; syncs all files

martin wrote:

Your code seems overcomplicated...

First thanks for the response and suggestions; much appreciated.

ETA2: What I am trying to do is copy via sFTP (not move nor delete from source) new files created within the last 10 minutes to a remote file server so analysts can pull the data from the files. Source files should remain intact and on the target server the script should create directories as needed and place the new (less than 10 minutes old) files into the path structure identified by the local server.

For example:
F:\SomeFolder\OldFile.txt
F:\SomeFolder2\AnotherFolder\NewFile.txt

On remote server only NewFile.txt shouild copy and create as needed the same folder strucure on the target server. Hope that helps.



I went ahead and cleaned up the code and now am solely relying ton the mask but it is still sync'ing the entire directory structure and files and not just those modified within the last 10 minutes. The log confirms this. Why is the mask not working with Sync? Do I need to use the Put method instead? I have attached the log.

ETA: Correction...its not recreating the full folder structure and instead is placing all the falsely identified new files/folders into the root of the destination folder. I believe this is why I had to use the earlier 'complicated' code to properly recreate the folder structure as the synch doesn't seem to do it on its own without some help.

Here is the new code:

#$ErrorActionPreference = "SilentlyContinue"

# Load WinSCP .NET assembly
Add-Type -Path "C:\Program Files (x86)\WinSCP\WinSCPnet.dll"

$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "Server"
UserName = "User"
Password = "PASSWORD"
GiveUpSecurityAndAcceptAnySshHostKey = $true
}

$session = New-Object WinSCP.Session
$session.SessionLogPath = "C:\Temp\sFTP.log"
$session.Open($sessionOptions)

# Synchronize files
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = "*>=10N"

$Folder_EGDMS_DIU = "F:\EGDMS_DIU"
$Folder_EGDMS_LOGS = "F:\EGDMS_LOGS"
$Folder_EGDMS_PROVISION = "F:\EGDMS_PROVISION"
$Folder_EGDMS_RAW = "F:\EGDMS_RAW"
$Folder_EGDMS_REPORTS = "F:\EGDMS_REPORTS"

$listOfFolders = $Folder_EGDMS_DIU, $Folder_EGDMS_LOGS, $Folder_EGDMS_PROVISION, $Folder_EGDMS_RAW, $Folder_EGDMS_REPORTS

ForEach ($Folder in $listOfFolders){

$synchronizationResult = $session.SynchronizeDirectories(
[WinSCP.SynchronizationMode]::Remote, $Folder, "/home/user/Miami/"

# Throw on any error
$synchronizationResult.Check()
}
# Disconnect, clean up
$session.Dispose()

Last edited by crashIO on 2018-01-15 21:41; edited 1 time in total
  • sFTP.log (2.63 MB, Private file)

Reply with quote

crashIO
Joined:
Posts:
3
Location:
Colorado

SOLVED!

Got it sorted. Seems like Put was the only way to get some of my logic to work. Thanks again for your help with the mask.

Final code:

$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "FileServer"
UserName = "User"
Password = "PASSWORD"
GiveUpSecurityAndAcceptAnySshHostKey = $true
}
$Folder_EGDMS_DIU = "F:\EGDMS_DIU"
$Folder_EGDMS_LOGS = "F:\EGDMS_LOGS"
$Folder_EGDMS_PROVISION = "F:\EGDMS_PROVISION"
$Folder_EGDMS_RAW = "F:\EGDMS_RAW"
$Folder_EGDMS_REPORTS = "F:\EGDMS_REPORTS"

$listOfFolders = $Folder_EGDMS_DIU, $Folder_EGDMS_LOGS, $Folder_EGDMS_PROVISION, $Folder_EGDMS_RAW, $Folder_EGDMS_REPORTS

$session = New-Object WinSCP.Session
$session.SessionLogPath = "C:\Temp\sFTP.log"
$session.Open($sessionOptions)

# Synchronize files
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = "*>=10N"

ForEach ($Folder in $listOfFolders){
# If the selected item is file, find all contained files and folders recursively
If (Test-Path $Folder -PathType container){
$files =
@($Folder) +
(Get-ChildItem $Folder -Recurse | Select-Object -ExpandProperty FullName)}
Else{
$files = $Folder}

$parentLocalPath = Split-Path -Parent (Resolve-Path $Folder)

foreach ($localFilePath in $files){
$remoteFilePath =
$session.TranslateLocalPathToRemote(
$localFilePath, $parentLocalPath, $remotePath)
$fullRemotePath = "/home/user/Miami/" + $remoteFilePath

if (Test-Path $localFilePath -PathType container){
# Create remote subdirectory, if it does not exist yet
if (!($session.FileExists("/home/user/Miami/" + $remoteFilePath))){
$session.CreateDirectory("/home/user/Miami/" + $remoteFilePath)}}
Else{
Write-Host "Moving file $localFilePath to $fullRemotePath..."
# Upload file and remove original
$session.PutFiles($localFilePath, $fullRemotePath, $False, $transferOptions).Check()
}
}
}

# Disconnect, clean up
$session.Dispose()

Reply with quote

Advertisement

You can post new topics in this forum