PowerShell - sFTP files less than 10 minutes old to remote directory not working; syncs all files

I am trying to only copy files 10 minutes old or less but the synch keeps grabbing all the files. What am I doing wrong? I don't see any errors returned nor in the log.

#$ErrorActionPreference = "SilentlyContinue"

# Load WinSCP .NET assembly
Add-Type -Path "C:\Program Files (x86)\WinSCP\WinSCPnet.dll"

#Current Date-Time minus:
$Age = (Get-Date).AddMinutes(-10)

$Folder_EGDMS_DIU = "F:\EGDMS_DIU"
$Folder_EGDMS_LOGS = "F:\EGDMS_LOGS"
$Folder_EGDMS_PROVISION = "F:\EGDMS_PROVISION"
$Folder_EGDMS_RAW = "F:\EGDMS_RAW"
$Folder_EGDMS_REPORTS = "F:\EGDMS_REPORTS"

$listOfFolders = $Folder_EGDMS_DIU#, $Folder_EGDMS_LOGS, $Folder_EGDMS_PROVISION, $Folder_EGDMS_RAW, $Folder_EGDMS_REPORTS

ForEach ($Folder in $listOfFolders){
$newFiles = Get-ChildItem -Recurse $Folder | Where-Object { $_.LastWriteTime -ge $Age}
$newFiles | Select Name, DirectoryName, LastWriteTime
}

$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "FileServer"
UserName = "User"
Password = "Password"
GiveUpSecurityAndAcceptAnySshHostKey = $true
}

$session = New-Object WinSCP.Session
$session.SessionLogPath = "C:\Temp\sFTP.log"
$session.Open($sessionOptions)

# Synchronize files
ForEach ($file in $newFiles){
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = "*>=$Age"
$newFolder = $file.DirectoryName
$destFolder = $newFolder.Trim("F:\")

$synchronizationResult = $session.SynchronizeDirectories(
[WinSCP.SynchronizationMode]::Remote, $file.DirectoryName, "/home/egdms_user/Miami/" + $destFolder -replace ("\\", "/"), $False, $transferOptions)

# Throw on any error
$synchronizationResult.Check()
}

# Disconnect, clean up
$session.Dispose()

Your code seems overcomplicated. What exactly are you trying to do?
Why do you first programmatically find the new files, but then instruct WinSCP to try find the new files again. You already, know what are the new files.

Anyway, the timetamp in FileMask need to have a fixed syntax yyyy-mm-dd hh:mm:ss.
So you need something like:
See https://winscp.net/eng/docs/script_formatting_timestamp_batch_file

Though actually, this is way easier:
See https://winscp.net/eng/docs/file_mask#size_time

martin wrote:

Your code seems overcomplicated...

First thanks for the response and suggestions; much appreciated.

ETA2: What I am trying to do is copy via sFTP (not move nor delete from source) new files created within the last 10 minutes to a remote file server so analysts can pull the data from the files. Source files should remain intact and on the target server the script should create directories as needed and place the new (less than 10 minutes old) files into the path structure identified by the local server.

For example:
F:\SomeFolder\OldFile.txt
F:\SomeFolder2\AnotherFolder\NewFile.txt

On remote server only NewFile.txt shouild copy and create as needed the same folder strucure on the target server. Hope that helps.



I went ahead and cleaned up the code and now am solely relying ton the mask but it is still sync'ing the entire directory structure and files and not just those modified within the last 10 minutes. The log confirms this. Why is the mask not working with Sync? Do I need to use the Put method instead? I have attached the log.

ETA: Correction...its not recreating the full folder structure and instead is placing all the falsely identified new files/folders into the root of the destination folder. I believe this is why I had to use the earlier 'complicated' code to properly recreate the folder structure as the synch doesn't seem to do it on its own without some help.

Here is the new code:

#$ErrorActionPreference = "SilentlyContinue"

# Load WinSCP .NET assembly
Add-Type -Path "C:\Program Files (x86)\WinSCP\WinSCPnet.dll"

$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "Server"
UserName = "User"
Password = "PASSWORD"
GiveUpSecurityAndAcceptAnySshHostKey = $true
}

$session = New-Object WinSCP.Session
$session.SessionLogPath = "C:\Temp\sFTP.log"
$session.Open($sessionOptions)

# Synchronize files
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = "*>=10N"

$Folder_EGDMS_DIU = "F:\EGDMS_DIU"
$Folder_EGDMS_LOGS = "F:\EGDMS_LOGS"
$Folder_EGDMS_PROVISION = "F:\EGDMS_PROVISION"
$Folder_EGDMS_RAW = "F:\EGDMS_RAW"
$Folder_EGDMS_REPORTS = "F:\EGDMS_REPORTS"

$listOfFolders = $Folder_EGDMS_DIU, $Folder_EGDMS_LOGS, $Folder_EGDMS_PROVISION, $Folder_EGDMS_RAW, $Folder_EGDMS_REPORTS

ForEach ($Folder in $listOfFolders){

$synchronizationResult = $session.SynchronizeDirectories(
[WinSCP.SynchronizationMode]::Remote, $Folder, "/home/user/Miami/"

# Throw on any error
$synchronizationResult.Check()
}
# Disconnect, clean up
$session.Dispose()

Got it sorted. Seems like Put was the only way to get some of my logic to work. Thanks again for your help with the mask.

Final code:

$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "FileServer"
UserName = "User"
Password = "PASSWORD"
GiveUpSecurityAndAcceptAnySshHostKey = $true
}
$Folder_EGDMS_DIU = "F:\EGDMS_DIU"
$Folder_EGDMS_LOGS = "F:\EGDMS_LOGS"
$Folder_EGDMS_PROVISION = "F:\EGDMS_PROVISION"
$Folder_EGDMS_RAW = "F:\EGDMS_RAW"
$Folder_EGDMS_REPORTS = "F:\EGDMS_REPORTS"

$listOfFolders = $Folder_EGDMS_DIU, $Folder_EGDMS_LOGS, $Folder_EGDMS_PROVISION, $Folder_EGDMS_RAW, $Folder_EGDMS_REPORTS

$session = New-Object WinSCP.Session
$session.SessionLogPath = "C:\Temp\sFTP.log"
$session.Open($sessionOptions)

# Synchronize files
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = "*>=10N"

ForEach ($Folder in $listOfFolders){
# If the selected item is file, find all contained files and folders recursively
If (Test-Path $Folder -PathType container){
$files =
@($Folder) +
(Get-ChildItem $Folder -Recurse | Select-Object -ExpandProperty FullName)}
Else{
$files = $Folder}

$parentLocalPath = Split-Path -Parent (Resolve-Path $Folder)

foreach ($localFilePath in $files){
$remoteFilePath =
$session.TranslateLocalPathToRemote(
$localFilePath, $parentLocalPath, $remotePath)
$fullRemotePath = "/home/user/Miami/" + $remoteFilePath

if (Test-Path $localFilePath -PathType container){
# Create remote subdirectory, if it does not exist yet
if (!($session.FileExists("/home/user/Miami/" + $remoteFilePath))){
$session.CreateDirectory("/home/user/Miami/" + $remoteFilePath)}}
Else{
Write-Host "Moving file $localFilePath to $fullRemotePath..."
# Upload file and remove original
$session.PutFiles($localFilePath, $fullRemotePath, $False, $transferOptions).Check()
}
}
}

# Disconnect, clean up
$session.Dispose()