How do I transfer new/modified files only?

When destination directory contains old files

When your destination directory already contains all the previously transferred files, synchronize source directory to destination directory, instead of simple transfer.

Advertisement

Basically use synchronize local instead of get and synchronize remote instead of put.

With WinSCP .NET assembly that means, use Session.SynchronizeDirectories, with argument mode set to SynchronizationMode.Remote or SynchronizationMode.Local instead of Session.GetFiles or Session.PutFiles respectively.

When destination directory does not contain old files

When the old files are removed from the destination directory, you can instead select the new files based on timestamp.

Relative time constraint

For that you can use file mask with time constraint. For example, to transfer only files created/modified since yesterday, use mask *>=1D (means all files modified in the last 24 hours).

In scripting, apply the mask using -filemask=<mask> switch of get or put commands.

For example:

put -filemask="*>=1D" *

In WinSCP .NET assembly, use TransferOptions.FileMask. PowerShell example:

$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = "*>=1D"
 
$session.PutFiles("d:\toupload\*", "/home/user/", $False, $transferOptions)

Absolute time constraint

If you need to select file specifically created/modified today (not simply in the last 24 hours), you need to generate a file mask with absolute timestamp constraint, like *>=2017-05-12.

In scripting, can also use %TIMESTAMP% syntax to generate “today” time constraint, e.g. *>=%TIMESTAMP#yyyy-mm-dd%.

In WinSCP .NET assembly, use System.DateTime class. For example in PowerShell:

$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.FileMask = ("*>=" + $(Get-Date -Format "yyyy-MM-dd"))
 
$session.PutFiles("d:\toupload\*", "/home/user/", $False, $transferOptions)

Remembering the last timestamp

In some cases you cannot rely on some artificially determined timestamp. E.g. because the files are being added continuously and your code needs to continuously (or very frequently) fetch them. In such case, you can remember the last run time, or even better the timestamp of the latest file processes in the last run, and use that timestamp as the threshold for the consecutive run.

The following snippet shows, how to implement a continuously running PowerShell script that fetches new files:

$sourcePath = "/remote/path/*"
$destPath = "C:\local\path\"
$lastTimestamp = $Null
 
while ($True)
{
    $transferOptions = New-Object WinSCP.TransferOptions
 
    if ($lastTimestamp -ne $Null) 
    {
        Write-Host "Downloading files modified after $lastTimestamp..."
        $transferOptions.FileMask = ("*>" + $lastTimestamp.ToString("yyyy-MM-dd HH:mm:ss"))
    }
    else
    {
        Write-Host "Downloading all files..."
    }
 
    $transferResult = $session.GetFiles($sourcePath, $destPath, $False, $transferOptions)
    $transferResult.Check()
 
    # Find the latest downloaded file
    $latestTransfer =
        $transferResult.Transfers |
        Sort-Object -Property @{ Expression = { (Get-Item $_.Destination).LastWriteTime } } `
                    -Descending |
        Select-Object -First 1
 
    if ($latestTransfer -eq $Null)
    {
        Write-Host "No files found."
    }
    else
    {
        $lastTimestamp = (Get-Item $latestTransfer.Destination).LastWriteTime
        Write-Host (
            "Downloaded $($transferResult.Transfers.Count) files, " +
            "latest being $($latestTransfer.FileName) with timestamp $lastTimestamp.")
    }
 
    Write-Host "Waiting..."
    Start-Sleep -Seconds 5
}

Though, the your code may run in separate scheduled runs, rather then continuously. Then you have to persist the last timestamp between the runs, for example like this:

# Read last timestamp of previous run, if any
$timestampFile = "timestamp.dat"
if (Test-Path $timestampFile)
{
    $lastTimestamp = [DateTime]::ParseExact((Get-Content $timestampFile), 'O', $Null)
}
else
{
    $lastTimestamp = $Null
}
 
# The inner code from loop of the above code snippet
...
 
# Persist timestamp for the next run
Set-Content -Path $timestampFile -Value $lastTimestamp.ToString("O")