Download only 'new' or 'modified' files

Advertisement

lintho
Guest

Download only 'new' or 'modified' files

Hello!

After some great help from you Martin last year i managed to get my scripts working in pipelies in devops(azure).

However, the downloading\uploading part is taking a loong time when im trying to deploy new versions of my websites.

This is my current script - which is downloading files from a folder on production web app, to a staging slot before swapping the slots.
$ftp = '$(hostname)'
$user = '$(produser)'
$pass = '$(prodpass)'
$folder = 'site/wwwroot/wwwroot/images/uploaded/*'
$sitenameFolder =  '$(sitename)'
$target = "C:\"+ $sitenameFolder+"\*"
Write-Host $target
 
# Load WinSCP .NET assembly
Add-Type -Path  "$(System.DefaultWorkingDirectory)/drop-yaml/drop-yaml/ftp/WinSCPnet.dll"
 
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = $ftp
    UserName = $user
    Password = $pass 
}
 
$session = New-Object WinSCP.Session
 
try
{
    # Connect
    $session.Open($sessionOptions)
 
    # Download files
 
    $transferResult = $session.GetFiles($folder, $target)
    $transferResult.Check()
 
    # Print results
    foreach ($transfer in $transferResult.Transfers)
    {
        Write-Host "Download of $transfer succeeded"
    }
}
finally
{
    # Disconnect, clean up
    $session.Dispose()
}
and for upload:
$ftp = '$(hostname)'
$user = '$(destinationuser)'
$pass = '$(desinationpass)'
$folder = 'site/wwwroot/wwwroot/images/uploaded/*'
 
$sitenameFolder =  '$(sitename)'
$fromFolder= "C:\"+ $sitenameFolder+"\*"
 
Write-Host $fromFolder
 
Add-Type -Path "$(System.DefaultWorkingDirectory)/drop-yaml/drop-yaml/ftp/WinSCPnet.dll"
 
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = $ftp
    UserName = $user
    Password = $pass 
}
 
$session = New-Object WinSCP.Session
 
try
{
    # Connect
    $session.Open($sessionOptions)
    $remotePath = $fromFolder
    $transferResult = $session.PutFiles(($fromFolder + "*.*"), ($folder + "*.*"), $False)
 
    foreach ($transfer in $transferResult.Transfers)
    {
        Write-Host "Uploaded of $transfer succeeded"
    }
}
finally
{
    # Disconnect, clean up
    $session.Dispose()
}
Is there any way to optimize this?

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
35,996
Location:
Prague, Czechia

Re: Download only 'new' or 'modified' files

This is quite broad question. And you didn't really explain what are you trying to achieve.
So I can only vaguely suggest that you may parallelize the transfer, if you need it to be completed faster.
It's not easy with PowerShell though.
We have only trivial example here:
https://winscp.net/eng/docs/library_example_parallel_transfers#powershell
I might write a more robust and generic example later.

Reply with quote

lintho
Guest

Hello.
Thanks for a quick reply! I will check it out.

I thought this was clear:
"downloading files from a folder on production web app, to a staging slot before swapping the slots."

But, on the website it its possible to upload files directly to a folder on the site. When i am deploying new versions of my website using CD/CI, i am using 'slots' in azure, so my staging slot is replacing the production website. If i do that without transfering files using winSCP, all files uploaded to the website ends up on the staging slot, and is NOT on production anymore. Therefor i am downloading all files from prod, and uploading them to staging slot, before i replace staging with prod.

Reply with quote

lintho
Guest

So, instead of downloading all files, is it possible\faster to check filesum on files or something so i dont have to download all files?

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
35,996
Location:
Prague, Czechia

lintho wrote:

But, on the website it its possible to upload files directly to a folder on the site. When i am deploying new versions of my website using CD/CI, i am using 'slots' in azure, so my staging slot is replacing the production website. If i do that without transfering files using winSCP, all files uploaded to the website ends up on the staging slot, and is NOT on production anymore. Therefor i am downloading all files from prod, and uploading them to staging slot, before i replace staging with prod.
I do not know anything about this, so I cannot advise.

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
35,996
Location:
Prague, Czechia

lintho wrote:

So, instead of downloading all files, is it possible\faster to check filesum on files or something so i dont have to download all files?
Theoretically yes, but most FTP servers do not support this. If your FTP server don't, you might build a service that provide the checksums and use it on your PowerShell script to choose the files to update. But in that case you might have the service automatically pull the updated files from the source, and you won't need the local intermediate PowerShell/WinSCP script.

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
35,996
Location:
Prague, Czechia

Re: Download only 'new' or 'modified' files

martin wrote:

This is quite broad question. And you didn't really explain what are you trying to achieve.
So I can only vaguely suggest that you may parallelize the transfer, if you need it to be completed faster.
It's not easy with PowerShell though.
We have only trivial example here:
https://winscp.net/eng/docs/library_example_parallel_transfers#powershell
I might write a more robust and generic example later.
I have added full implementation:
https://winscp.net/eng/docs/library_example_parallel_transfers#powershell

Reply with quote

Advertisement

You can post new topics in this forum