SFTP/FTPS file transfers in Microsoft Azure WebJob

WebJob is a package that contains your job executable (e.g. a batch file or a console application) along with all supporting files (script, WinSCP executable, WinSCP .NET assembly, etc). Azure platform automatically tries to detect what executable is the main one. To ease this, you should name the main executable run.*, e.g. run.bat for a batch file.

See implementation sections below for examples of the package contents.

To deploy your web job:

• Pack all the job files into a ZIP archive;
• Navigate to your web site page on Azure Management Portal;
• Switch to WebJobs tab;
• Use Add command on bottom bar;
• Give your WebJob a name; pick your ZIP archive and set up how the job is run.

After your WebJob is uploaded, it is extracted to /site/wwwroot/App_Data/jobs/type/name, where type is triggered for Scheduled and On demand jobs and continuous for Continuous jobs. From now on, you can manage/update the job files using FTPS.

In the WebJob environment, your web site data (as you see them over FTPS) are accessible at D:\home. The path is stored in an environment variable HOME. Other useful environment variables are:

When the WebJob is executed, its files are cloned to a temporary folder (see WEBJOBS_PATH) and the job is run there. So it makes no sense to store any data to job’s working directory, as that is not persistent and is not accessible remotely. Use WEBROOT_PATH to locate either job’s master directory (%WEBROOT_PATH%\App_Data\jobs\%WEBJOBS_TYPE%\%WEBJOBS_NAME%) or WEBJOBS_DATA_PATH to locate job’s data directory. See implementation sections below for examples.

The standard output of the job is redirected to a log file which you can display on Azure Management Portal.

It is useful to have WinSCP session log file included into the job’s log. For that as the last step of the job, print the WinSCP session log to the standard output. See implementation sections below for examples.

To display the job log, go to WebJobs tab of your web site page on Azure Management Portal and click on a link in Logs column of the job.

For a complex standalone transfer/synchronization tasks, you can use WinSCP .NET assembly from a PowerShell script.

Start with learning how to use WinSCP .NET assembly from PowerShell.

Comparing with generic WinSCP PowerShell examples, your Azure transfer job should:

• have a wrapper batch file named run.bat so that Azure correctly identifies it as the main executable (Azure WebJob cannot run PowerShell scripts directly);
• the PowerShell script should propagate any error from WinSCP code to an exit code, so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure);
• enable session logging to a unique job run directory (%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%);
• locate source or destination paths of the transfer using WebJob environment variables;
• print session log to a standard output, so that it is available from Azure Management Portal;
• cannot use Write-Host cmdlet in Azure WebJob as there is no console to write to, use Write-Output instead.

An example PowerShell script (backup.ps1) that backs up the WebSite to a remote SFTP server:

try
{
    # Load WinSCP .NET assembly
    Add-Type -Path "WinSCPnet.dll"
 
    # Setup session options
    $sessionOptions = New-Object WinSCP.SessionOptions
    $sessionOptions.Protocol = [WinSCP.Protocol]::Sftp
    $sessionOptions.HostName = "example.com"
    $sessionOptions.UserName = "user"
    $sessionOptions.Password = "mypassword"
    $sessionOptions.SshHostKeyFingerprint = "ssh-rsa 2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
    
    $sessionLogPath = "$env:WEBJOBS_DATA_PATH\$env:WEBJOBS_RUN_ID\session.log"
 
    $session = New-Object WinSCP.Session
 
    try
    {
        $session.SessionLogPath = $sessionLogPath
        # Connect
        Write-Output "Connecting..."
        $session.Open($sessionOptions)
 
        $backupPath = "/home/user/backup/$env:WEBJOBS_RUN_ID";
        $session.CreateDirectory($backupPath)
 
        Write-Output "Uploading..."
        $session.PutFiles("$env:WEBROOT_PATH\*", "$backupPath/*").Check()
    }
    finally
    {
        # Disconnect, clean up
        $session.Dispose()
    }
 
    $result = 0
}
catch [Exception]
{
    Write-Output $_.Exception.Message
    $result = 1
}
 
# Dumping session log to Azure log (standard output) when it exists
if (Test-Path $sessionLogPath)
{
    Write-Output "Session log:"
    Get-Content $sessionLogPath
}
 
# Reporting result to Azure
exit $result

An example wrapper batch file (run.bat):

powershell.exe -ExecutionPolicy Unrestricted -File backup.ps1

A complete WebJob package (to be deployed to Azure WebSite) consists of following files:

• winscp.exe executable;
• winscpnet.dll .NET assembly;
• backup.ps1 script;
• run.bat wrapper batch file;
• optionally a private key file for SSH authentication.

TODO

• Guide to connecting securely to a Microsoft Azure Web Site with FTPS;
• How to Deploy Azure WebJobs to Azure Websites;
• Use WebJobs to run background tasks in Microsoft Azure Websites.