SFTP/FTPS file transfers in Microsoft Azure WebJob

WebJob is a package that contains your job executable (e.g. a batch file or a console application) along with all supporting files (script, WinSCP executable, WinSCP .NET assembly, etc). Azure platform automatically tries to detect what executable is the main one. To ease this, you should name the main executable run.*, e.g. run.bat for a batch file.

See implementation sections below for examples of the package contents.

To deploy your web job:

• Pack all the job files into a ZIP archive;
• Navigate to your app service page on Azure Portal portal.azure.com;
• Switch to WebJobs page;
• Use Add command on top bar;
• Give your WebJob a name; pick your ZIP archive and set up how the job is run.

After your WebJob is uploaded, it is extracted to /site/wwwroot/App_Data/jobs/type/name, where type is triggered for Scheduled and On demand jobs and continuous for Continuous jobs. From now on, you can manage/update the job files using FTPS.

You can also deploy the job using Visual Studio, see C# Console Application section below.

If you update your WebJob frequently, or if you have multiple instances of similar WebJobs, you can automate the update using WinSCP scripting.

First, learn how to setup FTPS access to the WebJob (WebSite).

Example script:

open ftpes://winscp\winscp:mypassword@waws-prod-am1-234.ftp.azurewebsites.windows.net/
put C:\myjob\* /site/wwwroot/App_Data/jobs/type/name/
exit

Make sure you update the example with your actual FTP credentials, FTP host name, the job type (see above) and name.

Learn how to Automate file transfers (or synchronization) to FTP server or SFTP server.

In the WebJob environment, your web site data (as you see them over FTPS) are accessible at D:\home. The path is stored in an environment variable HOME. Other useful environment variables are:

When the WebJob is executed, its files are cloned to a temporary folder (see WEBJOBS_PATH) and the job is run there. So it makes no sense to store any data to job’s working directory, as that is not persistent and is not accessible remotely. Use WEBROOT_PATH to locate either job’s master directory (%WEBROOT_PATH%\App_Data\jobs\%WEBJOBS_TYPE%\%WEBJOBS_NAME%) or WEBJOBS_DATA_PATH to locate job’s data directory. See implementation sections below for examples.

The standard output of the job is redirected to a log file which you can display on Azure Portal.

It is useful to have WinSCP session log file included into the job’s log. For that as the last step of the job, print the WinSCP session log to the standard output. See implementation sections below for examples.

To display the job log, go to WebJobs page of your app service on Azure Portal, select the job and click on the Logs button on the top bar.

For a complex standalone transfer/synchronization tasks, you can use WinSCP .NET assembly from a PowerShell script.

Start with learning how to use WinSCP .NET assembly from PowerShell.

Comparing with generic WinSCP PowerShell examples, your Azure transfer job should:

• have a wrapper batch file named run.bat so that Azure correctly identifies it as the main executable (Azure WebJob cannot run PowerShell scripts directly);
• the PowerShell script should propagate any error from WinSCP code to an exit code, so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure);
• enable session logging to a unique job run directory (%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%);
• locate source or destination paths of the transfer using WebJob environment variables;
• print the session log to the standard output, so that it is available from Azure Portal;
• not use Write-Host cmdlet as there is no console to write to in the Azure WebJob, use Write-Output instead.

An example PowerShell script (backup.ps1) that backs up the app service/web site to a remote SFTP server:

try
{
    # Load WinSCP .NET assembly
    Add-Type -Path "WinSCPnet.dll"
 
    # Setup session options
    $sessionOptions = New-Object WinSCP.SessionOptions -Property @{
        Protocol = [WinSCP.Protocol]::Sftp
        HostName = "example.com"
        UserName = "user"
        Password = "mypassword"
        SshHostKeyFingerprint = "ssh-rsa 2048 xxxxxxxxxxx..."
    }
    
    $sessionLogPath = "$env:WEBJOBS_DATA_PATH\$env:WEBJOBS_RUN_ID\session.log"
 
    $session = New-Object WinSCP.Session
 
    try
    {
        $session.SessionLogPath = $sessionLogPath
        # Connect
        Write-Output "Connecting..."
        $session.Open($sessionOptions)
 
        $backupPath = "/home/user/backup/$env:WEBJOBS_RUN_ID"
        $session.CreateDirectory($backupPath)
 
        Write-Output "Uploading..."
        $session.PutFilesToDirectory($env:WEBROOT_PATH, $backupPath).Check()
    }
    finally
    {
        # Disconnect, clean up
        $session.Dispose()
    }
 
    $result = 0
}
catch
{
    Write-Output "Error: $($_.Exception.Message)"
    $result = 1
}
 
# Dumping session log to Azure log (standard output) when it exists
if (Test-Path $sessionLogPath)
{
    Write-Output "Session log:"
    Get-Content $sessionLogPath
}
 
# Reporting result to Azure
exit $result

An example wrapper batch file (run.bat):

powershell.exe -ExecutionPolicy Unrestricted -File backup.ps1

A complete WebJob package (to be deployed to Azure App Service) consists of following files:

• winscp.exe executable;
• winscpnet.dll .NET assembly;
• backup.ps1 script;
• run.bat wrapper batch file;
• optionally a private key file for SSH authentication.

If you are implementing your task using C# (console) application, you can use WinSCP .NET assembly to implement file transfer/synchronization. Implementing the task in C# may be useful for very complex and possibly continuous jobs.

Start with learning more about the WinSCP .NET assembly, how to install it and see some C# examples.

Comparing with generic WinSCP .NET assembly C# examples, your Azure transfer application should:

• have a wrapper batch file named run.bat so that Azure correctly identifies it as the main executable;1
• propagate any error from WinSCP .NET library to its exit code, so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure);
• enable session logging to a unique job run directory (%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%);
• locate source or destination paths of the transfer using WebJob environment variables;
• print the session log to the standard output, so that it is available from Azure Portal or provide other means for accessing the log.

An example C# code that backs up the app service/web site to a remote SFTP server:

using System;
using System.IO;
using WinSCP;
 
class Example
{
    public static int Main()
    {
        int result;
        string sessionLogPath =
            Path.Combine(
                Environment.GetEnvironmentVariable("WEBJOBS_DATA_PATH"),
                Environment.GetEnvironmentVariable("WEBJOBS_RUN_ID"),
                "session.log");
 
        try
        {
            // Setup session options
            SessionOptions sessionOptions = new SessionOptions
            {
                Protocol = Protocol.Sftp,
                HostName = "example.com",
                UserName = "user",
                Password = "mypassword",
                SshHostKeyFingerprint = "ssh-rsa 2048 xxxxxxxxxxx..."
            };
 
            using (Session session = new Session())
            {
                session.SessionLogPath = sessionLogPath;
 
                // Connect
                Console.WriteLine("Connecting...");
                session.Open(sessionOptions);
 
                string backupPath =
                    RemotePath.Combine(
                        session.HomePath,
                        "backup/" + Environment.GetEnvironmentVariable("WEBJOBS_RUN_ID"));
                session.CreateDirectory(backupPath);
 
                Console.WriteLine("Uploading...");
                string source = Environment.GetEnvironmentVariable("WEBROOT_PATH");
                session.PutFilesToDirectory(source, backupPath).Check();
            }
 
            result = 0;
        }
        catch (Exception e)
        {
            Console.WriteLine("Error: {0}", e);
            result = 1;
        }
 
        // Dumping session log to Azure log (standard output) when it exists
        if (File.Exists(sessionLogPath))
        {
            Console.WriteLine("Session log:");
 
            using (Stream sessionLog = File.OpenRead(sessionLogPath))
            using (Stream standardOutput = Console.OpenStandardOutput())
            {
                sessionLog.CopyTo(standardOutput);
            }
        }
 
        return result;
    }
}

An example wrapper batch file (run.bat) that simply runs your application executable:

ConsoleApplication1.exe

A complete WebJob package (to be deployed to Azure App Service) consists of following files:

• your application executable (e.g. ConsoleApplication1.exe);
• winscp.exe executable;
• winscpnet.dll .NET assembly;
• run.bat wrapper batch file;
• optionally a private key file for SSH authentication.

You can develop the WebJob (console) application as any other desktop console application. You can also start with Azure WebJob project template.

With Visual Studio, you can ease the deployment using command Publish as Azure WebJob, which is available in the project context menu in Solution Explorer. It opens Add Azure WebJob dialog that allows you to name your job and setup how it is run (including scheduling). Make sure you add all additional files needed for the job (i.e. winscp.exe, run.bat and private key, as shown above) to the project with Build Action set to Content to have them deployed (see also Using WinSCP .NET assembly from Visual Studio). Easier is to use the NuGet package. When you submit the dialog, a publish process starts on the background in Web Publish Activity pane. Next time you publish, after making changes to the project, only modified files are uploaded. See also Develop and deploy WebJobs using Visual Studio - Azure App Service.

• Connecting securely to Microsoft Azure service with SFTP or FTPS;
• Introducing Windows Azure WebJobs;
• Develop and deploy WebJobs using Visual Studio - Azure App Service;
• Run background tasks with WebJobs in Azure App Service.