Contents » Using WinSCP » Guides » Cloud Computing »

SFTP/FTPS file transfers in Microsoft Azure WebJob

If you need to export/backup data from your Microsoft Azure WebSite to a remote server or import data from the remote server to your WebSite, you can use WinSCP from Azure WebJob.

The Azure WebJob is a script or an application run on Azure WebSite server that has a read/write access to your web site. The job can be executed on demand or schedule. To implement file transfers for the WebJob, you can either run WinSCP in a scripting mode from a batch file; use WinSCP .NET assembly from a PowerShell script or develop a (console) application that uses the WinSCP .NET assembly.

You can also use the WebJob without having an actual web site as a cloud-hosted scheduled task.

WebJob structure

WebJob is a package that contains your job executable (e.g. a batch file or a console application) along with all supporting files (script, WinSCP executable, WinSCP .NET assembly, etc). Azure platform automatically tries to detect what executable is the main one. To ease this, you should name the main executable run.*, e.g. run.bat for a batch file.

Advertisements:

See implementation sections below for examples of the package contents.

Deploying WebJob

To deploy your web job:

  • Pack all the job files into a ZIP archive;
  • Navigate to your web site page on Azure Management Portal;
  • Switch to WebJobs tab;
  • Use Add command on bottom bar;
  • Give your WebJob a name; pick your ZIP archive and set up how the job is run.

After your WebJob is uploaded, it is extracted to /site/wwwroot/App_Data/jobs/type/name, where type is triggered for Scheduled and On demand jobs and continuous for Continuous jobs. From now on, you can manage/update the job files using FTPS.

You can also deploy the job using Visual Studio, see C# Console Application section below.

Automating WebJob Update

If you update your WebJob frequently, or if you have multiple instances of similar WebJobs, you can automate the update using WinSCP scripting.

First, learn how to setup FTPS access to the WebJob (WebSite).

Example script:

open ftps://winscp\winscp:mypassword@waws-prod-am1-234.ftp.azurewebsites.windows.net/
put C:\myjob\* /site/wwwroot/App_Data/jobs/type/name/
exit

Make sure you update the example with your actual FTP credentials, FTP host name, the job type (see above) and name.

Learn more about automating file transfers to FTPS server.

WebJob Environment

In the WebJob environment, your web site data (as you see them over FTPS) are accessible at D:\home. The path is stored in an environment variable HOME. Other useful environment variables are:

Variable name Description Example
HOME Web site root directory D:\home
WEBJOBS_DATA_PATH Job data directory D:\home\data\jobs\type\name.
WEBJOBS_NAME Job name
WEBJOBS_PATH Temporary directory, where job is running C:\DWASFiles\Sites\~1sitename\Temp\jobs\type\name\lpej4hrk.fks
WEBJOBS_RUN_ID An unique ID of the job run (an UTC timestamp of the run). There’s a data folder created for every run in %WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID% 201409090707416329
WEBJOBS_TYPE Job type triggered or continuous
WEBROOT_PATH Web site root directory D:\home\site\wwwroot
WEBSITE_SITE_NAME Web site name

When the WebJob is executed, its files are cloned to a temporary folder (see WEBJOBS_PATH) and the job is run there. So it makes no sense to store any data to job’s working directory, as that is not persistent and is not accessible remotely. Use WEBROOT_PATH to locate either job’s master directory (%WEBROOT_PATH%\App_Data\jobs\%WEBJOBS_TYPE%\%WEBJOBS_NAME%) or WEBJOBS_DATA_PATH to locate job’s data directory. See implementation sections below for examples.

Copying Session Log to Job Log

The standard output of the job is redirected to a log file which you can display on Azure Management Portal.

It is useful to have WinSCP session log file included into the job’s log. For that as the last step of the job, print the WinSCP session log to the standard output. See implementation sections below for examples.

To display the job log, go to WebJobs tab of your web site page on Azure Management Portal and click on a link in Logs column of the job.

Implementation

Using Scripting

If your transfer/synchronization task is simple, use WinSCP scripting. For complex tasks, see below for solutions using WinSCP .NET assembly from a PowerShell script or console application.

Start with learning how to automate file transfers (or synchronization) to FTP server or SFTP server.

Comparing with generic WinSCP script examples, your Azure transfer job should:

  • have a wrapper batch file named run.bat so that Azure correctly identifies it as the main executable;
  • the wrapper batch file should propagate WinSCP exit code, so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure, the same for WinSCP);
  • enable session logging to a unique job run directory (%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%);
  • locate source or destination paths of the transfer using WebJob environment variables;
  • print the session log to the standard output, so that it is available from Azure Management Portal.

An example script (script.txt) that backs up the WebSite to a remote SFTP server:

# Connect
open sftp://user:password@example.com/ -hostkey="ssh-rsa 2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
cd backup
# Use a job run ID (timestamp) as a backup ID
mkdir %WEBJOBS_RUN_ID%
# Upload a whole web site
put %WEBROOT_PATH%\* %WEBJOBS_RUN_ID%/
exit

An example wrapper batch file (run.bat):

@echo off
set LOG=%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%\session.log
 
echo Running script...
winscp.com /script=script.txt /log=%LOG%
set RESULT=%ERRORLEVEL%
echo Result code is %RESULT%
 
rem Dumping session log to Azure log (standard output) when it exists
if exist %LOG% (
    echo Session log:
    type %LOG%
)
 
rem Propagating WinSCP exit code to Azure
exit /b %RESULT%

A complete WebJob package (to be deployed to Azure WebSite) consists of following files:

  • winscp.com executable;
  • winscp.exe executable;
  • script.txt script;
  • run.bat wrapper batch file;
  • optionally a private key file for SSH authentication.

Using PowerShell

For a complex standalone transfer/synchronization tasks, you can use WinSCP .NET assembly from a PowerShell script.

Start with learning how to use WinSCP .NET assembly from PowerShell.

Comparing with generic WinSCP PowerShell examples, your Azure transfer job should:

  • have a wrapper batch file named run.bat so that Azure correctly identifies it as the main executable (Azure WebJob cannot run PowerShell scripts directly);
  • the PowerShell script should propagate any error from WinSCP code to an exit code, so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure);
  • enable session logging to a unique job run directory (%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%);
  • locate source or destination paths of the transfer using WebJob environment variables;
  • print the session log to the standard output, so that it is available from Azure Management Portal;
  • not use Write-Host cmdlet as there is no console to write to in the Azure WebJob, use Write-Output instead.

An example PowerShell script (backup.ps1) that backs up the WebSite to a remote SFTP server:

try
{
    # Load WinSCP .NET assembly
    Add-Type -Path "WinSCPnet.dll"
 
    # Setup session options
    $sessionOptions = New-Object WinSCP.SessionOptions -Property @{
        Protocol = [WinSCP.Protocol]::Sftp
        HostName = "example.com"
        UserName = "user"
        Password = "mypassword"
        SshHostKeyFingerprint = "ssh-rsa 2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
    }
    
    $sessionLogPath = "$env:WEBJOBS_DATA_PATH\$env:WEBJOBS_RUN_ID\session.log"
 
    $session = New-Object WinSCP.Session
 
    try
    {
        $session.SessionLogPath = $sessionLogPath
        # Connect
        Write-Output "Connecting..."
        $session.Open($sessionOptions)
 
        $backupPath = "/home/user/backup/$env:WEBJOBS_RUN_ID"
        $session.CreateDirectory($backupPath)
 
        Write-Output "Uploading..."
        $session.PutFiles("$env:WEBROOT_PATH\*", "$backupPath/*").Check()
    }
    finally
    {
        # Disconnect, clean up
        $session.Dispose()
    }
 
    $result = 0
}
catch [Exception]
{
    Write-Output ("Error: {0}" -f $_.Exception.Message)
    $result = 1
}
 
# Dumping session log to Azure log (standard output) when it exists
if (Test-Path $sessionLogPath)
{
    Write-Output "Session log:"
    Get-Content $sessionLogPath
}
 
# Reporting result to Azure
exit $result

An example wrapper batch file (run.bat):

powershell.exe -ExecutionPolicy Unrestricted -File backup.ps1

A complete WebJob package (to be deployed to Azure WebSite) consists of following files:

C# Console Application

If you are implementing your task using C# (console) application, you can use WinSCP .NET assembly to implement file transfer/synchronization. Implementing the task in C# may be useful for very complex and possibly continuous jobs.

Start with learning more about the WinSCP .NET assembly, how to install it and see some C# examples.

Comparing with generic WinSCP .NET assembly C# examples, your Azure transfer application should:

  • have a wrapper batch file named run.bat so that Azure correctly identifies it as the main executable1);
  • propagate any error from WinSCP .NET library to its exit code, so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure);
  • enable session logging to a unique job run directory (%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%);
  • locate source or destination paths of the transfer using WebJob environment variables;
  • print the session log to the standard output, so that it is available from Azure Management Portal or provide other means for accessing the log.

An example C# code that backs up the WebSite to a remote SFTP server:

using System;
using System.IO;
using WinSCP;
 
class Example
{
    public static int Main()
    {
        int result;
        string sessionLogPath =
            Path.Combine(
                Environment.GetEnvironmentVariable("WEBJOBS_DATA_PATH"),
                Environment.GetEnvironmentVariable("WEBJOBS_RUN_ID"),
                "session.log");
 
        try
        {
            // Setup session options
            SessionOptions sessionOptions = new SessionOptions
            {
                Protocol = Protocol.Sftp,
                HostName = "example.com",
                UserName = "user",
                Password = "mypassword",
                SshHostKeyFingerprint = "ssh-rsa 2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
            };
 
            using (Session session = new Session())
            {
                session.SessionLogPath = sessionLogPath;
 
                // Connect
                Console.WriteLine("Connecting...");
                session.Open(sessionOptions);
 
                string backupPath =
                    "/home/user/backup/" +
                    Environment.GetEnvironmentVariable("WEBJOBS_RUN_ID");
                session.CreateDirectory(backupPath);
 
                Console.WriteLine("Uploading...");
                string source = Path.Combine(Environment.GetEnvironmentVariable("WEBROOT_PATH"), "*");
                string target = backupPath + "/";
                session.PutFiles(source, target).Check();
            }
 
            result = 0;
        }
        catch (Exception e)
        {
            Console.WriteLine("Error: {0}", e);
            result = 1;
        }
 
        // Dumping session log to Azure log (standard output) when it exists
        if (File.Exists(sessionLogPath))
        {
            Console.WriteLine("Session log:");
 
            using (Stream sessionLog = File.OpenRead(sessionLogPath))
            using (Stream standardOutput = Console.OpenStandardOutput())
            {
                sessionLog.CopyTo(standardOutput);
            }
        }
 
        return result;
    }
}

An example wrapper batch file (run.bat) that simply runs your application executable:

ConsoleApplication1.exe

A complete WebJob package (to be deployed to Azure WebSite) consists of following files:

You can develop the WebJob (console) application as any other, even using Visual Studio Express for Windows Desktop. If you are using Visual Studio Express for Web, you can use Microsoft Azure WebJob project template. It’s basically the same template as a Console Application in the Desktop edition.2)

With Visual Studio Express for Web (2013 edition with Update 3)3) you can ease the deployment using command Publish as Azure WebJob, which is available in the project context menu in Solution Explorer. It opens Add Azure WebJob dialog that allows you to name your job and setup how it is run (including scheduling). Make sure you add all additional files needed for the job (i.e. winscp.exe, run.bat and private key, as shown above) to the project with Build Action set to Content to have them deployed (see also Using WinSCP .NET assembly from Visual Studio). When you submit the dialog, a publish process starts on the background in Web Publish Activity pane. Next time you publish, after making changes to the project, only modified files are uploaded. See also How to Deploy Azure WebJobs to Azure Websites.

Further Reading

1) There will be two .exe files, your application and winscp.exe, so Azure won’t know which one to pick up, unless you name your application run.exe.
2) Web edition does not have Console Application template.
3) For older versions you need Azure SDK.
 
  guide_microsoft_azure_webjob_sftp.txt · Last modified: by martin
 

Search Documentation

This page

Donate

About donations

$9   $19   $49   $99

About donations

Recommend

Associations

Site design by Black Gate