Differences
This shows you the differences between the selected revisions of the page.
| guide_microsoft_azure_webjob_sftp 2016-11-21 | guide_microsoft_azure_webjob_sftp 2025-02-14 (current) | ||
| Line 1: | Line 1: | ||
| ====== SFTP/FTPS file transfers in Microsoft Azure WebJob ====== | ====== SFTP/FTPS file transfers in Microsoft Azure WebJob ====== | ||
| - | If you need to export/backup data from your Microsoft Azure WebSite to a remote server or import data from the remote server to your WebSite, you can use WinSCP from Azure WebJob. | + | If you need to export/backup data from your Microsoft Azure App Service (Web Site) to a remote server or import data from the remote server to your App Service, you can use WinSCP from Azure WebJob. |
| - | The Azure WebJob is a script or an application run on Azure WebSite server that has a read/write access to your web site. The job can be executed on demand or schedule. To implement file transfers for the WebJob, you can either run WinSCP in [[scripting|a scripting mode]] from a batch file; use [[library|WinSCP .NET assembly]] from a [[#powershell|PowerShell script]] or develop [[#consoleapp|a (console) application]] that uses the WinSCP .NET assembly. | + | The Azure WebJob is a script or an application run on Azure App Service server that has a read/write access to your service (web site). The job can be executed on demand or schedule. To implement file transfers for the WebJob, you can either run WinSCP in [[scripting|a scripting mode]] from a batch file; use [[library|WinSCP .NET assembly]] from a [[#powershell|PowerShell script]] or develop [[#consoleapp|a (console) application]] that uses the WinSCP .NET assembly. |
| - | You can also use the WebJob without having an actual web site as a cloud-hosted scheduled task. | + | You can also use the WebJob without having an actual app service (web site) as a cloud-hosted scheduled task. |
| ===== [[structure]] WebJob structure ===== | ===== [[structure]] WebJob structure ===== | ||
| Line 15: | Line 15: | ||
| * Pack all the job files into a ZIP archive; | * Pack all the job files into a ZIP archive; | ||
| - | * Navigate to your web site page on [[https://manage.windowsazure.com/|Azure Management Portal]]; | + | * Navigate to your app service page on Azure Portal ''portal.azure.com''; |
| - | * Switch to //WebJobs// tab; | + | * Switch to //WebJobs// page; |
| - | * Use //Add// command on bottom bar; | + | * Use //Add// command on top bar; |
| * Give your WebJob a name; pick your ZIP archive and set up how the job is run. | * Give your WebJob a name; pick your ZIP archive and set up how the job is run. | ||
| - | After your WebJob is uploaded, it is extracted to ''/site/wwwroot/App_Data/jobs/type/name'', where ''type'' is ''triggered'' for //Scheduled// and //On demand// jobs and ''continuous'' for //Continuous// jobs. From now on, you can [[guide_microsoft_azure#website|manage/update the job files using FTPS]]. | + | After your WebJob is uploaded, it is extracted to ''/site/wwwroot/App_Data/jobs/type/name'', where ''type'' is ''triggered'' for //Scheduled// and //On demand// jobs and ''continuous'' for //Continuous// jobs. From now on, you can [[guide_microsoft_azure#appservice|manage/update the job files using FTPS]]. |
| //You can also deploy the job using Visual Studio, see [[#consoleapp|C# Console Application]] section below.// | //You can also deploy the job using Visual Studio, see [[#consoleapp|C# Console Application]] section below.// | ||
| Line 27: | Line 27: | ||
| If you update your WebJob frequently, or if you have multiple instances of similar WebJobs, you can automate the update using WinSCP [[scripting]]. | If you update your WebJob frequently, or if you have multiple instances of similar WebJobs, you can automate the update using WinSCP [[scripting]]. | ||
| - | First, learn how to [[guide_microsoft_azure#website|setup FTPS access to the WebJob (WebSite)]]. | + | First, learn how to [[guide_microsoft_azure#appservice|setup FTPS access to the WebJob (WebSite)]]. |
| Example script: | Example script: | ||
| <code winscp> | <code winscp> | ||
| - | open ftps://winscp\winscp:mypassword@waws-prod-am1-234.ftp.azurewebsites.windows.net/ | + | open ftpes://winscp\winscp:mypassword@waws-prod-am1-234.ftp.azurewebsites.windows.net/ |
| put C:\myjob\* /site/wwwroot/App_Data/jobs/type/name/ | put C:\myjob\* /site/wwwroot/App_Data/jobs/type/name/ | ||
| exit | exit | ||
| </code> | </code> | ||
| - | Make sure you update the example with your actual [[guide_microsoft_azure#website|FTP credentials]], %%FTP%% host name, the job type (see above) and name. | + | Make sure you update the example with your actual [[guide_microsoft_azure#appservice|FTP credentials]], %%FTP%% host name, the job type (see above) and name. |
| - | //Learn more about [[guide_automation|automating file transfers to FTPS server]].// | + | //Learn how to [[guide_automation|*]].// |
| ===== [[environment]] WebJob Environment ===== | ===== [[environment]] WebJob Environment ===== | ||
| Line 46: | Line 46: | ||
| ^ Variable name ^ Description ^ Example ^ | ^ Variable name ^ Description ^ Example ^ | ||
| - | | ''HOME'' | Web site root directory | ''D:\home'' | | + | | ''HOME'' | App service (web site) root directory | ''D:\home'' | |
| - | | ''WEBJOBS_DATA_PATH'' | Job data directory | ''D:\home\data\jobs\type\name''. | | + | | ''WEBJOBS_DATA_PATH'' | Job data directory | ''D:\home\data\jobs\type\name'' | |
| | ''WEBJOBS_NAME'' | Job name | | | | ''WEBJOBS_NAME'' | Job name | | | ||
| | ''WEBJOBS_PATH'' | Temporary directory, where job is running | ''C:\DWASFiles\Sites\~1sitename\Temp\jobs\type\name\lpej4hrk.fks'' | | | ''WEBJOBS_PATH'' | Temporary directory, where job is running | ''C:\DWASFiles\Sites\~1sitename\Temp\jobs\type\name\lpej4hrk.fks'' | | ||
| | ''WEBJOBS_RUN_ID'' | An unique ID of the job run (an UTC timestamp of the run). There's a data folder created for every run in ''%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%'' | ''201409090707416329'' | | | ''WEBJOBS_RUN_ID'' | An unique ID of the job run (an UTC timestamp of the run). There's a data folder created for every run in ''%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%'' | ''201409090707416329'' | | ||
| | ''WEBJOBS_TYPE'' | Job type | ''triggered'' or ''continuous'' | | | ''WEBJOBS_TYPE'' | Job type | ''triggered'' or ''continuous'' | | ||
| - | | ''WEBROOT_PATH'' | Web site root directory | ''D:\home\site\wwwroot'' | | + | | ''WEBROOT_PATH'' | App service (web site) root directory | ''D:\home\site\wwwroot'' | |
| - | | ''WEBSITE_SITE_NAME'' | Web site name | | | + | | ''WEBSITE_SITE_NAME'' | App service (web site) name | | |
| When the WebJob is executed, its files are cloned to a temporary folder (see ''WEBJOBS_PATH'') and the job is run there. So it makes no sense to store any data to job's working directory, as that is not persistent and is not accessible remotely. Use ''WEBROOT_PATH'' to locate either job's master directory (''%WEBROOT_PATH%\App_Data\jobs\%WEBJOBS_TYPE%\%WEBJOBS_NAME%'') or ''WEBJOBS_DATA_PATH'' to locate job's data directory. See [[#implementation|implementation sections]] below for examples. | When the WebJob is executed, its files are cloned to a temporary folder (see ''WEBJOBS_PATH'') and the job is run there. So it makes no sense to store any data to job's working directory, as that is not persistent and is not accessible remotely. Use ''WEBROOT_PATH'' to locate either job's master directory (''%WEBROOT_PATH%\App_Data\jobs\%WEBJOBS_TYPE%\%WEBJOBS_NAME%'') or ''WEBJOBS_DATA_PATH'' to locate job's data directory. See [[#implementation|implementation sections]] below for examples. | ||
| Line 59: | Line 59: | ||
| ===== [[log]] Copying Session Log to Job Log ===== | ===== [[log]] Copying Session Log to Job Log ===== | ||
| - | The standard output of the job is redirected to a log file which you can display on Azure Management Portal. | + | The standard output of the job is redirected to a log file which you can display on Azure Portal. |
| It is useful to have WinSCP session log file included into the job's log. For that as the last step of the job, print the WinSCP session log to the standard output. See [[#implementation|implementation sections]] below for examples. | It is useful to have WinSCP session log file included into the job's log. For that as the last step of the job, print the WinSCP session log to the standard output. See [[#implementation|implementation sections]] below for examples. | ||
| - | To display the job log, go to //WebJobs// tab of your web site page on Azure Management Portal and click on a link in //Logs// column of the job. | + | To display the job log, go to //WebJobs// page of your app service on Azure Portal, select the job and click on the //Logs// button on the top bar. |
| ===== [[implementation]] Implementation ===== | ===== [[implementation]] Implementation ===== | ||
| Line 70: | Line 70: | ||
| //If your transfer/synchronization task is simple, use WinSCP scripting. For complex tasks, see below for solutions using WinSCP .NET assembly from a [[#powershell|PowerShell script]] or [[#consoleapp|console application]].// | //If your transfer/synchronization task is simple, use WinSCP scripting. For complex tasks, see below for solutions using WinSCP .NET assembly from a [[#powershell|PowerShell script]] or [[#consoleapp|console application]].// | ||
| - | //Start with learning how to [[guide_automation|automate file transfers (or synchronization) to FTP server or SFTP server]].// | + | //Start with learning how to [[guide_automation|*]].// |
| Comparing with generic WinSCP script examples, your Azure transfer job should: | Comparing with generic WinSCP script examples, your Azure transfer job should: | ||
| * have a wrapper batch file named ''run.bat'' so that Azure correctly identifies it as [[#structure|the main executable]]; | * have a wrapper batch file named ''run.bat'' so that Azure correctly identifies it as [[#structure|the main executable]]; | ||
| - | * the wrapper batch file should propagate WinSCP [[scripting#checking_results|exit code]], so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure, the same for WinSCP); | + | * the wrapper batch file should propagate WinSCP [[scripting#result|exit code]], so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure, the same for WinSCP); |
| * enable session logging to a unique job run directory (''%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%''); | * enable session logging to a unique job run directory (''%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%''); | ||
| * locate source or destination paths of the transfer using WebJob [[#environment|environment variables]]; | * locate source or destination paths of the transfer using WebJob [[#environment|environment variables]]; | ||
| - | * [[#log|print the session log to the standard output]], so that it is available from Azure Management Portal. | + | * [[#log|print the session log to the standard output]], so that it is available from Azure Portal. |
| An example script (''script.txt'') that backs up the WebSite to a remote SFTP server: | An example script (''script.txt'') that backs up the WebSite to a remote SFTP server: | ||
| Line 84: | Line 84: | ||
| <code winscp> | <code winscp> | ||
| # Connect | # Connect | ||
| - | open sftp://user:password@example.com/ -hostkey="ssh-rsa 2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx" | + | open sftp://user:password@example.com/ -hostkey="ssh-rsa 2048 xxxxxxxxxxx..." |
| cd backup | cd backup | ||
| # Use a job run ID (timestamp) as a backup ID | # Use a job run ID (timestamp) as a backup ID | ||
| mkdir %WEBJOBS_RUN_ID% | mkdir %WEBJOBS_RUN_ID% | ||
| - | # Upload a whole web site | + | # Upload a whole app service/web site |
| put %WEBROOT_PATH%\* %WEBJOBS_RUN_ID%/ | put %WEBROOT_PATH%\* %WEBJOBS_RUN_ID%/ | ||
| exit | exit | ||
| Line 114: | Line 114: | ||
| </code> | </code> | ||
| - | A complete WebJob package (to be [[#deploying|deployed to Azure WebSite]]) consists of following files: | + | A complete WebJob package (to be [[#deploying|deployed to Azure App Service]]) consists of following files: |
| * ''winscp.com'' [[executables|executable]]; | * ''winscp.com'' [[executables|executable]]; | ||
| Line 133: | Line 133: | ||
| * enable session logging to a unique job run directory (''%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%''); | * enable session logging to a unique job run directory (''%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%''); | ||
| * locate source or destination paths of the transfer using WebJob [[#environment|environment variables]]; | * locate source or destination paths of the transfer using WebJob [[#environment|environment variables]]; | ||
| - | * [[#log|print the session log to the standard output]], so that it is available from Azure Management Portal; | + | * [[#log|print the session log to the standard output]], so that it is available from Azure Portal; |
| * not use ''[[ps>microsoft.powershell.utility/write-host|Write-Host]]'' cmdlet as there is no console to write to in the Azure WebJob, use ''[[ps>microsoft.powershell.utility/write-output|Write-Output]]'' instead. | * not use ''[[ps>microsoft.powershell.utility/write-host|Write-Host]]'' cmdlet as there is no console to write to in the Azure WebJob, use ''[[ps>microsoft.powershell.utility/write-output|Write-Output]]'' instead. | ||
| - | An example PowerShell script (''backup.ps1'') that backs up the WebSite to a remote SFTP server: | + | An example PowerShell script (''backup.ps1'') that backs up the app service/web site to a remote SFTP server: |
| <code powershell> | <code powershell> | ||
| Line 150: | Line 150: | ||
| UserName = "user" | UserName = "user" | ||
| Password = "mypassword" | Password = "mypassword" | ||
| - | SshHostKeyFingerprint = "ssh-rsa 2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx" | + | SshHostKeyFingerprint = "ssh-rsa 2048 xxxxxxxxxxx..." |
| } | } | ||
| Line 168: | Line 168: | ||
| Write-Output "Uploading..." | Write-Output "Uploading..." | ||
| - | $session.PutFiles("$env:WEBROOT_PATH\*", "$backupPath/*").Check() | + | $session.PutFilesToDirectory($env:WEBROOT_PATH, $backupPath).Check() |
| } | } | ||
| finally | finally | ||
| Line 178: | Line 178: | ||
| $result = 0 | $result = 0 | ||
| } | } | ||
| - | catch [Exception] | + | catch |
| { | { | ||
| - | Write-Output ("Error: {0}" -f $_.Exception.Message) | + | Write-Output "Error: $($_.Exception.Message)" |
| $result = 1 | $result = 1 | ||
| } | } | ||
| Line 201: | Line 201: | ||
| </code> | </code> | ||
| - | A complete WebJob package (to be [[#deploying|deployed to Azure WebSite]]) consists of following files: | + | A complete WebJob package (to be [[#deploying|deployed to Azure App Service]]) consists of following files: |
| * ''winscp.exe'' [[executables|executable]]; | * ''winscp.exe'' [[executables|executable]]; | ||
| Line 216: | Line 216: | ||
| Comparing with generic WinSCP .NET assembly C# examples, your Azure transfer application should: | Comparing with generic WinSCP .NET assembly C# examples, your Azure transfer application should: | ||
| - | * have a wrapper batch file named ''run.bat'' so that Azure correctly identifies it as [[#structure|the main executable]]((There will be two ''.exe'' files, your application and ''winscp.exe'', so Azure won't know which one to pick up, unless you name your application ''run.exe''.)); | + | * have a wrapper batch file named ''run.bat'' so that Azure correctly identifies it as [[#structure|the main executable]];((There will be two ''.exe'' files, your application and ''winscp.exe'', so Azure won't know which one to pick up, unless you name your application ''run.exe''.)) |
| * propagate any error from WinSCP .NET library to its exit code, so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure); | * propagate any error from WinSCP .NET library to its exit code, so that Azure correctly identifies failures (any non-zero exit code is a failure to Azure); | ||
| * enable session logging to a unique job run directory (''%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%''); | * enable session logging to a unique job run directory (''%WEBJOBS_DATA_PATH%\%WEBJOBS_RUN_ID%''); | ||
| * locate source or destination paths of the transfer using WebJob [[#environment|environment variables]]; | * locate source or destination paths of the transfer using WebJob [[#environment|environment variables]]; | ||
| - | * [[#log|print the session log to the standard output]], so that it is available from Azure Management Portal or provide other means for accessing the log. | + | * [[#log|print the session log to the standard output]], so that it is available from Azure Portal or provide other means for accessing the log. |
| - | An example C# code that backs up the WebSite to a remote SFTP server: | + | An example C# code that backs up the app service/web site to a remote SFTP server: |
| <code csharp> | <code csharp> | ||
| Line 249: | Line 249: | ||
| UserName = "user", | UserName = "user", | ||
| Password = "mypassword", | Password = "mypassword", | ||
| - | SshHostKeyFingerprint = "ssh-rsa 2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx" | + | SshHostKeyFingerprint = "ssh-rsa 2048 xxxxxxxxxxx..." |
| }; | }; | ||
| Line 261: | Line 261: | ||
| string backupPath = | string backupPath = | ||
| - | "/home/user/backup/" + | + | RemotePath.Combine( |
| - | ···················Environment.GetEnvironmentVariable("WEBJOBS_RUN_ID"); | + | ························session.HomePath, |
| + | "backup/" + Environment.GetEnvironmentVariable("WEBJOBS_RUN_ID")); | ||
| session.CreateDirectory(backupPath); | session.CreateDirectory(backupPath); | ||
| Console.WriteLine("Uploading..."); | Console.WriteLine("Uploading..."); | ||
| - | string source = Path.Combine(Environment.GetEnvironmentVariable("WEBROOT_PATH"), "*"); | + | string source = Environment.GetEnvironmentVariable("WEBROOT_PATH"); |
| - | string target = backupPath + "/"; | + | session.PutFilesToDirectory(source, backupPath).Check(); |
| - | session.PutFiles(source, target).Check(); | + | |
| } | } | ||
| Line 302: | Line 302: | ||
| </code> | </code> | ||
| - | A complete WebJob package (to be [[#deploying|deployed to Azure WebSite]]) consists of following files: | + | A complete WebJob package (to be [[#deploying|deployed to Azure App Service]]) consists of following files: |
| * your application executable (e.g. ''ConsoleApplication1.exe''); | * your application executable (e.g. ''ConsoleApplication1.exe''); | ||
| Line 310: | Line 310: | ||
| * optionally [[public_key|a private key file]] for SSH authentication. | * optionally [[public_key|a private key file]] for SSH authentication. | ||
| - | You can develop the WebJob (console) application as any other, even using //Visual Studio Express for Windows Desktop//. If you are using //Visual Studio Express for Web//, you can use //Microsoft Azure WebJob// project template. It's basically the same template as a //Console Application// in the Desktop edition.((Web edition does not have //Console Application// template.)) | + | You can develop the WebJob (console) application as any other desktop console application. You can also start with //Azure WebJob// project template. |
| - | With //Visual Studio Express for Web// (2013 edition with Update 3)((For older versions you need Azure SDK.)) you can ease [[#deploying|the deployment]] using command //Publish as Azure WebJob//, which is available in the project context menu in //Solution Explorer//. It opens //[[https://docs.microsoft.com/en-us/azure/app-service-web/websites-dotnet-deploy-webjobs#configure|Add Azure WebJob]]// dialog that allows you to name your job and setup how it is run (including scheduling). Make sure you add all additional files needed for the job (i.e. ''winscp.exe'', ''run.bat'' and private key, as shown above) to the project with //Build Action// set to //Content// to have them deployed (see also [[library_install#vs|Using WinSCP .NET assembly from Visual Studio]]). When you submit the dialog, a publish process starts on the background in //Web Publish Activity// pane. Next time you publish, after making changes to the project, only modified files are uploaded. See also [[https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-deploy-webjobs/|How to Deploy Azure WebJobs to Azure Websites]]. | + | With Visual Studio, you can ease [[#deploying|the deployment]] using command //Publish as Azure WebJob//, which is available in the project context menu in //Solution Explorer//. It opens //[[https://learn.microsoft.com/en-us/azure/app-service/webjobs-dotnet-deploy-vs#configure|Add Azure WebJob]]// dialog that allows you to name your job and setup how it is run (including scheduling). Make sure you add all additional files needed for the job (i.e. ''winscp.exe'', ''run.bat'' and private key, as shown above) to the project with //Build Action// set to //Content// to have them deployed (see also [[library_install#vs|Using WinSCP .NET assembly from Visual Studio]]). Easier is to use the [[library_install#nuget|NuGet package]]. When you submit the dialog, a publish process starts in the background in //Web Publish Activity// pane. Next time you publish, after making changes to the project, only modified files are uploaded. See also [[https://learn.microsoft.com/en-us/azure/app-service/webjobs-dotnet-deploy-vs|Develop and deploy WebJobs using Visual Studio - Azure App Service]]. |
| ===== Further Reading ===== | ===== Further Reading ===== | ||
| - | * Guide to [[guide_microsoft_azure#website|connecting securely to a Microsoft Azure Web Site with FTPS]]; | + | * [[guide_microsoft_azure#appservice|*]]; |
| - | * [[http://www.hanselman.com/blog/IntroducingWindowsAzureWebJobs.aspx|Introducing Windows Azure WebJobs]]; | + | * [[https://www.hanselman.com/blog/introducing-windows-azure-webjobs|Introducing Windows Azure WebJobs]]; |
| - | * [[https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-deploy-webjobs/|How to Deploy Azure WebJobs to Azure Websites]]; | + | * [[https://learn.microsoft.com/en-us/azure/app-service/webjobs-dotnet-deploy-vs|Develop and deploy WebJobs using Visual Studio - Azure App Service]]; |
| - | * [[https://azure.microsoft.com/en-us/documentation/articles/web-sites-create-web-jobs/|Use WebJobs to run background tasks in Microsoft Azure Websites]]. | + | * [[https://learn.microsoft.com/en-us/azure/app-service/webjobs-create|Run background tasks with WebJobs in Azure App Service]]. |