Hello,
First of all I want to describe our environment a little bit. In our environment we have a lot of external FTP sites where we get files from, most of them we poll the remote FTP servers every 5 / 10 minutes to see if there are new files available. I really hope anyone can help us out on this issue.
When we started this, we have created a couple of scripts in order to meet our demands (differs per interface). We now have around 80 scheduled tasks running (some start every 5 minutes and some start every hour) which polls remote servers to see if there are new files (mostly we are using the script that places all the filenames in a .txt to download only unique files).
Our scheduled tasks are run under the SYSTEM account and don't have highest privileges.
When we implemented around 20/30 scheduled tasks everything went perfect. Now we are at 80 we are noticing in our monitoring logs that we encounter some issue's:
This is a job that runs every 5 minutes. Sometimes when it runs it generates an error and ofcourse we cannot find the tempfile since it doesn't exist. Mostly after a couple of retry's it works, but we don't start these jobs for fun every 5 minutes.
2020-09-08-20-05-03,OK,0,JOB-72
2020-09-08-20-10-03,OK,0,JOB-72
2020-09-08-20-20-02,OK,0,JOB-72
2020-09-08-20-25-03,OK,0,JOB-72
2020-09-08-20-30-03,OK,0,JOB-72
2020-09-08-20-35-03,Error: Exception calling "Open" with "1" argument(s): "WinSCP process terminated with exit code 3. There was no output. Response log file C:\Windows\TEMP\wscp45D8.039CFF13.tmp was not created. This could indicate lack of write permissions to the log folder or problems starting WinSCP itself.",,JOB-72
2020-09-08-20-40-03,Error: Exception calling "Open" with "1" argument(s): "WinSCP process terminated with exit code 3. There was no output. Response log file C:\Windows\TEMP\wscp5124.01A453AA.tmp was not created. This could indicate lack of write permissions to the log folder or problems starting WinSCP itself.",,JOB-72
2020-09-08-20-45-04,Error: Exception calling "Open" with "1" argument(s): "WinSCP process terminated with exit code 3. There was no output. Response log file C:\Windows\TEMP\wscp2080.01A453AA.tmp was not created. This could indicate lack of write permissions to the log folder or problems starting WinSCP itself.",,JOB-72
2020-09-08-20-50-03,Error: Exception calling "Open" with "1" argument(s): "WinSCP process terminated with exit code 3. There was no output. Response log file C:\Windows\TEMP\wscp4D6C.01A453AA.tmp was not created. This could indicate lack of write permissions to the log folder or problems starting WinSCP itself.",,JOB-72
2020-09-08-20-55-04,OK,0,JOB-72
2020-09-08-21-05-04,Error: Exception calling "Open" with "1" argument(s): "WinSCP process terminated with exit code 3. There was no output. Response log file C:\Windows\TEMP\wscp3504.01A453AA.tmp was not created. This could indicate lack of write permissions to the log folder or problems starting WinSCP itself.",,JOB-72
2020-09-08-21-10-04,OK,0,JOB-72
2020-09-08-21-15-03,Error: Exception calling "Open" with "1" argument(s): "WinSCP process terminated with exit code 3. There was no output. Response log file C:\Windows\TEMP\wscp4CC4.01F4CB52.tmp was not created. This could indicate lack of write permissions to the log folder or problems starting WinSCP itself.",,JOB-72
2020-09-08-21-20-04,OK,0,JOB-72
Also we have implemented:
$sessionLogPath = "D:\Jobs\SessionLogs\JOB-72.log"
$session.SessionLogPath = $sessionLogPath
But when we receive an error in our monitoring logfile we don't see any new entry's in the sessionlogpath. So I don't have any more detailed logging.
The script is here below
param (
# Use Generate Session URL function to obtain a value for -sessionUrl parameter.
[Parameter(Mandatory = $True)]
$sessionUrl,
[Parameter(Mandatory = $True)]
$localPath,
[Parameter(Mandatory = $True)]
$remotePath,
[Parameter(Mandatory = $True)]
$jobName,
[Parameter(Mandatory = $True)]
$FileMask,
[Parameter(Mandatory = $True)]
$wildcard
)
#Pre-script variables
$username = ""
$pwdTxt = Get-Content "*****" | Select -First 1
$securePwd = $pwdTxt | ConvertTo-SecureString
$credential = New-Object System.Management.Automation.PSCredential -ArgumentList $username, $securePwd
try
{
#Set assemblypath
$assemblyPath = if ($env:WINSCP_PATH) { $env:WINSCP_PATH } else { "D:\FTPSync\Scripts" }
Add-Type -Path (Join-Path $assemblyPath "WinSCPnet.dll")
#Session Opions
$sessionOptions = New-Object WinSCP.SessionOptions
$sessionOptions.ParseUrl($sessionUrl)
$sessionOptions.GiveUpSecurityAndAcceptAnySshHostKey = "true"
$session = New-Object WinSCP.Session
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.ResumeSupport.State = [WinSCP.TransferResumeSupportState]::Off
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary
$transferOptions.FileMask = $FileMask
$date = Get-Date
$fulldate = $date.ToString("yyyy-MM-dd-HH-mm-ss")
$listPath = "D:\FTPSync\Unique\$jobname.log"
$monitoringPath = "D:\FTPSync\Monitoring\$jobname.log"
$scriptresultPath = "D:\FTPSync\ScriptResults\$jobname.log"
$sessionLogPath = "D:\FTPSync\SessionLogs\$jobname.log"
try
{
New-PSDrive -Name fileserver -Root $localPath -Credential $credential -PSProvider FileSystem
$session.SessionLogPath = $sessionLogPath
$listPath = [Environment]::ExpandEnvironmentVariables($listPath)
$listDir = (Split-Path -Parent $listPath)
if (Test-Path $listPath)
{
Write-Host "$fulldate Loading list of already downloaded files from $listPath..."
Add-Content -Path "$scriptresultPath" "$fulldate Loading list of already downloaded files from $listPath..."
$downloaded = @(Get-Content $listPath)
}
else
{
Write-Host "File $listPath with list of already downloaded files doesn't exist yet."
Add-Content -Path "$scriptresultPath" "$fulldate File $listPath with list of already downloaded files doesn't exist yet."
$downloaded = @()
}
Write-Host "Connecting..."
Add-Content -Path "$scriptresultPath" "$fulldate Connecting to $jobname"
$session.Open($sessionOptions)
Write-Host "Looking for new files..."
Add-Content -Path "$scriptresultPath" "$fulldate Looking for new files for $jobname"
$files =
$session.EnumerateRemoteFiles(
$remotePath, $wildcard, [WinSCP.EnumerationOptions]::None)
$count = 0
foreach ($fileInfo in $files)
{
$remoteFilePath = $fileInfo.FullName
if ($downloaded -notcontains $remoteFilePath)
{
$remoteFileLen = $fileInfo.Length
Write-Host `
"Found new file $remoteFilePath with size $remoteFileLen, downloading..."
Add-Content -Path "$scriptresultPath" "$fulldate Found new file $remoteFilePath with size $remoteFileLen, downloading... for $jobname"
New-PSDrive -Name fileserver9 -Root $localPath -Credential $credential -PSProvider FileSystem
$localFilePath =
[WinSCP.RemotePath]::TranslateRemotePathToLocal(
$remoteFilePath, $remotePath, $localPath)
$localFileDir = (Split-Path -Parent $localFilePath)
$source = [WinSCP.RemotePath]::EscapeFileMask($remoteFilePath)
$session.GetFiles($source, $localFilePath, $False, $transferOptions).Check()
Add-Content $listPath $remoteFilePath
$count++
Write-Host "Downloaded."
Add-Content -Path "$scriptresultPath" "$fulldate Downloaded. for $jobname"
#}
}
}
if ($count -gt 0)
{
Write-Host "Done, downloaded $count files."
Add-Content -Path "$monitoringPath" "$fulldate,OK,$count,$jobname"
}
else
{
Write-Host "Done, no new files found."
Add-Content -Path "$monitoringPath" "$fulldate,OK,$count,$jobname"
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
$result = 0
}
catch
{
Write-Host "Error: $($_.Exception.Message)"
Add-Content -Path "$monitoringPath" "$fulldate,Error: $($_.Exception.Message),$count,$jobname"
$result = 1
}
Exit $result