Optimizing a large script for FTP File downloads (~1100 lines)

Advertisement

abd1111
Joined:
Posts:
11

Optimizing a large script for FTP File downloads (~1100 lines)

I'm having a bit of trouble with running a .NET Assembly Powershell script. .NET Assembly and C code is very new to me, excuse my lack of knowledge. It takes ages and sometimes will timeout on various blocks of code, despite almost every script having a
$session.Timeout = New-TimeSpan -Seconds 600
or more.

I have a large FTP structure and I need certain files from certain subfolders. I have broken it up into many blocks like this, all with different filemasks and root/local folder parameters:
# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"
 
# Set up session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = "x"
    UserName = "x"
    Password = "x"
    FtpSecure = [WinSCP.FtpSecure]::Implicit
}
 
$session = New-Object WinSCP.Session
$remotePath = "/FTPFolder1/Subfolder1/Subf2/"
$localPath = "C:\Users\ABBBB\xxxxxx\xxxxxxx\xxxxxxx\xxxxxx\xxxxxxxx\"
 
try
{
    # Connect
    $session.Open($sessionOptions)
    $session.Timeout = New-TimeSpan -Seconds 600
 
    # Masking
    $transferOptions = New-Object WinSCP.TransferOptions 
    $transferOptions.FileMask = "Census Data Files/*/*.xlsx"  
 
    # Download files
    $transferOptions.AddRawSettings("ExcludeEmptyDirectories", "1")
    $session.GetFiles($remotePath, $localPath, $False, $transferOptions).Check()
}
finally
{
    # Disconnect, clean up
    $session.Dispose()
}
I am wondering how to make it more efficient. It takes a while for just one block to run, given the sheer scale of the subfolder and file structure, but, is there any way to make it more 'efficient'? Tips, ideas? .NET Assembly and C code are very new to me, excuse my lack of knowledge. I stashed all of them in a PowerShell ISE editor and run them all there.

Reply with quote

Advertisement

abd1111
Joined:
Posts:
11

Re: Optimizing a large script for FTP File downloads (~1100 lines)

I see, I will look into that, thank you. Yes, using a new login and end on each block.

Another question regarding timeout: With the given example of my original post, I've tried the filemask numerous times to get the same files (E.g. *Census Data Files/, Census Data Files/*/*.xlsx, Census Data Files/*, NamedCensusFileData.xlsx, etc.) and they often time out or endlessly run with no output into the local folder.

Is there something I could be doing differently? I tested those masks in WinSCP itself under the Find button, and it does take quite a while to get results. Is it just based on the structure size of the FTP folders?

Reply with quote

martin
Site Admin
martin avatar

Re: Optimizing a large script for FTP File downloads (~1100 lines)

Please post logs for your file mask problems.

Reply with quote

abd1111

Re: Optimizing a large script for FTP File downloads (~1100 lines)

martin wrote:

Please post logs for your file mask problems.
I may not be able to due to company data being displayed throughout the log, so I noticed.

Reply with quote

Advertisement

Guest

Re: Optimizing a large script for FTP File downloads (~1100 lines)

martin wrote:

Are you opening a new connection for each transfer? You better reuse it.
I tried some searches and can't determine how to implement a 're-use' of the connection for each transfer. Are you saying instead of having 3 different blocks of code that connect to the same FTP for different files to retrieve, group those 3 blocks under one $session.Open($sessionOptions), and before the $session.Dispose()?

Reply with quote

abd1111
Joined:
Posts:
11

Here is a log file I was able to create. The filemask is somewhat similar. Both masks have the same result (timeout error)
  • LogFileClsCensus.txt (1.81 MB, Private file)
Description: Here is a log file I was able to create for this.

Reply with quote

martin
Site Admin
martin avatar

Re: Optimizing a large script for FTP File downloads (~1100 lines)

Are you saying instead of having 3 different blocks of code that connect to the same FTP for different files to retrieve, group those 3 blocks under one $session.Open($sessionOptions), and before the $session.Dispose()?
Yes, that's what I mean.

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
41,440
Location:
Prague, Czechia

abd1111 wrote:

Here is a log file I was able to create. The filemask is somewhat similar. Both masks have the same result (timeout error)
In the log I see that WinSCP was trying for almost 30 minutes to find any file that matches your file mask. Then it was probably aborted somehow. I do not think it's "endless run". You just didn't give it enough time to complete.

Your file mask */Analyzer Census/*.* | ZZArchive/ is possibly incorrect. But you didn't tell us what you are trying to achieve. Are you trying to download only Analyzer Census folders, ignoring all others? The way you specify your file mask, you force WinSCP to walk though ALL the folders on your server to find those subfolders. That seems to take unbearably long. You need to give WinSCP more hints as to what folders to skip. Hard to help more, unless you tell us more about the directory structure.

Reply with quote

Advertisement

abd1111
Joined:
Posts:
11

Thank you for the response. Yeah, you've about summed it up. For example, I have a handful of 'Company Name' folders. Inside those folders are about 5 or 6 different folders full of files (one of them being the 'Analyzer Census' folder). I just need those files of the 'Analyzer Census' folders. Should I create the exclusion as
*/Analyzer Census/*.* | ZZArchive/; *PaycheckData*/; *PayTemplates*/; *Schedules*/; *GroupInfo*/
for example, to get rid of it looking through those other dense folders on the same level?

ALSO: Are those exclusion wildcards correct? Some folders have the company name before or after 'AA PaycheckData AMAZON', for example. Can you use double wildcards?

Reply with quote

Advertisement

You can post new topics in this forum