Forum » .NET assembly / COM library »

Post a reply

Options
Add an Attachment

If you do not want to add an Attachment to your Post, please leave the Fields blank.

(maximum 10 MB; please compress large files; only common media, archive, text and programming file formats are allowed)

Options

Topic review

abd1111

Thank you for your help.
martin

Yes that should do.
abd1111

Thank you for the response. Yeah, you've about summed it up. For example, I have a handful of 'Company Name' folders. Inside those folders are about 5 or 6 different folders full of files (one of them being the 'Analyzer Census' folder). I just need those files of the 'Analyzer Census' folders. Should I create the exclusion as 
*/Analyzer Census/*.* | ZZArchive/; *PaycheckData*/; *PayTemplates*/; *Schedules*/; *GroupInfo*/
for example, to get rid of it looking through those other dense folders on the same level?

ALSO: Are those exclusion wildcards correct? Some folders have the company name before or after 'AA PaycheckData AMAZON', for example. Can you use double wildcards?
martin

abd1111 wrote:

Here is a log file I was able to create. The filemask is somewhat similar. Both masks have the same result (timeout error)

In the log I see that WinSCP was trying for almost 30 minutes to find any file that matches your file mask. Then it was probably aborted somehow. I do not think it's "endless run". You just didn't give it enough time to complete.

Your file mask */Analyzer Census/*.* | ZZArchive/ is possibly incorrect. But you didn't tell us what you are trying to achieve. Are you trying to download only Analyzer Census folders, ignoring all others? The way you specify your file mask, you force WinSCP to walk though ALL the folders on your server to find those subfolders. That seems to take unbearably long. You need to give WinSCP more hints as to what folders to skip. Hard to help more, unless you tell us more about the directory structure.
martin

Re: Optimizing a large script for FTP File downloads (~1100 lines)

Are you saying instead of having 3 different blocks of code that connect to the same FTP for different files to retrieve, group those 3 blocks under one $session.Open($sessionOptions), and before the $session.Dispose()?

Yes, that's what I mean.
abd1111

Here is a log file I was able to create. The filemask is somewhat similar. Both masks have the same result (timeout error)
Guest

Re: Optimizing a large script for FTP File downloads (~1100 lines)

martin wrote:

Are you opening a new connection for each transfer? You better reuse it.

I tried some searches and can't determine how to implement a 're-use' of the connection for each transfer. Are you saying instead of having 3 different blocks of code that connect to the same FTP for different files to retrieve, group those 3 blocks under one $session.Open($sessionOptions), and before the $session.Dispose()?
abd1111

Re: Optimizing a large script for FTP File downloads (~1100 lines)

martin wrote:

Please post logs for your file mask problems.

I may not be able to due to company data being displayed throughout the log, so I noticed.
martin

Re: Optimizing a large script for FTP File downloads (~1100 lines)

Please post logs for your file mask problems.
abd1111

Re: Optimizing a large script for FTP File downloads (~1100 lines)

I see, I will look into that, thank you. Yes, using a new login and end on each block.

Another question regarding timeout: With the given example of my original post, I've tried the filemask numerous times to get the same files (E.g. *Census Data Files/, Census Data Files/*/*.xlsx, Census Data Files/*, NamedCensusFileData.xlsx, etc.) and they often time out or endlessly run with no output into the local folder.

Is there something I could be doing differently? I tested those masks in WinSCP itself under the Find button, and it does take quite a while to get results. Is it just based on the structure size of the FTP folders?
abd1111

Optimizing a large script for FTP File downloads (~1100 lines)

I'm having a bit of trouble with running a .NET Assembly Powershell script. .NET Assembly and C code is very new to me, excuse my lack of knowledge. It takes ages and sometimes will timeout on various blocks of code, despite almost every script having a 
$session.Timeout = New-TimeSpan -Seconds 600
or more.

I have a large FTP structure and I need certain files from certain subfolders. I have broken it up into many blocks like this, all with different filemasks and root/local folder parameters: 
# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"
 
# Set up session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = "x"
    UserName = "x"
    Password = "x"
    FtpSecure = [WinSCP.FtpSecure]::Implicit
}
 
$session = New-Object WinSCP.Session
$remotePath = "/FTPFolder1/Subfolder1/Subf2/"
$localPath = "C:\Users\ABBBB\xxxxxx\xxxxxxx\xxxxxxx\xxxxxx\xxxxxxxx\"
 
try
{
    # Connect
    $session.Open($sessionOptions)
    $session.Timeout = New-TimeSpan -Seconds 600
 
    # Masking
    $transferOptions = New-Object WinSCP.TransferOptions 
    $transferOptions.FileMask = "Census Data Files/*/*.xlsx"  
 
    # Download files
    $transferOptions.AddRawSettings("ExcludeEmptyDirectories", "1")
    $session.GetFiles($remotePath, $localPath, $False, $transferOptions).Check()
}
finally
{
    # Disconnect, clean up
    $session.Dispose()
}

I am wondering how to make it more efficient. It takes a while for just one block to run, given the sheer scale of the subfolder and file structure, but, is there any way to make it more 'efficient'? Tips, ideas? .NET Assembly and C code are very new to me, excuse my lack of knowledge. I stashed all of them in a PowerShell ISE editor and run them all there.

Documentation

Support

Associations

Follow Us