Folder Masks not working, or I'm dumb?

Advertisement

mobrien118
Joined:
Posts:
8
Location:
Saint Louis, MO

Folder Masks not working, or I'm dumb?

OK, I've read and searched and tried for hours now.

Trying to synchronize (MIRROR) remote-->local. Trying to only transfer a specific subset of files.
Remote source is /Shared/Active Projects/ local target is Z:\*.* (a mounted drive).
I only want to bring over the following:
/Shared/Active Projects/[anything]/DC/[everything]
I have tried "Include Directory" filters (being in /Shared/Active Projects/ as the root when I start) each with and without a / on the end:
*/DC
./*/DC
/Shared/Active Projects/*/DC
Every time, it quickly says "No differences found"
If I turn off the filter, there is no problem syncing, but that is far more than I want to bring over.

Reply with quote

Advertisement

mobrien118
Joined:
Posts:
8
Location:
Saint Louis, MO

Tried making a script that does this and it is logging that each folder (/Shared/Active Projects/[anything] is "excluded from synchronization"

It is actually, before that, saying that all of the existing Z:\[anything] folders (should be target since I specified local) are "excluded from synchronization", before it even does the remote. Why would it even check the local? Maybe this is part of the problem, but it doesn't seem like it is the whole problem.

Reply with quote

mobrien118
Joined:
Posts:
8
Location:
Saint Louis, MO

I think I figured it out...

I have to include *all* of:
Z:\
.\*.*
.\*.*\DC
I have a preview running now. It is an absolutely massive file structure so I'll check the log out tomorrow after it finishes running. The log is growing and showing activity now, though. whew...

Reply with quote

mobrien118
Joined:
Posts:
8
Location:
Saint Louis, MO

:-/

Not fixed yet. Still copying folders like /Shared/Active Projects/[Anything]/[Anything]/ and files like /Shared/Active Projects/[Anything]/[Anything]" but I am trying to limit it to the contents of /Shared/Active Projects/[Anything]/DC/

Why is this so hard? I have a simple regular expression that works for matching in C#, but this, in an attempt to be "simple" seems to be excessively complex...

How is it that I can't find other posts of people having this problem, either? Starting to think the "dumb" assessment of myself is the root cause here...

Reply with quote

mobrien118
Joined:
Posts:
8
Location:
Saint Louis, MO

I realize I haven't posted any script content yet, so here is what I'm testing right now, that is still not working...
open...(this all works)
cd "/Shared/TEST"
lcd Y:\
synchronize local -mirror -delete -preservetime=all -criteria=size -filemask="Y:/; .\*.*\; .\*.*\DC\ | *.rvt; Y:\Forms\; ./_Empower/" Y:\ "/Shared/TEST"
Without the .\*.*\ it does nothing. However, it is still including things like /Shared/TEST/1234 Project/Do Not Copy This.txt and all of the contents of /Shared/TEST/1234 Project/IgnoreFolder/, as well as /Shared/TEST/ThereIsNoDCInThisFolder/

It's either all or nothing. Maddening.

Reply with quote

Advertisement

mobrien118
Joined:
Posts:
8
Location:
Saint Louis, MO

New strategy. Use Robocopy. Actually, I couldn't get it to sync right, it is updating every...single...file even if it is identical or the datetime is newer. I even created a PowerShell wrapper that carves out the individual folders and then only robocopies them, which does an accurate copy, but just re-uploads every file, still because of the issue above.

So...new *new* strategy – PowerShell script creates a list of folders to synchronize and then runs the script file. Sounds brilliant, right? Well... the WinSCP script file it generates looks great – just as I would expect, but it doesn't seem to run and nothing ever shows up in the log...

The script file looks just like the old file I was using, except it contains about 200 synchronize commands, line by line. Is there some limitation on the number that can be listed? Is there somewhere else I should be looking for output other than the log file specified on the command line?

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
40,306
Location:
Prague, Czechia

Re: Folder Masks not working, or I'm dumb?

Such recursive selection criteria are indeed difficult to achieve in WinSCP.
If you want a help with your script generated by PowerShell, please post session log file.

Reply with quote

mobrien118
Joined:
Posts:
8
Location:
Saint Louis, MO

The issue was that PowerShell writes the file in ASCII instead of UTF-8, by default. I changed that to write UTF8 BOM and it is now working *however* new issue, with multiple synchronize commands, one after another, it was only executing the first sync command. I searched for documentation but couldn't find whether or not multiple synchronize commands should be supported, followed by a single exit, so I'm not sure if I was just mis-scripting or if it is not supported. Can you confirm?
Assuming it is not supported and attempting to forge ahead, since I was using the .exe for the scripts, I couldn't run them all sequentially (each one starts before the prior completes) so I was faced with 2 choices:
  1. Rewrite using the .com and create individual script files before running (possibly overwriting before running to not clog things up)
  2. Rewrite using the PowerShell wrapper
I chose option #2 and it is working, but there is one major drawback: logging
The wrapper logs an incredible amount of data. It seems like much more than the .exe. I have been running this initial sync for about a day now and my log file is over 50 GB. I may run out of hard disk space before the initial sync completes. Haven't decided how I will handle, but I already re-coded the script to allow me to pick a spot to resume instead of re-running all sub-folders, in case I do need to change something.
Would love your thoughts on this. Quite a journey I've been on here.
All said, it would be *awesome* if the synchronize logic could handle a Regular Expression (separately for both the local and remote would be even better). This entire exercise could have been avoided if it did. Obviously, great software, but if I had a feature request in mind, that would be it.

Reply with quote

mobrien118
Joined:
Posts:
8
Location:
Saint Louis, MO

Figured out that I needed to use SessionLogLocation instead of DebugLogLocation – now it's more manageable.
Would love to see regex functionality in the script interface, though. I think that would probably pay off, timewise, compared to fielding all of the requests I see on these forums for help with file masks. I know it would have saved me, as a user, a lot of time. Again, just my $.02
Glad to have this all working now!

Reply with quote

Advertisement

Advertisement

You can post new topics in this forum