Applying filters to directories in "Keep Remote Dir Updated" :: Support Forum

gavbaa Donor
Joined:: 2005-02-10
Posts:: 3

Applying filters to directories in "Keep Remote Dir Updated"

2005-02-10 23:03

First and foremost, thank you for this amazing product. I've already donated for my personal use, and want to bring it in for the business I've recently started at, pending a fix that may or may not be possible.

We use Subversion here, a source code revisioning tool. This tool leaves a ".svn" folder in every directory in our working source code tree (it's web development, so we have a working copy on our local machines, and we upload on save to the server for testing. We do it this way because it works both in the office and on the road, so emergency development can happen anywhere we can check out the source code from). So, when we're using "Keep Remote Directory Up to Date", every time we do something with the repository (checkout/update/delete/etc.), WinSCP goes insane, uploading lots of little files that are stored in those .svn directories. It doesn't crash, it just takes a long while to deal with the update.

To give you an example of the directory structure we're talking about:

(root)
 |->file_a.txt
 |->file_b.txt
 |->.svn
   |->entries
   |->...(more files here)...
 |->dir_c
   |->file_c.txt
   |->.svn
     |->entries
     |->...(more files here)...

Every subdirectory have its own ".svn" folder with stuff inside of it.

We don't really care about those files, and would prefer if the .svn folders and everything in them weren't even uploaded to the servers. We've already tried the filters significantly, and there does not seem to be a way to make the filters ignore .svn *directories*.

The other problem these directories create is that inside these .svn directories are many more directories that we also don't care about. They cause the entire site tree directory count to bloat severely. Now, I've already built my own slightly modified version from your source code that allows me to use many MANY more than 500 directories (and it works just fine, doesn't seem to bog the system down too much). If we didn't have to even check those .svn folders at all, the directory count would be sooo much lower, and definitely within the safety range of 500 directories. But all those .svn/* files changing all the time are causing definite lags of minutes before we can continue getting our code pushed up. Regular working and saving goes just fine and snappy.

If there is some way that we can get the filter to apply to directories as well, it's the last piece that I can find to make our development environment magical. :)

Thanks,
George

Reply with quote E-mail

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,518
Location:: Prague, Czechia

Re: Applying filters to directories in "Keep Remote Dir Updated"

2005-02-11

If you are willing to apply the patch yourself, here is quick and dirty fix to TCustomScpExplorerForm::DoSynchronize in CustomScpExplorer.cpp:

void __fastcall TCustomScpExplorerForm::DoSynchronize(
  TSynchronizeController * /*Sender*/, const AnsiString LocalDirectory,
  const AnsiString RemoteDirectory, const TSynchronizeParamType & Params)
{
  try
  {
    TCopyParamType CopyParam = GUIConfiguration->CopyParam;
    CopyParam.PreserveTime = true;

    AnsiString Path = LocalDirectory;
    bool Allow = true;
    while (Allow && !Path.IsEmpty())
    {
      Path = ExcludeTrailingBackslash(Path);
      Allow = CopyParam.AllowTransfer(ExtractFileName(Path));
      AnsiString ParentPath = ExtractFilePath(Path);
      if (Path == ParentPath)
      {
        Path = "";
      }
      else
      {
        Path = ParentPath;
      }
    }
    
    if (Allow)
    {
      Synchronize(LocalDirectory, RemoteDirectory, smRemote, CopyParam,
        Params.Params | TTerminal::spNoRecurse | TTerminal::spUseCache |
        TTerminal::spDelayProgress);
    }
  }
  catch(Exception & E)
  {
    ShowExtendedExceptionEx(Terminal, &E);
    throw;
  }
}

This way it will still watch for changes in filtered out directories (that's why it's dirty), but it won't upload them. BTW: I suppose that you are using GUI, not scripting, and this won't fix "keep up to date" in scripting.

Reply with quote

gavbaa Donor
Joined:: 2005-02-10
Posts:: 3

2005-02-11 17:19

First, I added the filter of ".svn". That didn't work, it still checked all the .svn folders. So I connected to the server, removed all the .svn folders on there. Be-yoooo-tiful! I don't know why that made a difference, but I believe that was the only change I had made up to that point.

I do notice that there's more drive activity than I would suspect out of the small file tree, but it's a small price for now for the functionality.

Thank you!
George

Reply with quote E-mail

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,518
Location:: Prague, Czechia

2005-02-12

gavbaa wrote:

First, I added the filter of ".svn". That didn't work, it still checked all the .svn folders. So I connected to the server, removed all the .svn folders on there. Be-yoooo-tiful! I don't know why that made a difference, but I believe that was the only change I had made up to that point.

Hm, strange, I do not think that it should make difference.

Reply with quote

plsnhat Guest

Watch more than 500 directories

2010-03-30 10:31

Hey there,

I am also very thankful for this great product (: At the moment I am looking for a possibility to keep more than 500 directories up to date and it seems that you, gavbaa, implemented a solution (: Would you mind to tell me how to do this?

You would do me a great favor!

Thanks,
Phil

Reply with quote

gavbaa Donor

Re: Watch more than 500 directories

2010-03-30 23:44

Unfortunately, I have since moved on from the company where I developed that, so I will be unable to produce any additional patches besides what was in this thread. However, I recall 99% of the work outside of applying the patch listed above just being finding the 500 constant in the source and increasing it, then re-compiling. Should be straightforward.

Reply with quote E-mail

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,518
Location:: Prague, Czechia

Re: Watch more than 500 directories

2010-03-31

plsnhat wrote:

I am also very thankful for this great product (: At the moment I am looking for a possibility to keep more than 500 directories up to date and it seems that you, gavbaa, implemented a solution (: Would you mind to tell me how to do this?

Where does the 500 limit restrict you? I believe WinSCP asks you if you want to continue nevertheless.

Reply with quote

Guest

Re: Watch more than 500 directories

2010-03-31 08:14

martin wrote:

plsnhat wrote:

I am also very thankful for this great product (: At the moment I am looking for a possibility to keep more than 500 directories up to date and it seems that you, gavbaa, implemented a solution (: Would you mind to tell me how to do this?
Where does the 500 limit restrict you? I believe WinSCP asks you if you want to continue nevertheless.

I store my projects locally and on a webserver which needs to be updated everytime I want to see an online version of a project. It may happen that just one file in any subdirectory of the project changes. So in order to not have to copy the whole project again I use WinSCP which determines the files which changed.

Well, it asks me whether I want to select other directories to be monitored and also offers the possibility to say "no", but doing this just closes the "keep up to date"-dialogue.

Thanks a lot for your help ! (:

Reply with quote