Synchronisation scripting headache

Advertisement

rclerkin
Joined:
Posts:
6
Location:
Ireland

Synchronisation scripting headache

I've been trying to get a script for synchronising working over the last few days and its proving a huge headache. It just doesn't seem to work as expected.

I'm synchronising a directory where there is multiple subdirectories. I only want some of these subdirectories so I'm using either option include or exclude.

Here's the script using option exclude;

============
option confirm off
open sftp://budgetscp:<passwd>@<ip>
cd /www_backup/OpenJaw
option exclude "E:\budget\Copy of tomcat\"
option exclude "E:\budget\tomcat\logs\*"
option transfer binary
synchronize remote -criteria=both E:\budget\
close
exit
=============

And its output;

=============
E:\budget>winscp.com /command /script=ftp.txt
batch on
confirm off
Searching for host...
Connecting to host...
Authenticating...
Using username "budgetscp".
Authenticating with pre-entered password.
Authenticated.
Starting the session...
Reading remote directory...
Session started.
Active session: [1] budgetscp@<ip>
/www_backup/OpenJaw
exclude E:\budget\Copy of tomcat\
exclude E:\budget\tomcat\logs\*
transfer binary
Comparing...
Local 'E:\budget' => Remote '/www_backup/OpenJaw'
Synchronizing...
Local 'E:\budget' => Remote '/www_backup/OpenJaw'
E:\budget\Copy of tomcat | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\Copy of tomcat\bin | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\bin\bootstrap.jar | 23 KiB | 1246.4 KiB/s | binary | 100%
E:\...\bin\catalina.bat | 7 KiB | 134.1 KiB/s | binary | 100%
E:\...\bin\catalina.sh | 9 KiB | 87.3 KiB/s | binary | 100%
E:\...\commons-daemon.jar | 9 KiB | 75.8 KiB/s | binary | 100%
E:\...\bin\cpappend.bat | 0 KiB | 53.3 KiB/s | binary | 100%
E:\...\bin\digest.bat | 1 KiB | 34.1 KiB/s | binary | 100%
E:\...\bin\digest.sh | 0 KiB | 25.8 KiB/s | binary | 100%
E:\...\bin\jasper.bat | 2 KiB | 21.1 KiB/s | binary | 100%
E:\...\bin\jasper.sh | 0 KiB | 18.8 KiB/s | binary | 0%
Terminated by user.
===========

As you can see its including the directory I told it to exclude. I've used lots of different combinations in the exclude;
E:\budget\web\Copy of tomcat*
E:\budget\web\Copy of tomcat\
E:\budget\web\Copy of tomcat\*
E:\budget\web\Copy of tomcat\*.*
The "Copy of tomcat" directory is still included no matter what.



I've also tried using the option include method, with slightly more success, but only slightly. Here's the script;

============
option confirm off
open sftp://budgetscp:<passwd>@<ip>
cd /www_backup/OpenJaw
option include "E:\budget\web\"
option transfer binary
synchronize remote -criteria=both E:\budget\
close
exit
=============


And here's the output;

=============

E:\budget>winscp.com /command /script=ftp.txt
batch on
confirm off
Searching for host...
Connecting to host...
Authenticating...
Using username "budgetscp".
Authenticating with pre-entered password.
Authenticated.
Starting the session...
Reading remote directory...
Session started.
Active session: [1] budgetscp@<IP>
/www_backup/OpenJaw
include E:\budget\web\
transfer binary
Comparing...
Local 'E:\budget' => Remote '/www_backup/OpenJaw'
Synchronizing...
Local 'E:\budget' => Remote '/www_backup/OpenJaw'
E:\budget\web | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\book.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\web\book.jsp_new | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\brochureSearch.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\config | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\css | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\error.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\home.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\images | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\index.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\old | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\packageResults.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\packageSearch.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\pages | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\popups | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\scripts | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\SessionAdmin.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\waitForResults.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\budget\web\WEB-INF | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\web\xRezError.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
E:\...\web\xRezReset.jsp | 0 KiB | 0.0 KiB/s | binary | 0%
Session 'budgetscp@193.95.150.26' closed.
No session.

============

This files should not be 0 KB and nothing shows up on the remote server except for the "web" subdirectory which is empty.



Is there any chance somebody could help me out here because I'm not sure of the syntax for the option commands? Shoud I be using windows directory notation or unix?

Some further examples of those would be helpful in the documentation, like having multiple "option exclude <dir>" in a row, or how the include/exclude pipe works mentioned on the last line of the File Mask section.

Thanks for any help,
R

Last edited by rclerkin on 2009-07-20 11:15; edited 1 time in total

Reply with quote

Advertisement

rclerkin
Joined:
Posts:
6
Location:
Ireland

I just realise by reading here that the second option exclude was overwriting the first. How can I exclude multiple directories? Do I seperate them with semi colons?

I'm still stuck with the include end of things tho.

Reply with quote

rclerkin
Joined:
Posts:
6
Location:
Ireland

Work around for include

I've managed to get some kind of functionality from the include statement.
I've added the following line;

option include "E:\budget\web\; E:\budget\web\*; E:\budget\web\*\*;

The way this seems to be working is that it synchronises the first directory (not the files in it but the directory itself)
It then synchronises the files in the first directory using the wildcard. This means if there's sub directories they will be sync'd but the files in them wont be.
The last entry will then sync everything 1 directory down.
If there were further sub directories you would need to continue to add entries with extra \* wildcards until you've reached the lowest level of sub directory.

If this is the expected way that the include statment works I think it should be changed. When a directory is included in a synchronisation I think the transfer should work recurrsively ... the way the exclude statement works.

Using the option Exclude is not a good way to do things because the contents of a directory may change over time ... a large log archive for example for grow by several GBs over a matter of a few days ... and these changes would be included in the sync. The include will only ever transfer what you want regardless of the other directory content.

Does anybody disagree with me?

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
27,459
Location:
Prague, Czechia

Re: Work around for include

Actually your last mask should include everything under "E:\budget\web\". Let me know if it does not.

The problem with 0kB files looks like a bug, I'll check it.
_________________
Martin Prikryl

Reply with quote

rclerkin
Joined:
Posts:
6
Location:
Ireland

I just tested it using only the last mask.
Here's my script;

=======================
option batch on
option confirm off
open sftp://budgetscp:<pwd>@<IP>
cd /www_backup/OpenJaw/test
option include " E:\budget\dpsearch\*\*; E:\budget\golf\*\*; E:\budget\web\*\*;"
option transfer binary
synchronize remote E:\budget\
close
exit
=======================

All the directories on the local server exist, containing non-zero files, and the remote directory is empty.
When I execute this I get the following result;

=======================
E:\budget>winscp /script=FTP_Commands.txt
batch on
confirm off
Searching for host...
Connecting to host...
Authenticating...
Using username "budgetscp".
Authenticating with pre-entered password.
Authenticated.
Starting the session...
Reading remote directory...
Session started.
Active session: [1] budgetscp@<IP>
/www_backup/OpenJaw/test
include E:\budget\dpsearch\*\*; E:\budget\golf\*\*; E:\budget\web\*\*;
transfer binary
Comparing...
Local 'E:\budget' => Remote '/www_backup/OpenJaw/test'
Nothing to synchronize.
Session 'budgetscp@<IP>' closed.
No session.
======================

Maybe somebody could try to replicate this?

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
27,459
Location:
Prague, Czechia

You have to do:
option include "E:\budget\dpsearch\; E:\budget\dpsearch\*; E:\budget\dpsearch\*\*; E:\budget\golf\; E:\budget\golf\*; E:\budget\golf\*\*; E:\budget\web\; E:\budget\web\*; E:\budget\web\*\*"
_________________
Martin Prikryl

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
27,459
Location:
Prague, Czechia

Re: Work around for include

martin wrote:

The problem with 0kB files looks like a bug, I'll check it.
This bug has been added to tracker.
_________________
Martin Prikryl

Reply with quote

Advertisement

You can post new topics in this forum