Topic "Synchronize failing for large directories"

Author Message
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
Using WinSCP 4.3.7 on Windows 7, connecting over straight FTP to a Windows 2008 server. Synchronize always fails when the directories are very out of sync. For example, perhaps more than X files exist remotely but not locally. I don't know what X is exactly, but I know it's more than 15. If it's a small number of differences, the error does not occur.

The file sizes range from 200KB up to 3MB, and the size of the directory is currently around 5,500 items, although I experienced the error as early as 4,500 items.

I use the synchronize function in a backup script like this:
Code:
option batch on
option confirm off
open "user@host"

cd Backups\Diffs
lcd Diffs
synchronize local

exit


Edit: to be more specific, the error is this:
Quote:
---------------------------
Error
---------------------------
Timeout waiting for external console to complete the command.
---------------------------
OK Help
---------------------------
Advertisements
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24530
Location: Prague, Czechia
How do you run the script?

Have you read
http://winscp.net/eng/docs/message_timeout_external_console ?
_________________
Martin Prikryl
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
Sorry, please ignore this reply. I was referring to something else here.
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
Sorry for the noise. I had read that page, but didn't think it applied because I was calling it from a batch file. Turns out the program with the problem wasn't the batch file program, it was indeed a .NET program, and I had confused the two. Thanks!
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
I adjusted the program using the recommendations in the documentation, but now something strange is going on. It seems to be copying over all the files that it had already copied. I have thousands of differential backup sets and winscp seems to start at the beginning every time I run it. Synchronizations are now taking forever.

They're both Windows machines, and I've checked the timestamps and they're the same on both ends as far as I can tell (even when viewed via FTP). Since this happened so recently, I suspect the recent DST change March 11th, but I really don't know how to proceed from here.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24530
Location: Prague, Czechia
So if you check the timestamps from WinSCP GUI, do you see the same for local and remote files?
_________________
Martin Prikryl
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
Yes, they were off by an hour. There was no setting I could find on the FTP server to match the DST change, so I just manually adjusted the offset by an hour in the WinSCP settings. Thanks.
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
The manual offset of one hour seems to have solved resyncing all existing files, however when WinSCP detected a new file out of sync, it downloaded it and then gave it the wrong timestamp. This causes it to resync the same file over and over again.

I had to remove the 1 hour offset, download the file manually, then add the manual offset back. Then WinSCP stopped resyncing the file.

WinSCP 4.3.7 build 1679.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24530
Location: Prague, Czechia
I have sent you an email with a debug version of WinSCP to address you have used to register on this forum.
_________________
Martin Prikryl
GaelenBurns
[View user's profile]

Joined: 2012-04-26
Posts: 2
Location: United States
Has this been resolved, or should I just stop pursuing Winscp? My directory that I want to synchronize has 500,000 files, not just 5,000. About 100-300 files are added per day.

Do I have a chance, or is this an inappropriate use?
GaelenBurns
[View user's profile]

Joined: 2012-04-26
Posts: 2
Location: United States
Well, I'm continuing with the project. Here's hoping I'm not wasting my time.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24530
Location: Prague, Czechia
GaelenBurns wrote:
Has this been resolved, or should I just stop pursuing Winscp? My directory that I want to synchronize has 500,000 files, not just 5,000. About 100-300 files are added per day.

Do I have a chance, or is this an inappropriate use?

There's no limit set. Definitely worth trying. Let us know.
Advertisements

You can post new topics in this forum






Search Site

What is WinSCP?

It is award-winning SFTP client, SCP client, FTPS client and FTP client integrated into one software program for file transfer to FTP server or secure SFTP server. [More]

And it's free!

Donate

About donations

$9   $19   $49   $99

About donations

Recommend

WinSCP Privacy Policy

WinSCP License