Topic "Synchronize failing for large directories"

Author Message
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
Using WinSCP 4.3.7 on Windows 7, connecting over straight FTP to a Windows 2008 server. Synchronize always fails when the directories are very out of sync. For example, perhaps more than X files exist remotely but not locally. I don't know what X is exactly, but I know it's more than 15. If it's a small number of differences, the error does not occur.

The file sizes range from 200KB up to 3MB, and the size of the directory is currently around 5,500 items, although I experienced the error as early as 4,500 items.

I use the synchronize function in a backup script like this:
Code:
option batch on
option confirm off
open "user@host"

cd Backups\Diffs
lcd Diffs
synchronize local

exit


Edit: to be more specific, the error is this:
Quote:
---------------------------
Error
---------------------------
Timeout waiting for external console to complete the command.
---------------------------
OK Help
---------------------------
Advertisements
martin◆
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 26754
Location: Prague, Czechia
How do you run the script?

Have you read
https://winscp.net/eng/docs/message_timeout_external_console ?
_________________
Martin Prikryl
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
Sorry, please ignore this reply. I was referring to something else here.
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
Sorry for the noise. I had read that page, but didn't think it applied because I was calling it from a batch file. Turns out the program with the problem wasn't the batch file program, it was indeed a .NET program, and I had confused the two. Thanks!
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
I adjusted the program using the recommendations in the documentation, but now something strange is going on. It seems to be copying over all the files that it had already copied. I have thousands of differential backup sets and winscp seems to start at the beginning every time I run it. Synchronizations are now taking forever.

They're both Windows machines, and I've checked the timestamps and they're the same on both ends as far as I can tell (even when viewed via FTP). Since this happened so recently, I suspect the recent DST change March 11th, but I really don't know how to proceed from here.
martin◆
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 26754
Location: Prague, Czechia
So if you check the timestamps from WinSCP GUI, do you see the same for local and remote files?
_________________
Martin Prikryl
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
Yes, they were off by an hour. There was no setting I could find on the FTP server to match the DST change, so I just manually adjusted the offset by an hour in the WinSCP settings. Thanks.
naasking
[View user's profile]

Joined: 2012-03-19
Posts: 7
The manual offset of one hour seems to have solved resyncing all existing files, however when WinSCP detected a new file out of sync, it downloaded it and then gave it the wrong timestamp. This causes it to resync the same file over and over again.

I had to remove the 1 hour offset, download the file manually, then add the manual offset back. Then WinSCP stopped resyncing the file.

WinSCP 4.3.7 build 1679.
martin◆
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 26754
Location: Prague, Czechia
I have sent you an email with a debug version of WinSCP to address you have used to register on this forum.
_________________
Martin Prikryl
GaelenBurns
[View user's profile]

Joined: 2012-04-26
Posts: 2
Location: United States
Has this been resolved, or should I just stop pursuing Winscp? My directory that I want to synchronize has 500,000 files, not just 5,000. About 100-300 files are added per day.

Do I have a chance, or is this an inappropriate use?
GaelenBurns
[View user's profile]

Joined: 2012-04-26
Posts: 2
Location: United States
Well, I'm continuing with the project. Here's hoping I'm not wasting my time.
martin◆
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 26754
Location: Prague, Czechia
GaelenBurns wrote:
Has this been resolved, or should I just stop pursuing Winscp? My directory that I want to synchronize has 500,000 files, not just 5,000. About 100-300 files are added per day.

Do I have a chance, or is this an inappropriate use?

There's no limit set. Definitely worth trying. Let us know.
Advertisements

You can post new topics in this forum

Search

What is WinSCP?

It is award-winning SFTP client, SCP client, FTPS client and FTP client integrated into one software program for file transfer to FTP server or secure SFTP server. [More]

And it's free!

Donate

About donations

$9   $19   $49   $99

About donations

Recommend

WinSCP Privacy Policy

WinSCP License