Synchronize to remote - resumesupport filepart on destination?

I'm working on a process that needs to update files across a high latency connection that does occasionally disconnect. Some of the files are as large as 50 GB. The destination is VShell providing SFTP. Here's the commands I'm using in a script:

option echo on
option batch on
option confirm off
# Connect Using pre-configured Profile
open user@Destination
lcd F:\CBA_QA_DEV_DBs\P\
synchronize remote -filemask=|ues*scramble* -transfer=binary -resumesupport=on F:\CBA_QA_DEV_DBs\P\ /CBA_Prod_Scramble_DBs/P/


It appears if the file does not exist on the destination it will write a new file and append .filepart, and will successfully rename when done. However if the file already exists on the destination and needs to be replaced with a newer version I do not see a .filepart file created, and if the connection breaks the download starts over. I am running with the 5.8.2-beta Portable version.

Please advise if this should create a new .filepart if the file already exists, if I've got anything incorrect in the above commands, or if I should be taking another approach, perhaps using PUT. Thank you.

It should not matter if the file exists or not.

Please attach a full log file showing the problem (using the latest version of WinSCP).

Thanks for the reply. The regular job does a lot of files. I want to narrow this down to a test example with a couple files and need to hold a previous version of the files for that testing. I'll provide a log in a few days.

I've sent logs as a private attachment from doing runs of the same WinSCP synchronize job. I've also included the log entries from the VShell side, in case you want to see that. Here's a summary of what was ran from what I understand.

Starting point - empty destination.

1. Ran job to send 2 files dated 4/16/2016
- Destination shows FILEPROC_PD_DY_FULL_BACKUP.bak.filepart with current date/time while being transferred.
- After file send completed FILEPROC_PD_DY_FULL_BACKUP.bak.filepart renamed to FILEPROC_PD_DY_FULL_BACKUP.bak and date/time got updated correctly.
- Destination shows SAFEDB_PD_DY_FULL_BACKUP.bak.filepart with current date/time while being transferred.
- After file send completed SAFEDB_PD_DY_FULL_BACKUP.bak.filepart renamed to SAFEDB_PD_DY_FULL_BACKUP.bak and date/time got updated correctly.

2. Updated source files with updates dated 4/23/2016 and ran the same job.
- Destination shows FILEPROC_PD_DY_FULL_BACKUP.bak with current date/time while being transferred. No filepart file.
- After file send completed FILEPROC_PD_DY_FULL_BACKUP.bak date/time got updated correctly.
- Destination shows SAFEDB_PD_DY_FULL_BACKUP.bak with current date/time while being transferred. No filepart file.
- After file send completed SAFEDB_PD_DY_FULL_BACKUP.bak date/time got updated correctly.

The 2nd job took longer, and the files are similar in size. There's plenty of variables for a high latency connection but there might be something different on how WinSCP sent the files on the 2nd run.
FILEPROC_PD_DY_FULL_BACKUP.bak
- 1st run 08:20:05 to 08:30:58 = about 11 minutes
- 2nd run 08:40:17 to 09:04:00 = about 24 minutes
SAFEDB_PD_DY_FULL_BACKUP.bak
- 1st run 08:30:58 to 08:36:13 = about 5 minutes
- 2nd run 09:04:00 to 09:17:48 = about 14 minutes

I appreciate your help and time with this. Thank you.

Thanks for the logs.

I'm sending you an email with a development version of WinSCP to the address you have used to register on this forum.

I ran this with the WinSCP you provided via the e-mail link and have attached a zip of the logs. This looks like the reason it's not sending the updated file with the resumable transfer:

. 2016-04-26 12:53:42.651 File exists: FILEPROC_PD_DY_FULL_BACKUP.bak;-;162123264;2016-04-16T04:05:20.000Z;3;"Administrators@localhost" [0];"Domain Users@mydomain.here.net" [0];---------;0

. 2016-04-26 12:53:42.651 Existing file is owned by another user [Administrators@localhost], not doing resumable transfer.

In this instance the source and destination sides are both Windows servers but are in different domains, so the owners are different. Please advise if there's something I can change on my side, or if something needs to be changed in the program. Thanks!

WinSCP does not do resumable transfers (= transfers to temporary file name) when overwriting files owned by another user.

Because to do so, the file has to be deleted and new one created. What changes ownership and that is usually undesirable.

In this case nothing is changing on the owner. It's the same user sending the 1st file as the 2nd one. For checking the owner on the source it's going through the local Windows system, while for the destination it's asking SFTP via the VSHell SFTP server who the file owner is.

From the log file:

1. Existing file is owned by [Administrators@localhost]
2. For the destination file that exists it shows owned by "Administrators@localhost" [0];"Domain Users@company.domain.net"

If my guess it correct despite administrators@localhost being listed ase owners of both files the destination also reports the domain group "Domain Users" and WinSCP logic determines the owners do not match. Why can't WinSCP evaluate owners and determine Administrators@localhost is on the source and destination, and determine the owner is the same and send the file with the resumeable option?

I can look at other options, such as a step to delete the files before WinSCP sends them, or see if the owner information returned from the destination can be modified, but it would be nice if WinSCP could handle this oddity.

The ownership of the local file does not matter.

What matters, is that you are logged in as the "Svc-INNH830294", while the existing file is owned by the "Administrators@localhost".

Thanks for the guidance. If I change the owner of the file to match what WinSCP connects with prior to the start of the WinSCP synchronize job it does create the filepart files as expected.

1 other minor thing - The log shows the following error occurs:
* 2016-05-14 10:03:14.505 (EScpSkipFile) **Upload of file 'DATA_PD_DP_FULL_BACKUP.bak' was successful, but error occurred while setting the permissions and/or timestamp.**

I am using the -nopermissions parameter. Here's the command as reflected in the WinSCP log:

> 2016-05-14 10:00:07.537 Script: synchronize remote -nopermissions -transfer=binary -filemask=|ues*scramble*;uescm* -resumesupport=on F:\CBA_QA_DEV_DBs\P\ /CBA_Prod_Scramble_DBs/P/

Is there a command parameter to use with synchronize to turn off "preserving time stamp" ?

seanm12 wrote:

Is there a command parameter to use with synchronize to turn off "preserving time stamp" ?

Only if you give up synchronization by timestamp.
See https://winscp.net/eng/docs/scriptcommand_synchronize

-nopreservetime -criteria=size