Time Offset Errors

I have just upgraded to the latest version of WinSCP and am trying to backup a remote website on a server to a local drive using synchronise by modification time. The remote server is in a different time zone (to the West, UTC-5), local file time zone is UTC (currently +1 for DST), but the files systems are Unix for remote, NTFS for local, so both - as I understand it - should 'correct' to UTC; however, the remote files show with a time five hours later ie +5 (eg local file modified time is shown as 01/01/2016 10:30:15, the (same) remote file shows as 01/01/2016 15:30:15. This makes it look newer (though it isn't - its the same file) so when I run Synchronise/Preview I get hundreds of files showing up as needing transfer (download) that don't need download. As there are hundreds (800+) going through manually and checking/unchecking to correct the errors isn't really an option. The server will only accept the FTP protocol so I can't manually set a time offset, and I am reluctant to turn off MLSD and use a deprecated command though I see it is an option in the documentation that might allow me to set the time offset. The session log by the way reports that MLSD is enabled.

Any advice on how to sort this out (so only files that really are new/newer show up in the preview)? BTB, WinSCP has always seemed to me to be the best program available for this type of backup - would be great to get it working 100%!

blueRay

Thanks for your report.

Please start here:
https://winscp.net/eng/docs/troubleshooting#timestamps

I did go through those pages, extensively, and am none the wiser.

I think or maybe I mean guess that the older versions (I most worked with v4.2.9) of WinSCP 'worked' by ignoring the remote time, so it became 00:00:00, while the local file always had a time say 12:30:15, so the local (identical) file always looked newer.

Using the deprecated (not really something I warm to) 'LIST' instead of MLSD does allow me to set a time offset, but the vast majority of remote files now have no time, only a date, so in effect WInSCP seems to be doing what it did before ie ignoring remote times.

It makes the whole thing feel like trying to stick the tail on the picture of a donkey with your eyes shut. Sometimes it ends up in the right place, sometimes it doesn't. What would be good is to see the remote file times, to allow a visual scan that everything makes sense after the analysis and before committing to the download (ie at the 'results' window stage).

Would it not be possible to allow manual time offset setting even when using MLSD? You could add suitable warnings that this is only for use when 'automatic' time setting/adjustment had failed.

bR

So can you post a session log file and other information mentioned in the documentation?

I've taken some screengrabs, comparing Filezilla and WinSCP reporting of file times. The file on both sides is identical:

Filezilla, no offset (FZ allows an offset to be set manually - hint hint): five hours difference.

Filezilla, -5hr offset: times match: Bingo. Files are seen as identical.

WinSCP, using (deprecated) LIST with or without offsets: no remote time reported. Left file is apparently seen as later when running synchronise, possibly for the reasons given earlier ie no time = 00:00:00?

WinSCP, using MLSD: six hours difference: as it is later, it appears newer (when it is not): no Bingo. (I guess the six rather than five hour difference may be a DST thing, possibly the older Windows DST thingy - I'm on XP SP3, or maybe the remote server or WinSCP doing something for DST).

Grrr... only three attachments per post, will have to add the last to another post... Also Grrr x 2 - the attachments are in the wrong order (last attachment is first in list... And Grrr x 3 - why not display the image inline?

bR

And here's the fourth image (we cross-posted).

Not a great fan of revealing all, usually poor SNR ratio, very happy to answer any specific questions.

bR

Log file, please.

It's over 8MB, max file attachment size is 10Mb, already nearly there...

Have given you most of the info you ask for already (OS, WinSCP versions, description of the problem, session log reports MLSD is enabled on the remote server, etc). What else would you like to know? I'm very happy to pull out any relevant bits and post them.

bR

I need the log. I need to see what information the server provides and how WinSCP interprets that. I also need to know what timezone WinSCP uses to convert the times. It's all in the log.

I've just done a long and thorough log based forensic analysis of what appears to be going on (bottom line was it's all very bizarre) and added an explanation as to why we punters don't like posting logs only to delete the post by a stupid mistake... 5-H-1-T!

FWIW, from what I remember, the unicode.inc times were (Filezilla FTP, no offset, WinSCP FTP mode, using MLSD so offset isn't used, though it can be set):
- local file (here): 05:09:08 (this is the real UTC time of the file, +/- Windows XP DST 'bug')
- Filezilla remote file time: 10:09:08
- WinSCP log remote file time: 09:09:08 and 09:09:08.000Z (latter = milliseconds/UTC marker added?)
- WinSCP GUI remote file time: 11:09:08

Even more bizarre, the remote server is to the West of here, ie a negative offset (UTC-5, ie local time here hh=05, local time there = hh=00, not hh=10 which is adding 5 (or 4 or 6...). Go figure...

Can't be bothered to do it all again. My final comment and suggestion was, as before, why not add the ability to manually set the time offset in FTP mode (using MLSD). That way the correct offset could be forced, and - it seems to me - problem solved. It might mean having to alter the offset twice a year (or it might not) for DST/non-DST changes, but that's not a great hardship if the result is synchronisation that works right, time in time out (sic)....

bR

Worth adding: why not use Filezilla anyway, since it can handle time offsets and do synchronised browsing? Because, although it can do both those things, it doesn't do the synchronisation itself, you have to set up the transfers manually = very tedious and boring. The beauty of WinSCP is that it sets up the transfer list for you (in Preview Mode) which you can then check, tweak if needed, and then, one click, and you're done.

bR

blueRay wrote:

Can't be bothered to do it all again. My final comment and suggestion was, as before, why not add the ability to manually set the time offset in FTP mode (using MLSD). That way the correct offset could be forced, and - it seems to me - problem solved. It might mean having to alter the offset twice a year (or it might not) for DST/non-DST changes, but that's not a great hardship if the result is synchronisation that works right, time in time out (sic)....

Because I believe that WinSCP behaves correctly and I haven't met a server that implements the MLSD incorrectly, yet. So there was no need for the offset option.

So if you want the offset option, post or email me a convincing log file.

It should be obvious from my previous posts that I want to use/like WinSCP but what has happened is that I have realised that what I thought was it working correctly was in fact it doing things not quite correctly behind the scenes, viz the earlier versions were silently setting the time to 00:00:00, thus making them usually appear older, and so not listing them for download. When I updated to the latest version, the quirks/bugs appeared, because WinSCP was now reporting hh:mm:ss, but for some reason these were offset from correct time, or so it seems.

Or so it seems, because getting my head round these times is complicated, what with time zones and DST, and then add in Windows DST 'bug', and possible MLSD/UNIX/server bugs/errors/whatever.

My understanding is that UTC is a 'universal' (though we probably mean global) time ie it is at the same instant (same hh:mm:ss.000...) anywhere in the world. We then have the problem that when it is noon in Greenwich, it is not noon in New York, so we add (to the east of Greenwich, because noon is earlier)/subtract (to the west of Greenwich) from the UTC to get a local time. Sometimes we add an hour or two for DST. So file created at noon in Greenwich in winter (so its UTC time is 12:00:00.000) has, to make it age correctly in different time zones, to have its UTC time adjusted to local time. Consider what would happen if you looked at that Greenwich noon file in New York at 10am/2pm New York time without any time offset...

So it should be simple enough to correct a remote time zone offset back to UTC, and for - for example - file comparing, always have the file timestamp in UTC ie what the time was/is in Greenwich. If the time zone is five hours behind (Greenwich/New York, noon in the latter happens five hours later), then we need to subtract 5 to go from UTC to New York time, add 5 to go from New York time to UTC. So the noon file will be reported in New York as 07:00:00 - making the file appear younger than it does in Greenwhich (time shows as 12:00:00), even though it is the same file. To fix this, at some point we apply an offset, in this case we either *add* to the Newy York time to get UTC (better, correct option - the file is always seen as 12:00:00 UTC) or equally we could *subtract* from the UTC time to get New York time (bad, times quite literally all over the place).

Now the work I did yesterday which sadly went down the pan established there was something wrong with timestamp reporting somewhere. Frankly it's a;; so freakin' complicated there is a limit to how long I am prepared to spend messing about with it, but the one clear observable fact was that (this all on WinXP, so there's the DST 'bug' to consider as well) Filezilla and WinSCP reported different timestamps on the remote file, despite both apparently using MLSD to get the listing and timestamp. The only conclusion is that at least one if not both are wrong.

So, for the file in question (unicode.inc, which appears in WinSCP as listed for transfer as being newer when it isn't, it's the same file):

Local timestamp in Windows Explorer (Props to get ss), FileZilla and WinSCP local pane: 05:09:08
FileZilla Remote (server) pane, no offset dialled in: 10:09:08
WinSCP GUI Remote (server) pane: 11:09:08
WinSCP Log entry for the listing: 09:09:08

The actual log entries (there are two for each file) for unicode.inc are:
. 2016-10-14 08:24:55.640 type=file;size=18180;modify=20150707090908;
. 2016-10-14 08:24:55.640 unicode.inc;-;18180;2015-07-07T09:09:08.000Z; (Z and .000 ie ms = UTC?)

So there's at least some time-warping going on. The first oddity is why are the New York (remote, to the west, so *behind* in time) times *ahead* in time? Crack that one, mate, and you may just be a Time Lord yet! The next oddity is why are FileZilla and WinSCP (both, remember, using MLSD, confirmed in the respective logs) showing different times??? And why is WinSCP showing different times in the log, and in the GUI???

On balance , it seems WinSCP has got its knickers in more of a twist than FileZilla. FileZilla shows a credible 5 hour offset (because that is, so far as we know, the real world offset), whereas WinSCP has two offsets, so both can't be right, and quite possibly both are wrong (4 and 6 hous offsets when it should be 5 hours).

Clearly something is wrong somewhere. It would be nice to have the code do it all correctly in the first place, but clearly it isn't - the times are all over the place. Sometimes it is easier just to dial in the correction manually. FileZilla allows this, but - for reasons I cannot really understand - you are reluctant to let WinSCP achieve this.

By the way, the time offset option is there in WinSCP (it can even be set, eg by changing to another mode then back to FTP), but greyed out - you have it appears deliberately turned it off. It even gets logged in the log file, but not used in plain FTP mode. This apparent 'I did it my way' approach is incidentally also apparent in the password visible/invisible fandango that appears from time to time here on the forum. You don't have to like your punters, but when they point our real problems, and propose possible solutions, it might be helpful to listen.

bR

PS I really dislike this forum implementation. Tight attachment number/size limits, logs you out too early etc. Confirmation codes when there shouldn't be. Not a friendly environment at all...

WinSCP adjusts the timestamps for Windows XP DST bug, while FileZilla does not. So they will show the timestamps differently in any case. That also explains the two offsets (in DST and out of DST).

WinSCP logs the timestamps in UTC, but in GUI, it shows a local time.

OK, that explains the WinSCP/Filezilla GUI difference, but doesn't explain the 2 hour offset between the WinSCP GUI and log times. We are currently still in DST, so is WinSCP doing some sort of DST correction plus Windows DST bug correction? Given the above data, what do we conclude is the real UTC time of the file?

And still the question remains: why not just enable setting time offsets in FTP mode? I don't understand why you are so reluctant to do this? It would enable a simple fix for these offset errors causing files to appear new when they are not new.

bR

Sorry, this is going nowhere.

If you are not willing to post a log file, I'm not willing to provide you a support on this.

I understand that you are concerned about privacy. But it would take you a fraction of the time, you spent with your lengthy post, to sanitize a log from all private information.

Please understand that there are dozens of other users that need help. If I had to spend so much time on every issue, as with this one, I would not be doing anything else.

I'm sorry you feel that way. WinSCP is deservedly popular and I and I am sure many others greatly appreciate your work and input.

Call me paranoid but I am! From what I can see, I have already provided all the key time data in posts above. The post above was lengthy, but it was me working my way through what was going on as far as I could myself. I can also write fast enough; sanitizing the logs would however take some time, unless I stripped everything out, which would pretty much leave what I have already covered above (or less!).

In the post that disappeared, I lamented the fact that some fora have a 'your logs (+OS/hardware/inside trouser leg measurement) or your life' culture and this is a double edged sword. I am sure I am not the only one reluctant to post logs online: there is just too much of a possibility something will slip through the 'sanitization'. Many will just walk away, when the problems didn't really need the logs to be fixed in the first place.

Really the crux of this thread is why not make the manual time offset (which is present but greyed out/not used in FTP mode) available in FTP mode?

WinSCP is your program and I fully respect your right to have it as you want it, with no hard feelings on my side and I hope none on yours. I was just asking a question,that's all.