keepuptodate limitation on number of folders? :: Support Forum

cawoodm
Joined:: 2017-03-01
Posts:: 5

keepuptodate limitation on number of folders?

2017-10-02 10:21

WinSCP v5.11.1 on Windows 2012 Server R2 Standard

We have seen that keepuptodate (FTP) does not work when (we think) the number of folders is too high.
We have a folder C:\foo which contains 150'000 subfolders
It contains a folder C:\foo\bar which contains 65'000 subfolders

When we use keepuptodate on the C:\foo folder it does not detect new files (we waited several minutes). If we watch C:\foo\bar it does (quite quickly) notice new files and changes etc.

Is there some limitation to how many folders it can watch?

The depth of files is 4 i.e C:\foo\bar\three\four\file.jpg

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,518
Location:: Prague, Czechia

Re: keepuptodate limitation on number of folders?

2017-10-03

There's no limitation as such.
But I can imagine that Windows suppresses the notifications, when there are so many.

Reply with quote

cawoodm
Joined:: 2017-03-01
Posts:: 5

Re: keepuptodate limitation on number of folders?

2017-10-16 07:27

martin wrote:

There's no limitation as such.
But I can imagine that Windows suppresses the notifications, when there are so many.

In my test scenario there would not have been "many" notifications as I only changed one file. WinSCP did not notice it however.

However we now see that WinSCO did not notice some changes on the smaller subfolder as well so it seems to be really unreliable. :(

Reply with quote

martin◆ Site Admin

Re: keepuptodate limitation on number of folders?

2017-10-16

Thanks for your report.
I have sent you an email with a debug version of WinSCP to the address you have used to register on this forum.

Reply with quote

cawoodm
Joined:: 2017-03-01
Posts:: 5

2017-10-16 11:15

Is it normal that with /loglevel=0 we get so many logs?

It looks like itis connecting again and again for each listing...

...
> 2017-10-16 11:40:39.367 LIST -a
. 2017-10-16 11:40:39.367 Connecting to 194.9.121.210:49608 ...
< 2017-10-16 11:40:39.367 150 Opening ASCII mode data connection.
. 2017-10-16 11:40:39.367 Data connection closed
. 2017-10-16 11:40:39.367 09-27-17  05:58PM                14643 8845427474462.jpg
. 2017-10-16 11:40:39.367 09-28-17  12:07AM                 9956 8849233477662.jpg
. 2017-10-16 11:40:39.367 09-28-17  12:29AM               120789 8852090847262.jpg
. 2017-10-16 11:40:39.367 09-28-17  05:55AM               127777 8857465815070.jpg
. 2017-10-16 11:40:39.367 09-28-17  07:44AM                  619 8859838742558.jpg
. 2017-10-16 11:40:39.367 09-28-17  12:36PM               126930 8864716324894.jpg
. 2017-10-16 11:40:39.367 09-28-17  12:42PM                19487 8864815087646.jpg
. 2017-10-16 11:40:39.367 09-28-17  08:14PM                24756 8872970387486.jpg
. 2017-10-16 11:40:39.367 09-28-17  11:54PM                 2561 8876915982366.jpg
. 2017-10-16 11:40:39.367 09-29-17  02:53AM                  754 8880301637662.jpg
. 2017-10-16 11:40:39.367 09-29-17  04:39AM                17980 8881821024286.jpg
< 2017-10-16 11:40:39.367 226 Transfer complete.
. 2017-10-16 11:40:39.367 Directory listing successful
. 2017-10-16 11:40:39.367 ..;D;0;1899-12-30T01:00:00.000Z;0;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.367 8845427474462.jpg;-;14643;2017-09-27T15:58:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.367 8849233477662.jpg;-;9956;2017-09-27T22:07:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 8852090847262.jpg;-;120789;2017-09-27T22:29:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 8857465815070.jpg;-;127777;2017-09-28T03:55:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 8859838742558.jpg;-;619;2017-09-28T05:44:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 8864716324894.jpg;-;126930;2017-09-28T10:36:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 8864815087646.jpg;-;19487;2017-09-28T10:42:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 8872970387486.jpg;-;24756;2017-09-28T18:14:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 8876915982366.jpg;-;2561;2017-09-28T21:54:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 8880301637662.jpg;-;754;2017-09-29T00:53:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 8881821024286.jpg;-;17980;2017-09-29T02:39:00.000Z;1;"" [0];"" [0];---------;0
. 2017-10-16 11:40:39.383 Remote file '/foo/bar/hc7/hbb/8849233477662.jpg' [2017-09-27T22:07:00.000Z] [9956] is modified comparing to local file 'C:\foo\bar\hc7\hbb\8849233477662.jpg' [2017-09-27T19:33:55.266Z] [9956]
. 2017-10-16 11:40:39.383 Remote file '/foo/bar/hc7/hbb/8857465815070.jpg' [2017-09-28T03:55:00.000Z] [127777] is modified comparing to local file 'C:\foo\bar\hc7\hbb\8857465815070.jpg' [2017-09-28T03:33:47.409Z] [127777]
. 2017-10-16 11:40:39.383 Collecting synchronization list for local directory 'C:\foo\bar\hc7\hbc' and remote directory '/foo/bar/hc7/hbc', mode = Remote, params = 0x3 (Delete, NoConfirmation), file mask = ''
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\.' [2017-09-28T22:43:10.146Z] [0] excluded from synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\..' [2017-09-28T22:43:10.146Z] [0] excluded from synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8842463739934.jpg' [2017-09-27T12:54:32.791Z] [40960] included to synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8853063598110.jpg' [2017-09-27T23:22:13.765Z] [71762] included to synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8863842336798.jpg' [2017-09-28T09:38:35.178Z] [4358] included to synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8864128073758.jpg' [2017-09-28T10:03:03.046Z] [94699] included to synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8864915947550.jpg' [2017-09-28T10:47:33.758Z] [100608] included to synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8868966793246.jpg' [2017-09-28T14:33:59.858Z] [24218] included to synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8869007687710.jpg' [2017-09-28T14:36:04.755Z] [17460] included to synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8870292389918.jpg' [2017-09-28T15:47:02.534Z] [11786] included to synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8876398510110.jpg' [2017-09-28T21:25:53.296Z] [22007] included to synchronization
. 2017-10-16 11:40:39.383 Local file 'C:\foo\bar\hc7\hbc\8877817430046.jpg' [2017-09-28T22:43:10.146Z] [14020] included to synchronization
. 2017-10-16 11:40:39.383 Retrieving directory listing...
> 2017-10-16 11:40:39.383 CWD /foo/bar/hc7/hbc
< 2017-10-16 11:40:39.383 250 CWD command successful.
> 2017-10-16 11:40:39.383 PWD
< 2017-10-16 11:40:39.383 257 "/foo/bar/hc7/hbc" is current directory.
> 2017-10-16 11:40:39.383 TYPE A
< 2017-10-16 11:40:39.383 200 Type set to A.
> 2017-10-16 11:40:39.383 PASV
< 2017-10-16 11:40:39.383 227 Entering Passive Mode (194,9,121,210,193,201).
> 2017-10-16 11:40:39.383 LIST -a
. 2017-10-16 11:40:39.383 Connecting to 194.9.121.210:49609 ...
< 2017-10-16 11:40:39.383 150 Opening ASCII mode data connection.
. 2017-10-16 11:40:39.383 Data connection closed
...

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,518
Location:: Prague, Czechia

2017-10-19

cawoodm wrote:

Is it normal that with /loglevel=0 we get so many logs?

It looks like itis connecting again and again for each listing...

That's how FTP protocol works.

Reply with quote

cawoodm
Joined:: 2017-03-01
Posts:: 5

2017-10-19 10:05

martin wrote:

cawoodm wrote:

Is it normal that with /loglevel=0 we get so many logs?

It looks like itis connecting again and again for each listing...
That's how FTP protocol works.

A new connection for each "ls" command? Check out FileZilla and you will see:

Status:   Resolving address of ecom-app01.buhlergroup.com
Status:   Connecting to 194.9.121.212:21...
Status:   Connection established, waiting for welcome message...
Status:   Insecure server, it does not support FTP over TLS.
Status:   Logged in
Status:   Retrieving directory listing of "/foo/bar/blah"...
Status:   Directory listing of "/foo/bar/tomcat" successful
Status:   Retrieving directory listing of "/foo/bar"...
Status:   Directory listing of "/foo/bar" successful
Status:   Retrieving directory listing of "/foo/bar/import"...
Status:   Directory listing of "/foo/bar/import" successful

My question however is if there is way of lessening the log so we only see relevant information like what is being written.[/code]

Reply with quote

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,518
Location:: Prague, Czechia

2017-10-19

A new connection for each "ls" command? Check out FileZilla and you will see:

While FileZilla does not display the connection in the message log in GUI, that does not mean it does not connect.
What you have posted is not a log file anyway. Enable real logging in FileZilla and you will see that it does connect too.

My question however is if there is way of lessening the log so we only see relevant information like what is being written.

Use XML log:
https://winscp.net/eng/docs/logging_xml

Reply with quote

keepuptodate limitation on number of folders?