Queue stalled/halted with "cannot create file" errors :: Support Forum

egt
Joined:: 2012-03-20
Posts:: 2
Location:: New Zealand

Queue stalled/halted with "cannot create file" errors

2012-03-20 09:36

Hi,
I am trying to download ~27GB spread over ~12,000 files overnight during my ISP's free off-peak time, 2-8am, while I am a sleep. However it appears that some of the files have a file-name path >~250 characters so that when winscp adds .filepart, to allow resuming on binary transferred files, the resulting in a file-name path is >260 characters normally supported by win32/NTFS. Naturally when that happens file creation fails and produces an error msgbox saying that it can't create the file. Sorry, didn't write down the error message this morning. [Update] see bottom for log sample and screen-shot.

When this error happens it stalls/blocks/halts that queue slot, causing two things to happen. 1) Total transfer speed drops as you now have n-1 active transfer queue slots and 2) When n-n occurs (all slots blocked) zero transfer speed. Which is the condition I see when I get up in the morning. Wasting hours of free download time and bandwidth capacity.

I have changed (ticked) 'continue on error (advanced users)'; to no effect and with no temp file. I also moved the download directory to the drive/volume root e.g. z:\ to reduce the number of affected files.

My current work-around is to download the problem files with an other program once WinSCP can't. So I know that the original file-name paths are short enough and that the files also pass integrity hash checks.

Tonight I am trying with resume disabled. (under Preferences->Endurance) and enable logging. Will update tomorrow with results.

Possible 'bugs'?:

.filepart making the resulting filename path too long. e.g. >260 characters
These errors stalling the queue.

Possible solutions:

fallback to using \\?\ prefix to allow ~32,767 length paths.
https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation
disable resume for that file and continue. (auto or option in error message box; current and/or future files eg 'auto disable on error' for this queue/session ?)
shorten .filepart extension to maybe .fp or .!fp in the hope it is now short enough.
'suspend'/pause problem file(s) to allow the rest of the queue to continue. So they remain visible at the top of the queue and the user knows they had problems.
'move' them to the bottom of the queue to allow the other files to continue.

If any one knows of any work around(s), please let me know.

Regards
Peter

Server: 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------

Local: Window 7 Pro, WinSCP 4.3.7, Interface: GUI-Commander

Session protocol = SSH-2
SSH implementation = OpenSSH_5.1p1 Debian-5
Encryption algorithm = aes
Compression = No
File transfer protocol = SFTP-3
------------------------------------------------------------
Server host key fingerprint
ssh-rsa 2048 ea:4a:9e:34:08:70:93:0f:14:98:25:4f:9d:63:e5:6b
------------------------------------------------------------
Can change permissions = Yes
Can change owner/group = Yes
Can execute arbitrary command = No
Can create symlink/hardlink = Yes/No
Can lookup user groups = No
Can duplicate remote files = No
Can check available space = No
Can calculate file checksum = No
Native text (ASCII) mode transfers = No
------------------------------------------------------------
Additional information
The server supports these SFTP extensions:
posix-rename@openssh.com="1"
statvfs@openssh.com="2"
fstatvfs@openssh.com="2"

Log sample: ~60,000 lines into the log of ~233,000 lines.

< 2012-03-21 03:33:17.928 [Background 1] Status code: 1
. 2012-03-21 03:33:17.928 [Background 1] 53 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.
> 2012-03-21 03:33:17.929 [Background 1] Type: SSH_FXP_CLOSE, Size: 13, Number: 20153860
< 2012-03-21 03:33:17.929 [Background 1] Type: SSH_FXP_STATUS, Size: 28, Number: 20148997
< 2012-03-21 03:33:17.929 [Background 1] Type: SSH_FXP_STATUS, Size: 28, Number: 20151557
< 2012-03-21 03:33:18.317 [Background 1] Type: SSH_FXP_STATUS, Size: 28, Number: 20153605
. 2012-03-21 03:33:18.318 [Background 1] Getting size of directory "Lynne Rossetto Kasper"
. 2012-03-21 03:33:18.319 [Background 1] Listing directory "/home/peter/EPUB Library (2012)/Lynne Rossetto Kasper".
> 2012-03-21 03:33:18.319 [Background 1] Type: SSH_FXP_OPENDIR, Size: 62, Number: 20156939
< 2012-03-21 03:33:18.319 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 20153860
. 2012-03-21 03:33:18.319 [Background 1] Discarding reserved response
< 2012-03-21 03:33:18.711 [Background 1] Type: SSH_FXP_HANDLE, Size: 13, Number: 20156939
> 2012-03-21 03:33:18.712 [Background 1] Type: SSH_FXP_READDIR, Size: 13, Number: 20160268
< 2012-03-21 03:33:19.111 [Background 1] Type: SSH_FXP_NAME, Size: 520, Number: 20160268
> 2012-03-21 03:33:19.111 [Background 1] Type: SSH_FXP_READDIR, Size: 13, Number: 20163084
< 2012-03-21 03:33:19.493 [Background 1] Type: SSH_FXP_STATUS, Size: 28, Number: 20163084
< 2012-03-21 03:33:19.493 [Background 1] Status code: 1
> 2012-03-21 03:33:19.493 [Background 1] Type: SSH_FXP_CLOSE, Size: 13, Number: 20165380
. 2012-03-21 03:33:19.493 [Background 1] Getting size of directory "/home/peter/EPUB Library (2012)/Lynne Rossetto Kasper/The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win (13666)"
. 2012-03-21 03:33:19.493 [Background 1] Listing directory "/home/peter/EPUB Library (2012)/Lynne Rossetto Kasper/The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win (13666)".
> 2012-03-21 03:33:19.493 [Background 1] Type: SSH_FXP_OPENDIR, Size: 171, Number: 20165643
< 2012-03-21 03:33:19.875 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 20165380
. 2012-03-21 03:33:19.875 [Background 1] Discarding reserved response
< 2012-03-21 03:33:20.260 [Background 1] Type: SSH_FXP_HANDLE, Size: 13, Number: 20165643
> 2012-03-21 03:33:20.260 [Background 1] Type: SSH_FXP_READDIR, Size: 13, Number: 20170508
< 2012-03-21 03:33:20.652 [Background 1] Type: SSH_FXP_NAME, Size: 561, Number: 20170508
> 2012-03-21 03:33:20.652 [Background 1] Type: SSH_FXP_READDIR, Size: 13, Number: 20173068
< 2012-03-21 03:33:21.038 [Background 1] Type: SSH_FXP_STATUS, Size: 28, Number: 20173068
< 2012-03-21 03:33:21.038 [Background 1] Status code: 1
> 2012-03-21 03:33:21.038 [Background 1] Type: SSH_FXP_CLOSE, Size: 13, Number: 20175876
. 2012-03-21 03:33:21.039 [Background 1] File: "/home/peter/EPUB Library (2012)/Lynne Rossetto Kasper"
. 2012-03-21 03:33:21.039 [Background 1] Listing directory "/home/peter/EPUB Library (2012)/Lynne Rossetto Kasper".
> 2012-03-21 03:33:21.039 [Background 1] Type: SSH_FXP_OPENDIR, Size: 62, Number: 20176139
< 2012-03-21 03:33:21.432 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 20175876
. 2012-03-21 03:33:21.432 [Background 1] Discarding reserved response
< 2012-03-21 03:33:21.827 [Background 1] Type: SSH_FXP_HANDLE, Size: 13, Number: 20176139
> 2012-03-21 03:33:21.827 [Background 1] Type: SSH_FXP_READDIR, Size: 13, Number: 20181004
< 2012-03-21 03:33:22.230 [Background 1] Type: SSH_FXP_NAME, Size: 520, Number: 20181004
> 2012-03-21 03:33:22.230 [Background 1] Type: SSH_FXP_READDIR, Size: 13, Number: 20183052
< 2012-03-21 03:33:22.614 [Background 1] Type: SSH_FXP_STATUS, Size: 28, Number: 20183052
< 2012-03-21 03:33:22.614 [Background 1] Status code: 1
> 2012-03-21 03:33:22.614 [Background 1] Type: SSH_FXP_CLOSE, Size: 13, Number: 20186116
. 2012-03-21 03:33:22.614 [Background 1] File: "/home/peter/EPUB Library (2012)/Lynne Rossetto Kasper/The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win (13666)"
. 2012-03-21 03:33:22.614 [Background 1] Listing directory "/home/peter/EPUB Library (2012)/Lynne Rossetto Kasper/The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win (13666)".
> 2012-03-21 03:33:22.614 [Background 1] Type: SSH_FXP_OPENDIR, Size: 171, Number: 20186379
< 2012-03-21 03:33:22.999 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 20186116
. 2012-03-21 03:33:22.999 [Background 1] Discarding reserved response
< 2012-03-21 03:33:23.389 [Background 1] Type: SSH_FXP_HANDLE, Size: 13, Number: 20186379
> 2012-03-21 03:33:23.389 [Background 1] Type: SSH_FXP_READDIR, Size: 13, Number: 20191244
< 2012-03-21 03:33:23.782 [Background 1] Type: SSH_FXP_NAME, Size: 561, Number: 20191244
> 2012-03-21 03:33:23.782 [Background 1] Type: SSH_FXP_READDIR, Size: 13, Number: 20195084
< 2012-03-21 03:33:24.167 [Background 1] Type: SSH_FXP_STATUS, Size: 28, Number: 20195084
< 2012-03-21 03:33:24.167 [Background 1] Status code: 1
> 2012-03-21 03:33:24.167 [Background 1] Type: SSH_FXP_CLOSE, Size: 13, Number: 20198404
. 2012-03-21 03:33:24.167 [Background 1] File: "/home/peter/EPUB Library (2012)/Lynne Rossetto Kasper/The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win (13666)/The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win - Lynne Rossetto Kasper.epub"
. 2012-03-21 03:33:24.167 [Background 1] Copying "/home/peter/EPUB Library (2012)/Lynne Rossetto Kasper/The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win (13666)/The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win - Lynne Rossetto Kasper.epub" to local directory started.
. 2012-03-21 03:33:24.167 [Background 1] Binary transfer mode selected.
* 2012-03-21 03:33:24.167 [Background 1] (EOSError) System Error.  Code: 3.
 
* 2012-03-21 03:33:24.167 [Background 1] The system cannot find the path specified
. 2012-03-21 03:33:24.167 [Background 1] Asking user:
. 2012-03-21 03:33:24.168 [Background 1] Can't create file 'V:\Lynne Rossetto Kasper\The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win (13666)\The Splendid Table's How to Eat Supper_ Recipes, Stories, and Opinions From Public Radio's Award-Win - Lynne Rossetto Kasper.epub'. ("System Error.  Code: 3.
 
. 2012-03-21 03:33:24.168 [Background 1] The system cannot find the path specified")

Can't create file system error - code 3.png (19.12 KB)

Description: Consolidated screen-shot showing error msgbox & queue stall effect.

martin◆ Site Admin
Joined:: 2002-12-10
Posts:: 41,441
Location:: Prague, Czechia

Re: Queue stalled/halted with"cannot create file"errors[updated]

2012-03-22

Thanks for your post.
It looks like we will need to implement a support for long file paths.
Added to the tracker:
Issue 821 – Support for long local paths

egt
Joined:: 2012-03-20
Posts:: 2
Location:: New Zealand

Re: Queue stalled/halted with"cannot create file"errors[updated]

2012-03-22 21:47

Hi Martin,
Thank you and your work-mates for working on this.
If there is any other info I can provide please ask, if you want it I can email you the full log . It is ~21MB text file (~1MB RARed).
Regards
Peter

PS. I am a computer programming student and the tutor covered, on Monday, how maintenance was often underappreciated and that it typically required wider and better skills than regular development as they do everything from; is it a real bug, analysis, design, coding, testing, deployment and maintenance; while others do just one or two.

fpabernard Guest

2012-07-16 13:13

Hello

I have the same problem. Folder synchronization failed, it seems to be because of local filenames too long.

Regards.
Fred

Queue stalled/halted with "cannot create file" errors