Bad News Folders

This may be a Linux issue, but I'm using WinSCP to connect so I'll ask it here.
When copying files from the Local (Windows) directory to the remote (Linux) FTP server, sometimes WinSCP loses the connection and is unable to reconnect. It keeps trying, but to no avail. However, after aborting, usually the connection is still there, or if it isn't, it's easy to reconnect manually.

Investigating further, I found is that there are certain "bad news" folders on the remote server. Simply clicking on a "bad news" folder to browse it in the WinSCP window causes WinSCP to lose the connection. Similarly, it can't delete these "bad news" folders. Interestingly, they have the same Unix permissions listed as the folders that work.

One workaround I tried that did work is this:
1) Rename the "bad news" folder.
2) Create a copy of the "bad news" folder on the local server WITH A DIFFERENT NAME.
3) Upload the renamed folder.
4) Rename the folder on the remote server to the original name.
5) Then delete the renamed "bad news" folders

Step #2 is particularly interesting here. If I try to upload a folder that has the same name that the "bad news" folder had previously (even though I renamed the bad news folder), then the uploaded folder is also "bad news." However, if I upload a folder with a different name, and then rename it to the name that the "bad news" folder had before, then the newly uploaded folder functions normally, and I can get inside it.

Moreover, after doing step #4, it's as if something "resets" because step #5 is possible, whereas it wasn't before. For example, before step #4, it's impossible to delete the bad news folders, even after they were renamed. But once the "good" folder is renamed to the original name, then the old "bad news" folders can be deleted.

I'm thinking this might have something to do with the fact that I deleted a bunch of files on the server previously (which I had put up earlier using the same account), and somehow some "ghosts" have remained? In any case, I wish WinSCP could somehow get around this, because my workaround takes a lot of extra work.

PS... Actually, my workaround doesn't always work either, unfortunately. Sometimes renaming a folder to the original name of the bad folders doesn't allow deleting bad folders. Furthermore, copying the files from certain subfolders into a folder can cause it to become bad. Since the bad folders are not accessible in WinSCP, it's hard to know which subfolder is causing the badness, except by copying them one at a time.

One big frustration is that often trying to delete a bad folder makes WinSCP completely lose the connection, so that I have to re-log back in and go through the long process of navigating to my local directory again.

Surely there's a better way...?

Further information: While it is possible to create a "good" version of some bad directories that have bad subfolders (using the renaming trick for the bad subfolders as well), it seems that if it is the subfolders that make them "bad," then it is never possible to delete any "bad" versions of those subfolders, even after the renaming trick. Apparently, when the problem is in the subfolders, simply getting a "good" version of the directory doesn't allow the system to reset itself so that the bad folders can be deleted.

Have you considered changing on your Linux box from an FTP server to an SSH server such as openssh (there are others that run on Linux that you can choose from)??

I have nothing against FTP servers, they have there place but connecting using SSH is so much more robust.

RobHam wrote:

Have you considered changing on your Linux box from an FTP server to an SSH server such as openssh (there are others that run on Linux that you can choose from)??

I have nothing against FTP servers, they have there place but connecting using SSH is so much more robust.

Sorry...I realize now that I wasn't logged in when I posted. I don't have control over how they set up the box. Personally I think FTP isn't such a good idea. But my role in this project is to put up the files, and this issue is driving me batty! Has anyone ever heard of such behavior? And is there anything I can do at the client/WinSCP end that would help?

There is a recent forum post that shows a similar behaviour when logged in using FTP and navigating to a directory that is restricted to not browsable on the server.

https://winscp.net/forum/viewtopic.php?t=11381

If you log into the server mentioned in the post and then navigate to the folder /edgar/data (which is somehow restricted on the server), Winscp seems to handle this situation badly and appears to disconnect, followed by an attempt to reconnect which then also fails because it is trying to list the same restricted directory contents.

Can you post a log file showing opening of the bad directory?
Are you able to open the bad directory with any other FTP client? If you are, please post it's log file.

martin wrote:

Can you post a log file showing opening of the bad directory?
Are you able to open the bad directory with any other FTP client? If you are, please post it's log file.

The problem appears to be more on the server side, where a directory listing is requested but the server cannot send the list because it is restricted, and hence it sends nothing. The clients then either wait indefinitely or assume a server disconnection, so attempt a re-connect which unfortunately then also fails.
Perhaps the current behaviour in WinSCP is OK in this situation.


1. Using filezilla client, the program will hang indefinitely with the last log entries as shown below. With this program it is possible to abort but the connection is lost.

2012-07-23 19:29:27 308 3 Command: PWD
2012-07-23 19:29:27 308 3 Response: 257 "/edgar" is the current directory
2012-07-23 19:29:27 308 3 Command: PASV
2012-07-23 19:29:27 308 3 Response: 227 Entering Passive Mode (162,138,185,31,148,16).
2012-07-23 19:29:27 308 3 Command: MLSD
2012-07-23 19:29:27 308 3 Response: 150 Opening ASCII mode data connection for MLSD
2012-07-23 19:29:28 308 3 Response: 226 Transfer complete
2012-07-23 19:29:28 308 3 Status: Directory listing successful
2012-07-23 19:29:33 308 3 Status: Retrieving directory listing...
2012-07-23 19:29:33 308 3 Command: CWD data
2012-07-23 19:29:36 308 3 Response: 250 CWD command successful
2012-07-23 19:29:36 308 3 Command: PWD
2012-07-23 19:29:36 308 3 Response: 257 "/edgar/data" is the current directory
2012-07-23 19:29:36 308 3 Command: PASV
2012-07-23 19:29:36 308 3 Response: 227 Entering Passive Mode (162,138,185,31,209,41).
2012-07-23 19:29:36 308 3 Command: MLSD
2012-07-23 19:29:36 308 3 Response: 150 Opening ASCII mode data connection for MLSD


2. When using wsftp client, it detects after about 60 seconds that there is no server response so assumes the connection is lost. Wsftp then waits perhaps another minute before trying to re-connect back into the same restricted directory hence this also fails. WinSCP behaves similar to this.

PWD
257 "/edgar" is the current directory
TYPE A
200 Type set to A
PASV
227 Entering Passive Mode (162,138,185,31,145,46).
connecting data channel to 162.138.185.31:145,46(37166)
data channel connected to 162.138.185.31:145,46(37166)
LIST
150 Opening ASCII mode data connection for file list
# transferred 2413 bytes in 0.031 seconds, 617.720 kbps ( 77.215 kBps), transfer succeeded.
226 Transfer complete
CWD data
250 CWD command successful
PWD
257 "/edgar/data" is the current directory
TYPE A
200 Type set to A
PASV
227 Entering Passive Mode (162,138,185,31,227,215).
connecting data channel to 162.138.185.31:227,215(58327)
data channel connected to 162.138.185.31:227,215(58327)
LIST
150 Opening ASCII mode data connection for file list
# transferred 0 bytes in 65.501 seconds, 0.000 bps ( 0.000 Bps), transfer failed.
Error reading response from server.
It appears that the connection is dead. Attempting reconnect...


3. This is the equivalent WinSCP log section to the above :-

> 2012-07-23 19:42:05.079 PWD
< 2012-07-23 19:42:05.189 257 "/edgar" is the current directory
. 2012-07-23 19:42:05.189 Got reply 1 to the command 16
. 2012-07-23 19:42:05.189 Retrieving directory listing...
> 2012-07-23 19:42:05.189 TYPE A
< 2012-07-23 19:42:05.314 200 Type set to A
> 2012-07-23 19:42:05.314 PASV
< 2012-07-23 19:42:05.423 227 Entering Passive Mode (162,138,185,31,183,26).
> 2012-07-23 19:42:05.423 LIST
< 2012-07-23 19:42:05.657 150 Opening ASCII mode data connection for file list
< 2012-07-23 19:42:06.251 226 Transfer complete
. 2012-07-23 19:42:06.251 Directory listing successful
. 2012-07-23 19:42:06.251 Reading symlink "Feed".
. 2012-07-23 19:42:06.251 Reading symlink "Oldloads".
. 2012-07-23 19:42:06.251 Reading symlink "index.htm".
. 2012-07-23 19:42:06.251 Reading symlink "xbrl.html".
. 2012-07-23 19:42:06.251 Got reply 1 to the command 2
. 2012-07-23 19:42:27.767 Changing directory to "data".
> 2012-07-23 19:42:27.767 CWD data
< 2012-07-23 19:42:28.033 250 CWD command successful
. 2012-07-23 19:42:28.033 Got reply 1 to the command 16
. 2012-07-23 19:42:28.033 Getting current directory name.
> 2012-07-23 19:42:28.033 PWD
< 2012-07-23 19:42:28.158 257 "/edgar/data" is the current directory
. 2012-07-23 19:42:28.158 Got reply 1 to the command 16
. 2012-07-23 19:42:28.158 Retrieving directory listing...
> 2012-07-23 19:42:28.158 TYPE A
< 2012-07-23 19:42:28.267 200 Type set to A
> 2012-07-23 19:42:28.267 PASV
< 2012-07-23 19:42:28.392 227 Entering Passive Mode (162,138,185,31,187,120).
> 2012-07-23 19:42:28.392 LIST
< 2012-07-23 19:42:28.627 150 Opening ASCII mode data connection for file list
. 2012-07-23 19:42:43.830 Timeout detected.
. 2012-07-23 19:42:43.830 Could not retrieve directory listing
. 2012-07-23 19:42:43.830 Got reply 1004 to the command 2
* 2012-07-23 19:42:43.830 (ESshFatal) Lost connection.
* 2012-07-23 19:42:43.830 Timeout detected.
* 2012-07-23 19:42:43.830 Could not retrieve directory listing
* 2012-07-23 19:42:43.830 Opening ASCII mode data connection for file list
* 2012-07-23 19:42:43.830 Error listing directory '/edgar/data'.
* 2012-07-23 19:42:43.830 Error changing directory to 'data'.
. 2012-07-23 19:42:48.987 Connecting to ftp.sec.gov ...