Topic ""Endurance" setting does not prevent "atomic write" problem"

Author Message
javasniffer
[View user's profile]

Joined: 2009-02-10
Posts: 6
Location: St Albans, UK
Problem: "Endurance" option does not prevent read error during transfer.

Description: Some of our PHP files are large, and take up to 5 seconds to upload to our web server. If someone visits the page during the upload, they retrieve only part of the file, as SFTP is writing it.

I understand that this used to be a known problem, and I see that the "Endurance" feature has now been added so that uploaded files are copied to a temporary file first, and only on completion of the upload are they copied/renamed, thereby reducing the time window of a partial read to almost zero.

However, if I set Endurance ON (all files) and request the file using HTTP during the upload, I still get a partial file read.

Settings: WinSCP 4.1.8, SFTP-3, Commander View (double-click and drag/drop both fail))

Any help would be very welcome.

WinSCP log:
----
Copying 1 files/directories to remote directory "/var/www/vhosts/mydomain.com/httpdocs/TEST/"
PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 01; Resume: Y (102400); CalcS: Yes; Mask: *.*
TM: M; ClAr: No; CPS: 0; ExclM(No):
AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml
File: "C:\www\mydomain\src\tableinfo.php"
Copying "C:\www\mydomain\src\tableinfo.php" to remote directory started.
Ascii transfer mode selected.
Opening remote file.
Type: SSH_FXP_OPEN, Size: 82, Number: 34563
Type: SSH_FXP_STATUS, Size: 24, Number: 34308
Discarding reserved response
Type: SSH_FXP_HANDLE, Size: 13, Number: 34563
Type: SSH_FXP_WRITE, Size: 4071, Number: 35078
10 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.
Type: SSH_FXP_CLOSE, Size: 13, Number: 36868
Type: SSH_FXP_SETSTAT, Size: 78, Number: 34825
Type: SSH_FXP_STATUS, Size: 24, Number: 36102
Type: SSH_FXP_STATUS, Size: 24, Number: 36358
Type: SSH_FXP_STATUS, Size: 24, Number: 36614
Type: SSH_FXP_STATUS, Size: 24, Number: 36868
Status/error code: 0
Type: SSH_FXP_STATUS, Size: 24, Number: 34825
Status/error code: 0
Listing directory "/var/www/vhosts/mydomain.com/httpdocs/TEST".
Type: SSH_FXP_OPENDIR, Size: 52, Number: 37131
Type: SSH_FXP_HANDLE, Size: 13, Number: 37131
Type: SSH_FXP_READDIR, Size: 13, Number: 37388
Type: SSH_FXP_NAME, Size: 12497, Number: 37388
Type: SSH_FXP_READDIR, Size: 13, Number: 37644
Type: SSH_FXP_NAME, Size: 12529, Number: 37644
Type: SSH_FXP_READDIR, Size: 13, Number: 37900
Type: SSH_FXP_NAME, Size: 12542, Number: 37900
Type: SSH_FXP_READDIR, Size: 13, Number: 38156
Type: SSH_FXP_NAME, Size: 12649, Number: 38156
Type: SSH_FXP_READDIR, Size: 13, Number: 38412
Type: SSH_FXP_NAME, Size: 12675, Number: 38412
Type: SSH_FXP_READDIR, Size: 13, Number: 38668
Type: SSH_FXP_NAME, Size: 12479, Number: 38668
Type: SSH_FXP_READDIR, Size: 13, Number: 38924
Type: SSH_FXP_NAME, Size: 12612, Number: 38924
Type: SSH_FXP_READDIR, Size: 13, Number: 39180
Type: SSH_FXP_NAME, Size: 13105, Number: 39180
Type: SSH_FXP_READDIR, Size: 13, Number: 39436
Type: SSH_FXP_NAME, Size: 12754, Number: 39436
Type: SSH_FXP_READDIR, Size: 13, Number: 39692
Type: SSH_FXP_NAME, Size: 12657, Number: 39692
Type: SSH_FXP_READDIR, Size: 13, Number: 39948
Type: SSH_FXP_NAME, Size: 12903, Number: 39948
Type: SSH_FXP_READDIR, Size: 13, Number: 40204
Type: SSH_FXP_NAME, Size: 13375, Number: 40204
Type: SSH_FXP_READDIR, Size: 13, Number: 40460
Type: SSH_FXP_NAME, Size: 6121, Number: 40460
Type: SSH_FXP_READDIR, Size: 13, Number: 40716
Type: SSH_FXP_STATUS, Size: 28, Number: 40716
Status/error code: 1
Type: SSH_FXP_CLOSE, Size: 13, Number: 40964
----

e.g.

-- time --> 1----------2----------3----------4----------5----------6----------7

1 = HTTP read file hello.php - OK
2 = start transfer of hello.php using SFTP-3 in WinSCP 4.1.8
3 = HTTP read file hello.php - ERROR
4 = HTTP read file hello.php - ERROR
5 = HTTP read file hello.php - ERROR
6 = finish transfer of hello.php using SFTP-3 in WinSCP 4.1.8
7 = HTTP read file hello.php - OK
Advertisements
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24553
Location: Prague, Czechia
This feature does not work for ascii transfers. Force binary transfer if possible. I have updated documentation accordingly.
_________________
Martin Prikryl
javasniffer
[View user's profile]

Joined: 2009-02-10
Posts: 6
Location: St Albans, UK
Thanks prikryl.

I have tried this, and it reduces the time window, but not to zero.

Is there *any* way to prevent a destination file being read while it is being moved/renamed remotely by SCP?
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24553
Location: Prague, Czechia
javasniffer wrote:
Is there *any* way to prevent a destination file being read while it is being moved/renamed remotely by SCP?

What difference that would make? You page still won't work.
_________________
Martin Prikryl
javasniffer
[View user's profile]

Joined: 2009-02-10
Posts: 6
Location: St Albans, UK
When WinSCP is being used to upload files to a live web site, there must be a mechanism - somewhere - to prevent partially written files from being read by the Web server program.

I was hoping that such a solution might be available using the SCP protocol and/or the WinSCP program - maybe I am wrong.

Perhaps the only way to achieve what I need (i.e. no read errors by the web server) is to implement a locking mechanism myself using a shell script? Currently, we upload files to the (live) web server directory using WinSCP, and this is causing problems.

As one of the main uses of WinSCP is to upload files to a web server, I think this is an important point that should be addressed here.

Does anyone else have any solutions?
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24553
Location: Prague, Czechia
Sorry I'm still not sure if I understand. So you would accept that the page does not load while uploading. While it is inacceptable that partial content is read?
_________________
Martin Prikryl
javasniffer
[View user's profile]

Joined: 2009-02-10
Posts: 6
Location: St Albans, UK
The requirement is that files in the web server's directory must always be read completely: before, during and after upload.

From your response so far, it looks like this problem will have to be solved outside of the scp transfer, e.g. by copying to a temporary directory which is not used by the web server:

a) scp files to a remote 'temp' directory, not used by the web server
b) lock the 'web' directory, the one used by the web server
c) copy the files locally from 'temp' to 'web' directory, using cp or rsync
d) unlock the 'web' directory

The web server would have to be configured to re-try for a period of time when it finds the 'web' directory locked. Does anyone know how to do this with Apache 2.0, or will Apache re-try automatically?
javasniffer
[View user's profile]

Joined: 2009-02-10
Posts: 6
Location: St Albans, UK
I have found an answer.

In order to guarantee that a file exists (in its entirety) to any process both before and after the file contents are changed, cp() function will not work, as it is not atomic. Instead, the mv() function must be used. mv() uses rename() which is guaranteed to be atomic.

So, to prevent partial files being served by Apache, WinSCP alone is not enough. The files must be moved/renamed locally using mv().

e.g.

1. using WinSCP, copy fileA.html from the local machine to /path/to/TEST directory on the remote web server machine (where we are not concerned about fileA.html being read during the copying operation)

2. using a local cp() command on the remote machine, create a temporary copy of the file which we will move to the 'live' directory - cp /path/to/TEST/fileA.html /path/to/temp/fileA.html

3. using a local mv() command on the remote machine, atomically overwrite fileA.html in the 'live' web directory - mv /path/to/temp/fileA.html /path/to/live/fileA.html

Your script (or shell) will need write access to both directories /path/to/temp and /path/to/live.

----

Note: there is some disagreement about whether mv() in a given implementation uses the library function rename(), a function that is guaranteed to be atomic. Some implementations may use link() and unlink() which are *not* atomic, and will therefore not prevent partial reading of the web server files. You are advised to check your version of *nix.
andipeh

Guest


Quote:
This feature does not work for ascii transfers. Force binary transfer if possible


i have tested this now, i switched everyting to binary and enabled this resume/transfer to temporay file for all files.
but winscp still writes directly into the files, and crashing the scripts Sad

please tell me what i can do
javasniffer
[View user's profile]

Joined: 2009-02-10
Posts: 6
Location: St Albans, UK
WinSCP cannot be used to transfer files to a live web server, if you cannot tolerate partial file reads.

You must use mv on the server to move a copy of the file into the live web directory.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24553
Location: Prague, Czechia
andipeh wrote:
i have tested this now, i switched everyting to binary and enabled this resume/transfer to temporay file for all files.
but winscp still writes directly into the files, and crashing the scripts Sad

Please post a log file again, with the new settings.
_________________
Martin Prikryl
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24553
Location: Prague, Czechia
javasniffer wrote:
WinSCP cannot be used to transfer files to a live web server, if you cannot tolerate partial file reads.

You must use mv on the server to move a copy of the file into the live web directory.

WinSCP does mv itself, if you "enable transfer to temporary file name".
_________________
Martin Prikryl
andipeh

Guest


thanks for the quick reply.

so here is my log. i have noticed already that the transfer is still done in ascii - so thats probably the reason why it does not create this filepart and do the mv afterwards.

i tried now everything to make the transfer binary but i couldnt. i selected it within the transfer selection on the top menu. then i changed every present and the default transfer type to binary. and finaly i removed .php from the text present. but it still takes this .php from somewhere else and starts a ascii transfer.

i dont know how much debug information you need. just tell me if you need more.

Code:

> 2009-03-20 17:59:06.343 Type: SSH_FXP_REALPATH, Size: 10, Number: 109328
< 2009-03-20 17:59:06.593 Type: SSH_FXP_NAME, Size: 23, Number: 109328
> 2009-03-20 17:59:07.343 [Background 1] Type: SSH_FXP_REALPATH, Size: 10, Number: 109584
< 2009-03-20 17:59:07.609 [Background 1] Type: SSH_FXP_NAME, Size: 23, Number: 109584
> 2009-03-20 17:59:14.359 Type: SSH_FXP_REALPATH, Size: 10, Number: 109840
< 2009-03-20 17:59:14.562 Type: SSH_FXP_NAME, Size: 23, Number: 109840
> 2009-03-20 17:59:15.359 [Background 1] Type: SSH_FXP_REALPATH, Size: 10, Number: 110096
< 2009-03-20 17:59:16.281 [Background 1] Type: SSH_FXP_NAME, Size: 23, Number: 110096
. 2009-03-20 17:59:21.875 [Background 1] Copying 1 files/directories to remote directory "/var/www/skins/unstable/phplib"
. 2009-03-20 17:59:21.875 [Background 1]   PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 01; Resume: N (102400); CalcS: Yes; Mask:
. 2009-03-20 17:59:21.875 [Background 1]   TM: A; ClAr: No; CPS: 0; ExclM(No):
. 2009-03-20 17:59:21.875 [Background 1]   AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml
. 2009-03-20 17:59:21.875 [Background 1] File: "C:\DOKUME~1\ROLAND~1.PFA\LOKALE~1\Temp\scp45062\var\www\skins\unstable\phplib\global_config.php"
. 2009-03-20 17:59:21.875 [Background 1] Copying "C:\DOKUME~1\ROLAND~1.PFA\LOKALE~1\Temp\scp45062\var\www\skins\unstable\phplib\global_config.php" to remote directory started.
. 2009-03-20 17:59:21.875 [Background 1] Ascii transfer mode selected.
. 2009-03-20 17:59:21.875 [Background 1] Opening remote file.
> 2009-03-20 17:59:21.875 [Background 1] Type: SSH_FXP_OPEN, Size: 73, Number: 110339
< 2009-03-20 17:59:22.125 [Background 1] Type: SSH_FXP_HANDLE, Size: 13, Number: 110339
> 2009-03-20 17:59:22.125 [Background 1] Type: SSH_FXP_WRITE, Size: 4013, Number: 110854
> 2009-03-20 17:59:22.859 Type: SSH_FXP_REALPATH, Size: 10, Number: 113680
< 2009-03-20 17:59:23.109 Type: SSH_FXP_NAME, Size: 23, Number: 113680
. 2009-03-20 17:59:30.828 [Background 1] 29 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.
> 2009-03-20 17:59:30.828 [Background 1] Type: SSH_FXP_CLOSE, Size: 13, Number: 115204
> 2009-03-20 17:59:30.828 [Background 1] Type: SSH_FXP_SETSTAT, Size: 69, Number: 110601
< 2009-03-20 17:59:31.312 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 114694
> 2009-03-20 17:59:31.359 Type: SSH_FXP_REALPATH, Size: 10, Number: 115472
< 2009-03-20 17:59:31.625 Type: SSH_FXP_NAME, Size: 23, Number: 115472
< 2009-03-20 17:59:32.453 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 114950
< 2009-03-20 17:59:32.453 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 115204
< 2009-03-20 17:59:32.453 [Background 1] Status/error code: 0
< 2009-03-20 17:59:32.453 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 110601
< 2009-03-20 17:59:32.453 [Background 1] Status/error code: 0
. 2009-03-20 17:59:32.453 [Background 1] Listing directory "/var/www/skins/unstable/phplib".
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24553
Location: Prague, Czechia
What does Transfer settings box on transfer settings dialog say before you upload?
_________________
Martin Prikryl
andipeh

Guest


i now tried the following:
i did not get the transfer option box for background transfers - so i tried to reenable it via preferences > Environment. i checked every confirmation box (exept the last one - continue on error)
also i removed the checkbox from Transfer > Background: Transfer on background by default and no confirmations for background transfers.

did not help - it still does the background transfers on its one - and does it still in ascii instead of binary.


here are some screenshots:



then i tried to drag&drop the files from outside into winscp - and then all works fine. it seems only the background transfers are able to resuming.



Code:

. 2009-03-23 12:19:39.328 Copying 1 files/directories to remote directory "/var/www/skins/unstable/phplib/"
. 2009-03-23 12:19:39.328   PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 01; Resume: Y (102400); CalcS: Yes; Mask: *.*
. 2009-03-23 12:19:39.328   TM: B; ClAr: No; CPS: 0; ExclM(No):
. 2009-03-23 12:19:39.328   AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml
. 2009-03-23 12:19:39.328 File: "C:\Dokumente und Einstellungen\roland.pfannhauser\Lokale Einstellungen\Temp\global_config.php"
. 2009-03-23 12:19:39.328 Copying "C:\Dokumente und Einstellungen\roland.pfannhauser\Lokale Einstellungen\Temp\global_config.php" to remote directory started.
. 2009-03-23 12:19:39.328 Binary transfer mode selected.
. 2009-03-23 12:19:39.328 Checking existence of file.
> 2009-03-23 12:19:39.328 Type: SSH_FXP_LSTAT, Size: 57, Number: 2231815
< 2009-03-23 12:19:39.421 Type: SSH_FXP_ATTRS, Size: 37, Number: 2231815
. 2009-03-23 12:19:39.421 Checking existence of partially transfered file.
> 2009-03-23 12:19:39.421 Type: SSH_FXP_LSTAT, Size: 66, Number: 2232071
< 2009-03-23 12:19:39.515 Type: SSH_FXP_STATUS, Size: 29, Number: 2232071
< 2009-03-23 12:19:39.515 Status/error code: 2
. 2009-03-23 12:19:39.515 Asking user:
. 2009-03-23 12:19:39.515 Remote file 'global_config.php' already exists. Overwrite?
. 2009-03-23 12:19:39.515 
. 2009-03-23 12:19:39.515 New:         227233 bytes, 23.03.2009 12:19:32
. 2009-03-23 12:19:39.515 Existing:    227227 bytes, 23.03.2009 12:14:56 ()
. 2009-03-23 12:19:44.171 Opening remote file.
> 2009-03-23 12:19:44.171 Type: SSH_FXP_OPEN, Size: 82, Number: 2232323
< 2009-03-23 12:19:44.265 Type: SSH_FXP_HANDLE, Size: 13, Number: 2232323
> 2009-03-23 12:19:44.265 Type: SSH_FXP_WRITE, Size: 4121, Number: 2232838
> 2009-03-23 12:19:46.515 [Background 1] Type: SSH_FXP_REALPATH, Size: 10, Number: 2235152
< 2009-03-23 12:19:46.625 [Background 1] Type: SSH_FXP_NAME, Size: 23, Number: 2235152
. 2009-03-23 12:19:47.250 19 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.
> 2009-03-23 12:19:47.250 Type: SSH_FXP_CLOSE, Size: 13, Number: 2235908
< 2009-03-23 12:19:47.546 Type: SSH_FXP_STATUS, Size: 24, Number: 2235398
< 2009-03-23 12:19:47.765 Type: SSH_FXP_STATUS, Size: 24, Number: 2235654
< 2009-03-23 12:19:47.765 Type: SSH_FXP_STATUS, Size: 24, Number: 2235908
< 2009-03-23 12:19:47.765 Status/error code: 0
> 2009-03-23 12:19:47.765 Type: SSH_FXP_REMOVE, Size: 57, Number: 2236173
< 2009-03-23 12:19:47.843 Type: SSH_FXP_STATUS, Size: 24, Number: 2236173
< 2009-03-23 12:19:47.843 Status/error code: 0
> 2009-03-23 12:19:47.843 Type: SSH_FXP_RENAME, Size: 118, Number: 2236434
< 2009-03-23 12:19:47.937 Type: SSH_FXP_STATUS, Size: 24, Number: 2236434
< 2009-03-23 12:19:47.937 Status/error code: 0
> 2009-03-23 12:19:47.937 Type: SSH_FXP_SETSTAT, Size: 73, Number: 2232585
< 2009-03-23 12:19:48.046 Type: SSH_FXP_STATUS, Size: 24, Number: 2232585
< 2009-03-23 12:19:48.046 Status/error code: 0
. 2009-03-23 12:19:48.078 Listing directory "/var/www/skins/unstable/phplib".


so ether "disabling background transfers" does not work this way or maybe its not meant to work this way? some with resuming/partfile-generating for background transfers.

thanks again for looking into it!
kind regards
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24553
Location: Prague, Czechia
Can you send me an email, so I can send you back a debug version of WinSCP to track the problem? Please include link back to this topic in your email. Also note in this topic that you have sent the email. Thanks.

You will find my address (if you log in) in my forum profile.
_________________
Martin Prikryl
andipeh
[View user's profile]

Joined: 2009-03-25
Posts: 5
email has just been sent to you.

just in addition: i am currently using WinSCP v4.2.1 Build 428

kind regards.
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24553
Location: Prague, Czechia
This issue has been added to tracker.
_________________
Martin Prikryl
Guest




prikryl wrote:
This issue has been added to tracker.


thank you again for looking into it and solving my problem! you really helped me out.
Guest




. 2009-03-23 12:19:39.328 Copying "C:\Dokumente und Einstellungen\roland.pfannhauser\Lokale Einstellungen\Temp\global_config.php" to remote directory started.


OK! What the hell! This certain name is my brothers name and it cant be any other. so whats going on with "remote directory..." in connection with his name?
please let me know!

thats what happens if you google family names...
kyra13
[View user's profile]

Joined: 2011-08-03
Posts: 1
Location: USA
I tried it as well and something went wrong unfortunately Sad
even don't know where to look for decedion...
martin
[View user's profile]
Site Admin
Joined: 2002-12-10
Posts: 24553
Location: Prague, Czechia
kyra13 wrote:
I tried it as well and something went wrong unfortunately Sad
even don't know where to look for decedion...

What went wrong? How?
_________________
Martin Prikryl
Advertisements

You can post new topics in this forum






Search Site

What is WinSCP?

It is award-winning SFTP client, SCP client, FTPS client and FTP client integrated into one software program for file transfer to FTP server or secure SFTP server. [More]

And it's free!

Donate

About donations

$9   $19   $49   $99

About donations

Recommend

WinSCP Privacy Policy

WinSCP License