Forum » Support and Bug Reports »

Post a reply

Before posting, please read how to report bug or request support effectively.

Bug reports without an attached log file are usually useless.

Options
Add an Attachment

If you do not want to add an Attachment to your Post, please leave the Fields blank.

(maximum 10 MB; please compress large files; only common media, archive, text and programming file formats are allowed)

Options

Topic review

martin

test12567 wrote:

I use a 5.17.7 client on Windows and can confirm that there is NOT atomic copying php files. Is it supposed to be so or can be overcome somehow via changing config or whatsoever?

Sorry, but this is quite vague. What protocol are you using? What are your transfer settings? A log file?
test12567

I use a 5.17.7 client on Windows and can confirm that there is NOT atomic copying php files. Is it supposed to be so or can be overcome somehow via changing config or whatsoever?
martin

kyra13 wrote:

I tried it as well and something went wrong unfortunately :(
even don't know where to look for decedion...

What went wrong? How?
kyra13

I tried it as well and something went wrong unfortunately :(
even don't know where to look for decedion...
Guest

. 2009-03-23 12:19:39.328 Copying "C:\Dokumente und Einstellungen\roland.pfannhauser\Lokale Einstellungen\Temp\global_config.php" to remote directory started.


OK! What the hell! This certain name is my brothers name and it cant be any other. so whats going on with "remote directory..." in connection with his name?
please let me know!

thats what happens if you google family names...
Guest

martin wrote:

This issue has been added to tracker.


thank you again for looking into it and solving my problem! you really helped me out.
martin

This issue has been added to tracker.
andipeh

email has just been sent to you.

just in addition: i am currently using WinSCP v4.2.1 Build 428

kind regards.
martin

Can you send me an email, so I can send you back a debug version of WinSCP to track the problem? Please include link back to this topic in your email. Also note in this topic that you have sent the email. Thanks.

You will find my address (if you log in) in my forum profile.
andipeh

i now tried the following:
i did not get the transfer option box for background transfers - so i tried to reenable it via preferences > Environment. i checked every confirmation box (exept the last one - continue on error)
also i removed the checkbox from Transfer > Background: Transfer on background by default and no confirmations for background transfers.

did not help - it still does the background transfers on its one - and does it still in ascii instead of binary.


here are some screenshots:

<invalid hyperlink removed by admin> (<invalid hyperlink removed by admin>)

then i tried to drag&drop the files from outside into winscp - and then all works fine. it seems only the background transfers are able to resuming.

<invalid hyperlink removed by admin> (<invalid hyperlink removed by admin>) 

. 2009-03-23 12:19:39.328 Copying 1 files/directories to remote directory "/var/www/skins/unstable/phplib/"

. 2009-03-23 12:19:39.328   PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 01; Resume: Y (102400); CalcS: Yes; Mask: *.*

. 2009-03-23 12:19:39.328   TM: B; ClAr: No; CPS: 0; ExclM(No): 

. 2009-03-23 12:19:39.328   AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml

. 2009-03-23 12:19:39.328 File: "C:\Dokumente und Einstellungen\roland.pfannhauser\Lokale Einstellungen\Temp\global_config.php"

. 2009-03-23 12:19:39.328 Copying "C:\Dokumente und Einstellungen\roland.pfannhauser\Lokale Einstellungen\Temp\global_config.php" to remote directory started.

. 2009-03-23 12:19:39.328 Binary transfer mode selected.

. 2009-03-23 12:19:39.328 Checking existence of file.

> 2009-03-23 12:19:39.328 Type: SSH_FXP_LSTAT, Size: 57, Number: 2231815

< 2009-03-23 12:19:39.421 Type: SSH_FXP_ATTRS, Size: 37, Number: 2231815

. 2009-03-23 12:19:39.421 Checking existence of partially transfered file.

> 2009-03-23 12:19:39.421 Type: SSH_FXP_LSTAT, Size: 66, Number: 2232071

< 2009-03-23 12:19:39.515 Type: SSH_FXP_STATUS, Size: 29, Number: 2232071

< 2009-03-23 12:19:39.515 Status/error code: 2

. 2009-03-23 12:19:39.515 Asking user:

. 2009-03-23 12:19:39.515 Remote file 'global_config.php' already exists. Overwrite?

. 2009-03-23 12:19:39.515  

. 2009-03-23 12:19:39.515 New:         227233 bytes, 23.03.2009 12:19:32

. 2009-03-23 12:19:39.515 Existing:    227227 bytes, 23.03.2009 12:14:56 ()

. 2009-03-23 12:19:44.171 Opening remote file.

> 2009-03-23 12:19:44.171 Type: SSH_FXP_OPEN, Size: 82, Number: 2232323

< 2009-03-23 12:19:44.265 Type: SSH_FXP_HANDLE, Size: 13, Number: 2232323

> 2009-03-23 12:19:44.265 Type: SSH_FXP_WRITE, Size: 4121, Number: 2232838

> 2009-03-23 12:19:46.515 [Background 1] Type: SSH_FXP_REALPATH, Size: 10, Number: 2235152

< 2009-03-23 12:19:46.625 [Background 1] Type: SSH_FXP_NAME, Size: 23, Number: 2235152

. 2009-03-23 12:19:47.250 19 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.

> 2009-03-23 12:19:47.250 Type: SSH_FXP_CLOSE, Size: 13, Number: 2235908

< 2009-03-23 12:19:47.546 Type: SSH_FXP_STATUS, Size: 24, Number: 2235398

< 2009-03-23 12:19:47.765 Type: SSH_FXP_STATUS, Size: 24, Number: 2235654

< 2009-03-23 12:19:47.765 Type: SSH_FXP_STATUS, Size: 24, Number: 2235908

< 2009-03-23 12:19:47.765 Status/error code: 0

> 2009-03-23 12:19:47.765 Type: SSH_FXP_REMOVE, Size: 57, Number: 2236173

< 2009-03-23 12:19:47.843 Type: SSH_FXP_STATUS, Size: 24, Number: 2236173

< 2009-03-23 12:19:47.843 Status/error code: 0

> 2009-03-23 12:19:47.843 Type: SSH_FXP_RENAME, Size: 118, Number: 2236434

< 2009-03-23 12:19:47.937 Type: SSH_FXP_STATUS, Size: 24, Number: 2236434

< 2009-03-23 12:19:47.937 Status/error code: 0

> 2009-03-23 12:19:47.937 Type: SSH_FXP_SETSTAT, Size: 73, Number: 2232585

< 2009-03-23 12:19:48.046 Type: SSH_FXP_STATUS, Size: 24, Number: 2232585

< 2009-03-23 12:19:48.046 Status/error code: 0

. 2009-03-23 12:19:48.078 Listing directory "/var/www/skins/unstable/phplib".


so ether "disabling background transfers" does not work this way or maybe its not meant to work this way? some with resuming/partfile-generating for background transfers.

thanks again for looking into it!
kind regards
andipeh

thanks for the quick reply.

so here is my log. i have noticed already that the transfer is still done in ascii - so thats probably the reason why it does not create this filepart and do the mv afterwards.

i tried now everything to make the transfer binary but i couldnt. i selected it within the transfer selection on the top menu. then i changed every present and the default transfer type to binary. and finaly i removed .php from the text present. but it still takes this .php from somewhere else and starts a ascii transfer.

i dont know how much debug information you need. just tell me if you need more. 

> 2009-03-20 17:59:06.343 Type: SSH_FXP_REALPATH, Size: 10, Number: 109328

< 2009-03-20 17:59:06.593 Type: SSH_FXP_NAME, Size: 23, Number: 109328

> 2009-03-20 17:59:07.343 [Background 1] Type: SSH_FXP_REALPATH, Size: 10, Number: 109584

< 2009-03-20 17:59:07.609 [Background 1] Type: SSH_FXP_NAME, Size: 23, Number: 109584

> 2009-03-20 17:59:14.359 Type: SSH_FXP_REALPATH, Size: 10, Number: 109840

< 2009-03-20 17:59:14.562 Type: SSH_FXP_NAME, Size: 23, Number: 109840

> 2009-03-20 17:59:15.359 [Background 1] Type: SSH_FXP_REALPATH, Size: 10, Number: 110096

< 2009-03-20 17:59:16.281 [Background 1] Type: SSH_FXP_NAME, Size: 23, Number: 110096

. 2009-03-20 17:59:21.875 [Background 1] Copying 1 files/directories to remote directory "/var/www/skins/unstable/phplib"

. 2009-03-20 17:59:21.875 [Background 1]   PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 01; Resume: N (102400); CalcS: Yes; Mask: 

. 2009-03-20 17:59:21.875 [Background 1]   TM: A; ClAr: No; CPS: 0; ExclM(No): 

. 2009-03-20 17:59:21.875 [Background 1]   AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml

. 2009-03-20 17:59:21.875 [Background 1] File: "C:\DOKUME~1\ROLAND~1.PFA\LOKALE~1\Temp\scp45062\var\www\skins\unstable\phplib\global_config.php"

. 2009-03-20 17:59:21.875 [Background 1] Copying "C:\DOKUME~1\ROLAND~1.PFA\LOKALE~1\Temp\scp45062\var\www\skins\unstable\phplib\global_config.php" to remote directory started.

. 2009-03-20 17:59:21.875 [Background 1] Ascii transfer mode selected.

. 2009-03-20 17:59:21.875 [Background 1] Opening remote file.

> 2009-03-20 17:59:21.875 [Background 1] Type: SSH_FXP_OPEN, Size: 73, Number: 110339

< 2009-03-20 17:59:22.125 [Background 1] Type: SSH_FXP_HANDLE, Size: 13, Number: 110339

> 2009-03-20 17:59:22.125 [Background 1] Type: SSH_FXP_WRITE, Size: 4013, Number: 110854

> 2009-03-20 17:59:22.859 Type: SSH_FXP_REALPATH, Size: 10, Number: 113680

< 2009-03-20 17:59:23.109 Type: SSH_FXP_NAME, Size: 23, Number: 113680

. 2009-03-20 17:59:30.828 [Background 1] 29 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.

> 2009-03-20 17:59:30.828 [Background 1] Type: SSH_FXP_CLOSE, Size: 13, Number: 115204

> 2009-03-20 17:59:30.828 [Background 1] Type: SSH_FXP_SETSTAT, Size: 69, Number: 110601

< 2009-03-20 17:59:31.312 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 114694

> 2009-03-20 17:59:31.359 Type: SSH_FXP_REALPATH, Size: 10, Number: 115472

< 2009-03-20 17:59:31.625 Type: SSH_FXP_NAME, Size: 23, Number: 115472

< 2009-03-20 17:59:32.453 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 114950

< 2009-03-20 17:59:32.453 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 115204

< 2009-03-20 17:59:32.453 [Background 1] Status/error code: 0

< 2009-03-20 17:59:32.453 [Background 1] Type: SSH_FXP_STATUS, Size: 24, Number: 110601

< 2009-03-20 17:59:32.453 [Background 1] Status/error code: 0

. 2009-03-20 17:59:32.453 [Background 1] Listing directory "/var/www/skins/unstable/phplib".
martin

javasniffer wrote:

WinSCP cannot be used to transfer files to a live web server, if you cannot tolerate partial file reads.

You must use mv on the server to move a copy of the file into the live web directory.

WinSCP does mv itself, if you "enable transfer to temporary file name".
martin

andipeh wrote:

i have tested this now, i switched everyting to binary and enabled this resume/transfer to temporay file for all files.
but winscp still writes directly into the files, and crashing the scripts :(

Please post a log file again, with the new settings.
javasniffer

WinSCP cannot be used to transfer files to a live web server, if you cannot tolerate partial file reads.

You must use mv on the server to move a copy of the file into the live web directory.
andipeh

This feature does not work for ascii transfers. Force binary transfer if possible


i have tested this now, i switched everyting to binary and enabled this resume/transfer to temporay file for all files.
but winscp still writes directly into the files, and crashing the scripts :(

please tell me what i can do
javasniffer

I have found an answer.

In order to guarantee that a file exists (in its entirety) to any process both before and after the file contents are changed, cp() function will not work, as it is not atomic. Instead, the mv() function must be used. mv() uses rename() which is guaranteed to be atomic.

So, to prevent partial files being served by Apache, WinSCP alone is not enough. The files must be moved/renamed locally using mv().

e.g.

1. using WinSCP, copy fileA.html from the local machine to /path/to/TEST directory on the remote web server machine (where we are not concerned about fileA.html being read during the copying operation)

2. using a local cp() command on the remote machine, create a temporary copy of the file which we will move to the 'live' directory - cp /path/to/TEST/fileA.html /path/to/temp/fileA.html

3. using a local mv() command on the remote machine, atomically overwrite fileA.html in the 'live' web directory - mv /path/to/temp/fileA.html /path/to/live/fileA.html

Your script (or shell) will need write access to both directories /path/to/temp and /path/to/live.

----

Note: there is some disagreement about whether mv() in a given implementation uses the library function rename(), a function that is guaranteed to be atomic. Some implementations may use link() and unlink() which are *not* atomic, and will therefore not prevent partial reading of the web server files. You are advised to check your version of *nix.
javasniffer

The requirement is that files in the web server's directory must always be read completely: before, during and after upload.

From your response so far, it looks like this problem will have to be solved outside of the scp transfer, e.g. by copying to a temporary directory which is not used by the web server:

a) scp files to a remote 'temp' directory, not used by the web server
b) lock the 'web' directory, the one used by the web server
c) copy the files locally from 'temp' to 'web' directory, using cp or rsync
d) unlock the 'web' directory

The web server would have to be configured to re-try for a period of time when it finds the 'web' directory locked. Does anyone know how to do this with Apache 2.0, or will Apache re-try automatically?
martin

Sorry I'm still not sure if I understand. So you would accept that the page does not load while uploading. While it is inacceptable that partial content is read?
javasniffer

When WinSCP is being used to upload files to a live web site, there must be a mechanism - somewhere - to prevent partially written files from being read by the Web server program.

I was hoping that such a solution might be available using the SCP protocol and/or the WinSCP program - maybe I am wrong.

Perhaps the only way to achieve what I need (i.e. no read errors by the web server) is to implement a locking mechanism myself using a shell script? Currently, we upload files to the (live) web server directory using WinSCP, and this is causing problems.

As one of the main uses of WinSCP is to upload files to a web server, I think this is an important point that should be addressed here.

Does anyone else have any solutions?
martin

javasniffer wrote:

Is there *any* way to prevent a destination file being read while it is being moved/renamed remotely by SCP?

What difference that would make? You page still won't work.
javasniffer

Thanks prikryl.

I have tried this, and it reduces the time window, but not to zero.

Is there *any* way to prevent a destination file being read while it is being moved/renamed remotely by SCP?
martin

Re: "Endurance" setting does not prevent "atomic write" problem

This feature does not work for ascii transfers. Force binary transfer if possible. I have updated documentation accordingly.
javasniffer

"Endurance" setting does not prevent "atomic write" problem

Problem: "Endurance" option does not prevent read error during transfer.

Description: Some of our PHP files are large, and take up to 5 seconds to upload to our web server. If someone visits the page during the upload, they retrieve only part of the file, as SFTP is writing it.

I understand that this used to be a known problem, and I see that the "Endurance" feature has now been added so that uploaded files are copied to a temporary file first, and only on completion of the upload are they copied/renamed, thereby reducing the time window of a partial read to almost zero.

However, if I set Endurance ON (all files) and request the file using HTTP during the upload, I still get a partial file read.

Settings: WinSCP 4.1.8, SFTP-3, Commander View (double-click and drag/drop both fail))

Any help would be very welcome.

WinSCP log: 
Copying 1 files/directories to remote directory "/var/www/vhosts/mydomain.com/httpdocs/TEST/"

  PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 01; Resume: Y (102400); CalcS: Yes; Mask: *.*

  TM: M; ClAr: No; CPS: 0; ExclM(No): 

  AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml

File: "C:\www\mydomain\src\tableinfo.php"

Copying "C:\www\mydomain\src\tableinfo.php" to remote directory started.

Ascii transfer mode selected.

Opening remote file.

Type: SSH_FXP_OPEN, Size: 82, Number: 34563

Type: SSH_FXP_STATUS, Size: 24, Number: 34308

Discarding reserved response

Type: SSH_FXP_HANDLE, Size: 13, Number: 34563

Type: SSH_FXP_WRITE, Size: 4071, Number: 35078

10 skipped SSH_FXP_WRITE, SSH_FXP_READ, SSH_FXP_DATA and SSH_FXP_STATUS packets.

Type: SSH_FXP_CLOSE, Size: 13, Number: 36868

Type: SSH_FXP_SETSTAT, Size: 78, Number: 34825

Type: SSH_FXP_STATUS, Size: 24, Number: 36102

Type: SSH_FXP_STATUS, Size: 24, Number: 36358

Type: SSH_FXP_STATUS, Size: 24, Number: 36614

Type: SSH_FXP_STATUS, Size: 24, Number: 36868

Status/error code: 0

Type: SSH_FXP_STATUS, Size: 24, Number: 34825

Status/error code: 0

Listing directory "/var/www/vhosts/mydomain.com/httpdocs/TEST".

Type: SSH_FXP_OPENDIR, Size: 52, Number: 37131

Type: SSH_FXP_HANDLE, Size: 13, Number: 37131

Type: SSH_FXP_READDIR, Size: 13, Number: 37388

Type: SSH_FXP_NAME, Size: 12497, Number: 37388

Type: SSH_FXP_READDIR, Size: 13, Number: 37644

Type: SSH_FXP_NAME, Size: 12529, Number: 37644

Type: SSH_FXP_READDIR, Size: 13, Number: 37900

Type: SSH_FXP_NAME, Size: 12542, Number: 37900

Type: SSH_FXP_READDIR, Size: 13, Number: 38156

Type: SSH_FXP_NAME, Size: 12649, Number: 38156

Type: SSH_FXP_READDIR, Size: 13, Number: 38412

Type: SSH_FXP_NAME, Size: 12675, Number: 38412

Type: SSH_FXP_READDIR, Size: 13, Number: 38668

Type: SSH_FXP_NAME, Size: 12479, Number: 38668

Type: SSH_FXP_READDIR, Size: 13, Number: 38924

Type: SSH_FXP_NAME, Size: 12612, Number: 38924

Type: SSH_FXP_READDIR, Size: 13, Number: 39180

Type: SSH_FXP_NAME, Size: 13105, Number: 39180

Type: SSH_FXP_READDIR, Size: 13, Number: 39436

Type: SSH_FXP_NAME, Size: 12754, Number: 39436

Type: SSH_FXP_READDIR, Size: 13, Number: 39692

Type: SSH_FXP_NAME, Size: 12657, Number: 39692

Type: SSH_FXP_READDIR, Size: 13, Number: 39948

Type: SSH_FXP_NAME, Size: 12903, Number: 39948

Type: SSH_FXP_READDIR, Size: 13, Number: 40204

Type: SSH_FXP_NAME, Size: 13375, Number: 40204

Type: SSH_FXP_READDIR, Size: 13, Number: 40460

Type: SSH_FXP_NAME, Size: 6121, Number: 40460

Type: SSH_FXP_READDIR, Size: 13, Number: 40716

Type: SSH_FXP_STATUS, Size: 28, Number: 40716

Status/error code: 1

Type: SSH_FXP_CLOSE, Size: 13, Number: 40964


e.g.

-- time --> 1----------2----------3----------4----------5----------6----------7

1 = HTTP read file hello.php - OK
2 = start transfer of hello.php using SFTP-3 in WinSCP 4.1.8
3 = HTTP read file hello.php - ERROR
4 = HTTP read file hello.php - ERROR
5 = HTTP read file hello.php - ERROR
6 = finish transfer of hello.php using SFTP-3 in WinSCP 4.1.8
7 = HTTP read file hello.php - OK

Documentation

Support

Associations

Follow Us