Multi million file upload to Google Cloud Storage occasionally file fails stalls process.

Advertisement

bendecko
Joined:
Posts:
2
Location:
Andorra

Multi million file upload to Google Cloud Storage occasionally file fails stalls process.

I'm uploading a lot of individual small files to Google Cloud Storage (S3 type) using Version: 5.19.4 – see attachment.

I started it Wednesday, and it is now Sunday.

There are a lot of files, and it should take approximately 1 day I guess.

The reason it is taking so long is that apparently randomly and periodically Google servers return
503 service unavailable
6gb WinSCP log file so far.... I've just copied the end of it:
. 2021-11-21 04:04:57.621 Header Name: [content-length], Value: [19]
. 2021-11-21 04:04:57.621 Header Name: [date], Value: [Sun, 21 Nov 2021 04:04:57 GMT]
. 2021-11-21 04:04:57.621 Header Name: [server], Value: [UploadServer]
. 2021-11-21 04:04:57.621 Header Name: [connection], Value: [close]
. 2021-11-21 04:04:57.621 End of headers.
. 2021-11-21 04:04:57.622 sess: Closing connection.
. 2021-11-21 04:04:57.622 sess: Connection closed.
. 2021-11-21 04:04:57.622 Request ends, status 503 class 5xx, error line:
. 2021-11-21 04:04:57.622 503 Service Unavailable
< 2021-11-21 04:04:57.622 Service Unavailable
. 2021-11-21 04:04:57.622 Request ends.
. 2021-11-21 04:04:57.622 sess: Destroying session.
* 2021-11-21 04:04:57.622 (ExtException) **XmlParseFailure**
. 2021-11-21 04:04:57.622 Asking user:
. 2021-11-21 04:04:57.622 Error transferring file 'D:\landcycle\Tiles\landregistry\16\32392\21852.pbf'. (XmlParseFailure)
So WinSCP then halts the upload process with a dialogue (attached)

If I click "Retry", it always works on this attempt and thousand more files are uploaded until Google throws the 503 again. This, of course, happens at 2am and I don't see the error until 9am. So those intervening hours have been lost.

Eventually I will be able to upload all these files. But what would be better if there was a setting and countdown. "Automatically retry failed uploads after X seconds" and WinSCP would continue and automatically close the dialogue and retry itself.

I've attached a short video of the clicking "Retry", and that the file that previously failed then works no problem on the reattempt.

number_of_files.png

dialog.png

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
36,510
Location:
Prague, Czechia

Re: Multi million file upload to Google Cloud Storage occasionally file fails stalls process.

Thanks for your report.
I will take a look at this issue and will contact you in several days.

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
36,510
Location:
Prague, Czechia

Re: Multi million file upload to Google Cloud Storage occasionally file fails stalls process.

I have added this issue to the tracker:
https://winscp.net/tracker/2047

I'm sending you an email with a development version of WinSCP to the address you have used to register on this forum.

Reply with quote

Advertisement

You can post new topics in this forum