Download restarts from the beginning after connection error

Advertisement

KJS_vt
Guest

Download restarts from the beginning after connection error

Hello,

I just started having an issue pulling a file from an SFTP site. This issue started just after the WinSCP software was updated to the latest version (as of a couple weeks ago) from a much older (unknown) version.

Description of the process.
I am calling WinSCP.com from a SQL Server Integration System package in order to either pull a full SQL backup file or a differential backup file depending on the day of the week. The full backup is downloaded one day a week (friday) because it is > 11gb and takes over 3 hours to download. The previous issue we had was that a connection issue would occur during this time and the process would error out and quit. After upgrading to the latest version of WinSCP when the connection issue occurs the process reconnects and starts the download from the beginning. This is an issue because we have had this job run for more than a day and a half on the weekend before it finally errored out. This same process occurs for three similar jobs that each have an 11+gb file. Each of these jobs has been reporting the issue where the download will start at the beginning when the connection breaks and restarts.

Question:
1. Is there a setting that can reconnect and not start the download from the beginning?
2. If #1 is not possible, is there a better combination of settings that can be used to better accomodate connection issues? Via automation?

I have attached the log file that was created this past Friday night. I have obfiscated the file paths, site, and user/password names. As you can see it downloads 99% of the file and times out. It then attempts to reconnect or maybe it downloads again until it gets to 99% (that part isn't clear to me since there are no timestamps). It then shows that it errors out at 19%, meaning it started to download the file from the beginning.

Thanks for your help.

Transfer protocol: SFTP
Scripting/automations
Windows: 10 Enterprise

KS
  • WinSCP_Log.txt (2.19 KB, Private file)
Description: WinSCP log file

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
35,731
Location:
Prague, Czechia

Re: Download restarts from the beginning after connection error

What you have attached is not a log file, but just a transcript of the script output. Please attach an actual session log file.

To generate the session log file, use /log=C:\path\to\winscp.log command-line argument. Submit the log with your post as an attachment. Note that passwords and passphrases not stored in the log. You may want to remove other data you consider sensitive though, such as host names, IP addresses, account names or file names (unless they are relevant to the problem). If you do not want to post the log publicly, you can mark the attachment as private.

Reply with quote

KJS_vt
Guest

Log file

You're absolutely right. I'm not sure where or how I created that file. Real Log file attached. Sorry about that.
  • WinSCP_log_20210212_221501.log (60.45 KB, Private file)
Description: Real Log file.

Reply with quote

KJS_vt
Guest

Log file

I made the update to create the debug log file. I started the process about four hours ago. I have already seen that the file was downloaded and then started to download it again. I will let the process run until it eventually errors so that you get as much info as possible. I will post the log tomorrow morning (my time).

Reply with quote

Advertisement

KJS_vt
Guest

Log file

Here is the updated log file. It ran for about 22 hours.
  • WinSCP_log_20210223_082336.log (6.16 MB, Private file)

Reply with quote

martin
Site Admin
martin avatar
Joined:
Posts:
35,731
Location:
Prague, Czechia

Re: Log file

I should have noticed this before: You do not have transfer resume enabled:
< 2021-02-23 08:23:37.991 Script: Using configured transfer settings different from factory defaults.
...
. 2021-02-23 08:23:38.632 Copying 2 files/directories to local directory "\\AHS\AHSSOFT\BACKUP\SQL\AHSSQLW01D\Backups\OMS" - total size: 12,193,277,952
. 2021-02-23 08:23:38.632 PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 0100; Resume: N (102400); CalcS: No; Mask:
You should use /ini=nul to isolate your script from the GUI configuration changes:
https://winscp.net/eng/docs/scripting#configuration

Reply with quote

KJS_vt
Guest

Log file

Sorry, I've had Mon-Wed off this week. I will look into setting that inline parameter. Unfortunately, we cannot currently connect to the FTP site. I believe the host of the site changed the password and did not notify us. Once I am able to test I will capture another log file and see if the issue persists. Thanks!

Reply with quote

KJS_vt
Guest

Log file

I was able to run the process last night and it only ran for about 3 hours and 20 minutes, opposed to 5 days and one hour the last time it ran. The log file shows that it disconnected once and then restarted and finished. I think it looks good. I updated the other two similar processes that we have to create the log files and use the /ini=nul option. They are all scheduled to run Friday night so I will see Monday if this was completely successful. I'm hopeful with last night's success. I am attaching last night's log file as proof.
  • WinSCP_log_20210304_221501.log (906.32 KB, Private file)
Description: Log file: 3/5/2021

Reply with quote

Advertisement

KJS_vt
Guest

Log file

I think that did the trick. Of the seven jobs that download (FTP) a large file Friday night, only one failed and I think that was an unrelated issue. I would like to have these run one more time (Friday 3/12) before declaring this issue resolved. I will reply back on 3/15 with a verdict. Thanks.

Reply with quote

KJS_vt
Guest

Log file

I reviewed the seven jobs that run Friday night, and each weekend night, and they were all successful. I'm confident that the issue is resolved and can be closed. Thanks so much for your help!

Reply with quote

Advertisement

You can post new topics in this forum