Transfer Speed of Numerous Small Files

Advertisement

Guest

Transfer Speed of Numerous Small Files

Hello,

When putting files onto a SFTP server with the .NET assembly, what is the expected speed when it comes to small files (1 KB or less)? For example, I had 18 1KB files take around 4 minutes to upload, and that is with multiple Session Tasks running. It seems to vary in the amount of time, but it's generally longer than 2 minutes.

I watched the files process a few times, and it seems like it's 1 file every 10 seconds or so, which feels like a long time, even for small files.

Is there anything I can look at to improve this speed? The files, once uploaded, do get copied to an archive folder, and that does have over 100,000 files in it already if that makes a difference.

Let me know if you need me to provide more information, and I will try to get that to you.

Thanks
  • 990_ACCEPT_10_7125742E_2002150702.txt (250 Bytes, Private file)
Description: Example file being uploaded. The files are of type EDI, but since that file type isn't allowed I changed it to a txt.
  • log202108131239.log (1.83 MB, Private file)
Description: Earlier Session log with Key information removed

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
41,276
Location:
Prague, Czechia

Re: Transfer Speed of Numerous Small Files

Your server takes long time to respond to every request. Do you have a different experience with any other SFTP client?

Reply with quote

jacobgralinski
Joined:
Posts:
6

Hi Martin, there are a couple different SFTP clients. One of them completes uploads of a similar size in about half the time with the same number of files. If the issue is server response time, is there anything that can be done? I have noticed that the download side also appears to suffer from slowness.

At the moment this issue isn't a huge issue, but in the past it has conflicted with other tasks that run at the same time.

Reply with quote

jacobgralinski
Joined:
Posts:
6

Here is a session log for another SFTP client. The files being upload are roughly the same size, so this is a good comparison.
  • log202109201117.log (48.6 KB, Private file)
Description: Session log for another SFTP client

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
41,276
Location:
Prague, Czechia

You have again attached a log from WinSCP:
. 2021-09-20 11:17:09.261 --------------------------------------------------------------------------
. 2021-09-20 11:17:09.261 WinSCP Version 5.18.1 beta (Build 11089) (OS 6.2.9200 - Windows Server 2012 Standard)

Reply with quote

jacobgralinski

I understand now, sorry about that. We do not currently use any other SFTP clients besides WinSCP, sorry for that confusion. We have numerous connections that all use WinSCP, and the speed appears to vary between them, so I was curious if there is any way to resolve the slower connections.

Reply with quote

Advertisement

jacobgralinski
Joined:
Posts:
6

At the moment I don't believe I can, but as of late I believe the issues have been on the other end of the connection. What I do have is one of our tasks uses a Windows batch file to conduct the downloads and uploads, and that typically takes 10 seconds to run, even with over 300 files to download and a few larger files to upload (3KB to 10KB). In that task, the C#/.NET project creates the files and puts them in a folder for the batch file to pick up and upload. The downloads and uploads still go through WinSCP. Could the batch file be more efficient than creating sessions through .NET?

Reply with quote

jacobgralinski

Okay, I thought so. We are going to be restructuring our project in the very near future to break out some of the larger tasks, so I think for now this won't be a huge issue. Thank you for your help.

Reply with quote

Advertisement

You can post new topics in this forum