Over 2000 folders not recognised with S3 protocol

Advertisement

willship
Donor
Joined:
Posts:
5
Location:
Australia

Over 2000 folders not recognised with S3 protocol

I'm copying from local PC source with 2400 folders to Backblaze bucket using S3 protocol, with drag & drop in the Commander GUI interface. Once I reach 1999 folders in the bucket, any extra folder that I transfer means the earliest folder transferred is no longer shown in the destination window. I never see more than 1999 folders, and if I transfer a file then that doesn't appear in Commander view either, but it does successfully transfer. The status bar on the remote pane also shows only 1999 items.

If I try the Synchronize function, WinSCP reports the earliest folders as missing. But if I access the bucket directly via a web browser separately from WinSCP then I can see all folders and files (approx 2400). I couldn't see this issue mentioned in doco or forum posts, so thanks for reviewing.

WinSCP 5.19.2 on Windows 10 Home (fully patched). S3 protocol (no compression) using the Commander interface. I haven't tried earlier versions. Log file attached.
  • B2.log (2.41 MB, Private file)
Description: Log file for transferring a single text file using Commander and S2 into a bucket with 2400 folders.

Reply with quote

Advertisement

martin
Site Admin
martin avatar
Joined:
Posts:
41,440
Location:
Prague, Czechia

Re: Over 2000 folders not recognised with S3 protocol

Thanks for your report. Please post a log that only lists a contents of a folder that has over 2000 files. And give us a names of some of the files that are missing in the panel. Can you see the missing files in any other S3 client?

Reply with quote

willship
Donor
Joined:
Posts:
5
Location:
Australia

Hi, I created a folder and uploaded 2100 files, and was able to see all 2100 in WinSCP. So it looks like files are displayed ok, it's only folders that are limited? I attached the log file for viewing 2100 files.

An example of a folder I can see in other browsers but I cannot see in WinSCP is "Ultra". I cannot see folders with names beginning with T or any later letter of the alphabet.

I tried the Cyberduck S3 client and I could see all the 2380 folders including the folders that do not display in WinSCP.

Reply with quote

martin
Site Admin
martin avatar

Sorry for my late answer. It took me while to investigate this. But I do not see any indication in the log that WinSCP is missing some contents. Can you share some test bucket for us to test? Or share contents and steps for us to replicate the bucket?

Reply with quote

willship
Donor

No problem. I will set up another bucket that demonstrates the problem that I can share with you. It may take a few days to get to it but I will reply here.

Reply with quote

Advertisement

willship
Donor
Joined:
Posts:
5
Location:
Australia

Over 2000 folders not recognised with S3 protocol

Hi, I've now created a test bucket that contains 2401 folders, each with two subfolders and a few small test files. This is like my original structure.

Endpoint: s3.us-west-002.backblazeb2.com
BucketID: 641bb7c85cc341e274c2081c (if needed)
BucketName: ws-test-files
Read-only KeyID: 0024b78c312428c0000000006
Secret Key: K002JVOWv1kByizlQFNCIqbQiv6KweM

The folders are named dir0000 to dir2400. When I log in with WinSCP explorer view I can only see the first 1000 folders in this bucket. Cyberduck and the bucket administrative interface both show all 2401 folders.

Reply with quote

Advertisement

You can post new topics in this forum