We have a problem upstream with someone sending us data where they constantly resend old files with updated time stamps. So the folder is full of tons of old files, and then our FTP job sometimes times out and doesn't finish. I've been reporting the issue upstream for months but they haven't fixed it and keep doing it, which causes problems for my application so I need to look for a solution on my end.
I'm using:
However, I still see random .filepart files and then full files later, so I'm not sure it is resuming, partially because if the vendor upstream updates the file with a new timestamp again, the client may not think it can resume because it assumes it is a new/updated file. And even with
-neweronly.
The old files don't have new content, the timestamp is just being touched/updated for some reason.
For a given file naming scheme, can I ONLY download the file with the most recent timestamp or sort by file name DESCENDING and start my job downloading that way so start with the most recent?
For example if I have a file named:
agentScheduleDetail_2_1031210405
I've got the date in the filename so if I tell it to download the one with the largest numbers in it first (or exclusively) I can get the most recent file.
Right now I'm literally just using a Windows batch file that launches WinSCP and then executes a text script. I see there are scripts to try and connect via PowerShell and then only download the most recent file, but I don't know how to connect via PowerShell in my scenario where we don't use a password at all.
Instead we're connecting with a private key and passphrase and I don't know how to connect in the PowerShell script that way.
Thanks!