Using Script to Copy Files from Remote FTP to Local - Only Want New Files Starting From Today

Good Afternoon,

I am trying to use the WinSCP.com script to download files from a remote FTP site.

The remote site has old files that have not been removed and I do not want to grab these files. I want to set this up as an automated to process to run multiple times a day and check for any new files that were added within a day from the time of the script running.

I have been trying to comb through the documentation but still have not been able to find what I am looking for.

My script so far looks like this:

"C:\Program Files (x86)\WinSCP\WinSCP.com" CooperSFTP /command "synchronize local -filemask=">=today" E:\Cooper\SRO\ /caso/outbound/SRO" exit

Please bare with me as I am new to scripting with WinSCP and scripting in general. Can anyone try and point me in the right direction of what changes I would need to make?

Thanks!

You didn't really tell us what particular problem you have without command.

All I can see that you have quotes wrong.

It should be like:
"C:\Program Files (x86)\WinSCP\WinSCP.com" CooperSFTP /command "synchronize local -filemask="">=today"" E:\Cooper\SRO\ /caso/outbound/SRO" exit

See https://winscp.net/eng/docs/commandline#syntax

Or you actually do not need the inner quotes, so this will work too:
"C:\Program Files (x86)\WinSCP\WinSCP.com" CooperSFTP /command "synchronize local -filemask=>=today E:\Cooper\SRO\ /caso/outbound/SRO" exit

Hi Martin,

Thank you for the response! I did remove the quotes as you suggested and the command ran without a problem but the older files were still downloaded to the local location.

The remote location contains some old folders from February and I added some new text files today to ensure that the command would run correctly. But when running the command it is downloading the older folders as well as the text files from today.

So it seems like there is something not working correctly with the date criteria.

Do you have any other suggestions I could try?

I want to set this for automation, and don't want the script to end up downloading all of the old files that we won't need, just the newly added files that are not present in the local directory.

Thanks!

Please attach a full session log file showing the problem (using the latest version of WinSCP).

Hi Martin,

Please see my log file below.

For your info, in the remote destination I had four folders with files and 4 text files. The folders and images inside of them were already present in my local directory.

The text files were being used for testing and only one of them was created and added to the remote location today, so I expected only this particular file to be copied over to the local destination.

However still even with the "today" filemask it grabbed all of the text files, not just the
one from today's date.

Please let me know what you think!

Thanks!

Hi Martin!

Did you have a chance to review the logs I attached and see if you could identify the problem?

Thanks!

You are using three years old version of WinSCP. It does not support the today keyword. Please upgrade.

Hi Martin,

I had no idea we were using an out of date version. I updated to the most recent version of WinSCP and the script worked as needed, only copying over a file that had been added from today.

Thank you for the assistance!

I did a test and it seemed to work, but after trying a bit more I am still having trouble.

It looks like the today mask is working for individual files, but not for directories in the remote location. Is there something I can do to discriminate the synchronize command against directories as well?

I am attempting to grab multiple directory folders from the remote directory but only want to acquire the new ones (IE from today or within 24 hours). It seems that the today mask will work if I add old text files to the remote location, those are not transferred. But when going against directories it is still transferring everything.

Do you have any suggestions for this?

Sorry, this is bit vague.
Do you mean that WinSCP transfers all files (even old ones) which are in directories?
As previously, the best would be if you post a log file of the problem, with some comments.

Hi Martin,

I have re-ran the WinSCP script with logging enabled and attached the log file. I also added some screenshot to help clarify what I am trying to illustrate. The CooperFTPRemote.png attachment shows the remote directory. As you can see there are a few different folders in here as well as the text files.

When running the script the 'TODAY' mask seems to only be affecting the text files. If you take a look at the CooperFTPLocal.png file this is my location where the files are being synced to.

After running the script all of the folders and files within those folders are transferred, and then the text file - only from today - is transferred. I am wanting to use the 'TODAY' mask to ideally transfer only the folders from today as well, not just the text files. As it is now, it appears that it is syncing everything, and then only using the today mask on the text files that are present. All of the folders are still being synced even if they do not have a date from today.

Hopefully this makes more sense with this additional information! Please let me know if you need anything else!

I can see only one file downloaded in your log:
/caso/outbound/SRO/041720.txt
The file had timestamp 2020-04-17, what was the day the script ran.
So I believe it works correctly.

Hi Martin,

Can you please see my attached log again from something I just ran today. I changed the remote and local directories to something new and only added one file from today which was Test1.txt.

But the other folders in that location were synchronized to my local directory even when using the today filemask.

Thanks!

There are two sessions in your log. Which one should I look at?

Anyway I see only one file downloaded in each session – /caso/outbound/SRO/041720.txt and /caso/outbound/Capital/Test1.txt – both 2020-04-17

If you believe another file(s) was transferred, give me some names. I cannot investigate 100 thousand lines long log file.

Anyway, isn't what bothers you that possibly some empty directories were created on the local machine? If you want to avoid that, set ExcludeEmptyDirectories:
https://winscp.net/eng/docs/rawtransfersettings

Hi Martin,

I do notice now that it just seems to be copying over the empty folders, not any of the actual files that are contained inside of them.

I will try and use your suggestion regarding the empty file exclusions and get back to you.

Thank you!

I need to copy files from FTP to local directory using the WinSCP.
Can you help me to find the best command.

@tony.akiki: See Automate file transfers (or synchronization) to FTP server or SFTP server