Thanks for your report.

This issue has been added to the tracker:
https://winscp.net/tracker/1781

Meanwhile, you can workaround it by escaping the < and > explicitly by doubling them:
Koala[?][?]<<>>>>:::[*][*][*]|[?].jpg
See https://winscp.net/eng/docs/file_mask#size_time

Though note that the above is an invalid Windows filename anyway. It will get "escaped" to
Koala%3F%3F%3C%3E%3E%3A%3A%3A%2A%2A%2A%7C%3F.jpg

Hi.

I'm trying to download a file "Koala??<>>:::***|?.jpg".

It's always the same error message:
Mask is invalid near '>>:::***|?.jpg'.

With RemotePath.EscapeFileMask:
Mask is invalid near '>>:::[*][*][*]|[?].jpg'.

Is there a way to disable this masking stuff?