1) WinSCP seems to list all dataset for the selected user/prefix and then filter against wildcard pattern on the client side. This can result in huge lists being sent over the net for no reason.
That's correct. There's no (at least no standard) way to get the server filter the files. So they have to be filtered on client side.
2) Allthough I can see (with Wireshark) that the server returns the z/OS specific information like Dsorg,Recfm, Lrecl there doesn't seem to be a way to retrieve the information?
I downloaded the source, cloned the ListDirectory method and dependencies to a new ListMainframeFiles method, but it seems that the assembly communicate with WinSCP.exe through an XML log-file and the data I need is not in the log-file either.
No, WinSCP does not support retrieving those fields. All you can possibly do, is to parse WinSCP log file.