This request is being tracked already:
https://winscp.net/tracker/1797
You can vote for it there.

Would be nice if WinSCP could do the conversion on the fly.

Because this effectively forces me to manage multiple versions of the private key on my harddrive, one format for AWS commands, one format for WinSCP etc.

WinSCP supports PuTTY key format and can automatically convert the other formats.

- PKCS#1 / OpenSSL
- PKCS#8
- OpenSSH
- Putty Private Key

https://superuser.com/questions/1515261/how-to-quickly-identify-ssh-private-key-file-formats