First, you do not authenticate with public key, but with private key.
You specify the private key on
Authentication page of Advanced site settings dialog:
https://winscp.net/eng/docs/ui_login_authentication
Though the "timeout" error can actually indicate a completely different problem. If the above does not help, please attach a full session log file showing the problem (using the latest version of WinSCP).
To generate the session log file, enable
logging, log in to your server and do the operation and only the operation that causes the error. Submit the log with your post as an attachment. Note that passwords and passphrases not stored in the log. You may want to remove other data you consider sensitive though, such as host names, IP addresses, account names or file names (unless they are relevant to the problem). If you do not want to post the log publicly, you can mark the attachment as private.