I added an attachment with three logfiles by WinSCP. First, there is the decryption error, in the second part the "no such file" error (since I didn't expect rename to be called after I sent permission denied, I already removed the file) and lastly the hoped for "permission denied" error. All cases are reached with the same code on the server side.
If you are wondering what I am doing in the openssh sftp-server code, here is a small summary:
1. process_open: If the flag for writing is set, I check whether there already exists a file with the given name or the name minus ".filepart". If this is so, I rename that file to have a backup in case the mime type of the new file is not on the whitelist. Then I write the name of the backup into the handle.
2. process_write: On every process_write I check the handle if it is locked (variable initiated with false). Then I write the bytes and check the mime type via libmagic. If it is not on my whitelist, I immediately truncate the file, set the lock and rename the backup file (if it exists) to it's original name. If the truncated file has ".filepart" in it's name, I unlink it. If there was no backup in the first place, I just unlink the truncated file. I send the status SSH2_FX_PERMISSION_DENIED.
If process_write is called and the lock is set, I just send the permission denied status.
3. process_close: If the handle wasn't locked and there is a backup file, I unlink it.