OK, it's due to how an SCP protocol works. When downloading a folder, it deletes the whole folder only after it is completely retrieved. If some files fail to download or are excluded (your case), the folder is not deleted.
If you switch to an SFTP protocol, it would work as your expect.
If you switch to an SFTP protocol, it would work as your expect.