https://winscp.net/eng/docs/ui_pref_editor
Maybe you didn't put your jpg/png rule above the *.* internal editor rule.

---

Or, if you want WinSCP to open the files in application associated with their extension, set Operation to perform on double-click to Open:
https://winscp.net/eng/docs/ui_pref_panels

I tried to set an external viewer for jpg and png files but it just opens them in the internal text editor. A reasonable default is to use Windows' system default which is the Photos app on Windows to display an image. You could use it for the many other times of images as well. Similar if you were going to try and open a movie file.

What I ended up doing was just download the directory and double clicking files to open them but this becomes inconvenient when there are a lot of files and you might end up having to download individual files and then click on them, in 2 steps as opposed to one.