Editing file over 500MB is well beyond capabilities of WinSCP internal editor. Please use external editor capable of editing such a huge files.
Out of memory
<Try to describe precise steps that lead to the problem (where do you click, what keys do you press, what do you see, etc.)>
<If relevant, consider attaching a session log file or a screenshot)>
Try to edit file with 500 mb