On the "Options" dialog, when clicking "OK: button, hold down Ctrl+Shift keys. WinSCP will not execute the extension, but only copy it's command to a clipboard.
See also https://winscp.net/eng/docs/custom_command#debugging
Then go to Windows command-line (cmd.exe) and paste the command there. That will allow you to see the error message. Please post it here.

Thanks for pointing me to Batch Rename...

I can get as far seeing in a shell window what it's going to change my filesnames to, and it looks fine, but when I hit "y" for winscp to proceed, it instead generates an error and closes the shell window before I can read what the error is. I'm on a windows 7 enterprise machine, 64 bit, WINSCP 5.13.2, Sftp login, port 22. Appreciate your help on this. kate

You cannot add a prefix using operation mask.
You can use "Batch rename" extension (installed in WinSCP):
https://winscp.net/eng/docs/library_example_advanced_rename

Trying to follow instructions at:
https://winscp.net/eng/docs/operation_mask but must not be understanding it right.

I'm trying to add a prefix AA- in front of existing filenames on a remote server, so to change number of files like:

1740.txt to AA-1740.txt

so I do FILES --> MOVE TO... --> /AA-*.txt

and that gets me

1740.txt changed to AA-0.txt

I also tried this:

FILES --> MOVE TO... --> /AA-????.txt

same result, 1740.txt changed to AA-0.txt

thanks for any advice.