While WinSCP code has strictly separated core functionality and the GUI, which allows existence of plugins to file managers, there is no usable API. It means, that the interface of the core library:

• is not documented;
• changes significantly with every release;
• uses proprietary classes of Borland libraries (VCL) and proprietary features of the Borland C++ compiler.

You can of course use scripting functionality of WinSCP, if it suits your needs.

Note that you also cannot use the WinSCP filemanager plugins from you application. These are designed to work from within the respective filemanager environment only.

Otherwise, I would suggest you to take a look at source code of PuTTY. WinSCP uses its SSH code and SFTP and SCP code is particularly inspired by it as well. PuTTY does not have the API too, but it at least uses plain C and is has also separate core and the GUI.