Need some help with the source code

I'm developing a program at the university and wish to use plink and pscp as an integrated part of my project.
I'm going to explain the main goals of my project. The program I'm
developping is a GUI for a command based application, so that it must
adquire data, generate a script file with that data, and launch the
command based program and feed it with the script file generated.
Then, the gui have to read the result files generated by the command
based app and display it graphically. The problem is that the command
based app must be launched in a cluster linux, so that you can perform
multiple processes at the same time, one or more in each machine. Here
is where your client enters the action: First I decided to use the rsh
protocol to launch the command based app in the linux machines, but
the rsh protocol is very unsecure and is discontinued, also most linux
machines have deactivated it, so I have to use ssh protocol.
I have been looking for a port of openssh for windows and I have found
a few. I would like to integrate plink and pscp in my GUI.
Since that, I have been looking for a program that have putty
integrated in it as a reference for integrating it in my project and I have found WinSCP, that is written in Borland C++ Builder and have
putty integrated in it, so that it doesn't make calls based on
commands, but a comunication with direct calls to
putty's functions.
I have put an eye on the winscp code and I think
the communication is based on transferring data via windows registry, but i'm not sure. That's why I'm asking you.

I would be very thankfull if you give me some clues on how to integrate pscp into my project and communicate with it (based on your experience in WinSCP). What parts of code from pscp should I change and what functions should I call to make the comunnication between pscp and the gui. The gui is not inteded to transfer files interactively so it should be much easier than in winscp.
Also there is the multithreading issue. I'm not sure at all, but I think that it's not
possible to make more than one connection at the same time using
direct calls to putty functions: the data structures used to stablish
and maintain a communication by putty should be overwritten when I try to make
a new one and there's another link in progress. For this issue I
thought in porting the plink code to C++ classes so I could create as
much objects as links I wanted to make, so that the data from one
comunication wont interfere with another.
I have tried to change some things but I'm afraid it's a very hard
task and would take almost a year to complete it. I don't have so much
time so I think there should be another way to fix the multithreading
issue without porting the code to C++ classes.
So the question is:
is there a way to stablish multiple connections at
the same time if the code is integrated in the gui and ussing only one instance of the gui?
I guess winscp does this data interchange by using the registry, but this doesn't solve this problem.

By the way I'm writting the code in VC++ NET as I have never worked with BCB, that's why I'm asking you for help and not looking at the code itself (I have never coded in BC++ Builder).

I you have the answer for my questions I would be very thankfully if
you give me some clues on the way of doing it.

Thank you very much in advance and keep up the good work.

1) Which parts of Putty you need: Have a look at what WinSCP links (Putty library project file). Or just take a look on what PSCP or PSFTP links. These are only source files you need. The rest is Putty GUI.

2) How to integrate: Putty is well designed, it has strictly separated GUI from SSH core. So you do not need to change any line from its code. You have just to implement set of functions that Putty calls back. Some of them will be never called, you just have to have their empty implementation, otherwise it would not link. See PuttyIntf.c from WinSCP, which contains them. Or again look at PSCP/PSFTP, which does the same.

3) Multithreading: I'm not sure. You may create multiple connections at once. Each connection have its own structure holding its data. WinSCP does it. But it have only one thread. So I have no experiences with using Putty in multithreaded application.

4) IFAIK Putty developers use VC++

Thank you very much for your help!

1) I have already split the files I need into a new project to test it. I need only to launch executables and transfer files (with the results) between some unix machines and a windows machine (where is running the gui), so I guess I need the project files from plink and pscp, which I have already integrated in a test project. (this is not interactive, the gui must do it by itself).

2) Thank you for pointing the callback functions, I thought I had to change some code. The problem is that the source version I have from putty is different from the version that comes with the last WinSCP release, so I had been comparing the putty files with a compare and merge tool and have found some code in putty.h that I thought you added it for integrating with WinSCP. I'll take a deep look at those callback functions. Thanks.

3) I didn't know that you could create multiple connections at once with winscp, I thought that putty used the same data structure for all connections, I think that if it's possible to do multiple connections in one session, there shouldn't be problems with multithreading programming.

I have to recognice that I have been so gratefully surprised when trying WinSCP that now I use it and I'm sure I'll do it for long.

Thank you very much and keep up the good work!

Link39 wrote:

Thank you very much for your help!

You're welcome :-)

The problem is that the source version I have from putty is different from the version that comes with the last WinSCP release, so I had been comparing the putty files with a compare and merge tool and have found some code in putty.h that I thought you added it for integrating with WinSCP.

My putty.h includes some hacking macros that enables me to read some internal data from putty, which are not otherwise available. These are mostly things like actual SSH protocol version, encryption algorithm, etc. I use it only to display information to user. It is in fact not necessary for WinSCP to compile/link/work. Next thing it does is that it overrides standard RegOpenKey and RegCreateKey, so Putty reads from WinSCP registry key, instead its own one.

Thank you again!

You have the best support I have seen ever! :P

martin wrote:

Next thing it does is that it overrides standard RegOpenKey and RegCreateKey, so Putty reads from WinSCP registry key, instead its own one.

What's the purpose of using the same registry keys? Does putty stores some function results in the registry?
If I need to launch binaries and transfer files is correct to use plink and pscp project files? or maybe it's enought with the pscp files?

Thanks! :D

You have the best support I have seen ever! :P

Thanks :-)

What's the purpose of using the same registry keys? Does putty stores some function results in the registry?

No. Putty caches public keys of the servers in the registry. See verify_host_key() and store_host_key() in WINSTORE.C

If I need to launch binaries and transfer files is correct to use plink and pscp project files? or maybe it's enought with the pscp files?

Sorry, I do not undertand this one :-(

I have been away some days so I couldn't read your post, sorry.

Sorry for that, it was an understandable question.
What I mean with the thirth question is that in the gui that I'm developping I need to execute binaries in remote machines and transfer files from them, so I ask if with the pscp project files would be enough for this 2 features or may I need to include also the plink source files in the project?

[quote=prikryl]
Putty caches public keys of the servers in the registry. See verify_host_key() and store_host_key() in WINSTORE.C
[/quote]
Ah, so this is a treak for saving the public keys and not request them in each future connection. Good.

Thanks.

Link39 wrote:

Sorry for that, it was an understandable question.
What I mean with the thirth question is that in the gui that I'm developping I need to execute binaries in remote machines and transfer files from them, so I ask if with the pscp project files would be enough for this 2 features or may I need to include also the plink source files in the project?

It depends on how you want to execute the files. You have two options, either to launch one command per session, the same way as PSCP launches SCP command. Specify the command in Config.remote_cmd_ptr. Or you may let SSH to execute shell (leave Config.remote_cmd_ptr blank) and then you may execute several commands in row, just by sending raw command followed by 'enter' character (just as if user types it on command line). This is what WinSCP does (see TSessionData::StoreToConfig in core\SessionData.cpp). Both these approaches are core functionality of SSH, so you do not need pscp nor plink for it.

Sorry 4 that question. The problem is that I don't have any version of BC++ Builder, so I haven't seen what files WinSCP links and if it compiles putty as a library (I guess it does). I have asked a friend who have a demo from version 6 and then I will try to check the code deeply and trace it to see how it works and how those callback functions works and what files from putty should I link.

Thank you very much, you have helped me a lot.

Finally I have installed BC++ Builder 6 (enterprise) that a friend lend to me and I have tried to get in depply into the winscp code.
First I tried to compile all the projects, opening one each time, compiling it, and doing the same with the next (leaving the main project for the end), but when I tried to compile the putty library project it gave me a bunch of errors like "unix.h" not found, some errors in putty.org.h and so on (I don't know what I did wrong).
Then I used the make tool with the "makefile" and finally all projects were compiled. After that I opened the WinSCP3.bpr project and tried to trace the code from the start, but the "trace into" command launched the executable without letting me trace a single line of code, and the "step over" command started tracing machine code instead of the c++ code.
As I was not able to start at "WINAPI WinMain(HINSTANCE, HINSTANCE, LPSTR, int)" function, I tried to trace it setting a lot of breakpoints all over the code of the different units that take part on that project, but it ignored all the breakpoints and launched the executable again without letting me trace the code. I think it should be important for me to trace the code to learn how some functions works and see what they return and with what data they are called, specially the functions you have implemented in PuttyIntf.c that putty calls back with results of some operations.
As the winscp solution is a very large project and my knowledge with BC++Builder is limited to run and compile projects, I really need some help with it. I have seen what files you link, they are mostly the same I took to separate pscp from the rest of the putty files.
I think I know some tasks to do:
- Separate the files I need (for example the files that pscp links expect scp.cpp of course). Make a library project with this.
- Implement some funcions that are in scp.cpp that the ssh core calls back with results (as you did in PuttyIntf.c). This uses tha above library.
- Implement a class which wraps the Config struct and some functions to stablish the connection, transfer, etc. These functions should call the ssh core functions with the data in the Config struct (one struct per object, so I should work with multiple connections and multithreading).

But I'm not sure at all, it's only the idea I have formed in my mind with your answers and a short look at the code. I'm not an expert programmer and reading and understanding code not written by me is a hard work for me.
I would apreciate your opinnion about the above issues and also if you point what functions I have to call in order to stablish a connection, transfer files and launching commands.
About the integration issue: I think it should be easier for me to start from zero and take the pscp project files unmodiffied from the putty package than starting over with the putty project you use in you r code, but I'm not sure and I don't know I will need some more files that the ones that pscp links.
About the launching commands issue: I only whant to launch an executable per session, so I think that the first method should be enough, but I think that some times I will need to launch more than one command per session, for example before retrieving files checking the time and date to not transfer the file if it haven't changed since the last time. So I think I will need the second method, but... for this method is necesary to link any more files than pscp does?

I'm lost in a bunch of code and it seems to me very complex. Winscp is a very complex project, very large and with a lot of modules, so I think I wouldn't get a clear sight of the integration if someone doesn't point me in which files of winscp I have to look for the integration or calls to the putty code.
I thought I should be a good start tracing the code after the logging button was pressed in the login formulary and tracing all the connection and transfer processes after that, but I haven't been able to trace a single line of code (I don't know how to do it with BC++B as I have never worked with BC++B before).

I would apreciate any help.

Thanks

Link39 wrote:

but when I tried to compile the putty library project it gave me a bunch of errors like "unix.h" not found, some errors in putty.org.h and so on (I don't know what I did wrong).

There is a bug in C++ Builder that causes it. But as you have succeeded in compiling it already, I would not write you workaround now.

tried to trace the code from the start, but the "trace into" command launched the executable without letting me trace a single line of code, and the "step over" command started tracing machine code instead of the c++ code.

Debug info is probably stripped from EXE file. In Project options dialog, click "full debug" button, save the project and make it again (using makefile). You probably need to do this for every library as well.

- Separate the files I need (for example the files that pscp links expect scp.cpp of course). Make a library project with this.

That's what Putty library of WinSCP is for. It contains only the files needed. You probably need the same.

- Implement some funcions that are in scp.cpp that the ssh core calls back with results (as you did in PuttyIntf.c). This uses tha above library.

OK.

- Implement a class which wraps the Config struct and some functions to stablish the connection, transfer, etc. These functions should call the ssh core functions with the data in the Config struct (one struct per object, so I should work with multiple connections and multithreading).

OK. TSecureShell class (core\SecureShell.cpp) in WinSCP is the wrapper around the Putty functionality.

But I'm not sure at all, it's only the idea I have formed in my mind with your answers and a short look at the code.

You have just good idea.

and also if you point what functions I have to call in order to stablish a connection, transfer files and launching commands.

Estabilish connection: Look what PSCP does. In scp.c see Init() function. It calls back->init(). And then back->sendok() in loop in function ssh_scp_init(). Transfer files: Again take pscp as example. Keep one thing in mind, PSCP despite its name in both SCP and SFTP client, so it's code looks now rather complex comparing with time it was SCP-only client. I don't know what protocol do you want to use. SCP is simpler, while SFTP is much better. With SCP you cannot interrupt file transfer in middle without closing the connection, it does not allow transfer resume, and much more.
Launching commands: See below.

I don't know I will need some more files that the ones that pscp links.

I believe that you do not need any other file than PSCP need. Maybe you need little less, if you decide to use SCP only. PSCP links sftp.c. WinSCP does not link this file, because I have implemented SFTP by myself.

About the launching commands issue: I only whant to launch an executable per session, so I think that the first method should be enough, but I think that some times I will need to launch more than one command per session, for example before retrieving files checking the time and date to not transfer the file if it haven't changed since the last time. So I think I will need the second method, but... for this method is necesary to link any more files than pscp does?

You do not need anything more as I have already wrote once. To use second method: leave Config.remote_cmd_ptr blank and after estabilishing connection, putty opens default user shell. If you want to force particular shell (like BASH), set Config.remote_cmd_ptr to that shell (e.g. '/bin/bash'). Than send command to execute via back->send() and back->sendbuffer(), see TSecureShell::Send().

Wow, you have really impressed me with this so good support.

martin wrote:

There is a bug in C++ Builder that causes it. But as you have succeeded in compiling it already, I would not write you workaround now.

Thank you very much, I was really confused about it, I thought I was doing something wrong and I felt like a dumb. I'm looking for a patch to solve it, but this does no bother me as I have no plans of using BCB in any other projects.

martin wrote:

Debug info is probably stripped from EXE file. In Project options dialog, click "full debug" button, save the project and make it again (using makefile). You probably need to do this for every library as well.

I have done it and now it traces the code as normal. You saved me!. Thank you very much. Yesterday I thought I was already compiling a debug binarie due to the size of the executable file, also I didn't know where those debug options was.

martin wrote:

TSecureShell class (core\SecureShell.cpp) in WinSCP is the wrapper around the Putty functionality.

Thank you for pointing it, I'll take a look at that class.

martin wrote:

Estabilish connection: Look what PSCP does. In scp.c see Init() function. It calls back->init(). And then back->sendok() in loop in function ssh_scp_init(). Transfer files: Again take pscp as example. Keep one thing in mind, PSCP despite its name in both SCP and SFTP client, so it's code looks now rather complex comparing with time it was SCP-only client. I don't know what protocol do you want to use. SCP is simpler, while SFTP is much better. With SCP you cannot interrupt file transfer in middle without closing the connection, it does not allow transfer resume, and much more.

Thank you very much. I was confused with those two protocols scp and sftp and I didn't know well the differences nor they were both integrated in pscp.
For now I have planned to work with the last stable putty version which is much previous than the one you are using in the last winscp package. So I guess it has still the scp and sftp protocols splitted in pscp and psftp, and also it have many less files that the latest snapshoot release and in fact i should be easier to integrate/understand/use.
I think I don't need the sftp protocol as I don't need to resume/stop, I think I can manage without the advantaged of the sftp protocol.

martin wrote:

You do not need anything more as I have already wrote once. To use second method: leave Config.remote_cmd_ptr blank and after estabilishing connection, putty opens default user shell. If you want to force particular shell (like BASH), set Config.remote_cmd_ptr to that shell (e.g. '/bin/bash'). Than send command to execute via back->send() and back->sendbuffer(), see TSecureShell::Send().

Thank you very much, now I understand: two methods with the same files, I don't need adittional files for the second method. It's a simple way of choosing between the two methods.

You have helped me a lot, I will thank you also in the book of my final career project when I finish it, and also I will recommend your powerfull WinSCP as the best scp/sftp windows client.

Now that I have a general idea in my mind I'm going to start with the hard work of integrating the library and designing the wrapper class. I think now it should be easier as I have your wrapper class as a reference and I can trace it for better understand.

Thank you again a good luck.

I have been playing with the code for a while, looking at TSecureShell class and I think that I can use this wrapper class, the puttyinf and the putty lib you have developped to save some days of developping my own wrapper class. But I don't know what would be easier:
- Start from zero
- Use your wrapper class and adapt it to VC++ style and to my needs.

But I don't know if using your class involves using also TTerminal and TTerminalManager classes (maybe more) or it can be adapted to work alone.

So the question is: What do you think that should be easier? Adapt your class/es or build from zero my own one with your class as a reference?

Thanks

Link39 wrote:

So the question is: What do you think that should be easier? Adapt your class/es or build from zero my own one with your class as a reference?

I would suggest you to start from zero, using TSecureShell as hint.

Hi prikryl,

I have been working hard during these days and I have advanced a lot with your help, but now I have realiced that I was working for nothing and now I must trow all my work to the trash and start again, just because I was working with the latest stable version (that still have the static-variables session deppendant). Browsing the wishes list from the putty project I found this:
https://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/remove-statics.html

For heaven shake! I was stupid for not starting working with the latest snap-shoot release (that I think that have this problem fixed).

I was programing the interface and the wrapper class like yours but with some diferences:
- The interface is in the same files that the wrapper class.
- The class is merged and compiled with the library instead of compiling the library and then using it in the class. I think this simplifies the interface and the communication method (not so much extern functions required).

What do you think about this interface/wrapper class method? Is better to implement it like in your program or is this way ok?

And finally another question: I have been working a lot, but still I have a lot of job to do, and I have thought of using pscp and plink as command line processes, just deffining pippes and redirecting it to the pscp and plink stdin, stdout and stderr and then send commands through the stdin pipe and receive results and errors throught the others. I think it should be much easier but not so clean and efficient. I want to know your oppinion and suggests about this question.


Thank you very much!

Link39 wrote:

What do you think about this interface/wrapper class method? Is better to implement it like in your program or is this way ok?

In my opinion, it does not matter. Both is same good, it just depends on what you like better.

And finally another question: I have been working a lot, but still I have a lot of job to do, and I have thought of using pscp and plink as command line processes, just deffining pippes and redirecting it to the pscp and plink stdin, stdout and stderr and then send commands through the stdin pipe and receive results and errors throught the others. I think it should be much easier but not so clean and efficient. I want to know your oppinion and suggests about this question.

Problem with PSCP is AFAIK that for every single transfer you need to launch new instance of program and to complete the authentication, which may take several seconds.

martin wrote:

Problem with PSCP is AFAIK that for every single transfer you need to launch new instance of program and to complete the authentication, which may take several seconds.

But as long as I have to work also with SSH2 protocol and I can't execute shell commands by the way you can do it with the SCP protocol in SSH1 I also should need to add some more files and extend the class to be able to make SSH2 terminal connections like in plink.

One more question: is there any other fundamental static varible that I must wrapp in the class like the Config and Backend structs?

Thanks

Problem with PSCP is AFAIK that for every single transfer you need to launch new instance of program and to complete the authentication, which may take several seconds.

I think that with the wildcards trick I can manage to transfer if not all at least half of the files I need in each moment.

And now I need for sure to integrate also with the plink and pscp files (double work, I can't make the trick of the scp protocol to launch commands in SSH2),
so with all these information what do you recommend me now?

Thanks

Link39 wrote:

And now I need for sure to integrate also with the plink and pscp files (double work, I can't make the trick of the scp protocol to launch commands in SSH2)

Why you cannot use the trick for SSH2?

so with all these information what do you recommend me now?

I do not know about what you ask for recommendation :-)

Why you cannot use the trick for SSH2?

Because SSH2 does not support SCP protocol, and with SFTP protocol you can't execute commands, so I will need to integrate also the plink files in order to execute commands with SSH2.
I don't know at all, as you can see I know nothing from SSH protocol, I'm only guessing. I don't have the time to go in deeper into the SSH (1 & 2) protocol.


Thanks.

P.S.: MERRY CHRISTMAS AND HAPPY NEW YEAR TO EVERYBODY!

Link39 wrote:

Because SSH2 does not support SCP protocol, and with SFTP protocol you can't execute commands, so I will need to integrate also the plink files in order to execute commands with SSH2.
I don't know at all, as you can see I know nothing from SSH protocol, I'm only guessing. I don't have the time to go in deeper into the SSH (1 & 2) protocol.

SCP can work even in SSH2. Just place scp and scp1 binaries in path. Also as I have already wrote, you do not need Plink sources to execute commands! See some ports above.

SCP can work even in SSH2. Just place scp and scp1 binaries in path. Also as I have already wrote, you do not need Plink sources to execute commands! See some ports above.

Then why WinSCP can't loggin with SCP protocol in an SSHv2 server?

Then why WinSCP can't loggin with SCP protocol in an SSHv2 server?

It can. See my last post.

Hi Martin,

Finally I have managed to implement a wrapper class for putty with SFTP functionality.
I think that SSH v2 is supported by more servers than SSH v1, and almost all servers that supports SSH v1 also supports SSH v2 nowadays, so I have made only SFTP and no SCP implementation.

I linked the files from psftp project except psftp.c/h and sftp.c/h of course.
Then I started refinning the class creating a set of functions to replace the ones in console.c and cmdline.c for providing multisession support to my class and non-interactive login: (verify_ssh_host_key,
old_keyfile_warning, askcipher, etc). Then I removed cmdline and console linkage as I already implemented the basic functions to replace that ones.

All ok and working till now. Putty seems to be designed for multisession but I have found a critical problem:
"The console_get_line problem".
I think the password asking module is a bit crappy. I'm gona explain this:
When you login, the ssh core calls the function pointed by ssh_get_line to provide the password:
As I did my own implementation of the function console_get_line, I pointed ssh_get_line to it, so it's called to provide a password to the ssh core.
Well, as passwords are session dependant atributes, it must be stored as a NON-STATIC class member.
When the console_get_line function is called it has no way to know the password for this session as it's not called with the frontend pointer, so you can't do "return ((class*)frontend)->GetLine(...)" as in the from_backend function.
So I made the console_get_line a member method of the wrapper class. As the function must be static in order to be assigned to the ssh_get_line pointer, I made it static. And the big problem comes now: as the function is now static, it can't access non-static members of the class, so the password string atribute has to be static, which will force all session to have the same password.
When the password entry is interactive this works ok, as the password is provided when needed, so the console_get_line function doesn't have to access any class atributes.
I don't understand why the password is not stored in the cfg struct so the ssh core uses it and don't have to call that crappy function, as it does with the login name.

This is a big problem and I don't know how to solve it. I have been looking to the WinSCP code and I have seen that you use a global pointer variable for solving this problem: CurrentSSH. When you are going to init the session you point the variable to the current TSecureShell object, so when the get_line function is called by the ssh core you do this: Result = CurrentSSH->GetPassword(Password);
After the initialization you point the variable back to NULL.
This method works well for your program as the user can't start more than one session at a time, so the CurrentSSH variable doesn't suffer the exclussion problem. As my program is multithreading, I have a critical problem with this, as if two sessions are created at a time, there will be an exclussion problem with the CurrentSSH variable, as one session will point the variable CurrentSSH to its object and if the
other session accesses the same variable for the same purposes before the first session provides the password, the same password will be sent for the two sessions.
The only way I know to solve this is by using semaphores for accessing the critical section (variable CurrentSSH). But this may carry problems and I consider that it's a waste of time to only permit one session init at a time.
The ssh core should call the get password procedure with the frontend parameter in order to be able to do multiple sessions at a time and should be totally session-independant and not global.
This could be done modding the variable ssh_get_line as this:
and adding the ssh->frontend parameter in each call to ssh_get_line within the putty core.

Personally I think that's better to add the frontend parameter to the function than using mutexes, but this leads to a ssh core modiffication and have to be made every time you upgrade the putty core.

If you know another method not based in a global object pointer (CurrentSSH), please help me.

-----------------------------------------------------

Another question not related to the problem beside.

I want to launch a command per session as I told you some posts before. Thank's to you I know there are two ways of doing this:
- specify the command in cfg.remote_cmd (as psftp and plink does)
- leave blank cfg.remote_cmd and start a remote shell

I have been trying the first method with the psftp wrapper class, specifing "notepad" instead of "sftp" in cfg.remote_cmd and it worked, after changing some cfg parameters and only with commands that not produce output to stdout and stderr. For example, notepad works, but a simple batch file with a line "echo Hello" doen't works, the ssh core throws a connection_fatal with this error:
(I think this is related to F-Secure server, but not sure)

With the second method it also works.

Now there is the problem: the commands I need to execute produce output and I need to log the stdout and stderr outputs of the command and show them in the interface.
I have been testing the plink module and I have seen that it logs the stderr and stdout of a command, but it doesn't parse the output in search of escape characters, so that it prints the output strings and also garbage (escape characters not interpreted).
I think I will need to implement another from_backend function for the "Launching" module of the class to process the data in different way that psftp's from_backend does.

Am I wrong about this? So from_backend should differ the session method (sftp and shell) and call different members of the wrapper class in order to process the data received in a sftp connection and the data from a shell.
Is this ok?


Another questions talking about the cfg:

*What are cfg.ssh_subsys and cfg.ssh_subsys2 for? Concretelly I had to change the cfg.ssh_subsys parameter in order to be able to execute the notepad remotelly.
If I assign TRUE to cfg.ssh_subsys it throws the above error. And it works if I assign FALSE.

*What is the use of cfg.remote_cmd2 if there is cfg.remote_cmd?
I know cfg.remote_cmd_ptr is for providing long commands, but what is cfg.remote_cmd2 for?

*When clossing a session all commands and programs launched in that session are killed. Is there any way to prevent closing them? Does this happen with all servers or this is F-Secure related?


P.S: All tests where done against a F-Secure server (ssh2) (not supports ssh1)


Thank you very much for your help. You have helped me a lot and I couldn't do what I have done without your help.
Best regards.
(Take your time to reply, it's a very long post (I'm sorry for that, but you are the only one that I know that can help me))

Link39 wrote:

Finally I have managed to implement a wrapper class for putty with SFTP functionality.
I think that SSH v2 is supported by more servers than SSH v1, and almost all servers that supports SSH v1 also supports SSH v2 nowadays, so I have made only SFTP and no SCP implementation.

From the text below, I've understood that you are going to only to execute some commands of your own. Do you? Why do you than talk about SFTP? Or would you like to transfer file as well? Sorry, I've lost a track of your problem and I do not have time to read your previous posts :-(

Personally I think that's better to add the frontend parameter to the function than using mutexes, but this leads to a ssh core modiffication and have to be made every time you upgrade the putty core.

I would suggest you to add frontend parameter to the function and send the modification to Simon (author of Putty) to ask him to include it to official source code. I think he would do it, unless he has some good reason not to do it :-)

Am I wrong about this? So from_backend should differ the session method (sftp and shell) and call different members of the wrapper class in order to process the data received in a sftp connection and the data from a shell.
Is this ok?

Sure. That's one of the points where my classes TSFTPFileSystem and TSCPFileSystem differ.

*What are cfg.ssh_subsys and cfg.ssh_subsys2 for? Concretelly I had to change the cfg.ssh_subsys parameter in order to be able to execute the notepad remotelly.
If I assign TRUE to cfg.ssh_subsys it throws the above error. And it works if I assign FALSE.

cfg.ssh_subsys = TRUE means that string in cfg.remote_cmd is name of SSH2 subsystem and not a command. SSH2 subsystem is kind of symbolic name for executable. Particulary I know only one subsystem now, which is "sftp". So if you set cfg.ssh_subsys = TRUE and cfg.remote_cmd = "sftp", it is same as cfg.ssh_subsys = FALSE and cfg.remote_cmd = "/usr/lib/sftp-server". But you do not have to know exact location of SFTP server executable.

*What is the use of cfg.remote_cmd2 if there is cfg.remote_cmd?
I know cfg.remote_cmd_ptr is for providing long commands, but what is cfg.remote_cmd2 for?

You may specify two commands/subsystems. Putty tries to execute the first and when it fails, it tries the second. PSCP and WinSCP in "SFTP (allow SCP fallback)" tries to execute SFTP subsystem and when it fails it opens shell (WinSCP) or executes command "scp" (PSCP).

*When clossing a session all commands and programs launched in that session are killed. Is there any way to prevent closing them? Does this happen with all servers or this is F-Secure related?

I do not know, but I suppose that cannot control this from the client.

(Take your time to reply, it's a very long post (I'm sorry for that, but you are the only one that I know that can help me))

No problem. Now I see that you understand the things finally and the questions get to the point. So it is much easier to answer :-)

PS: I hope I have answered all your questions. If I have missed something, please ask again :-)

martin wrote:

From the text below, I've understood that you are going to only to execute some commands of your own. Do you? Why do you than talk about SFTP? Or would you like to transfer file as well? Sorry, I've lost a track of your problem and I do not have time to read your previous posts :-(

I need to transfer files and execute commands. First I need to transfer the "simulator" binarie to a powerfull machine and execute it. Then transfer back from that machine files containig the results of the simulation produced by the "simulator" binarie.

martin wrote:

No problem. Now I see that you understand the things finally and the questions get to the point. So it is much easier to answer.

It's easy to learn when your teacher is a master ;)
Thank you very much for you help.

martin wrote:

Sure. That's one of the points where my classes TSFTPFileSystem and TSCPFileSystem differ.

I will code another class in order to provide a secure shell and be able to launch a command a get it's output.

Best regards.

Hi Martin,

Another time I'm bothering you, I'm sorry for that.

I'm codding a class for launching commands and executable files on the remote server. I told you before that I need a class for transfers and another for the shell. I finished the transfers wrapper class some time ago and I have been creating the shell class.
To be precise, I need to execute a remote file and catch its output (both stdout and stderr). Ok, I followed your explanations for launching remote files and I did what you told:
- Leave cfg.remote_cmd empty in order to open the shell
- Open a connection
- Launch a file doing: back->send(backhandle,cmd,sizeof(cmd))
Where cmd is a pointer to a string containing the command (for example: "b.bat\r")
- Wait for the output data to be received

Ok, this works fine, the from_backend function is called with the output of the command about 4 or 6 times, BUT after those 4 or 6 times, it's not called anymore (so I stop receiving the output from the command). This is the problem I have and I don't know what I'm doing wrong.
As I'm testing it, in the from_backend function I do nothing, as I only want to see what data I receive.
Perhaps I need to empty the buffer in the from_backend function, I don't know. I have been looking at plink.c but it's code is a big mess. I don't understand all those events and the way he uses the threads for writting and reading data. It's impossible to understand it. I think that he does some flushing of the buffer that the core uses to send the data via the from_backend function but I don't know how and when.

Sorry for bothering you with my questions, but you have helped me a lot and you are the only one that can help me with this.

I know it's hard to understand my problem without the code, so if you need it, please ask me to send you it.
For testing purposes I have done the simplest thing, openning the connection (as always), doing a loop until we are connected, and the send the command. It's executed and then I do nothing, only waiting for the calls to from_backend function and watching the data sent to that function.

Thank you very much in advance.

Best regards.

P.S.: the "b.bat" is a bucle that prints five lines infinite times.

P.S.S.: I mailed Simon some days ago about the do_select and get_ssh_line problems (read about it some posts above) but he haven't replyed me neither he haven't made the changes in the code.

Wow, at this very moment I'm testing emptying the buffer in the from_backend function and not it doesn't stop receiving data.

I guess that the value returned by the from_backend function is related to resizing the buffer in order to transfer more data in each call to from_backend, but I'm only guessing. Is this true? For what is used the value returned by that function?

As the shell class is now totally syncronous, I don't see how to interrupt the command and disconnect. Perhaps creating a thread for the Receive() function and checking within it a bool value, and if it's true, stop receiving data and exit the connection. That bool value would be modiffied within a method: AbortReceive() for example.
What do you think about this?

Another question:
How to close the shell and terminate the connection? If I send the "exit\r" command to the shell, it does nothing, why? I only know the brute force method, but I want to know the clean method.

The last question:
If I send a command consisting of more than 5 characters, it's not sended complete, only the first 3 chars are sended. Do I need to use sendbuffer() for large commands?

Thank you very much.
Regards.

Take your time to repply.

Sorry for all those posts. When I wrote them I was planning to stop computting and get a rest till tomorrow, but I continued testing things and some doubts became solved and others appeared.

About the way of exiting with the "exit" command: It's because only the 4 first characters of the command are sent to the server, "exit\r", the return carriage is not send and the command is not executed. Why only the first 4 chars of a command are sent to the server? Do I have to send one character of the command at a time?

The finall question:
The trick of emptying the buffer (data) in the front_end function doen't works propertly. I only receive half of the data that I should receive. If I send a "dir" command, I receive only half of the directories, and also if I send the "help" command I only receive the half or less of the command output's.
But if I execute a batch file with the help command in it in an infinite bucle, the from_backend function doen't stop receiving data, but it always receives the same incomplete output.
Before (before emptying the data buffer), when executing a batch file with an infinite echoing, it completelly stopped receiving data after a while. Now it doen't stop, but I don't receive the entire message if the message is about more than 12 lines (I receive 12 of those lines, and always the same lines -> not always, in the first 2 calls to from_backend I receive only 1 or 2 lines of the message, then I receive about 12 lines which contains the former 2 lines).

Why this strange thing happens?

I have tested the plink client and there is also another strange thing: If I type dir, then the command is executed about 3 or 4 times, but only in the last time I receive all the lines (It's like if the data buffer is growing until it meets the size of the entire output of the dir command, and each time the command is re-executed)

I'm a bit lost. This is very strange and this is also related to plink command.

Do you know why this happens?

Regards.

Of course, take your time to repply.

I have been implementing features you struggle with a long time ago. So I may not help you. However here are some hints:

1) Receiving data:
You cannot just wait for from_backend() to be called. You must do something like sftp_recvdata() from psftp.c does (there are several such functions in Putty, at least one for each application). sftp_recvdata() is implementation for synchronous applications, so you need to know how much data you are going to read. I solve this in WinSCP, by sending commands like 'ls ; echo "this is end"' and then I read line by line until I find 'this is end'. Then I know that I should not expect any more output and I can send another command.

There maybe asynchronous implementation in plink or putty itself, if you need it. I have not, so I do not know.

sftp_recvdata() just does some repetitious calls to Win API functions, which in turn causes yours from_backend() to be called.

2) Sending data.
Yes, you need to call sendbuffer() too. See my TSecureShell::Send(). Important here is TSecureShell::WaitForData() too, which is used both for sending and receiving. It is the same as for example ssh_sftp_loop_iteration() from winsftp.c does. It is used for sftp_recvdata(), but I do not see it being used for sending. Maybe something has changed and it may not be necessary to call it for sending anymore. The same may apply for my WaitForData(), but it seems still working as it is.

3) Terminating connection.
If you are going to execute only one command (I do not remember, sorry), then you may specify it in cfg.remote_cmd. Than the server will terminate connection once the command is finished.

If you need to open shell, than you may terminate it using "exit\n". However I do not do it in WinSCP :-)
For the sample you wrote about, it should not work, if I understood correctly that b.bat is running endlessly. Then "exit\n" cannot be received by the shell, as the control is never returned to the shell.

4) Simon.
Simon has very long response times. So be patient. I have sent him some bug fixes for Putty at end of November and he has included them to Putty at end on January, without any reply :-)

I hope this helps. I also (again) hope I have answered all your questions. If not, please ask again. :-)

Thank you very much for your fast reply.

I was doing more test this morning and I have discovered many things. Forget all what I asked you yesterday.

I was stupid because when I received data in the from_backend function I didn't paid attention to the "len" parameter and as I was debugging with vs.net, I was coppying the data buffered from the debugger whatches and pasting it in a blank document for checking what I had received. Ok, I was receiving packets of about 2000 bytes of len, and I didn't paid attention to it, so when I was coppying it from the debugger's watches I thought that it was all data that I was receiving, but I was pasting about 600 bytes, that is the maximun data that the vs.net debugger shows and allows to copy in the watches window. So I was really receiving all data.
BUT it was overlapped, for example:
If I send the "help" command, in the first packet I received from the start of the output command till ECHO's help paragraph. In the second packet I received from DISKCOPY till FOR (so that the ECHO's help sentence was also present in this packet), and so on... So I was receiving packets with the data overlapped. I was very stranged with it and I decided to do the same test with an OpenSSH server (I was doing tests with a F-Secure server) and I received packets of about 80 bytes but without data overlapped.
In conclussion, the f-secure server is not compatible at all with the putty core. With the openSSH server I had no problems.

martin wrote:

1) Receiving data:
You cannot just wait for from_backend() to be called. You must do something like sftp_recvdata() from psftp.c does (there are several such functions in Putty, at least one for each application). sftp_recvdata() is implementation for synchronous applications, so you need to know how much data you are going to read. I solve this in WinSCP, by sending commands like 'ls ; echo "this is end"' and then I read line by line until I find 'this is end'. Then I know that I should not expect any more output and I can send another command.

There maybe asynchronous implementation in plink or putty itself, if you need it. I have not, so I do not know.

I was already calling a function I derived from a blucle in plink and I called it Receive(). (I mentioned it yesterday) It does something similar to sftp_recvdata(), but as I don't know exactly how much data I'm going to read, I exits the function when all sockets are idle.
Thank you very much for the "echo this is end" tip. I was getting problems with the sincronization, as if I send a command and then another, the second one is sent before the first one finishes and then the second one is not executed.

martin wrote:

2) Sending data.
Yes, you need to call sendbuffer() too. See my TSecureShell::Send(). Important here is TSecureShell::WaitForData() too, which is used both for sending and receiving. It is the same as for example ssh_sftp_loop_iteration() from winsftp.c does. It is used for sftp_recvdata(), but I do not see it being used for sending. Maybe something has changed and it may not be necessary to call it for sending anymore. The same may apply for my WaitForData(), but it seems still working as it is.

I have been testing with sendbuffer() call after send() and without it and I haven't seen any difference.

martin wrote:

3) Terminating connection.
If you are going to execute only one command (I do not remember, sorry), then you may specify it in cfg.remote_cmd. Than the server will terminate connection once the command is finished.
If you need to open shell, than you may terminate it using "exit\n". However I do not do it in WinSCP :-)
For the sample you wrote about, it should not work, if I understood correctly that b.bat is running endlessly. Then "exit\n" cannot be received by the shell, as the control is never returned to the shell.

I open the shell and then I send commands. If I send any command ending with "\n" it does nothing, as it's only a new line char. I need to send "\r" (carriage return). At least "\n" doesn't works for me.
Is there any chance to interrupt a command by sending ctrl+c or anything to the server?(as the server does not process the "exit" command untill the former command is finished)

Finally, what are these functions/attributes used for?
back->unthrottle()
back->sendbuffer()
noise_ultralight()
cfg.nopty
cfg.tcp_nodelay
nodelay in the call to back->init()


Thank you very much.

Regards

Link39 wrote:

In conclussion, the f-secure server is not compatible at all with the putty core. With the openSSH server I had no problems.

I have no experience with F-secure.

I open the shell and then I send commands. If I send any command ending with "\n" it does nothing, as it's only a new line char. I need to send "\r" (carriage return).

I'm sending '\n'. And it works. See my TSecureShell::SendLine().

Is there any chance to interrupt a command by sending ctrl+c or anything to the server?(as the server does not process the "exit" command untill the former command is finished)

OK, this one is tough. I have gained following knowledge using long testing period, so I do not know the terminology. But I'll try to be clear :-)

You may open SSH session in two modes. With TTY (lets call it 'interactive') and without TTY (non-interactive). The first mode is typically used in regular SSH terminals (like Putty itself). In this mode some characters have special meaning (like Ctrl+C = break, Ctrl+Z = put process to backgound, etc). In the second (raw, non-interactive) more, no special meaning is given to characters with ASCII code 0-27. You need to use this mode for SFTP, because it is binary mode and you generaly need to send bytes 0-27 directly to process (SFTP server) without being interpreted by terminal (maybe shell process does it, I do not know).

In Putty code you control the modes using Config.nopty (=true for raw/non-interactive mode). In WinSCP I always set nopty to true.

If you set it to false, than you may send Ctrl+C to terminate the process. But it has other consequences as well. I do not know all. Among others the shell command-line prompt (like /home/prikryl>) will be displayed. So you have to implement its detecting not to mistake it with command output.

The better is probably to se nopty=true and implement some way how to terminate the remote command inside the command (if it is your own program).

Finally, what are these functions/attributes used for?
back->unthrottle()
back->sendbuffer()
noise_ultralight()
cfg.nopty
cfg.tcp_nodelay
nodelay in the call to back->init()

I know these:
noise_ultralight(): it shuffles a random number generator. You should call it to make the connection more secure. In fact I do call only noise_regular() in regular intervals.
cfg.nopty: see above
cfg.tcp_nodelay: This is linked to Putty's option "Disable Nagle's algorithm (TCP_NODELAY option)". See Putty's help for more details.

Thank you very much!

martin wrote:

I'm sending '\n'. And it works. See my TSecureShell::SendLine().

Ok, you're right. It works on OpenSSH server but not on F-Secure servers. I was doing all tests under an f-secure server.

Now I'm using a thread that is always waiting for a network event and then recurse the sockets for receive/send data. I think that's better than calling after any command the receive() proc.
I also use mutexes in order to know when the connection is stablished/finished and when the command output is finished, for this I do the "echo" trick you commented yesterday. But as the "command ; command" syntax is only valid in some shells, I'm sending the command and inmediately I send the "echo" command. It works pretty well in openssh servers, but in f-secure it doen't works as it overlaps the output and the "echo" appears itself ("echo message", not only the message, because it have not been executed yet by the remote shell) twice or more times mixed within the output of the first command.

Regards.

Finally I have discovered what was the problem with f-secure server and putty's core. The problem is that by default the "terminal emulation" setting in the server is set to colour, which emulates a colour console, so that putty is not able to understand the colours scape secuences and it gets crazy. So that the terminal emulation must be set to "stream" as it doesn't work either with bw neither with raw.

And now I use a different way for detecting the end of a command: after stablishing connection I change the prompt (PS1 in unix-like and prompt in windows) so that every time I get the prompt means that the command is finished.

Regards.

Link39 wrote:

And now I use a different way for detecting the end of a command: after stablishing connection I change the prompt (PS1 in unix-like and prompt in windows) so that every time I get the prompt means that the command is finished.

So you have set nopty to false?
Keep in mind, that you have to set it to true, for transfer connection.

martin wrote:

So you have set nopty to false?
Keep in mind, that you have to set it to true, for transfer connection.

Yes, but only for the remote shell class.

I have been doing tests with the nopty to false and here are the results:

Under OpenSSH: If I send a ctrl+c (03), then I stop receiving output data from the command, but I don't get the command prompt.
Under F-Secure: The same, but it displays a message asking if I want to interrupt the batch command.

So why is it not working?

Another tests:
I have done some tests with 2 threads, one of them instances a class of the sftp wrapper and the other thread instances another class. Then both attemp to stablish a connection and it get a CRASH (not my fault). I did the same test but launching one thread and waiting some seconds before launching the other thread: so that the first thread stablish a connection and then start transferring a large file. While this thread transfers the file, the other thread is launched and another object of the sftp wrapper class is instanced and attemp to make a connection, but once the connection is stablished, the first thread stops receiving data (WHY?), and only the second thread is able to finish transferring the file and disconnect (but with a crash). I have done all ok, I don't use any static variable/function in my class, so I don't see the point on how this doesn't work. I link the same files that psftp links except: cmdline.c, console.c, winsftp.c, sftp.c/h, psftp.c/h, I call the frontend in almost all the callback functions (even in do_select and ssh_get_line), so I'm confused.
I did another test, because the static variables from winnet.c were suspicious to me, so I started tracing the code from the connect method of the class, in with I call "sk_init". I traced it and I saw that it assigns a new tree to the "sktree" static variable each time the fu***ng function is called. I did the same test as before but preventing that the second class calls "sk_init", and it worked: the both classes didn't stop receiving data until both files were transferred. But in disconnection, while doing some cleanup it crashed, but the important thing is that I could transfer both files at the same time in two sessions running at the same time (with a 2 seconds delay within the launch of the threads).
I have been looking at your code and I have seen that you call "sk_init" for each object you create, but as long as I know in WinSCP it's not possible to transfer 2 files simulataneous in two different sessions, as while one file is transferring you can't select another session.
I thought that putty's core was really multisession, that you could maintain more than one connection at a time and you could transfer files at the same time in two different sessions and also launch commands at the same time. If I have been working during the last two months for nothing I think I'm going to kill myself.
Please tell me that I have done something wrong, or that I have only to call sk_init only once in the whole program or something like this, because I'm desperated with this. I was so glad when all was working fine until I did the multisession test. As far as I know, all variables are shared within winnet.c, which means that the same sockets are used for all the sessions?? Is the "sktree" structure the cause?, and the Actual_Socket?

Please help me.

Thank you.

Link39 wrote:

I have been doing tests with the nopty to false and here are the results:

Under OpenSSH: If I send a ctrl+c (03), then I stop receiving output data from the command, but I don't get the command prompt.
Under F-Secure: The same, but it displays a message asking if I want to interrupt the batch command.

So why is it not working?

I do not know. I have never worked with nopty = false.

Please tell me that I have done something wrong, or that I have only to call sk_init only once in the whole program or something like this, because I'm desperated with this. I was so glad when all was working fine until I did the multisession test. As far as I know, all variables are shared within winnet.c, which means that the same sockets are used for all the sessions?? Is the "sktree" structure the cause?, and the Actual_Socket?

Again, I'm afraid that I cannot help you. As you have written in WinSCP you cannot work with two connections at the same time, so I have never tried this :-(

However I intend to add support for multithreading to the next version. So if you succeed, please send me some hint. Thanks.

Good luck :-)

martin wrote:

However I intend to add support for multithreading to the next version. So if you succeed, please send me some hint. Thanks.

It was the problem I thought. The sktree structure contains ALL sockets used by the applicattion, so if you call sk_init one time per connection, a new structure will be created and the pointer to the last structure will point to the newer one, so you loose controll over all pending data to be received/transmited in that moment, as you can't access anymore those old sockets. The solution is very simple, you have to call sk_init and sk_cleanup only once within the whole program. Simply add a mutex counter, so the first object initialized calls sk_init and the last one destroyed calls sk_cleanup.
This also fixes the sktree memory leak in you code, as every time you call sk_init, a new "tree234" structure is created and the last one won't be freed.

I'm happy that finally I can help you in something. I was in debt with you as you have helped me a lot.

If I can help you with something I would be happy.

Regards.

I'm happy that finally I can help you in something. I was in debt with you as you have helped me a lot.

If I can help you with something I would be happy.

Thanks for your info. Can you send me your email address (to my email), so I can contact you easier when I encounter some problem? Thanks.

I have mailed you to the sourceforge address. (I don't know if you refer to that one)
Please, feel free to contact with me whenever you want. I'll be glad if I can help you with something.

Greets.

Link39 wrote:

I have mailed you to the sourceforge address. (I don't know if you refer to that one)
Please, feel free to contact with me whenever you want. I'll be glad if I can help you with something.

I haven't received anything :-( See my forum profile for email address (link is on the left).

I received an undelivered mail response from the sourceforge server, so I have mailed you again to the other address.

Hope you receive it this time ;)

I have noticed that uploading files with sftp subsystem takes exactly double time than downloading the same files. I have been doing some tweaks to the upload mechanism but I was not able to reduce in more than 1 second the upload time. I have also incremented the upload buffer but it doesn't affect at all. I get double upload time with WinSCP, my library and psftp. With other clients such as f-secure sftp client the upload time was exactly the same as the download time (as it should be).

All tests where done in local, so the network connection doesn't affect. I did the tests with compression and without it and as I was transferring a compressed file, timings with compression were greater than without it.

With large files this is a real problem, and I think that's a putty core problem.

Do you know any tweak that could be applyed to the sftp upload core to reduce the upload time?
As the cipher used for uploading and downloading is the same, the timings should be also the same.
I'm trying to find the cause but I haven't found it for the moment.

Could the encrypting functions be the responsible of this issue (for example if encryption is slower than decryption)?

By the way, how could I select the cipher to use in each session?

Thanks.

Oh, I'm sorry for the last question. I didn't looked for it before posting.
I guess that the cfg.ssh_cipherlist is used for setting the preferred order of the ciphers to use.

So I redefine the question: If the user selects for example AES and the server only supports ssh1, will the putty's core fallback to the next cipher in the list?

Thanks.

Link39 wrote:

So I redefine the question: If the user selects for example AES and the server only supports ssh1, will the putty's core fallback to the next cipher in the list?

Yes it will. And if it falls below CIPHER_WARN then askcipher() is called.