SFTP Directory retrieval very very slow

For some reason when I specify SFTP for transfers the directory listing always times out. I have also tried the SSH non-commercial version, which works, but takes a long time to list directory contents, though it does so eventually. This is on a local 100mb connection, so I'm a bit concerned turning this loose for users. I'm using openssh on a linux server. Is there something obvious on the server side that might be causing this?

Thanks

Sorry, I have no clue for you :-(