Custom Transfer Settings Dialog is used to configure custom transfer settings for functions Synchronize and Keep remote directory up to date. To customize the settings, go to Transfer settings > Custom on the respective options dialog (you can also double-click the Transfer settings box).
The controls for setting transfer options are documented in separate chapter Transfer Settings.