To untar a archive you need to use: tar -xvzf "source" "destination".
The predefined custom command in the preferences menu is tar -xz --directory="!?&Extract to directory:?.!" -f "!"
I do not know about the variables that have to be used here:
How can i integrate the unrar command that it can remotely extract rar files ?