I'm just trying this out for the first time, and getting a feel for it.
In Options|Preferences|Panels there's a checkbox labelled "Show Inaccessable Directories". I tried clearing this checkbox, clicking OK, and the ONLY directory to disappear from the right panel was the one directory other than my own home directory that was accessable to me via group membership. All the other user directories appear, but give me a 'permission denied' error when I attempt to enter any of them.