So is this considered a bug, or is it just the way it works?
It is the way it works. Of course, it's bit a bug that you are allowed to turn the option off, if it is considered on nevertheless. Also it may be reasonable to let it be off, if size is used as comparison criteria. So there are things to improve. Thanks for bringing my attention to it :-)