It does not slow the transfer at all. While the operation presents as "size calculation", the ultimate goal is to collect a list of files to transfer, so they can be split among parallel connections. The only difference is that it's collecting all files upfront. While normally, it would list one directory, transfer its files, list another directory, transfer its files, etc. Overall time is the same in both cases. Indeed for a normal non-parallel transfer, the size calculation slows the transfer, as there it really only calculates size and for actual transfer, directories are listed again. But that's not the case for a parallel transfer.
I've documented this:
Maybe I can change it, that for parallel background transfers, the "size calculation" is forced (even if disabled globally).
This issue has been added to the tracker: