For me the buttons are ordered exactly the same way as in the preferences. If you see something different, please post a screenshot (showing the toolbar with preferences dialog next to it).

What determines the order of commands in the Custom Commands toolbar?

They seem to change in a very odd fashion. If I use command X, it might reorder itself to be first. If I then use command Y, instead of reordering itself to be first, it reorders itself to be second.

Also, when I restart, the order is never maintained. It always reverts so some odd order, and that order is never the order specified in the preferences (as is the case for the context-menu listing of custom commands).

Very quirky. I think it should be fixed.