One speed loader on your belt is in a covered holder isn't going to attract attention, and it is much faster than a speed strip. Of course you never want to have to you more than one or two shots during a SD situation.
I have speed loaders, however, as I carry a small digital camera a lot of the time, the case for it attaches to my belt. It has room for several loaders, but many times I just have one reload loose in the case. It does not look anything like an ammo case.