With any holster, it will depend upon the internal finishing of the Holster. Some are internally lined with a softer material, such as Suede Leather. The Leather Holsters that are hard on finishes are very coarsely finished internally and usually cheap. Those particular leather holsters use leather that has only been smoothly finished on one side instead of both sides and are not lined with any other fabric.
Finish wear can also happen with some Kydex and Nylon Holsters with rough internal finish or improper fit. Research your intended purchase carefully for fit and finish.
These days I tend to prefer properly fitted Kydex for CCW, particularly a Kydex IWB carried just behind my right hip.
Well made leather holsters are the best. They do take some time to break in, however, most people damage their holsters using everything under the sun. Answer: Mitch Rosen Gun Leather has a product called "leather lightning". It will break in your holster and make it a very smoothe draw. It will not damage leather.