I've noticed this as well.  I made a bow hand guard out of buckskin a few months ago with the suede side in.  It was so soft at first, but it got matted down over time.  I have new bow with a handle suede side out and it looks great, but then I haven't used it much.