Exactly the same with my wife! With her only other bf before me. A real jerk! He'd mix it in with regular sex every couple of weeks (for two years) But on the good side, when we started, she was used to it and with lube and me taking a little time, she liked it and she just opened right up. Even during regular p.i.v. sex she liked a finger moving in there. Unfortunately, we don't do it much any more.
I tried it once, a long time ago with an ex girlfriend. No fear or hesitation for either of us. Neither one of us liked it very much. She said it was too painful and I didn't get any special or unexpected enjoyment from it.
It took a while to convince my wife when we first did it but it was great for both of us. One of my friends loves it but we have to take our time getting it in back there. I love a toy in my ass but again, I have to take my time. I've had someone give my a prostate massage to the finish and it was wildly wonderful but not exactly sexy.