It depends on how far to the extremes you're willing to go.
Until you get your own seductions down to the point where you can pick up and sleep with women in minutes, and the women you sleep with are all mostly slept with quite fast, you'll have a hard time really knowing if you were a woman's fastest or not... and it might even be an unrealistic goal to BE a girl's fastest. However, if you're willing to devote the time to getting good at quick seductions and rapid escalation, it's easy to eventually be in a place where you're liable to be the fastest lover that most of the women you've been with have had.
If other men have bedded her faster than you, it's always there a little bit in the back of her head. It affects respect, how she thinks about you, where she slots you in the relationship, etc. How MUCH it affects the relationship depends on you, the girl, and the health of the relationship. e.g., if you're a super manly, attractive guy, and she's pretty mentally healthy and doesn't spend much time dwelling on the past, and the relationship is healthy and strong, it's a non-issue. However, if you start acting weak, or if she has some kind of emotional problem, or the relationship becomes less healthy, this can be one of the gas cans that throws more fuel on the fire.
Saying you'll break up with a girl because you've discovered she's slept with someone faster than you've slept with her is going to be unrealistic for most men in most relationships older than a week or two. It's just not a pressing enough reason to nix a relationship with a girl you dig. This one's more a preventative measure - be her fastest, or don't make her your girlfriend if you're not, is good form in setting up your relationships.
Things can still be fine even if you were a little slow compared to some of her lovers. But the more of them there are who "beat your score," and the more they beat it by, the more problems you'll (potentially - if things aren't perfect) have down the road, so keep it in mind.