How can u make a man who was a virgin not wonder about what its like to sleep with other women?

Im having trouble deciding on whether to keep this man figure around due to the fact that he expressed that he was curious of what other women are like. He expressed this only after 2 yrs of knowing and dating each other. I love him a lot. But I'm a mommy also. Is it right to have a man wonder what other women are like if he's already spent 2 yrs convincing you to fall inlove with him? He said that the reason why he was a virgin for so long was that he was waiting for the right girl. Shoudn't that mean that i'm the right girl? The longer this relationship goes the more he is short and abrupt on the phone, the less sweet things he does to impress me, its almost like he's doing it to get rid of me. Are these insecure feelings common? What should I do? Do I keep him with the hope that devotion to me will come, or do i "face reality" and let this guy go?