Is there something wrong with not wanting to lose my virginity to someone I've dated for a week?

That's moving too fast. No normal and non ahole would say or do that to you. Move on. He might have just told you he had a crush on you for his benefit!