My boyfriend often rejects me when i try to have with him?

I get rejected almost every time I try to touch my boyfriend ually. But when he touches me, I try to touch back and he won't let me. it's so confusing. It's like it's only ok when he wants to but not when I want to. We live together and we are happy. I'm not worried about infidelity , I just want to have more often. Right now it's only on the weekends and we do go weeks with having . He often says he can't turn his mind off from stress. But helps me release stress,but he says it doesn't work the same for him. However he jerks off almost everyday sometimes more than once a day. And when we do have it's great and we both really enjoy it. Sometimes afterwards he'll say it nice it was and I say, yeah we could do and feel that way more often, but he'll say, can't we just enjoy the moment. And when I say the sex is good when we have it, it really is. Sometimes very sensual and sometimes rough. It's always nice and we both enjoy it. I don't understand why he turns me down so much. But it really bothers me. When I try to talk to him about it I say how would you feel if I was always rejecting you? He usually just gets upset. I just don't understand he's loving and we have a great relationship. Help?!