Is your libido that much better when you don't exercise? How do you feel in general when you don't exercise? I do believe that exercise can enhance libido, so if I were you I would be looking for another reason. Not saying you're crazy or don't know your own body but I do think this is one thing the medical profession is (probably) right about. Even if you don't take the libido factor into account, you should feel better with regular exercise, not worse.