Brad DeLong, in the course of something completely different, suggests that the theory of relativity really isn't all that hard. At least, if your standard of comparison is quantum mechanics. He's completely right, of course. While relativity has a reputation for being intimidatingly difficult, it's a peculiar kind of difficulty. Coming at the subject without any preparation, you hear all kinds of crazy things about time dilating and space stretching, and it seems all very recondite and baffling. But anyone who studies the subject appreciates that it's a series of epiphanies: once you get it, you can't help but wonder what was supposed to be so all-fired difficult about this stuff. Applications can still be very complicated, of course (just as they are in classical mechanics or electrodynamics or whatever), but the basic pillars of the theory are models of clarity. Quantum mechanics is not like that. The most on-point Feynman quote is this one, from The Character of Physical Law:
There was a time when the newspapers said that only twelve men understood the theory of relativity. I do not believe there ever was such a time. There might have been a time when only one man did, because he was the only guy who caught on, before he wrote his paper. But after people read the paper a lot of people understood the theory of relativity in some way or other, certainly more than twelve. On the other hand, I think I can safely say that nobody understands quantum mechanics.
"Hardness" is not a property that inheres in a theory itself; it's a statement about the relationship between the theory and the human beings trying to understand it. Quantum mechanics and relativity both seem hard because they feature phenomena that are outside the everyday understanding we grow up with. But for relativity, it's really just a matter of re-arranging the concepts we already have. Space and time merge into spacetime; clocks behave a bit differently; a rigid background becomes able to move and breathe. Deep, certainly; inscrutable, no. In the case of quantum mechanics, the sticky step is the measurement process. Unlike in other theories, in quantum mechanics "what we measure" is not the same as "what exists." This is the source of all the problems (not that recognizing this makes them go away). Our brains have a very tough time separating what we see from what is real; so we keep on talking about the position of the electron, even though quantum mechanics keeps trying to tell us that there's no such thing.