I've been playing both.
Guitar Hero 3 is hard for the sake of being hard.It's not very true to the songs. However, the guitar controllers are better quality. (Every person I've spoken to who's gotten the full Rock Band package has said that they ditched the RB guitars for the GH controllers in the first 5 mins). The 'strum' switch is pressure sensitive, rather than having the recognizable 'click' that lets you know that you're strummed the note. The 'strumless' frets on the neck for solos aren't useful at all.
However there are some things about the Rock Band game that I really don't like. The note board is transparent, making it sometime difficult to see notes depending on what the camera is focused on. The hammer on and pull off notes are almost impossible to see, and theres no online Co-Operative play... only Versus and Band play... I forget how they phrase it. I thought the whole idea of Rock Band was supposed to be to meet up with other players online and to start a 'Band' together... how can you find people who play other instruments if you can only play competitively? Their marketing campaign was REALLY misleading for this fact...
My recommendation: Buy Guitar Hero 3 with the wireless guitar, and then just buy the Rock Band game itself... without the full kit. You'll have a big library of songs that you can play, and you can always go back to GH2 if you want more. Play the guitar sections of the game, and wait for Harmonics to start selling the Drum kit seperately in the spring. Don't pay a premuim price for a badly designed guitar controller.