I just spoke to a friend who has an inside scoop at Google. The N1 has better graphics, and yes it's faster. Not enough of a difference for me to switch. Even he is telling me to stick to my droid.
I would have asked my friend if he's seen the 3D benchmark vid that showd the Nexus achieving a very modest 26.7 FPS. That puts it on par with a G1 running a cyanogen ROM or Hero.
I Have yet to see any real evidence of the Nexus outpacing the Droid at all. I've only heard people saying "well..uh...its smoother and faster like a little bit kinda sorta" That says absolutely nothing. Over all other Android devices, I would call the Nexus a huge leap forward, compared to the Droid, I consider it more of a "sidegrade" than anything.