I've created a few Android apps, there is no hardware specific coding, you use openGL and Renderscript to do high-end graphics.
I believe the reason some high-end games are released on a per-device basis are to ensure a good user experience. Ie. they test them on the phones, make sure it works good, and then allow users with those devices to install the game.
This avoids comments like "thizzzz game issss so laggiiieeee" the comments....
Some phones are just to slow to allow them to run these high-end games (like GTA3 coming out soon), if people want to call that "fragmentation" then so be it, it's not really Google's fault. The bigger problem is the speed in which phones get the new updates (that's what needs to be improved)