Roughly three months later ... maybe someone else will find my ramblings useful too?
So far, letting chrooted Linux to write directly to the framebuffer was the smoothest option for me (there are other options like the X server inside Android, but I doubt it could be more fluid than pure FB output), just make sure you have framebuffer access (eventually HW acceleration provided by a proprietary driver for your SoC/GPU) and a functional input method (OTG mouse, keyboard, or some X driver compatible with your TS).
Also, Ubuntu worked the best out of the box for me (Arch, Gentoo etc. users, feel free to disagree, this is only my opinion).