There is no point in trying to port it. We will have Jelly Bean source in about two weeks. If ICS drivers work with JB, like everyone seems to think, then it should be easy to make it work.
But if you just want to play with it... What kernel and which Rom are you using as base? Did you extract the boot image and use our ICS kernel and rc files in ramdisk? Did you use libs from our ICS rom?
HTC One X (Stock Android 4.0.4)
Samsung Galaxy Nexus (Stock Android 4.2)
LG Nexus 4 (Stock Android 4.2)
Asus Nexus 7 (Stock Android 4.2)
Samsung Galaxy Tab 7.7 3G (Stock Android 4.0.4)
Samsung Galaxy Tab 8.9 3G (Android 4.1.2 AOSP ROM)