Apparently the latest 4.0.3 AOSP can directly be built for the Xoom now:
https://groups.google.com/forum/#!topic/android-building/mjEzwvc_J4k
So, it just depends on how soon is "soon" regarding the proprietary binaries. But I have a feeling we have a good chance of having ICS to play with in time for Xmas.
Are these the Xoom binaries we need?
https://groups.google.com/forum/#!topic/android-building/mjEzwvc_J4k
https://dl.google.com/dl/android/aosp/nvidia-stingray-iml74k-96830dd7.tgz
https://dl.google.com/dl/android/aosp/nvidia-wingray-iml74k-e5226417.tgz