I got Google Voice working.
edit: Can probably ignore most of this and use this APK instead:
I've been bothered by Google Voice not working since I got my Xoom on launch day, but have been using the web portal... Until now.
I loaded up my Droid, backed up Google Voice+settings with Titanium Backup, restored it on my Xoom and it came up without doing the setup, since it already had my account settings installed. It force closed the first time but I noticed an error in the notification area... Error was asking for permission to use that email address as my Google Voice account, I accepted, re-launched Google Voice and success, full app, working as it would on the phone. Gives notifications and everything.
edit: Attaching screenshots