Thanks for the tips, I got it working!
The problem was with the complied binaries, somehow, my installer script didn't work.
Anyways, It works all the time. No need to unplug, replug the cable, or the keyboard multiple times, or switch usb host mode on-off-on to work. I just enable it with the mode changer button, and it works as expected.
Unfortunately, USB mouse gets recognized, but only as a keyboard device. I mean, it only issues UP, and DOWN commands. I guess we're missing drivers for this at the moment.
Is there anybody here, developing usb host drivers for our device?
Mouse with a pointer, and mass storage support would be awesome.