There is no difference if installing it manually or over the air. The issue is that Google seems to have paused rolling out 4.0.4 for the time being.Thank you for your answer.
I went back to Yakju 4.02. I don't have a problem and I don't seem to get the OTA to 4.04. I wonder if I would get the same problem with the regular OTA instead of using fastboot to install it.