The way I'd answer your question post-Asus Prime is different than I would have earlier. The "why aren't all phones affected" is a mystery. All the HT23 One X's were built on the same assembly line, using the same parts, within weeks of each other and are running the same s/w. There's no logical reason why they should perform differently.
There's something funky related to Teg3 that I'm guessing is inconsistency between the chips. Software is developed to expect a range of responses and if the h/w is reacting out of that range it'll cause the s/w to flake out. When the Prime started having similar Teg3 related issues I was firmly convinced s/w wouldn't be able to overcome them. But, updates (it took four) eventually did. If my theory is correct and it was different responses being sent by different Teg3 chips the fix(es) were modifying the s/w to anticipate them. When all's said and done the Prime owners that were unhappy are now happy. Neither Asus nor Nvidia confirmed the cause of the issues and once they went away everyone stopped caring.
So short answer; I think there's a h/w component but the One X's issues can be resolved to users satisfaction via s/w alone. It would be nice if HTC gave more color on the root cause of the issues but they'll probably go the way of Asus. While we bought HTC phones and they're 100% responsible for our satisfaction these issues are almost guaranteed to be with Nvidia h/w, s/w, or a combination of both. After following the Prime so closely I bought the One X Teg3 version anyway. My hope was that lessons learned would make the launch smoother. So much for that.