I changed over from a One to an X and I don't have any regrets (aside from some annoyances with 4.4 but that's a different subject), the X is a fantastic phone. Both have their strengths and weaknesses.
I agree with jellmoo on the camera, the X takes some pretty good pictures in decent lighting, better than the One but the One takes better shots in low light plus it has many more options and settings. The One has the better speakers though the X's speaker is clear and loud as well. The One's screen is very good but I personally prefer the X's OLED; if you value absolute clarity the One's screen is better, if you value color and contrast the X's is better. I personally prefer some of the Sense apps over the X's/Google's/alternatives such as the dialer, SMS, email and the customized Sense browser is far far superior to Chrome in every way. It also has the IR port and TV app plus radio. The X has active notifications which I love, the always listening search, better battery life, stock for all that entails good and bad as well as some useful stuff like Assist. The main reason I got the X over the One is because of its size as well as the fact that the One CANNOT be repaired in any way, once its battery dies it becomes an expensive paperweight which I think HTC should be flogged for personally. I did love my One though.