Interesting... You're correct that YouTube grabs the stream directly and determines the best settings. But if you have a 1080p TV and YouTube is pulling a 480p stream, it'll definitely be noticeable - especially on things like text.
My Chrome is Version 32.0.1700.107 m
and Google Cast Extension is 18.104.22.168
My system is relatively old, but it was a powerhouse in its day and still fine for what I do with it.
Windows 7 Professional 64-bit
Dual Quad-Core AMD Opteron 8389 2.9 GHz
32 GB RAM
AMD Radeon HD 7750
I just tried installing regular chrome and had the same results.
your computer is def more powerful than mine, but I don't think thats the issue with such a low cpu utilization.