To more directly answer the question: No, a downgrade is not possible.
Once updated, there is no mechanism on the CC to flash back to "stock ROM" like there is with most Android phone recoveries. Google probably didn't see the point in a stock recovery method: If a customer's Chromecast bricks...