Well, I think I figured it out if anyone's wondering about this. When you first set up Chromecast you have to connect to it via WiFi and then after setup it will put you back on your normal WiFi connection. I went in the list of networks on my phone and deleted the Chromecast's WiFi setup network and now I haven't lost a connection since.
Samsung Galaxy S IV
Stock ME7 rooted