I have the same issue (Samsung Galaxy Note 8 casting to built-in Chromecast in my Toshiba TV).
The Issue: I could connect via "Smart View" once, but then after disconnecting, wasn't able to connect again.
I found a temporary solution - at least, a better temporary solution (for me) than restarting my phone each time I want to cast. I have a Google Home in the same room.
The Solution: After connecting my Note 8 to my TV successfully, when I'm ready to disconnect, rather than disconnecting directly, I push the Smart View bubble and press "change device" to my Google Home. Once connected to my Google Home, I can disconnect. When I turn on Smart View again, it auto-connects to my Google Home first, and then I can change devices to my TV again. No issue and no need to restart the phone now. I just need to remember to move to the Google Home again to disconnect each time. Annoying? Definitely. But better than restarting the phone each time, at least.