Default [Q] Hands Free Mode wont turn off

Hi, I'm having an annoying problem with the hands free mode, in that it doesn't turn off automatically when it looses connection. It turns on S-Voice and enables hands free mode just fine when it establishes a connection to my car Bluetooth but I have to manually turn it off every time.

Surely if it can turn on automatically it can turn off that way too no?

Can't seem to find any help on the matter, every question I can find is either about how to stop hands free mode working at all or to make it turn on automatically, but nothing about turning it off.