I haven't disabled the stock YouTube app. So may be that's the reason. I have only set it to not open any links.This isn't a Vanced issue, because I have the non-root version working as you describe, clicking a Google Play video opens it in Vanced.
Unfortunately, I can't tell you exactly what I did to get it working. Getting Play videos to play was the most difficult, I remember.
I do have stock YouTube disabled and then it is a case of setting up default apps correctly.
Below is how I have YouTube Vanced set up. I am surprised to see that I have Vanced set to always ask, because it doesn't.
View attachment 5297731
View attachment 5297733
Despite setting Always Ask, some links open directly in the Vanced app without asking. I guess it happens when you click on a YouTube link from the Google app or one of Google's apps. It is the same that I see happening on my device too.