If you still want to use Voice Match while being able to record audio in any app, I found this workaround:
1. Wipe Google app data.
2. Go to Settings/Apps & Notifications/Default apps and set Google as Assist app.
3. Press back and enter the 'Assist app' settings ⚙, enable 'Voice Match', enable 'Hey Google' and follow the steps to set the voice command.
4. Go back to Default apps/Assist app,
select 'None' and then select 'Google' again.
5. Go back to 'Voice Match', disable 'Hey Google' and re-enable it again.
It sounds weird but it works. The only 'little' downside is that this is a temporary workaround because every reboot will mute your audio records and you will have to repeat steps 4 and 5.
Hope this helped.