I want to setup VAT rules so they only apply to specific store

We are running multiple stores off Magento 2. I have setup VAT rates and VAT rules per country. Is there a way to control which stores the VAT rate is applied to? For example i have a retail store and i want VAT to be applied here but not on my wholesale store.

I can see how i achieve this by changing vat depending on customer group but can’t see how to control the tax rules and how they are applied on per store basis.

thanks